Information Retrieval: A Health and Biomedical Perspective, Second Edition
William R. Hersh
Springer
Health Informatics (formerly Computers in Health Care)
Kathryn J. Hannah Marion J. Ball Series Editors
This page intentionally left blank
William R. Hersh
Information Retrieval A Health and Biomedical Perspective Second Edition
With 77 Illustrations
Springer 1
William R. Hersh, MD Professor and Head Division of Medical Informatics & Outcomes Research School of Medicine Oregon Health & Science University Portland, OR 97201-3098, USA
[email protected] Series Editors: Kathyrn J. Hannah, PhD, RN Adjunct Professor, Department of Community Health Science Faculty of Medicine The University of Calgary Calgary, Alberta, Canada T2N 4N1
Marion J. Ball, EdD Vice President, Clinical Solutions Healthlink 2 Hamill Road Quadrangle 359 West Baltimore, MD 21210, USA and Adjunct Professor The Johns Hopkins University School of Nursing Baltimore, MD 21205, USA
Cover illustration: “Technology Doctor” by Nikolai Punin/images.com Library of Congress Cataloging-in-Publication Data Hersh, William R. Information retrieval : a health and biomedical perspective / William R. Hersh. p. cm.—(Health informatics) First ed. has subtitle: a health care perspective. Includes bibliographical references and index. ISBN 0-387-95522-4 (alk. paper) 1. Medical informatics. 2. Information storage and retrieval systems—Medicine. I. Title. II. Series. R858 .H47 2003 025.0661—dc21 2002075762 ISBN 0-387-95522-4
Printed on acid-free paper.
© 2003, 1996 Springer-Verlag New York, Inc. All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer-Verlag New York, Inc., 175 Fifth Avenue, New York, NY 10010, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of going to press, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Printed in the United States of America. 987654321
SPIN 10882153
www.springer-ny.com Springer-Verlag New York Berlin Heidelberg A member of BertelsmannSpringer ScienceBusiness Media GmbH
To Sally, Becca, and Alyssa
MEDLINE, MeSH, PubMed, MEDLINEPlus, ClinicalTrials.gov, Metathesaurus, and UMLS are registered trademarks of the United States Library of Medicine.
This page intentionally left blank
Series Preface
This series is directed to healthcare professionals who are leading the transformation of health care by using information and knowledge. Launched in 1988 as Computers in Health Care, the series offers a broad range of titles: some addressed to specific professions such as nursing, medicine, and health administration; others focusing on special areas of practice such as trauma and radiology. Still other books in the series treat interdisciplinary issues, such as the computer-based patient records, electronic health records, and networked healthcare systems. Renamed Health Informatics in 1998 to reflect the rapid evolution in the discipline now known as health informatics, the series will continue to add titles that contribute to the evolution of the field. In the series, eminent experts, serving as editors or authors, offer their accounts of innovations in health informatics. Increasingly, these accounts go beyond hardware and software to address the role of information in influencing the transformation of healthcare delivery systems around the world. The series also increasingly focuses on “peopleware” and the organizational, behavioral, and societal changes that accompany the diffusion of information technology in health services environments. These changes will shape health services in this new millennium. By making full and creative use of the technology to tame data and to transform information, health informatics will foster the development of the knowledge age in health care. As coeditors, we pledge to support our professional colleagues and the series readers as they share advances in the emerging and exciting field of health informatics. Kathryn J. Hannah Marion J. Ball
vii
This page intentionally left blank
Preface
The main goal of this book is to provide an understanding of the theory, implementation, and evaluation of information retrieval (IR) systems in health and biomedicine. There are already a number of excellent “how-to” volumes on searching healthcare databases (Hutchinson, 1998, Katcher, 1999). Similarly, there are also a number of high-quality basic IR textbooks (Grossman and Frieder, 1998, Baeza-Yates and Ribeiro-Neto, 1999, Belew, 2000). This volume is different from all of the above in that it covers basic IR as do the latter books, but with a distinct focus on the health and biomedicine domain. The first edition of this book was written from 1994 to 1995 and published in 1996. Although subsequent editions of books in many fields represent incremental updates, this edition is profoundly rewritten and is essentially a new book. As everyone who is reading this knows, the IR world has changed substantially in the 8 to 9 years since the first edition. In that edition, the Internet was a “special topic” in the very last chapter of the book. Here, on the other hand, it is introduced at the beginning of the first chapter. At the time of the first edition, using a search engine was something that only a minority of healthcare practitioners had ever done. Now, not only have virtually all physicians used the World Wide Web, but a majority of their patients have used it for seeking personal health information as well. The Web also profoundly altered the way this edition was researched and written. When preparing the first edition, finding an article not in my own collection or the Oregon Health & Science University (OHSU) Library was a chore that entailed either driving to a nearby library at another university or ordering it through interlibrary loan. For this edition, I was usually able to find articles on the Web, either because OHSU or I had a subscription or because the article was freely available. In writing this edition I really learned firsthand the value of scientific publishing on the Web. The only downside to the feast of articles I found was the ease with which I found additional articles to consider for inclusion. Another way the Web has impacted and will continue to impact this edition is through the maintenance of a Web site for errata and updates. The Web site www. irbook.org will identify all errors in the book text as well as provide updates on important new findings in the field as they become available. ix
x
Preface
The work on this edition also drove home the quality of the IR systems I was using. I must give particular mention to the following resources that provided fast and accurate access to a great deal of information: the National Library of Medicine (NLM) PubMed MEDLINE system, the Google search engine, ResearchIndex, and the multitude of journals that have opted to electronically publish their full texts. As in the first edition, the approach is still to introduce all the necessary theory to allow coverage of the implementation and evaluation of IR systems in health and biomedicine. Any book on theoretical aspects must necessarily use technical jargon, and this book is no exception. Although jargon is minimized, it cannot be eliminated without retreating to a more superficial level of coverage. The reader’s understanding of the jargon will vary based on their backgrounds, but anyone with some background in computers, libraries, health, and/or biomedicine should be able to understand most of the terms used. In any case, an attempt to define all jargon terms is made. Another approach is to attempt wherever possible to classify topics, whether discussion types of information or models of evaluation. I have always found classification useful in providing an overview of complex topics. One problem, of course, is that everything does not fit into the neat and simple categories of the classification. This occurs repeatedly with information science, and the reader is forewarned. This book had its origins in a tutorial taught at the former Symposium on Computer Applications in Medicine (SCAMC) meeting. The content continues to grow each year through my annual course taught to medical informatics students in the on-campus and disease-learning programs at OHSU. (They often do not realize that next year’s course content is based in part on the new and interesting things they teach me!) The book can be used in either a basic information science course or a health and biomedical information science course. It should also provide a strong background for others interested in this topic, including those who design, implement, use, and evaluate IR systems. Interest continues to grow in health and biomedical IR systems. I entered a fellowship in medical informatics at Harvard University in the late 1980s, when the influence of medical artificial intelligence was still strong. I had assumed I would take up the banner of some aspect of that area, such as knowledge representation. But along the way I came across a reference from the field of “information retrieval.” It looked interesting, so I looked at the references of that reference. It did not take long to figure out that this was where my real interests lay, and I spent many an afternoon in my fellowship tracing references in the Harvard University and Massachusetts Institute of Technology libraries. Even though I had not yet heard of the field of bibliometrics, I was personally validating all its principles. Like many in the field, I have been awed to see IR become “mainstream” with the advent of the Web in recent years. The book is divided into three sections. The first section covers the basic concepts of information science. The first chapter provides basic definitions and models that will be used throughout the book. The next chapter gives an overview of
Preface
xi
health and biomedical information, covering issues related to its generation and use. The third chapter discusses the evaluation of IR systems, highlighting the methods and their limitations. The evaluation chapter is deliberately placed at the beginning of the book to emphasize the fundamental importance of this topic. The second section covers the current state-of-the-art in commercial and other widely used retrieval systems. The first chapter in this section gives an overview of the great deal of content that is currently available. Next come chapters on the two fundamental intellectual tasks of IR, indexing and retrieval. The predominant paradigms of each are discussed in detail. The final chapter covers evaluation of these systems, providing a justification for the work described in the following section on research efforts. The third section covers the major threads of research and development in efforts to build better IR systems. The focus is initially on details of indexing and retrieval, with a chapter each on the two major thrusts, which are lexicalstatistical and linguistic systems. In the next chapter, a survey of various efforts to augment other systems is described. This is followed by a chapter on information extraction, a topic of growing importance. Throughout this section, a theme of implementational feasibility and evaluation is maintained. Within each chapter, the goal is to provide a comprehensive overview of the topic, with thorough citations of pertinent references. There is a preference to discuss health and biomedical implementations of principles, but where this is not possible, the original domain of implementation is discussed. Several chapters make use of a small sample database in Appendix 1 to illustrate the principles being discussed, which is further described at the end of Chapter 1. This book would not have been possible without the influence of various mentors, dating back to high school, who nurtured my interests in science generally and/or medical informatics specifically, and/or helped me achieve my academic and career goals. The most prominent include: Mr. Robert Koonz (then of New Trier West High School, Northfield, IL), Dr. Darryl Sweeney (University of Illinois at Champaign-Urbana), Dr. Robert Greenes (Harvard Medical School), Dr. David Evans (Clairvoyance Corp.), Dr. Mark Frisse (Express Scripts Corp.), Dr. J. Robert Beck (then of OHSU), Dr. David Hickam (OHSU), Dr. Brian Haynes (McMaster University), and Dr. Lesley Hallick (OHSU). I must also acknowledge the contributions of the late Dr. Gerard Salton (Cornell University), whose writings initiated and sustained my interest in this field. I would also like to note the contributions of institutions and people in the federal government that aided the development of my career and this book. While many Americans increasingly question the abilities of their government to do anything successfully, the National Library of Medicine (NLM), under the directorship of Dr. Donald A. B. Lindberg, has led the growth and advancement of the field of medical informatics. The NLM’s fellowship and research funding have given me the skills and experience to succeed in this field. Likewise, the Agency for Healthcare Research and Quality (AHRQ) deserves mention for its contributions to my own growth as well as others in the field of medical informatics. I would also like to acknowledge retired Oregon Senator Mark O. Hat-
xii
Preface
field through his dedication to biomedical research funding that aided myself and many others. Finally, this book also would not have been possible without the love and support of my family. All of my “extended” parents, Mom and Jon, Dad and Gloria, as well as my grandmother Baubee, brother Jeff, sister-in-law Myra, motherin-law Marjorie, and father-in-law Coop supported, sometimes grudgingly, the various interests I developed in life and the somewhat different career path I chose. (I think they still cannot understand why I decided not to be a “regular doctor.”) And last, but most importantly, has been the contribution of my wife, Sally, and two children, Becca and Alyssa, whose unlimited love and support made this undertaking so enjoyable and rewarding. William R. Hersh
Contents
Series Preface vii Preface ix
PART I BASIC CONCEPTS CHAPTER 1 CHAPTER 2 CHAPTER 3
Terms, Models, and Resources 3 Health and Biomedical Information 22 System Evaluation 83
PART II STATE OF THE ART CHAPTER CHAPTER CHAPTER CHAPTER
4 5 6 7
Content 117 Indexing 146 Retrieval 183 Evaluation 219
PART III RESEARCH DIRECTIONS CHAPTER CHAPTER CHAPTER CHAPTER
8 9 10 11
Lexical-Statistical Systems 265 Linguistic Systems 310 Augmenting Systems for Users 356 Information Extraction 397
xiii
xiv
Contents
APPENDIX 1 APPENDIX 2 APPENDIX 3 APPENDIX 4
Ten Sample Documents to Illustrate Indexing and Retrieval 431 Inverted File of Words from Documents of Appendix 1 435 Document Vectors for Words from Documents of Appendix 1 with Weighting Measures 443 Inverted File of MeSH Terms from Documents of Appendix 1 with Term Frequency and Postings 448
References 450 Index 503
Part I Basic Concepts
This page intentionally left blank
Chapter 1 Terms, Models, and Resources
The goal of this book is to present the field of information retrieval (IR), with an emphasis on the health and biomedical domain. To many, “information retrieval” implies retrieving information of any type from a computer. However, to those working in the field, it has a different, more specific meaning, which is the retrieval of information from databases that predominantly contain textual information. A field at the intersection of information science and computer science, IR concerns itself with the indexing and retrieval of information from heterogeneous and mostly textual information resources. The term was coined by Mooers in 1951, who advocated that it be applied to the “intellectual aspects” of description of information and systems for its searching (Mooers, 1951). The advancement of computer technology, however, has altered the nature of IR. As recently as the 1970s, Lancaster (1978) stated that an IR system does not inform the user about a subject; it merely indicates the existence (or nonexistence) and whereabouts of documents related to an information request. At that time, of course, computers had considerably less power and storage than today’s personal computers. As a result, systems were only sufficient to handle bibliographic databases, which contained just the title, source, and a few indexing terms for documents. Furthermore, the high cost of computer hardware and telecommunications usually made it prohibitively expensive for end users to directly access such systems, so they had to submit requests that were run in batches and returned hours to days later. In the twenty-first century, however, the state of computers and IR systems is much different. End-user access to massive amounts of information in databases and on the World Wide Web is routine. Not only can IR databases contain the full text of resources, but they may also contain images, sounds, and even video sequences. Indeed, there is continued development of the digital library, where books and journals are replaced by powerful file servers that allow high-resolution viewing and printing, and library buildings are augmented by far-reaching computer networks (Lesk, 1997; Arms, 2000; Witten and Bainbridge, 2003). One of the early motivations for IR systems was the ability to improve access to information. Noting that the work of early geneticist Gregor Mendel was undiscovered for nearly 30 years, Vannevar Bush called in the 1960s for science to 3
4
I. Basic Concepts
create better means of accessing scientific information (Bush, 1967). In current times, there is equal if not more concern with “information overload” and how to find the more pertinent information within “data smog” (Shenk, 1998). However, important information is still missed. For example, a patient who died in a clinical trial in 2000 might have survived if information from the 1950s (before the advent of MEDLINE) about the toxicity of the agent being studied had been more readily accessible (McLellan, 2001). Indeed, a major challenge in IR is helping users find “what they don’t know” (Belkin, 2000). Nevertheless, IR systems are a unique type of computer application, and their growing prevalence demands a better understanding of the principles underlying their operation. This chapter gives an overview of the basic terminology of IR, some models of systems and their interactions with the rest of the world, a discussion of available resources, and a view of IR on the Internet and the World Wide Web.
1.1 Basic Definitions There are a number of terms commonly used in IR. An IR system consists of content, computer hardware to store and access that content, and computer software to process user input in order to retrieve it. Collections of content go by a variety of terms, including database, collection, or—in modern Web parlance—site. In conventional database terminology, the items in a database are called records. In IR, however, records are also called documents, and an IR database may be called a document database. In modern parlance, items in a Web-based collection may also be called pages, and the collection of pages called a Web site. A view of the IR system is depicted in Figure 1.1. The goal of the system is to enable access by the user to content. Content consists of units of information, which may themselves be an article, a section of a book, or a page on a Web site. Content databases were easier to describe in the pre-Web era, as the boundaries of nonlinked databases and the persistence of paper documents were, in general, more easy to delineate. The scope of Web content, however, varies widely: some sites organize information into long pages covering a great deal of matter, while others break it down into numerous short pages. The picture is further complicated by multimedia elements, which may be part of a page or may be found in their own separate file. In addition, Web pages may be composed of frames, each of which in turn may contain its own Web page of information. Users seek content by the input of queries to the system. Content is retrieved by matching metadata, which is meta-information about the information in the content collection, common to the user’s query and the document. As will be seen in Chapter 5, metadata consist of both indexing terms and attributes. Indexing terms represent the subject of content, that is, what it is about. They vary from words that appear in the document to specialized terms assigned by professional indexers. Indexing attributes can likewise be diverse. They can, for example, consist of information about the type of a document or page (e.g., a jour-
1. Terms, Models, and Resources
5
FIGURE 1.1. A model of the IR system.
nal article reporting a randomized controlled trial) or its source (e.g., citation to its location in the Journal of the American Medical Informatics Association and/or its Web page address). A search engine is the software application that matches indexing terms and attributes from the query and document to return to the user. There are two major intellectual or content-related processes in building and accessing IR systems: indexing and retrieval. Indexing is the process of assigning metadata to items in the database to facilitate and make efficient their retrieval. The term indexing language, sometimes used, refers to the sum of possible terms that can be used in the indexing of terms. There may be, and typically are, more than one set of indexing terms, hence indexing languages, for a collection. In most bibliographic databases, for example, there are usually two indexing procedures and languages. The first indexing procedure is the assignment of indexing terms from a controlled vocabulary or thesaurus by human indexers. In this case, the indexing language is the controlled vocabulary itself, which contains a list of terms that describe the important concepts in a subject domain. The controlled vocabulary may also contain nonsubject attributes, such as the document publication type. The second indexing procedure is the extraction of all words that occur (as identified by the computer) in the entire database. Although one tends typically not to think of word extraction as indexing, the words in each document can be viewed as descriptors of the document content, and the sum of all words that occur in all the documents is an indexing language. Retrieval is the process of interaction with the IR system to obtain documents. The user approaches the system with an information need. Belkin et al. (1978) have described this as an anomalous state of knowledge (or ASK). The user (or a specialized intermediary) formulates the information need into a query, which most often consists of terms from one or more of the indexing
6
I. Basic Concepts
vocabularies, sometimes (if supported by the system) connected by the Boolean operators AND, OR, or NOT. The query may also contain specified metadata attributes. The search engine then matches the query and returns documents or pages to the user.
1.2 Comparisons with Computer Applications of Other Types One way to understand a computer system is to compare it with applications of other types. An IR system is not, for example, the same as a database management system (DBMS). A traditional DBMS typically provides access to highly structured data, such as a zip code or blood sugar level. In a DBMS, the existence of an answer to a query tends to be an either/or situation. Either the zip code is there or it is not. In an IR system, on the other hand, an answer to a specific question may not exist, and even if it does exist, the user may not be able to find it easily. Another difference is that the records of an IR database are not the same as those of a DBMS. In a DBMS, the database record has one or more fields, each consisting of a specific type of information. For example, a database of patient demographic information has one record for each patient, with fields for items such as name, address, zip code, gender, and race. The field for name is typically a string of characters with fixed length (even if there may be many blank characters for shorter names), while the field for zip code will be numerical, with five (or nine) digits. A record in an IR database may simply have two fields, such as a title and body of text. Or, as is seen in some of the specialized bibliographic databases, it may have numerous fields (title, abstract, source, indexing terms, publication type, etc.). While some of these fields may be DBMS-like in having a fixed length or data type, most of the fields have text of varying length. Another difference between a DBMS and an IR system is how the database is indexed. Besides the assignment of descriptors to represent the content in record and fields, another purpose of indexing is to allow rapid access to records or documents, based on their content. In a DBMS, the index for a record is one or more keys, each of which is typically derived from the entire contents of a single field, such as the last name or zip code. In an IR system, on the other hand, indexing may involve breaking out certain terms within a field (such as the indexing terms) or even all the words. Indexing may also involve more complicated procedures, such as using natural language processing techniques to allow synonyms or map text in various fields to terms in a controlled vocabulary. Nonetheless, the distinction between the DBMS and IR systems is blurring. Many DBMS applications allow indexing of all the words in its fields that contain free text (e.g., FileMaker Pro, from Apple Computer Inc., www.apple.com, and the Oracle Database, from Oracle Corp., www.oracle.com). Likewise, some IR systems are built on top of a DBMS and use its functionality (e.g., BasisPlus, from Information Dimensions Inc., www.idi.oclc.org).
1. Terms, Models, and Resources
7
The nature of an IR system can also be demonstrated by comparing it with an expert system. An expert system uses rules and other procedures (knowledge) to manipulate data for the purpose of making inferences (Musen, Shahar et al., 2000). Like a DBMS, an expert system also uses highly structured data, but its goal is to make new inferences based on data entered into the system that interacts with its underlying knowledge. Some of the distinctions between expert systems and IR systems are also blurring. For example, as will be seen in Part III, expert system techniques have been utilized to improve IR systems.
1.3 Models of IR Another way to understand a field is to look at models of the processes that are studied. A model of the IR system has already been presented. In this section, three models are discussed that depict the overall information world, the user’s interaction with an IR system, and—in keeping with the book’s focus on health— the factors that influence decision making in healthcare.
1.3.1 The Information World Figure 1.2 depicts the cyclic flow of information from its producers, into the IR system, and on to its users (Meadow, 1992). Starting from the creation of information, events occur in the world that lead to written observations in the form of books, periodicals, scientific journals, and so on. These are collected by database producers, who may create a bibliographic database of references to these sources or, as is happening with greater frequency, may create electronic ver-
FIGURE 1.2. A model of information flow in the world. (Meadow, 1992.)
8
I. Basic Concepts
sions of the full text. However a database is constructed, it is the organized into records and loaded into the IR system. In the system, a file update program stores the data physically. Users, either directly or through trained intermediaries, query the database and retrieve content. Those users not only use the information but may also add value to it, some of which may make its way into new or other existing content. In addition, users also feed back observations to the database producers, who may correct errors or organize it better.
1.3.2 Users Figure 1.3 shows the information-seeking functions of the user (Marchionini, 1992). The central component is the user defining the problem (or information need). Once this has been done, the user selects the source for searching and articulates the problem (or formulates the query). The user then does the search, examines the results, and extracts information. Many of these tasks are interactive. Any step along the way, for example, may lead the user to redefine the problem. Perhaps some results obtained have provided new insight that changes the information need. Or, examination of the results may lead the user to change the search strategy. Likewise, information extracted may cause the user to examine the rest of the searching results in a different manner.
1.3.3 Health Decision Making The ultimate goal of searching for health information is often to make a decision, such as whether to order a test or prescribe a treatment. The decision maker may be a patient, his or her family, or a healthcare professional. A variety of factors go into making a health-related decision. The first of these categories is the scientific evidence, which answers the question of whether there has been objective-as-possible science to support a decision. A system for finding and appraising this sort of information most effectively is called evidence-based med-
FIGURE 1.3. A model of the IR user. (Marchionini, 1992.)
1. Terms, Models, and Resources
9
icine (EBM). As discussed in more detail in the next chapter, EBM provides a set of tools that enable individuals to more effectively find information and apply it to health decisions. Evidence alone, however, is not sufficient for decisions. Both patients and clinicians may have personal, cultural, or other preferences that influence how evidence will be applied. The healthcare professional may also have limited training or experience to be able to apply evidence, such as a physician not trained as a surgeon or not trained to perform a specific treatment or procedure. There are other constraints on decision making as well. There may be legal or other restrictions on what medical care can be provided. There may also be constraints of time (patient far away from the site at which a specific type of care can be provided) or financial resources (patient or entity responsible for paying for the care cannot afford it). Figure 1.4, adapted from Mulrow et al. (1997), depicts the relationships among the factors that influence health decision making. The intersection of evidence and preferences provides the knowledge that can be used to make decisions. The intersection of evidence and constraints leads to guidelines. There is a growing
FIGURE 1.4. A model of health decision making. (Mulrow, C., Cook, D., et al., (1997). Systematic reviews: critical links in the great chain of evidence. Annals of Internal Medicine, 126:389–391. Reprinted with permission.)
10
I. Basic Concepts
TABLE 1.1. Information Science Professional Organizations Organization
Web address
American Library Association (ALA) American Medical Informatics Association (AMIA) American Society for Information Science and Technology (ASIST) American Society of Indexers Association for Computing Machinery, Special Interest Group on Information Retrieval (ACM/SIGIR) Medical Library Association (MLA) National Library of Medicine (NLM) Special Libraries Association (SLA)
www.ala.org www.amia.org www.asis.org www.asindexing.org www.acm.org/sigir www.mlanet.org www.nlm.nih.gov www.sla.org
interest in practice guidelines, and this specialized type of content will be discussed in Chapter 4. The ethical dimensions of healthcare lie at the intersection of preferences and constraints. Finally, the intersection of all three represents everything that is considered in a health decision.
1.4 IR Resources It has already been noted that IR is a heterogeneous, multidisciplinary field. This section describes the field’s organizations and publications. Because of the diversity of this field, the focus is on the organizations and publications most centrally committed to IR. Also included are those from the healthcare field that may not have IR as a central focus but do have an interest in it. The Web addresses of all the organizations, journals, and tools mentioned in this section are listed in Tables 1.1, 1.2, and 1.3, respectively.
1.4.1 People The field most centrally concerned with IR is information science, a multidisciplinary field that studies the creation, use, and flow of information. Information TABLE 1.2. Journals Noted for Coverage of Information Retrieval Journal ACM Transactions on Information Systems (ACM TOIS) British Medical Journal (BMJ) Computers in Biology and Medicine (CBM) Journal of Biomedical Informatics (JBI) Journal of the American Medical Informatics Association (JAMIA) Journal of the American Society for Information Science and Technology (JASIST) Journal of the Medical Library Association (JMLA) Information Processing and Management (IP&M) Information Retrieval (IR) Methods of Information in Medicine (MIM)
Web address www.acm.org/tois www.bmj.com www.elsevier.nl/locate/compbiomed www.academicpress.com/jbi www.jamia.org www.asis.org/Publications/JASIS/jasis.html www.mlanet.org/publications/bmla www.elsevier.nl/locate/infoproman www.wkap.nl/journals/ir www.schattauer.de/zs/methods/main.asp
1. Terms, Models, and Resources
11
TABLE 1.3. Information Science Tools Name and description Bow: library of code for writing statistical text analysis, language modeling and information retrieval programs Frakes and Baeza-Yates: source code from Frakes and Baeza-Yates (1992) freeWAIS-sf: Wide Area Information Searcher Internet Archive: archive of the Internet, including various “snapshots” from specific points in time IR Framework: object-oriented framework for information retrieval (IR) applications written in Java IRTools: Retrieval testbed for various techniques The Lemur Toolkit for Language Modeling and Information Retrieval: supports indexing of large-scale text databases, the construction of simple language models for documents, queries, or subcollections, and the implementation of retrieval systems based on language models as well as a variety of other retrieval models MG: public domain indexing and retrieval system for text, images, and textual images NCBI: most of the genomic databases of the National Center for Biotechnology Information of the National Library of Medicine are freely available NLM: the entire MEDLINE database, MeSH vocabulary, Unified Medical Language System, and other resources are available from the National Library of Medicine PLS: free retrieval system from America On-Line PRISE: prototype indexing and search engine developed by National Institute of Standards and Technology Okapi: indexing and retrieval system implementing Okapi weighting scheme SMART: current version of IR system based on Salton’s algorithm
Web address www.cs.cmu.edu/%7Emccallum/bow
ftp://ftp.vt.edu/pub/reuse/IR.code ls6-www.informatik.unidortmund.de/ir/ projects/freeWAIS-sf www.archive.org
www.itl.nist.gov/iaui/894.02/projects/irf/irf.html
www.ils.unc.edu/tera/Projects/projects.html www-2.cs.cmu.edu/lemur/
www.kbs.citri.edu.au/mg www.ncbi.nlm.nih.gov/ftp/index.html
www.nlm.nih.gov/databases/leased.html
www.pls.com www.itl.nist.gov/iaui/894.02/works/papers/ zp2/zp2.html www.soi.city.ac.uk/andym/OKAPI-PACK/ ftp://ftp.cs.cornell.edu/pub/smart
scientists come from a wide variety of backgrounds, including information science itself, library science, computer science, systems science, decision science, and many professional fields. Some consider library science and information science to be part of the same field, which they call library and information science.
12
I. Basic Concepts
There are also fields that study health information in various contexts and have some interest in IR. The health-related informatics fields are concerned with the use of computers and other information technology in areas related to health and biomedicine. While there is some disagreement about the adjectives that precede the word “informatics” (e.g., medical informatics, health informatics, nursing informatics, bioinformatics), all these fields are concerned with the storage, acquisition, and use of information in health and biomedicine. They all go beyond IR, and all include many other applications of information technology in health and biomedicine.
1.4.2 Organizations A number of specialty organizations have interests that overlap with IR. There is one organization devoted specifically to information science, called the American Society for Information Science and Technology (ASIST). The library science field is closely related to information science, and there is much overlap in personnel, but its central issues are more related to the structure and functioning of libraries. The major library science organization is the American Library Association (ALA). Another organization more focused on technical libraries is the Special Libraries Association (SLA). There is also a professional group devoted specifically to document indexing, called the American Society of Indexers (ASI). In addition, computer science’s largest professional organization, the Association for Computing Machinery (ACM), sponsors the Special Interest Group in Information Retrieval (SIGIR). A number of health information organizations have IR among their interests. The American Medical Informatics Association (AMIA) is devoted to all facets of healthcare information technology, including IR. The Medical Library Association (MLA) is concerned with the needs and issues of medical libraries, which of course include IR systems. Another organization that is not a professional society per se but is heavily involved in health-related IR is the National Library of Medicine (NLM), the U.S. government agency that not only maintains many important medical databases but also funds research and training in medical informatics. An additional group of organizations involved in IR are the companies that comprise the growing marketplace for search and retrieval products. A review of this market sector showed these companies to make up a $350 million industry in 2000 that is estimated to grow to over $2.5 billion by 2005 (Anonymous, 2001q). The leading companies at the current time include Vertity (www. verity.com), Convera (www.convera.com), and Open Text (www.opentext.com).
1.4.3 Journals The three premiere English-language information science journals are Journal of the American Society for Information Science and Technology (JASIST), Information Processing and Management (IP&M), and Information Retrieval (IR), all
1. Terms, Models, and Resources
13
of which cover a wide variety of theoretical and practical issues. JASIST is an official publication of ASIST, while the latter two are not affiliated with any society. ACM publishes the journal Transactions on Information Systems, which covers computer-science-oriented IR issues. MLA and AMIA publish their own journals, Journal of the Medical Library Association (JMLA, formerly Bulletin of the Medical Library Association) and Journal of the American Medical Informatics Association (JAMIA) respectively. Other English-language medical informatics journals that sometimes publish IR articles include Journal of Biomedical Informatics, Computers in Biology and Medicine, and Methods of Information in Medicine. General medical journals also occasionally publish IR articles. Probably the most notable of these is the British Medical Journal (BMJ), which not only publishes articles on the topic but also uses innovative Web technology to the fullest extent in electronic publishing of the journal.
1.4.4 Texts There are a variety of IR texts. As noted in the Preface, some are of the “how-to” variety for searching the medical literature (Hutchinson, 1998; Katcher, 1999). Another category of text consist of general IR books. Among the most recent are those by Baeza-Yates and Ribeiro-Neto (1999), Belew (2000), and Grossman and Frieder (1998). Some “classic” texts in the field include Salton and McGill (1983), which was written by a pioneer of “statistical” techniques in IR (Salton), and vanRijsbergen (1979), whose text is maintained on the Web (www.dcs.gla.ac.uk/ Keith/Preface.html). A text designed for those interested in actual computer implementation of IR systems (Frakes and Baeza-Yates, 1992) describes actual algorithms and provides source code in the C programming language for many of them. The book is actually a complement to other IR texts for courses oriented toward the actual implementation of IR systems. The final category of texts is those describing the operation of Web search engines (Sonnenreich and MacInta, 1998; B. Cooper, 2000; Glossbrenner and Glossbrenner, 2001).
1.4.5 Tools When the first edition of this book was written, access to IR systems was expensive and in many cases required specialized software. Now, however, a plethora of search engines are just a mouse click away in anyone’s Web browser. As will be described in Chapter 6, there are search engines that attempt to cover all topics (e.g., Google and Yahoo) as well as those that are health specific (e.g., Medical Matrix and MEDLINEplus). Many commercial medical publishers whose wares were described as CD-ROM offerings in the first edition now have subscriptionbased products on the Web. A number of these will be described in Chapter 4. For those actually wanting to experiment implementing IR systems, there are many options, some of which are given in Table 1.3, which lists a variety of IR systems, most of which run on the Unix or Linux platforms. While some are purely in the public domain, others have some licensing restrictions, such as lim-
14
I. Basic Concepts
iting use to noncommercial purposes. In addition to these systems, there are a number of commercial text retrieval packages available, some of which run on microcomputers and are fairly inexpensive.
1.5 The Internet and World Wide Web This chapter has already alluded to the profound impact on IR of the Internet and the World Wide Web. Indeed it is telling that in the first edition of this book, the Internet and the Web were described in the last chapter in the “special topics” section. In this edition, however, they are introduced here in the first chapter and discussed widely throughout. These technologies have transformed IR from a task done by information professionals and a small number of other computer-savvy individuals to one done by people of all ages, levels of education, and, increasingly, geographic locations. Few readers of this book are likely to need a description of the Internet and the Web. However, it does help to define the key terms and point out some of their attributes relevant to IR. The Internet is the global computer network that connects machines of varying sizes and capacities using a communications protocol called TCP/IP. The Web is a software application that runs on the Internet, with servers making available pages coded in the Hypertext Mark-Up Language (HTML) that are downloaded and displayed on client computers running a Web browser. In the hardware/software distinction of computers, the Internet is the hardware and the Web is the software. But the Web is more than a simple computer application; essentially, it is a platform from which virtually any computer function can be done, including database access and transactions. Because of its distributed nature (i.e., there is no central server or authority), the total size of the Web cannot be known, nor can it be indexed in its entirety, since it is constantly changing. The total size of the Web has been estimated by various authors. In 1999, Lawrence and Giles estimated the Web contained at that time 800 million pages, with a total of six terabytes (TB) of text residing on 2.8 million servers (Lawrence and Giles, 1999). This study also found that Web search engines covered less than half of all the available pages on the Web, and there was considerable lack of overlap across different search engines. Extrapolating from more recent estimates of search engine size and coverage, there are now likely well over 2 billion Web pages accessible (Glossbrenner and Glossbrenner, 2001). Travis and Broder (2001) have noted that the Web is used for purposes beyond traditional IR tasks and as such must be studied and evaluated in a broader context. They classify three general tasks for which the Web is used: • Information tasks—traditional IR seeking of information in “documents” • Navigational tasks—finding something, such as the Web site of an institution or a resource such as a database
1. Terms, Models, and Resources
15
TABLE 1.4. Web Sites Covering Aspects of Web Searching Name and description SearchEngineWatch The Megasite Project SearchEngineShowdown
Web address www.searchenginewatch.com www.lib.umich.edu/megasite www.searchengineshowdown.com
• Transactional tasks—accessing a service, data (such as an airline schedule), or shopping Table 1.4 lists some Web sites related to Web searching. SearchEngineWatch contains a wealth of information on Web search engines, such as how they work and how managers can be sure their sites are included in such online facilities. SearchEngineShowdown compares different search engines, and its results will be referenced in more detail in Chapter 7. The Megasite Project is a compendium of all major health-related search engines and their features.
1.5.1 Hypertext and Hypermedia Another way to define the Web is to call it “Internet-based hypermedia.” While the terms hypertext and hypermedia are not used much any more, understanding them can be beneficial. In both the paper and electronic information worlds, there are two ways of finding information: searching and browsing. In searching, information is sought by finding terms in an index that point to locations where material about that term may be. In books, for example, searching is done by looking up topics in the index in the back. Searching in electronic resources is carried out by means of an IR system. Browsing, on the other hand, is done by delving into the text itself, navigating to areas that are presumed to hold the content that is sought. In books, browsing is usually started by consulting the table of contents, but the reader may also follow references within the text to other portions of the book. Electronic browsing in early computer systems was difficult if not impossible but has been made easier with the advent of hypertext. The majority of the chapters of this book focus on searching as the means to find information. But computers also allow a unique form of information seeking that recognizes the nonlinearity of most text, especially scientific and technical reference information. Most paper-based resources allow some nonlinearity by referring to other portions of the text (e.g., “see Chapter 14”). Computers allow these linkages to be made explicit. Text linked in a nonlinear fashion is termed hypertext. The person most often credited with originating this notion is Vannevar Bush, who over 50 years ago proposed that the scientist of the future would carry a device called a memex that linked all his or her information (Bush, 1945). Another pioneer in the hypertext area was Ted Nelson, who implemented the first systems in the 1970s (T. Nelson, 1987). The popularity of hypertext did not take hold until the widespread
16
I. Basic Concepts
proliferation of computers that used a graphical user interface (GUI) and a mouse pointing device. These systems allowed simple and easy-to-use hypertext interfaces to be built. Although not a true hypertext system, Apple Computer’s Hypercard application, released in 1987, brought the concepts of hypertext to the mainstream. Another change brought by computers with GUIs was the ability to display nontextual information, such as images, sounds, video, and other media, often integrated with text. The term hypermedia is often used to describe systems that employ hypertext combined with other nontextual information. The Web brought Internet-based hypermedia to the mainstream. Each object in a hypertext system, whether a chunk of text, a graphic, or a sound, is called a node. For text documents, either the whole document or a portion of it may be a node. The unique feature of hypermedia is the linking of objects. The start or end of a link is called an anchor. The portion of the anchor that can be clicked on to initiate a link is called a hot spot, and is usually represented in a distinct format—underlined and/or in a different color. The hot spot may be a button or a string of text. The text within an anchor is sometimes called anchor text. The end point of the link may be a node itself or a specific location within a node. In a hypermedia system, the user reads the text or views the image in the node of interest. When a hot spot appears, the user clicks a pointing device (e.g., mouse) on it, whereupon a new node is displayed on the screen. Links may be between portions of the same or different documents. Figure 1.5 depicts the elements of a hypermedia collection. There are different types of links in a hypertext system. Conklin (1987) has noted that links can be referential or organizational. Referential links connect two points in hypertext explicitly, whereas organizational links support the hierarchical structure of the database. The equivalents of these links in books are, respectively, the cross-reference and the chapter–section–subsection hierarchy. DeRose (1989) has defined other types of hypermedia systems links. He di-
FIGURE 1.5. A model of hypermedia.
1. Terms, Models, and Resources
17
vides them into two broad categories, extensional and intensional, which represent explicit and implicit links, respectively. DeRose notes that extensional links tie hypermedia objects together in idiosyncratic if not unpredictable ways. The category of extensional links subsumes Conklin’s referential and organizational links. Intensional links, on the other hand, represent the names of nodes and indexing items they contain. These links are not explicit in the hypermedia system, though they are typically available in systems, especially if they support any type of searching capability. Marchionini (1994), who has noted that indexing documents can be thought of as a form of linking, furthermore has found that hypermedia databases are best built upon an indexing framework. There are advantages and limitations to hypermedia systems. On the positive side, they allow browsing through information in a way that is not possible in paper sources. Disparate but linked areas of text can be traversed in an instant. Since much scientific and reference text is richly linked, both hierarchically and referentially, moving quickly from one place to another is very easy. Links between documents, which may be time-consuming to pursue in a library, can be usable instantaneously if both are accessible by computer. For reference systems of certain types, hypermedia systems offer dynamic ways of viewing information. Consider as an example a hypermedia neurology textbook. Many neurological conditions, such as the abnormal gait seen in Parkinson’s disease, are much better viewed than described narratively. Furthermore, since the pharmacological treatment of this disease can be complex, this textbook may be linked to other sources, such as a pharmacology textbook that described the use and side effects of the medications in more detail than could a neurology textbook. Another valuable link could be to the primary medical literature, where clinical trials with these medications could be found. There are, however, disadvantages that hypermedia systems must address. First, even though systems provide navigational abilities, the user can still become “lost in hyperspace,” unable to get back to the main path of the information being pursued. Some of the navigational aids described earlier can assist when this happens. A more serious problem, however, is getting lost in a cognitive sense, where the amount of information itself has overwhelmed the user. The searching capabilities of IR systems can play a very complementary role in such cases, as will be described shortly. Another problem occurs with the creators of hypermedia documents, who cannot necessarily anticipate all the links users will want or need. In most systems, links are designated explicitly by human authors, whose views of what links are important may differ from those of users. One solution that has been advocated is the use of automated methods for creating additional linkages, to be described in Chapter 8.
1.5.2 The Web and Healthcare The Web has transformed health care and will likely continue to do so. A majority of households in most cities of the United States are connected to the Internet. The most connected city in the United States in 2001 was Portland, Ore-
18
I. Basic Concepts
gon, with nearly 70% of households accessing the Internet (Anonymous, 2001t). A survey of American adults who used the Internet found that 80% search for health information (Taylor, 2002). Although physicians have in the past been considered laggards in the use of the computers, they have embraced the Internet as well. A survey in late 2001 found that nearly 90% of physicians use the Internet (Taylor and Leitman, 2001). This survey found that the average physician user spends about eight hours on-line, with three of those hours devoted to clinical activities. The most common activities included researching clinical matters (90%), reading journal articles (78%), communicating with colleagues (61%), and participating in continuing medical education (45%). The physicians who were busiest seeing patients spent the most time on-line, i.e., those who put in the most hours each week caring for patients were the heaviest users. A previous survey by the American Medical Association in 2000 showed that over 70% of physicians use the Web, up from 37% in 1999 and 20% in 1997 (Anonymous, 2001a). The survey also showed the most common activities were sending and receiving non-patient-related e-mail (done by 96% of users), accessing medical information sources (86%), collecting travel information (85%), obtaining product information (77%), and communications with professional associations (68%). A survey of internists by the American College of Physicians in 1998 found that 81% had the ability to connect to the Internet from home and 65% from the office (Lacher, Nelson et al., 2000). Internists under age 50 and those with academic affiliations were more likely to use the Internet. The respondents were more likely to use the Internet for non-medical purposes (e.g., email, non-medical information) than medical purposes. Only 27% reported doing database literature searches on a weekly or daily basis. Family physicians are equally likely to own computers, with a 1998 survey showing 71% had a computer either in the office or at home (Ely, Levy et al., 1999). Rural family physicians are just as likely to use the Internet as those in urban areas (Kalsman and Acosta, 2000). The amount of health information on the Web is unknown. The Open Directory Web catalog (dmoz.org, see chapter 4) lists over 51,000 sites in its Health subdirectory. A 1999 estimate suggested that there were at least 100,000 healthrelated Web sites (Eysenbach, Su et al., 1999). As will be seen in Chapter 4, virtually all types of health information, from journals to textbooks to patient handouts, are now available on the Web.
1.5.3 Models of the World Wide Web Research about the Web has defined various models for its structure. Most of these have emerged from analysis of the Web’s link structure. Broder et al. (2000) have exhaustively studied the mathematical properties of Web links, the results of which will be described in more detail in Chapter 8. Their analysis finds the Web can be divided into four roughly equal parts, as shown in Figure 1.6a: 1. The “strongly connected core” (SCC) of pages that are highly connected to each other 2. IN pages, which reach the SCC but cannot be reached from it
1. Terms, Models, and Resources
19
3. OUT pages, which have links from the SCC but do not link back to it 4. Tendrils, which are pages that have no links to or from the other components Kleinberg and Lawrence (2001) have proposed an alternative model for the Web, noting that it consists mainly of hubs and authorities (see Figure 1.6b). Hubs are catalogs and resource lists of pages that point to authorities on given topics. They also observe that Web communities can be discerned by the observation that pages within such communities tend to be more densely linked to each other than to other pages (see Figure 1.6c). Kleinberg and Lawrence (2001) have proposed an alternative model for the Web, noting that the Web consists mainly on hubs and authorities (see Figure 1.6b). Hubs are catalogs and resource lists of pages that point to authorities on given topics. They also observe that Web communities can
a
b FIGURE 1.6. Models of the World Wide Web. (a) The four major groups of pages. (Broder, Kumar et al., 2000.) (b) Hubs and authorities. (Kleinberg and Lawrence, 2001.) Continued
20
I. Basic Concepts
c
d FIGURE 1.6. (Continued) (c) Web communities. (Kleinberg and Lawrence, 2001.) (d) the visible vs. invisible Web. (Sherman and Price, 2001.)
be discerned by the observation that pages within such communities tend to be more densely linked to each other than other pages (see Figure 1.6c). Another view of the Web is to divide into the visible and invisible Web, as depicted in Figure 1.6d (Sherman and Price, 2001). The former contains all of the Web content that can be found by fixed or static URLs, while the latter contains content “hidden” behind password-protected sites or in databases (Anonymous, 2002d). In general, the visible Web is searched via general Web search engines. On the other hand, most of the commercial on-line databases to be described in later chapters reside on the invisible Web.
1.6 A Sample Document Database for Examples The final introductory note in this chapter concerns the sample database in Appendix 1, which is used for both examples and exercises throughout the text. The database contains 10 “documents” on hypertension and related topics. Each is
1. Terms, Models, and Resources
21
relatively simple, containing just a title, a sentence of text, and some manually assigned indexing terms. The database is deliberately small, to make the examples and exercises tenable. Using the database does not require any sophisticated medical knowledge. The documents are designed more to demonstrate issues in IR than to impart any specific medical knowledge. Subsequent appendices provide indexed representations of the document database. Appendix 2 contains an inverted file of the words in the database, which will be described in Chapter 5. Appendix 3 contains document vectors for the database, derived by a process also to be introduced in Chapter 5. Appendix 4 contains an inverted file of the subject headings for the database.
Chapter 2 Health and Biomedical Information
Chapter 1 defined the basic terminology of information retrieval (IR) and presented some models of the use of IR systems. Before proceeding with the details of IR systems, however, it is worthwhile to step back and consider the more fundamental aspects of information, especially as it is used in the health and biomedical domain. In this chapter, the topic of information itself will be explored by looking at what it consists of and how it is produced and used. Consideration of this topic allows a better understanding of the roles as well as the limitations of IR systems.
2.1 What Is Information? The notion of information is viewed differently by different people. Webster’s dictionary (Gove, 1976) provides seven different definitions of information. These include the following: • “The communication or reception of knowledge or intelligence” • “Facts and figures ready for communication or use as distinguished from those incorporated in a formally organized branch of knowledge” • “The process by which the form of an object of knowledge is impressed upon the apprehending mind so as to bring about the state of knowing” • “A numerical quantity that measures the uncertainty of in the outcome of an experiment to be performed” Pao (1989) states that information can be viewed in many ways. She notes that some related it to a “mysterious act of the intellect,” while others view it as a commodity bought and sold on demand. Others still view it as a utility, supplied on a pay-as-you-go basis. Whether a commodity or utility, however, information is unlike other forms of capital because it is often in surplus, with a great effort going into its management. Others have attempted to define information by comparing it on a spectrum containing data, information, and knowledge (Blum, 1984). Data consists of the observations and measurements made about the world. Information, on the other hand, is data brought together in aggregate to demonstrate facts. Knowledge is 22
2. Health and Biomedical Information
23
what is learned from the data and information, and what can be applied in new situations to understand the world. Whatever the definition of information, its importance cannot be overemphasized. This is truly the information age, where information (or access to it) is an indispensable resource, as important as human or capital resources. Most corporations have a chief information officer (CIO) who wields great power and responsibility. Some of best-known very wealthy Americans (e.g., Bill Gates, Paul Allen, Larry Ellison, Ross Perot) each made their fortunes in the information industry. Information is important not only to managers, but to workers as well, particularly those who are professionals. Many healthcare professionals spend a significant proportion of their time acquiring, managing, and utilizing information. Two studies have shown that healthcare personnel, including physicians, spend about one-third of their time handling and using information (Jydstrup and Gross, 1966; Mamlin and Baker, 1973), while the costs of personal and professional communication have been estimated to consume over 35% of the average hospital’s budget (Richart, 1970). Although these studies are several decades old, and their results have not been updated, it is likely that the time and cost devoted to managing information in health care are at least as large now and perhaps larger.
2.2 Theories of Information One way of understanding a complex concept like information is to develop theories about it. In particular, one can develop models for the generation, transmission, and use of information. This section will explore some of the different theories of information, which provide different ways to view information. More details in all the theoretical aspects of information can be found in a book by Losee (1990).
2.2.1 Shannon and Weaver The scientists generally credited with the origin of information theory are Claude Shannon and Warner Weaver (1949). Shannon was an engineer, most concerned with the transmission of information over telephone lines. His theory, therefore, viewed information as a signal transmitting across a channel. His major concerns were with coding and decoding the information as well as minimizing transmission noise. Weaver, on the other hand, was more focused on the meaning of information, and how that meaning was communicated. Figure 2.1 depicts Shannon and Weaver’s model of communication. In information communication, the goal is to transfer information from the source to the destination. For the information to be transmitted, it must be encoded and sent by the transmitter to a channel, which is the medium that transmits the message to the destination. Before arriving, however, it must be captured and decoded by the receiver. In electronic means of communication, the signal is composed of
24
I. Basic Concepts
FIGURE 2.1. Shannon and Weaver’s model of communication. (Shannon and Weaver, 1949. Courtesy of University of Illinois Press.)
either waves (i.e., the analog signals of telephone or radio waves) or binary bits (i.e., in a digital computer). From the standpoint of the sender, the goal is to deliver information as efficiently as possible. Therefore, information is a measure of uncertainty or entropy. Shannon actually defines this quantitatively. The simplest form of this expression is
1 I log2 log2(p) p
(1)
where p is the probability of a message occurring. If base two is used for the logarithm, then information can be measured in terms of bits. An alternative view is that the quantity of information is measured by q, the logarithm of the number of different forms that a message can possibly take. In this case, the measurement of information is expressed as I log2(q)
(2)
Obviously messages of greater length have a higher number of possible forms. As an example of the measure of information, consider the transfer of a single letter. If each letter has an equal probability of occurring, then the chance of any one letter occurring is 1/26. The information contained in one of these letters, therefore, is log2(1/26) 4.7 bits. This can be alternatively expressed as having 26 different forms, with the information contained in a letter as log2(26) 4.7 bits. Similarly, the information in a coin flip is log2(1/2) log2(2) 1 bit. Therefore, there is more information in a single letter than a coin flip. These examples indicate that the more likely a message is to occur, the less information it contains. Shannon’s measure is clearly valuable in the myriad of engineering problems in transmitting messages across electronic media. But one may question whether it has any other value, especially related to the transmission of medical information. In fact it does, and Heckerling (1990) used Shannon’s theory to demonstrate, based on prior probabilities of disease, that the information in diagnostic tests is often insufficient to overcome diagnostic uncertainty.
2. Health and Biomedical Information
25
Weaver, as mentioned before, was more concerned with the transmission of meaning (Shannon and Weaver, 1949). He noted that Shannon’s view of communication was only one of three levels of the communication problem, and that the other two levels also must be considered in the communication of information. These levels were as follows: 1. The technical level—issues of engineering, such as how to efficiently encode information and move it across a channel with a minimum of noise. 2. The semantic level—issues of conveying meaning, such as whether the destination understands what the source is communicating. 3. The effectiveness level—issues of whether information has the desired effect at the destination level. A well-engineered communication system may have good semantic representation, but if it does not provide proper behavioral outcomes at other end, then the system is not effective.
2.2.2 Other Models of Information Many others have attempted to refine and extend Shannon and Weaver’s model. Bar Hillel and Carnap (1953) added a layer of semantics to the measurement of information. They noted that information does not consist just of isolated bits; it actually contains objects with relationships (or predicates) between them. The objects and relationships can be encoded in logical forms, and therefore, information can be defined as the set of all statements that can be logically excluded from a message. In other words, information increases as statements become more precise. Belis and Guiasi (1968) worked at Weaver’s effectiveness level by adding values of utility of messages for both the sender and receiver. Certainly a message over a paramedic’s radio that a patient in cardiac arrest is on the way to the Emergency Room has a great deal more utility for sender and receiver than one announcing that someone with a fractured wrist is coming. Belis and Guiasi added factors based on utilities of these types to Shannon’s original equations.
2.2.3 Information Theory and Information Science Whereas information science is concerned with these theoretical notions of information and communication, most work has a more practical basis. In particular, information scientists are most concerned with written communication, which plays an important role in the dissemination of historical events as well as scholarly ideas. Information scientists focus on written information, from both archival and retrieval perspectives. Written information has been viewed not only from theoretical perspectives, such as the measuring of the “productivity” of scientists, but also from practical standpoints, such as deciding what books and journals to put on library shelves and, more recently, how to build and disseminate IR systems.
26
I. Basic Concepts
2.3 Properties of Scientific Information As just noted, information scientists study many facets of information but are usually most concerned with the written form. In the course of this work, they have identified many properties of information. Since the focus of information science is also usually based on scholarly and scientific information, most of these properties turn out to be quite pertinent to health information. This section will explore the growth, obsolescence, fragmentation, and linkage of scientific information.
2.3.1 Growth Scientific information has been growing at an exponential level for several centuries and shows no signs of abating. Price (1963) found that from the first known scientific journals in the 1600s, the doubling time of the scientific literature was about 15 years. Pao (1989) noted that Price’s model predicted an accumulation of 2.3 million scientific papers by 1977 (based on an initial paper in 1660), which was very close to the 2.2 million documents that were indexed by members of the National Federation of Abstracting and Indexing Services in that year (Molyneux, 1989). In the medical field, the growth of scientific information has had profound consequences. It is cited as one of the reasons why physicians choose subspecialties over primary care (Petersdorf, 1989), which in turn is known to be a cause of escalating healthcare costs. Indeed, each year over 300,000 references are added to the MEDLINE database. While no practitioner will read all those articles, there is clearly a large amount of scientific information any individual will not acquire, potentially compromising patient care. Will the exponential growth in scientific information continue? There are some practical issues, such as whether there will be enough trees to produce the paper to print the increasing numbers of journals on, although as trees become more scarce and electronic media more developed and affordable, there could just be a shift from print to electronic publication. Another factor that may slow the growth of scientific information is the diminished funding of science by government agencies. With fewer scientists, especially those funded by public means, who are more likely to publish in the scientific literature, there could be a leveling off of the growth of scientific literature. But even if the rate of information growth slows, there will still be plenty of new information for scientists and professionals to assimilate.
2.3.2 Obsolescence Despite its exponential growth in size, another property of scientific information is that it becomes obsolete, sometimes rather quickly. Newer literature not only reports on more recent experiments, but is also more likely to provide a more
2. Health and Biomedical Information
27
up-to-date list of citations to recent work. Furthermore, new experimental findings often cause underlying views of a topic to change over time. As new results are obtained, older experiments may be viewed in a different light. A classic example of how views change over time because of new experimental results is seen with respect to the role of serum cholesterol in the heart disease (Littenberg, 1992). When the link between serum cholesterol and coronary artery disease was first discovered, there was no evidence that lowering the cholesterol level was beneficial. But as experimental studies began to demonstrate that benefits could occur, the earlier beliefs were displaced (though such assertions remained in the literature in the form of outdated papers). Even more recently, however, it has become clear that not everyone benefits from lowering serum cholesterol. In particular, for primary prevention of heart disease, many individuals must be treated to obtain benefit in a relatively few, and those likely to benefit cannot be predicted. Some phenomena, such as medical diseases, change over time. For example, the presentation of many infectious diseases has changed drastically since the beginning of the antibiotic era, while the incidence of coronary artery disease continues to decline. Even phenomena from chemistry and physics, which themselves do not change, are seen in a different light when methods of measuring and detecting them are refined. These changes over time indicate that more recent literature is clearly advantageous and that some information becomes obsolete. The actual rate of information obsolescence varies by field. Price (1963) found that half of all references cited in chemistry papers were less than 8 years old, while half of those in physics papers were less than 5 years old. This type of observation is not just theoretical; it has practical implications for those designing libraries and IR systems. For the former, there are issues of shelves to build and librarians to hire, while for the latter there are issues of how much data to store and maintain. Another aspect of the information obsolescence problem is the long lead time in the dissemination of information. A common dictum in the healthcare field is that textbooks are out of date the moment they are published. As it turns out, they may be out of date before the authors even sit down to write them. Antman et al. (1992) have shown that the information of experts as disseminated in the medical textbooks, review articles, and practice recommendations they produce often lags far behind the edge of accumulated knowledge. As a result, important advances go unmentioned and/or ineffective treatments are still advocated. Of course, just because literature is old does not mean it is obsolete. When a healthy volunteer died after being given hexamethonium in a recent trial, it was noted that this compound’s toxicity had been documented in earlier literature (McLellan, 2001). Since, however, the pertinent literature was published before the advent of MEDLINE in the 1950s and early 1960s, it was not included in the MEDLINE database. This case led to the call for more systematic searching of older literature by researchers.
28
I. Basic Concepts
2.3.3 Fragmentation Ziman (1969) noted another property of the scientific literature, namely, that a single paper typically reports only on one experiment that is just small part of overall picture. Ziman observed that the scientific literature is mainly written for scientists to communicate with their peers and thus presumes a basic understanding of the concepts in the field. Ziman also maintained that the literature is not only fragmented, but also derivative, in that it relies heavily on past work and is edited, which provides a quality control mechanism. Part of the reason for the fragmentation of the scientific literature is the scientist’s desire to “seed” his or her work in many different journals, where it may be seen by a larger diversity of readers. In addition, the academic promotion and tenure process encourages scientists in academic settings to “publish or perish.” A common quip among academicians is to slice results into “minimal publishable units” so that the maximum number of publications can be obtained for a body of work. The extent to which this is done is not clear, but to the extent that the practice exists, more fragmentation of the scientific literature is a result.
2.3.4 Linkage A final property of scientific information is linkage, which occurs via the citation. The study of citations in scientific writing is a field unto itself called bibliometrics. This field is important in a number of ways, such as measuring the importance of individual contributions in science, indicating the likely places to find information on a given topic, and, as will be discussed in Chapter 10, offering potential ways to enhance IR systems. 2.3.4.1 Citations The bibliography is an important part of a scientific paper. It provides background information, showing the work that has come before and indicating what research has motivated the current work. It also shows that the author is aware of others working in the field. Authors also use citations to substantiate claims. Thus, a scientific paper on a new treatment for a disease will usually cite papers describing the disease, its human toll, and the success of existing therapies. An author arguing for a certain experimental approach or a new type of therapy, for example, may cite evidence from basic science or other work to provide rationale for the novel approach. Citations can be viewed as a network, or as a directed acyclic graph. Although reasons for citation can often be obtuse (i.e., a medical paper may cite a statistical paper for a description of an uncommon method being used), networks can give a general indication of subject relationship. One of the early workers in bibliometrics was Garfield (1964), who originated the Science Citation Index (Institute for Scientific Information, Philadelphia), a publication that lists all citations of every scientific paper in journals. Being cited is important for scientists. Academic promotion and tenure com-
2. Health and Biomedical Information
29
mittees look at, among other things, how widely cited an individual’s work is, as do those who review the work of candidates for grant funding. In a study of several different fields, Price (1965) found that in certain fields, half of all citations formed a core of a small number of papers representing authors and publications with major influence on a given subject. An example of this is seen in information science. Virtually any writing on the topic of automated indexing, which will be covered in several subsequent chapters, contains one or more citations to the work of the late Gerard Salton, whose pioneering work in this area continues to influence one field strongly. Adam (2000) recently reviewed the pros and cons of using citation indexes to judge the work of scientists. As mentioned, the field of bibliometrics is concerned with measuring the individual contributions in science as well as the distribution of publications on topics. This field has also generated two well-known laws that deal with author productivity and subject dispersion in journals, Lotka’s law and Bradford’s law, respectively. It has also developed measures that attempt to assess the impact of journals. 2.3.4.2 Author Productivity: Lotka’s Law Most readers who work in scientific fields know that there is a small core of authors who produce a large number of publications. A mathematical relationship describing this has been described by Lotka and verified experimentally by Pao (1986). Lotka’s law states if x is the number of publications by a scientist in a field and y is the number of authors who produce x publications each, then: xn * y C
(3)
where C is a constant. For scientific fields, the value for n is usually near 2.0. Thus in scientific fields, the square of the number of papers published by a given author is inversely proportional to the number of authors who produce that number of papers. Lotka’s law is also known as the inverse square law of scientific productivity (Pao, 1986). If the number of single-paper authors is 100, then number of authors producing 2 papers is 100/22 25 and the number of authors producing 3 papers is 100/32 11, etc. In general, 10% of the authors in a field produce half the literature in a field, while 75% produce less than 25% of the literature. 2.3.4.3 Subject Dispersion: Bradford’s Law Bradford (1948) observed, and several others have verified (Urquhart and Bunn, 1959; Trueswell, 1969; Self, Filardo et al., 1989), a phenomenon that occurs when the names of journals with articles on a topic are arranged by how many articles on that topic each publication contains. The journals tend to divide into a nucleus of a small number of journals followed by zones containing n, n2, n3, etc. journals with approximately the same number of articles. This observation is known as Bradford’s law of scattering. Its implication is that as a scientific field grows, its literature becomes increasingly scattered and difficult to organize. But Bradford’s law also indicates that most articles on a given topic are found
30
I. Basic Concepts
in a core of journals. This fact is of importance to libraries, which must balance the goal of comprehensiveness with space and monetary constraints. Bradford’s law has been found to apply to other phenonoma in IR, such as distribution of query topics to a database (Bates, 2002). Pao (1989) demonstrated that dispersion occurs according to Bradford’s law in the computational musicology literature. She found that the top eight journals in the field (in terms of numbers of papers on the subject) produced 26% of literature. The top 26 journals produced half the literature, with the remaining half in 248 other journals. Pao divided these into zones of equal numbers of articles, with the “Bradford multiplier” holding constant at 1.66. This phenomenon has also been demonstrated more recently with the literature on the acquired immunodeficiency syndrome (AIDS) (Self, Filardo et al., 1989). In 1982, shortly after the disease was identified, only 14 journals had literature on AIDS. By 1987, this had grown to over 1200. The authors plotted the cumulative percent of journal titles versus journal articles for AIDS (Figure 2.2) and found a Bradford distribution, with the first third of articles in 15 journals, the second third in 123 journals (1.5 8.2), and the final third in 1,023 journals (15 8.22). Another implication of both Lotka’s law and Bradford’s law is that scientists and journals that are already successful in writing and attracting articles are likely to continue to be so in the future. In Section 2.5.2 on peer review, aspects of the scientific publishing process that indicate why successful scientists in a field continue their good fortune are explored. 2.3.4.4 Journal Importance: Impact Factor Linkage information can also be used to measure the importance of journals, based on the impact factor (IF), which was introduced by Garfield (1994). The
FIGURE 2.2. The Bradford distribution for articles on AIDS. (From Self, Filardo et al., 1989, Acquired immunodeficiency syndrome (AIDS) and the epidemic growth of its literature. Scientometrics, 17:49–60. Reprinted with kind permission of Kluwer Academic Publishers.)
2. Health and Biomedical Information
31
TABLE 2.1. Impact Factors of Selected Journals from 2000 Journal name
Impact factor
Major medical journals New England Journal of Medicine JAMA Lancet Annual Reviews of Medicine Annals of Internal Medicine Archives of Internal Medicine American Journal of Medicine British Medical Journal Medicine Amyloid Medical informatics journals Journal of the American Medical Informatics Association Medical Decision Making Artificial Intelligence in Medicine Statistics in Medicine Medical Informatics and the Internet in Medicine Medical & Biological Engineering & Computing International Journal of Technology Assessment Methods of Information in Medicine IEEE Engineering in Medicine & Biology Magazine International Journal of Medical Informatics
29.5 15.4 10.2 9.89 9.83 6.10 5.96 5.33 4.62 2.96 3.09 2.15 1.79 1.72 1.18 1.00 0.98 0.93 0.86 0.70
Source: Institute for Scientific Information, www.isi.com.
assumption underlying the IF is that the quantity of citations of a journal’s papers is associated with that journal’s importance. As with other linkage measures, IF can be used to make decisions about the journals to which a library should subscribe. IF is usually measured with a formula that is the ratio of the number of citations in a given time period to the number of articles published. The formula also has a variant that adjusts for self-citations within a journal. The IF for current year citations of articles published over the last two years would thus be: IF
Citations in current year to articles published in prior to years Number of articles published in prior to years
(4)
Table 2.1 shows the IF for the top 10 journals in general medicine and in medical informatics. Not everyone agrees that IF is the best determinant of journal quality. West (1996) points out that the importance of scientific articles is influenced by other factors, such as the nature of the underlying research, variations in the number of references different publications include, journal editorial policies that limit the number of references per article, different-sized readerships, which may lead to differences related to audience size, scientists’ conformity in often citing papers that are currently cited, authors’ tendencies to cite their own work, and referees’ tendencies to recommend inclusion of references to their work. However,
32
I. Basic Concepts
Lee et al. (2002) have found that IF and other measures of journal quality (e.g., citation rate, acceptance rate, listing in MEDLINE, circulation) are associated with higher methodological quality of a given publication. Likewise, Callaham et al. (2002) have found IF to be more important in the subsequent citing of a scientific paper than even the quality of the study methodology. Just as scientific literature is linked, so is information on the Web. Indeed, Ingwersen (1998) has proposed a Web impact factor measure that replaces citations in the IF formula with Web links and number of articles with number of pages on a site, with a correction for links to within the same site. The Web search engine most often associated with exploiting linkage information is Google (www.google.com), which uses a technique called page rank to give higher ranking to pages that have more links to them (Brin and Page, 1998). The rationale is that these pages are more likely to represent authoritative pages. Page rank will be described in detail in Chapter 6.
2.4 A Classification of Textual Health Information Now that some basic theories and properties of information have been described, attention can be turned to the type of information that is the focus of this book, textual health information. It is useful to classify it, since not only are varying types used differently, but alternative procedures are applied to its organization and retrieval. Table 2.2 lists a classification of textual health information. Patient-specific information applies to individual patients. Its purpose is to tell healthcare providers, administrators, and researchers about the health and disease of a patient. This information comprises the patient’s medical record. Patient-specific data can be either structured, as in a laboratory value or vital sign measurement, or in the form of free (narrative) text. Of course, many notes and reports in the medical record contain both structured and narrative text, such as the history and physical report, which contains the vital signs and laboratory values. For the most part, this book does not address patient-specific information, although Chapter 11 covers the processing of clinical narrative text. As will be seen, the goals and procedures in the processing of such text are often different from other types of medical text. The second major category of health information is knowledge-based information. This is information that has been derived and organized from observational
TABLE 2.2. A Classification of Textual Health Information Patient-specific information Structured—lab results, vital signs Narrative—history and physical, progress note, radiology report Knowledge-based information Primary—original research (in journals, books, reports, etc.) Secondary—summaries of research (in review articles, books, practice guidelines, etc.)
2. Health and Biomedical Information
33
or experimental research. In the case of clinical research, this information provides clinicians, administrators, and researchers with knowledge derived from experiments and observations, which can then be applied to individual patients. This information is most commonly provided in books and journals but can take a wide variety of other forms, including computerized media. Of course, some patientspecific information does make it into knowledge-based information sources, but with a different purpose. For example, a case report in a medical journal does not assist the patient being reported on, but rather serves as a vehicle for showing the knowledge gained from the case with other practitioners. Knowledge-based information can be subdivided into two categories. Primary knowledge-based information (also called primary literature) is original research that appears in journals, books, reports, and other sources. This type of information reports the initial discovery of health knowledge, usually with original data. Revisiting the earlier serum cholesterol and heart disease example, an instance of primary literature could include a discovery of the pathophysiological process by which cholesterol is implicated in heart disease, a clinical trial showing a certain therapy to be of benefit in lowering it, a cost–benefit analysis that shows which portion of the population is likely to best benefit from treatment, or a metaanalysis combining all the original studies evaluating one or more therapies. Secondary knowledge-based information consists of the writing that reviews, condenses, and/or synthesizes the primary literature. The most common examples of this type of literature are books, monographs, and review articles in journals and other publications. Secondary literature also includes opinion-based writing such as editorials and position or policy papers. It also encompasses clinical practice guidelines, systematic reviews, and health information on Web pages. In addition, it includes the plethora of pocket-sized manuals that are a staple for practitioners in many professional fields. As will be seen later, secondary literature is the most common type of literature used by physicians.
2.5 Production of Health Information Since the main focus of this book is on indexing and retrieval of knowledgebased information, the remainder of this chapter will focus on that type of information (except for Chapter 11, where patient-specific information is revisited, but only in the context of processing the text-based variety). This section covers the production of health information, from the original studies and their peer review for publication to their summarization in the secondary literature.
2.5.1 The Generation of Scientific Information How is scientific information generated? It ultimately begins with scientists themselves, who make and record observations, whether in the laboratory or in the real world. These observations are then submitted for publication in the primary literature. If they pass the test of peer review, as described shortly, they are pub-
34
I. Basic Concepts
lished. Eventually, the writer of a book or review article may deem them important enough to be included in a secondary literature publication. The best-known description of the scientific process is Thomas Kuhn’s The Structure of Scientific Revolutions (Kuhn, 1962). Kuhn noted that science proceeds in evolutions and revolutions. In the evolutionary phase of a science, there is a stable, accepted paradigm. In fact, Kuhn argued, a field cannot be a science until there is such a paradigm that lends itself to common interpretation and agreement on certain facts. A science evolves as experiments and other observations are performed and interpreted under the accepted paradigm. This science is advanced by publication in peer-reviewed journals. In the revolutionary phase, however, evidence in conflict with the accepted paradigm mounts until it overwhelmingly goes against the paradigm, overturning it. The classic example of this described by Kuhn came from the work of Copernicus, who contributed little actual data to astronomy but showed how the astronomical observations of others fit so much better under a new paradigm in which the planets revolve around the sun rather than around the earth. Scientists communicate their findings and theories via the primary literature. When they complete new research, they write up the methods, results, and conclusions in a paper and submit it to a journal for publication. The paper is reviewed by peer scientists, who decide whether it is worthy of publication. If not, it is either rejected outright or returned for revision. The goal of this process is to ensure that the appropriate experimental methods were used, that the findings represent a new and informative contribution to the field, and that the conclusions are justified by the results. Of course, what is acceptable for publication varies with the scope of the journal. Journals covering basic biomedical science (e.g., Cell) will tend to publish papers focusing on laboratory-based research that is likely to focus on mechanisms of diseases and treatments, whereas clinical journals (e.g., Journal of the American Medical Association will tend to publish reports of large clinical trials and other studies pertinent to providing clinical care. Specialized journals are more likely to publish preliminary or exploratory studies. Another phenomenon is the tendency of technology-oriented journals (e.g., those in medical informatics and telemedicine) to publish evaluation studies that may be less rigorous and/or more preliminary than work that would be found in a clinical journal. Hersh et al. (2001) have found, for example, that many studies published in the telemedicine literature tend to couple descriptions of the development and use of the technology with evaluative studies that are poor in their methodological quality. The peer-reviewed journal is not the only vehicle for publication of original science. Other forums for publication include the following: 1. Conference proceedings—usually peer reviewed; publish either full papers or just abstracts 2. Technical reports—may be peer reviewed; frequently provide more detail than journal papers 3. Books—may be peer reviewed
2. Health and Biomedical Information
35
In general, however, nonjournal primary literature does not carry the scientific esteem accorded to journal literature. Many authors, such as Ziman (1969), have noted that the scientific method is the best method humans have devised for discerning the truth about their world. While a number of limitations with the peer review process and the scientific literature itself will be seen in ensuing sections, the present author agrees that there is no better method for understanding and manipulating the phenomena of the world than the scientific method. Flaws in science are usually due more to flaws in scientists and the experiments they devise than to the scientific method itself. To standardize the publishing process, the editors of the major medical journals have formed a committee (International Committee of Medical Journal Editors, ICMJE, www.icjme.org) to provide general recommendations on the submission of manuscripts to journals. They have defined the so-called Vancouver format for publication style, which journals will agree to accept upon submission even if their own formats vary and will require editing later. Equally important, however, they have defined a number of additional requirements and other statements for biomedical publishing (Anonymous, 2001o): • Redundant or duplicate publication occurs when there is substantial overlap with an item already published. One form of redundant publication that is generally acceptable is the publication of a paper whose preliminary report was presented as a poster or abstract at a scientific meeting. Acceptable instances of secondary publication of a paper include publication in a different language or in a journal aimed at different readers. In general, editors of the original journals must consent to such publication, and the prior appearance of the material should be acknowledged in the secondary paper. • Authorship should be given only to those who make substantial contribution to study conception or design and data acquisition, analysis, or interpretation; drafting of the article or revising it critically for important intellectual content; and giving final approval for publication. • A peer-reviewed journal is one that has most of its articles reviewed by experts who are not part of the editorial staff. • While journal owners have the right to hire and fire editors, the latter must have complete freedom to exercise their editorial judgment. • All conflicts of interest from authors, reviewers, and editors must be disclosed. Financial support from a commercial source is not a reason for disqualification from publishing, but it must be properly attributed. • Advertising must be kept distinguishable from editorial content and must not be allowed to influence it. • Competing manuscripts based on the same study should generally be discouraged, and this requires careful intervention by editors to determine an appropriate course of action. A recent editorial from the ICMJE (Davidoff, DeAngelis et al., 2001) elucidated further the issue of conflicts of interest and authorship. Noting recent instances of the sponsors of clinical trials attempting to influence or even withhold publi-
36
I. Basic Concepts
cation (e.g., Rennie, 1997), they advocated that sponsors be allowed input into the publishing process but that authors be considered to own the intellectual property of the publication and thus have the final say.
2.5.2 Study of the Peer Review Process Although peer review had its origins in the nineteenth century, it did not achieve widespread use until the mid-twentieth century (Burnham, 1990). As noted earlier, the goal of peer review is to serve as a quality filter to the scientific literature. Theoretically, only research based on sound scientific methodology will be published. In reality, of course, the picture is more complicated. Awareness that what constitutes acceptable research may vary based on the quality or scope of the journal has led to the realization by some that the peer review process does not so much determine whether a paper will be published as where it will be published. Most journals have a two-phase review process. The manuscript is usually reviewed initially by the editor or an associate editor to determine whether it fits the scope of the journal and whether there are any obvious flaws in the work. If the manuscript passes this process, it is sent out for formal peer review. The results of the peer review process vary widely. JAMA’s Instructions for Authors (jama.ama-assn.org/info/auinst.html) states that only 10% of submitted papers are accepted. Among smaller journals, the rate of acceptance varies widely, from 13 to 91% (Hargens, 1990). The recommendations of acceptance or rejection by reviewers vary equally widely. The peer review process serves other purposes besides assessing the quality of science. For example, review of papers by peers also leads to improvement in the reporting of results and conclusions. Purcell et al. (1998) found that peer review identified five types of problem with papers: too much information, too little information, inaccurate information, misplaced information, and structural problems. These can be corrected during the editorial process. Even if the paper is not accepted, peer review can be beneficial to authors who are likely to implement suggested changes before submitting their material to a different journal (Garfunkel, Lawson et al., 1990). Some research has attempted to identify the characteristics of good peer reviewers. The only consistent factor associated with high-quality reviewing in all studies done has been younger age. One writer (Stossel, 1985) found that the best reviews came from faculty of junior academic status, while another (Evans, McNutt et al., 1993) showed that the best reviews came from younger faculty working at top academic institutions or who were known to the editors. Nylenna et al. (1994) found that younger referees with more experience refereeing had a better chance at detecting flaws in problematic papers. Black et al. (1998) also showed that younger reviewers and those who had training in epidemiology or statistics produced better reviews, though their study found in general that reviewer characteristics could explain only 8% of the variation in review quality. Callaham et al. (1998) found that subjective ratings of reviewers, correlated with the ability to detect flaws in manuscripts.
2. Health and Biomedical Information
37
Another question relating to peer review is whether the process is biased by institutional prestige or geographic location of the reviewers. Garfunkel et al. (1994) observed that institutional prestige did not influence acceptance of major manuscripts at the Journal of Pediatrics, though it was found to correlate positively with brief reports. Link (1998) noted that U.S.-based and non-U.S.-based reviewers had comparable rates of recommendation of acceptance when reviewing for the journal Gastroenterology. Other research has addressed ways of improving the process so that peer reviewers can do their job more effectively. One intervention that has been shown to be effective is to provide reviewers with abstracts and preprints of related papers (Hatch and Goodman, 1998). Another intervention has been to blind reviewers to the identity of the authors of the paper being reviewed. It is presumed this reduces bias in reviews. Of course, complete blinding of reviewers can be difficult. Even if author identities are stripped from manuscripts, references to past work, location of the study, funding source, or other aspects may reveal their identity. Indeed, one study of masking found that it was successful only 68% of the time, and less often for well-known authors (Justice, Cho et al., 1998). While one study has shown that blinding produced better reviews (McNutt, Evans et al., 1990), several more recent studies have provided evidence that it does not (Godlee, Gale et al., 1998; Justice, Cho et al., 1998; van Rooyen, Godlee et al., 1998). Two of these studies also assessed whether unmasking the identity of reviewers to authors led to higher quality reviews, with both showing that it did not (Godlee, Gale et al., 1998; van Rooyen, Godlee et al., 1998). These studies have led the British Medical Journal (BMJ) to adopt an “open review” policy, where the names of peer reviewers are disclosed to authors. Most editors of peer-reviewed journals are not trained in editorial practices, tending to come from the ranks of accomplished academicians and clinicians. Most of these individuals are likely to have had prior experience with publishing and/or peer reviewing, however. While the editors of the major journals devote full-time effort to editing, editors of specialty journals usually devote parttime effort. Most specialist clinical medical journals tend to be edited by practicing clinicians who are self-taught, part-time editors (Garrow, Butterfield et al., 1998). Of course, the peer review process is not without imperfections. Even the relatively sterile world of science is susceptible to human tendencies toward competitiveness, arrogance, and even dishonesty. The ensuing discussion of the problems should inform the reader of the limitations of the process, rather than leading to rejection of its merits. It was seen that Lotka’s law indicates that current success in a scientific field is a good predictor of future success. Certainly those who have already produced good work are likely to continue to do so. However, there may also be an unfair bias toward those who are already successful. Evidence for this was shown most strikingly in an experiment by two psychologists, Peters and Ceci (1982), who took 12 psychology articles that were already published in prestigious psychol-
38
I. Basic Concepts
ogy journals and resubmitted them with different author names and slight rewording of titles. These articles were eventually disseminated to 38 reviewers, only three (8%) of whom detected that the article was a resubmission. For the remaining nine articles, 16 of 18 reviewers recommended against acceptance, and all but one of the articles were rejected for publication. The most common reason for rejection of these previously accepted papers was “serious methodologic flaw.” Peters and Ceci’s paper was published with a large number of rebuttals from various psychology journal editors and other psychologists. A variety of limitations of the study were proposed, including its small sample size and possibility that the results represented a regression toward the mean. Peters and Ceci acknowledged the small sample size but refuted the assertion that the results were due to chance or some other statistical anomaly. Clearly the rejection of findings previously deemed suitable for publication but now described by unknown authors indicated that already esteemed authors have a better chance of publication. The findings of Peters and Ceci are not the only ones to indicate problems with the peer review process. Ingelfinger (1974), a former editor of the New England Journal of Medicine (NEJM), noted that for nearly 500 consecutive papers submitted to that journal, the concordance between the two reviewers for each article was only slightly better than chance. Among the problems he cited in the peer review process were reviewers wrongly assumed to be knowledgeable on a particular topic on the basis of their stature in the field as a whole, reviewers not skilled in detecting poor writing that obscured the quality of an underlying message, and reviewer bias toward or against others in the individual’s field. A more recent study showed that in secondary review of accepted manuscripts, although there was high concordance among reviewers for accepting or rejecting the paper, there was a wide divergence in the identification of problems deemed to warrant further revision (Garfunkel, Ulshen et al., 1990). Another study assessing consistency of peer reviewers focused on studies in medical education (Bordage, 2001). For 151 papers submitted to a medical education conference, 19% were recommended for acceptance unanimously by an average of 4.1 reviews. Of the remainder of the papers, rejection was recommended by 15% unanimously, 34% by a majority of the reviewers, 11% by half the reviewers, and 40% by less than half of the reviewers. The top reasons for rejection included statistical problems, overinterpretation of results, problems with instrumentation, inadequate or biased sample size, writing that was difficult to follow, insufficiently detailed statement of the research problem, inaccurate or inconsistent data reported, inadequate review of the literature, insufficient data presented, and problems with tables or figures. The main strengths identified in the accepted manuscripts were the importance or timeliness of the problem studies, excellence of writing, and soundness of study design. The author concluded that while some of the problems (e.g., overstating the results and applying the wrong statistics) could be fixed, others (e.g., ignoring past literature, poor study design, use of inappropriate instruments, poor writing) were likely to be fatal flaws that warranted rejection.
2. Health and Biomedical Information
39
Commenting on inadequacies in the peer review process for grant proposals, Stumpf (1980) noted a number of problems, which also occur with journal peer review: 1. For scientific pioneers, there are often few peers who are knowledgeable enough to adequately review their work. 2. For all scientists, the closest peer is a competitor, who may not be appropriate as a reviewer. 3. While reviewers have the opportunity to criticize every aspect of the submitter’s work, there is little if any chance for rebuttal. 4. Reviewers are anonymous, hence are shielded from their own deficiencies and/or biases. Stumpf and Inglefinger both call into the question the anonymous peer review, though most scientific journals still operate under this principle. Some believe that the peer review system tends to serve to keep control of science in the hands of those who already have it. Readings (1994) argues that peer reviewers “take exciting, innovative, and challenging work by younger scholars and reject it.” Because those who do peer reviewing have already passed into the inner circle, they have incentive to keep others out. Agger (1990) states that there is not enough room for everybody in the prestigious journals, so those on the inside are likely want to minimize the number of new entrants. Given the growing competitiveness for research grants, those who have already achieved success in science do have incentives to minimize new competition. Roberts advocates a more moderate view, arguing that peer review is beneficial and that the Internet opens up a new means to make it more open and efficient (P. Roberts, 1999). One question that could be asked is whether peer review has been shown to improve the quality of publications or, better yet, the advancement of human health or scientific knowledge. In one systematic review, Jefferson et al. (2002) found that all studies to date have focused on surrogate and intermediate measures and none have compared peer review with other methods.
2.5.3 Primary Literature and Its Limitations As already noted, the primary literature consists of reports of original research. Key features of primary literature are that it reports on new discoveries and observations, describes earlier work to acknowledge it and place the new findings in the proper perspective, and draws only conclusions that can be justified by the results. Another feature of most primary literature is that it has not been published elsewhere, especially in non-peer-reviewed forums. Indeed, most journals adhere to the “Ingelfinger rule,” which states that a manuscript will be accepted for publication only if it has not been published elsewhere (Ingelfinger, 1969). Exceptions are made for articles that have been presented at scientific meetings, situations in which early publication would have a major impact on public health, and cases in which findings have been released for government deliberations (An-
40
I. Basic Concepts
gell and Kassirer, 1991). The ease of posting research results on the Web has challenged the Ingelfinger rule, but most medical journals still adhere to it (Altman, 1996). This discussion on peer review indicates that good science sometimes does not make it through the peer review process for reasons not related to the quality of the work. The converse occurs as well, with poor and even invalid science sometimes getting published. Furthermore, medical journals, online databases, and libraries are not well equipped to handle fraudulent science. A number of analyses in the late 1970s and early 1980s found that studies using weak or inappropriate methods were being published. Fletcher and Fletcher (1979) noted that weak methods were still quite prevalent in the literature, such as studies using nonrandomized designs or very small sample sizes. Glantz (1980) found that nearly half of all studies in medical journals utilized statistics incorrectly, with the most common error being the inappropriate use of the t-test in comparing more than two groups of means. Freiman et al. (1978) noted that many studies inadequately reported the statistical power of the methods used to discern a difference when one exists. More recently, Moher et al. (1994) found that many published clinical trials still do not have a large enough sample size to be able to detect clinically meaningful relative differences. Halpern et al. (2002) have deemed the continued performing and reporting of underpowered clinical trials as an ethical dilemma. There have been some efforts to improve statistical reporting. In an assessment of an attempt to improve statistical reporting at the BMJ, it was found that the rate of papers considered statistically acceptable improved from 11% at submission to 84% by publication (Gardner and Bond, 1990). It should be noted that even if statistical methods and results are reported correctly, readers will not necessarily understand them. A number of studies have found that professionals who read scientific journals tend to understand natural frequencies much better than probabilities (Hoffrage, Lindsey et al., 2000). Thus, these readers can calculate the risk of a disease better by using natural frequencies (e.g., 10 out of 100) than probabilities (e.g., 10%). Another problem has been inadequate reporting of methods. Der Simonian et al. (1982) identified 11 factors deemed important in the design and analysis of studies, such as eligibility criteria for admission to the trial, method of randomization used, and blinding, and found that only slightly over half of all studies in four major medical journals handled these factors adequately. Others also found that randomization methods were poorly described in the obstetrics and gynecology literature (Schulz, Chalmers et al., 1994). Bailar (1986) has lamented that some scientific practices border on the deceptive, such as the selective reporting of results in some experiments, which is sometimes done to improve chances for publication. Reporting of methods and results has been found to be particularly problematic in the area of the randomized controlled trial (RCT). As will be further described in Section 2.8, the RCT is considered to be the best type of evidence for the effectiveness of a healthcare intervention, whether in the treatment of a disease or its prevention. Furthermore, the data in RCTs are used in meta-analysis,
2. Health and Biomedical Information
41
which is the aggregation of the results of many similar trials to obtain a more statistically powerful result (see Section 2.5.4). It is therefore imperative that RCTs report their data in a way that can be understood by readers and used by other researchers. Authors and journals, however, appear to be falling short. For example, one review of the use of selective serotonin-reuptake inhibitors for depression found that only one of 122 RCTs described the randomization process unequivocally (Hotopf, Lewis et al., 1997). This is important, since one of the most important predictive factors of the quality of an RCT is whether the randomization process is concealed from those involved in care of the patient (Schulz, Chalmers et al., 1995). The problems in the reporting of methods and results in RCTs has led to the formation of the Consolidated Standards of Reporting Trials (CONSORT) statement, which provides a checklist of 22 items to include when reporting an RCT (Moher, Schulz et al., 2001) (see Table 2.3). Most of these elements are included because it has been found that their omission is associated with biased evidence favoring the treatment being studied. For example, studies not using masked assessments of the outcome (i.e., the person judging whether a patient got better did not know whether the patient had received the experimental or control intervention), studies rated as low quality, and studies not concealing the allocation of the patients to the experimental or control intervention, all have been found to have a higher average treatment effect than studies of better quality (Moher, Pham et al., 1998). This finding was also observed in studies of selective digestive decontamination, where there was an inverse relationship between methodological quality score of RCTs and the benefit of treatment (van Nieuwenhoven, Buskens et al., 2001). It has also been noted that smaller studies are more likely to have their intervention effects exaggerated by inadequate randomization and double-blinding, leading to discrepancies in their results in comparison to larger trials (Kjaergard, Villumsen et al., 2001). A more recent analysis, however, has found that quality measures do not always correlate with treatment effects (Balk, Benis, et al. 2002). Use of the CONSORT statement has been shown to result in improvements in the quality of reports of RCTs (Moher, Jones et al., 2001), although others (Huwiler, Juni et al., 2002) have found that good reporting is not strictly correlated with good quality of the study itself. RCTs are not the only type of study considered to be problematic in the medical literature. The reporting of studies of diagnostic test research has also been criticized (Reid, Lachs et al., 1995). Similar to the case of RCTs, it has been found that inadequate methodology used in evaluating diagnostic tests tend to overestimate their accuracy (Lijmer, Mol et al., 1999). Likewise, Udvarhelyi et al. (1992) found that studies of cost-effectiveness and cost–benefit similarly have reporting problems. Also found to be a problem in journal articles is incomplete reporting of past studies. Gotzsche and Olsen (2001) found that subsequent references to seven RCTs of mammography screening tended to omit important limitations of the trials. They advocated that study protocols remain available on the Web after the results have been published. Clarke et al. (2002) noted that only two of 25 RCTs where prior
42
I. Basic Concepts
TABLE 2.3. Elements in the CONSORT Statement Checklist Title and abstract: How participants were allocated to interventions (e.g., “random allocation,” “randomized,” or “randomly assigned”) Introduction Background—Scientific background and explanation of rationale. Methods Participants—Eligibility criteria for participants and the settings and locations where the data were collected. Interventions—Precise details of the interventions intended for each group and how and when they were actually administered. Objectives—Specific objectives and hypotheses. Outcomes—Clearly defined primary and secondary outcome measures and, when applicable, any methods used to enhance the quality of measurements (e.g., multiple observations, training of assessors). Sample size—How sample size was determined and, when applicable, explanation of any interim analyses and stopping rules. Randomization Sequence generation—Method used to generate the random allocation sequence, including details of any restriction (e.g., blocking, stratification). Allocation concealment—Method used to implement the random allocation sequence (e.g., numbered containers or central telephone), clarifying whether the sequence was concealed until interventions were assigned. Implementation—Who generated the allocation sequence, who enrolled participants, and who assigned participants to their groups? Binding (masking)—Whether participants, those administering the interventions, and those assessing the outcomes were blinded to group assignment. If done, how was the success of blinding evaluated? Statistical methods—Statistical methods used to compare groups for primary outcome(s); methods for additional analyses, such as subgroup analyses and adjusted analyses. Results Participant flow—Flow of participants through each stage (a diagram is strongly recommended). Specifically, for each group report the numbers of participants randomly assigned, receiving intended treatment, completing the study protocol, and analyzed for the primary outcome. Describe protocol eviations from study as planned, together with reasons. Recruitment—Dates defining the periods of recruitment and follow-up. Baseline data—Baseline demographic and clinical characteristics of each group. Numbers analyzed—Number of participants (denominator) in each group included in each analysis and whether the analysis was by “intention to treat.” State the results in absolute numbers when feasible (e.g., 10 of 20, not 50%). Outcomes and estimation—For each primary and secondary outcome, a summary of results for each group and the estimated effect size and its precision (e.g., 95% confidence interval). Ancillary analyses—Address multiplicity by reporting any other analyses performed, including subgroup analyses and adjusted analyses, indicating those prespecified and those exploratory. Adverse events—All important adverse events or side effects in each intervention group. Discussion Interpretation—Interpretation of the results, taking into account study hypotheses, sources of potential bias or imprecision, and the dangers associated with multiplicity of analyses and outcomes. Generalizability—Generalizability (external validity) of the trial findings. Overall evidence—General interpretation of the results in the context of current evidence.
2. Health and Biomedical Information
43
trials existed described the new results in the proper context of prior trials. A related problem is follow-up weaknesses identified in follow-up correspondence. Horton (2002) found that half the criticisms in postpublication correspondence of the three RCTs went unanswered. In addition, criticisms of the studies raised in such correspondence were not noted in subsequent practice guidelines. Authorship is another area that can be problematic in reporting. Although it probably does not affect the quality of the information being reported, authorship is important nonetheless in terms of academic promotion and funding. As many as 19% of articles in major journals may have authors who do not meet the ICMJE criteria for authorship, that is, are “honorary” authors (Flanagin, Carey et al., 1998). Conversely, up to 11% show evidence of having “ghost” authors. Laine et al. (2001) have found that that while 93% of first authors of papers in Annals of Internal Medicine and Radiology satisfy the ICMJE criteria, fewer authors in other positions do so (72% of second authors, 51% of third authors, 54% of last authors, and 33% of authors in other positions). Van Rooyen et al. (2001) found that only 17% of papers in the BMJ adhere to the reporting standards of the ICMJE criteria. A related concern to authorship is conflict of interest. ICMJE guidelines do not prevent authors with a financial interest from publishing, requiring only that they disclose such interests. While the explicit financial interest that authors have is not well reported (Gupta, Gross et al., 2001), studies having such interests have actually been found to be associated with a higher quality of methodological reporting (Olson, Rennie et al., 2001). There is also no association between trial quality or outcome, although the authors of this study noted that their analysis was limited because the reporting of these interests was voluntary (Clifford, Moher et al., 2001). Even when the methods are adequately described, the writing may be problematic. As mentioned earlier, scientists who serve as peer reviewers may not be skilled at ensuring that a paper will describe its findings and conclusions as clearly and succinctly as possible (Ingelfinger, 1974). Even when the body of a paper is written soundly, the abstract may not accurately convey the nature of the results. A recent study of six major medical journals found that 18 to 68% of abstracts contained data that either were inconsistent or were not found in the body of the article (Pitkin, Branagan et al., 1999). This problem is of increased gravity when practitioners access bibliographic databases (such as MEDLINE) that have only titles and abstracts. These users may not have the time or motivation to seek the primary reference and thus may be misled by an inaccurate abstract. These problems have motivated the use in virtually all major medical journals of structured abstracts, which require information about the objective, design, setting, participants, intervention, main outcome, results, and conclusions from a study (Haynes, Mulrow et al., 1990). Journal articles may also have inaccuracies in citations and quotations. Unless noted by peer reviewers, these errors usually pass through the editorial process. In a study of three surgical journals, Evans et al. (J. Evans, Nadjari et al., 1990) found that almost half of all references had errors such as misspelling of author
44
I. Basic Concepts
names or partial omissions of titles and authors, although most of the errors were deemed to be minor. A more serious problem was the number of major errors in quotation, such as an article being referenced that did not substantiate, was unrelated to, or contradicted the authors’ assertions. Each journal issue had over 10 major quotational errors. A decade later, citation errors are still found in 7 to 60% of journal articles, with 1 to 24% being so significant that the articles cannot be located based on the information given (Riesenberg and Dontineni, 2001; Wager and Middleton, 2001). An additional problem with the primary literature is the phenomenon of publication bias. Given studies of equal methodological rigor, those with “positive” results are more likely to be published (Dickersin, 1990). Publication bias tends to result because scientists want to report positive results and journal editors (and presumably their readers) want to read them. As will be seen in the next section, publication bias is particularly problematic with the growing use of metaanalysis, where the aggregation of studies presumes that individual studies represent the full spectrum of results. There is strong evidence for publication bias. Sterling (1959) noted several decades ago that studies yielding statistically significant results were more likely to be published than those that did not, raising the possibility that studies with significant results may never be further verified. Dickersin (Dickersin and Min, 1993) noted this problem and clinical trials, observing that studies approved by various institutional review boards and/or National Institutes of Health (NIH) funding agencies were more likely to be published if statistically significant results were achieved. Others have looked at a cohort of studies approved by a hospital review board and found that those with positive results were 2.3 times more likely to be published than those with negative results (Stern and Simes, 1997). The likelihood of publication for clinical trials was even higher (3.1 times more likely). Studies with indeterminate results were even less likely to be published, while those measuring qualitative outcomes did not seem to reflect the influence of publication bias. Similar publication bias also occurs with studies submitted for presentation at scientific meetings (Callaham, Wears et al., 1998). Scherer and Langenberg (2001) performed a systematic review of studies assessing subsequent publication of abstracts presented at scientific meetings, with an average of only 44.8% ultimately achieving publication as a full paper. Few are published more than 3 years after their presentation. While abstracts showing a preference for the experimental treatment were not associated with full publication, those having statistically significant results were, indicating some publication bias. There are other manifestations of publication bias. Some studies with positive outcomes tend to be published sooner than those with negative or equivocal outcomes. An additional finding from Stern and Simes (1997) was that the median time to publication for studies with positive results was significantly shorter (4.7–4.8 vs 8.0 years). Likewise, Ioannidis (1998) found that clinical trials not achieving statistical significance of the results in the treatment of human immunodeficiency virus (HIV) were likely to be published later than those that did.
2. Health and Biomedical Information
45
A contrary result was found at JAMA, where studies with positive results were not found to be published more quickly than those with negative results (Olson, Rennie et al., 2002). Another facet of publication bias is that researchers from non-English-speaking countries tend to publish their positive results more in English-language journals (Egger, Zellweger-Zahner et al., 1997). Negative results or the inability to obtain statistical significance are not reasons not to publish, since in the case of clinical trials, for example, it is just as important to know when a therapy is not more effective than the current standard or none at all. This is even more crucial with the current widespread use of meta-analysis, since data that should be part of a meta-analysis might not be used because it had never been published. Indeed, Chalmers (1990) has called failure to publish a clinical trial a form of “scientific misconduct.” In recognition of this problem, more than 100 journals announced an “amnesty for unpublished trials” in September 1997, and authors of clinical trials not yet published were invited to report their register and publish their results over the Web (Smith and Roberts, 1997). Friedman and Wyatt (2001) also raise concern about publication bias in fields like medical informatics, where system developers often perform evaluations and are less likely to publish unfavorable results about their systems, especially when such data may detract from future grant funding. Another concern with the primary literature is low relevance to the average practitioner: as will be seen in Section 2.7, it is not the most preferred source for meeting information needs. This is partially due to the fragmentation and relative inaccessibility of the primary literature, since most practitioners have access to only a very small personal library and perhaps a somewhat larger hospital library. The primary literature also reflects divergence between what experts and readers believe are the most important topics to be covered. A study at JAMA found that only three of the topics rates as the 10 most important by experts were rated in the top 10 by readers (Lundberg, Paul et al., 1998). Likewise, only one of the top 10 topics rated by readers was in the experts’ top 10 (see Table 2.4). A final issue associated with primary literature is the handling of invalid or fraudulent science. While the NLM has the means to retract incorrect or fraud-
TABLE 2.4. Experts’ and readers’ Top 10 Subjects of Interest in JAMA Survey Experts 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.
Managed care Death and dying Genetics Quality of care Violence Aging Care for uninsured/underinsured Outcomes improvement/research HIV/AIDS Cancer
Source: Lundberg, Paul et al. (1998).
Readers 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.
Aging Cancer Computers Atherosclerosis and heart disease Obesity Clinical practice guidelines Alternative medicine Children’s health Managed care Critical care medicine
46
I. Basic Concepts
ulent literature from its databases (Colaianni, 1992), removing it from library shelves and the Web is considerably more difficult. A survey of 129 academic medical libraries in North America found that 59% have no policy or practices for calling retracted publications to the attention of their patrons (Hughes, 1998). Another 9% have no formal policies but attempt to notify users by one means or another. Among the means for identifying such research include tagging articles on the first page of the issue in which each one occurs or keeping lists of such articles at reference or circulation desks. Friedman has found that journals are inconsistent in how they identify retracted publications (P. Friedman, 1990). Specific instances of identified research fraud have been well studied. In an analysis of the work of John Darsee, a Harvard researcher who was later found to have fabricated experimental results, two researchers found that the other publications of Darsee, whose validity may never be known, are still cited by other researchers in a positive light (Kochen and Budd, 1992). Others (Whitely, Rennie et al., 1994) did a similar analysis of the publications of Robert Slutsky, another scientist guilty of fraud, and found a similar phenomenon, though the rate of citation diminished as the case was publicized in the press. Likewise, another scientist known to publish fraudulent work, Stephen Breuning, was also found to have positive citations of his work long after he had pleaded guilty to deception (Garfield and Welljams-Dorof, 1990). A related problem is that of plagiarism. A group of case studies from the Bulletin of Environmental Contamination and Toxicology identified instances of selfplagiarism (title, coauthors, and data were changed), a paper published in Spanish that was translated into English and submitted, and the copying of figures without acknowledgment (Nigg and Radulsecu, 1994). Plagiarism may now be even easier with the growth of electronic publication on the Web. Kock (1999) reported a case of a paper of his that was copied from the Web and then submitted for publication to a journal. A case of a medical journal paper being assembled by plagiarism from a number of Web-based sources has also been identified (Eysenbach, 2000b).
2.5.4 Meta-Analysis and Its Limitations As noted earlier in the chapter, the scientific literature is fragmented, perhaps purposefully (Ziman, 1969). As will be seen in subsequent chapters, even experienced searchers have difficulty identifying all the relevant articles on a given topic. The proliferation of clinical trials, particularly when they assess the same diseases and treatments, has led to the use of meta-analysis, where the results of similar trials are pooled to obtain a more comprehensive picture of the intervention with greater statistical power. The term meta-analysis was coined by Glass (1976), and the technique is used increasingly in assessing the results of clinical trials. Meta-analyses are not limited to RCTs and have been used with diagnostic test studies (Glasziou and Irwig, 1998) and observational studies (Stroup, Berlin et al., 2000). Meta-analyses are often part of systematic reviews, which are distinguished from traditional review articles, described in the next
2. Health and Biomedical Information
47
TABLE 2.5. Express of RCT Data in a Meta-Analysis Intervention Event
Experimental
Control
Present Absent
a c
b d
section, by their focused questions, exhaustive review of the literature, and use of evidence-based techniques (Petticrew, 2001). They are a major aspect of evidence-based medicine (EBM), to be discussed in Section 2.8. The most common use of meta-analysis in health care is to combine the results of RCTs. The most common approach is to express the data as adverse event rates and compare the odds ratio (OR) between the control and experimental groups (Bland and Altman, 2000). The data is set up as a contingency table as seen in Table 2.5. From this table, the OR can be calculated: a/c OR b/d
(5)
The OR calculates the odds of an event in a patient in an experimental group relative to that of a patient in a control group. When the OR 1, the experimental treatment is favored. In addition to knowing the point estimate of the treatment effect, one must know the precision of the value. That is, how likely is it that the point estimate from the patient sample represents the true value for the entire population? To determine the precision, the confidence interval (CI) is calculated (Bland and Altman, 2000). The 95% CI represents the range of values in which the true value of the population has a 95% likelihood of falling and is calculated by: a b c d CI eln(OR)1.96* 1 1 1 1
(6)
The OR and CI are usually displayed graphically. When the OR point falls to the left of the OR 1 line, then the treatment is beneficial; and if the 95% CI does not touch the OR 1 line, the difference is statistically significant. Usually the individual studies are displayed in rows, with the meta-analysis summary statistic in the last row. Sometimes the studies are displayed chronologically in a cumulative meta-analysis, where the treatment effect value and confidence interval are shown cumulatively as each new study is added. This type of display allows investigators to determine when the result has achieved statistical significance. This approach has been used to show that many clinical interventions show statistically significant evidence of benefit long before experts begin to recommend them in textbooks and review articles (Antman, Lau et al., 1992). A sensitivity analysis can also be applied to meta-analysis, where the studies are sorted by those of highest quality, and as the lower quality studies are added to the analysis it becomes possible to state whether the aggregate result holds. These concepts and the value of the meta-analysis are best demonstrated
48
I. Basic Concepts
FIGURE 2.3. Logo of the Cochrane Collaboration showing result of meta-analysis demonstrating benefit for use of corticosteroids in preterm labor. (Courtesy of the Cochrane Collaboration steering group.)
in the logo of the Cochrane Collaboration (see Figure 2.3), a group devoted to production of systematic reviews (Levin, 2001). The logo demonstrates a metaanalysis done to assess whether steroids are beneficial to the fetus in premature labor. Of the seven trials identified at the time the logo was created, five showed statistically insignificant benefit (i.e., their CI crossed the OR 1 line). However, when all seven trials were included in a meta-analysis, the results unequivocally demonstrated benefit, with the CI well to the left of the OR 1 line. The Cochrane Collaboration has given inspiration to a similar effort in the social sciences, the Campbell Collaboration (campbell.gse.upenn.edu). Not everyone accepts the value of meta-analysis. Feinstein has called it “statistical alchemy” and warns that relying it on too heavily ignores such other factors in clinical decision making as physiological reasoning and clinical judgment (Feinstein, 1995; Feinstein and Horwitz, 1997). (Although as was seen in Figure 1.4, even in the context of EBM, evidence is not the sole criterion for making clinical decisions.) Hopayian (2001) is less critical, but does note that meta-analyses on the same topic do often reach different conclusions, usually because important clinical details are overlooked by those researchers focused purely on methodology. He recommends that they be conducted from the clinical as well as methodological viewpoint. Even those who perform meta-analyses have noted pitfalls in the approach. As already described, RCTs of lower quality (Moher, Pham et al., 1998) and of smaller size (Kjaergard, Villumsen et al., 2001) tend to show a larger benefit for the treatment studied. The problem of publication bias described above is likely to be amplified in meta-analysis, since the analysis relies on the full spectrum of results for a given research question to be published. Indeed, at least one out of 36 meta-analyses produced different results when non-English language studies were added to an original English-only analysis (Gregoire, Derderian et al., 1995).
2. Health and Biomedical Information
49
One particular challenge in performing meta-analyses is identifying all the appropriate RCTs to include. As will be seen in Chapter 7, the literature searching process requires assessing many articles to find the few that should be included. A possible source of additional studies beyond conventional database searching is the Internet (Eysenbach, Tuische et al., 2001). Another limitation is that some studies may not be published in the regular medical literature. One analysis found that the exclusion of gray (unpublished) literature from meta-analyses was likely to exaggerate the benefit of interventions (McAuley, Tugwell et al., 2000). In addition, some meta-analyses are reported better than others. Jadad et al. (1998) have found that systematic reviews from the Cochrane Collaboration tend to use better methodologic rigor and are updated more frequently than those published in traditional medical journals. One concern always with meta-analyses is whether the results are biased in some way, such as through studies of low quality or via publication bias. A simple approach to detecting bias is through the use of funnel plots, which are scatter plots of the treatment effect on the horizontal axis against a measure of the sample size on the vertical axis (Egger, Smith et al., 1997). In general, an unbiased meta-analysis should show studies with small sample sizes having more scattered effect sizes than those with larger samples, that is, a funnel plot would appear as a symmetrical inverted funnel. Biased meta-analyses, however, are more likely to show asymmetrical funnel plots. This approach has been shown to explain why subsequent large RCTs contradict meta-analyses (i.e., the new large study added to the meta-analysis shows an asymmetrical funnel plot) and to demonstrate publication bias in meta-analyses (Egger, Smith et al., 1997). Two hypothetical funnel plots are shown in Figure 2.4.
2.5.5 Secondary Literature and Its Limitations Because the primary literature is fragmented, its use is prohibitively difficult, and there is strong impetus to present the information in different formats. Another problem is that in many professional fields, healthcare included, practitioners apply scientific information but do not necessarily generate it and are therefore less likely to understand the scientific esoterica described by researchers. This is especially so for nonexpert professionals, such as clinicians, who must make decisions that are, however, based on specialized scientific information. The need for overviews of primary literature is one of the motivations for the secondary literature. This literature consists of review articles (which are often published in the same journals that contain primary literature), books, editorials, practice guidelines, and other forms of publication in which original research information is reviewed. There are other motivations for generating this literature besides clinical use, such as for policy making and administration. The secondary literature, in the form of textbooks and review articles, has always been the major source of literature used by clinicians. Some of the review articles occur in the voluminous literature of the “throw-away” journals. These journals often serve as vehicles for pharmaceutical or other advertising, but
50
I. Basic Concepts
FIGURE 2.4. Hypothetical funnel plots showing potentially (a) unbiased and (b) biased meta-analyses. (Courtesy of the Cochrane Collaboration steering group.)
nonetheless often feature well-written and concise articles on clinically pertinent topics. In fact, one study found that although such publications are of lower methodological quality from a research perspective, they communicate their message better via use of tables, pictures, larger fonts, and easier readability (Rochon, Bero et al., 2001). Of course, these publications are often justly criticized
2. Health and Biomedical Information
51
because it is unknown how the vested interests of the advertisers influence the editorial content. Unfortunately, there are as many problems stemming from poor methodology and writing making it into secondary literature as there are for primary literature. It has also been found, for example, that the rigor of review articles, even in esteemed medical journals, can be lacking. Mulrow (1987) looked for eight criteria (purpose, data identification, data selection, validity assessment, qualitative synthesis, quantitative synthesis, summary, and future directives) in 50 review articles in major medical journals. For the categories of data identification, data selection, validity assessment, and quantitative synthesis, virtually all papers were inadequate. Mulrow argued that review papers were not complete without the details of the literature search as well as a quantitative synthesis such as a metaanalysis. A recent follow-up study, which added the criteria of addressing heterogeneity and addressing generalizability of data, showed that most review articles still were not measuring up (McAlister, Clark et al., 1999). This study found that less than one-fourth of the articles assessed described how evidence was identified, evaluated, or integrated. It also noted that articles in journals with higher IF did not meet the criteria any better than those in journals with lower IF. The problem of inadequate literature identification in review articles has been further addressed by others. As will be described in more detail in Chapter 7, Dickersin et al. (1994) have found that even exhaustive searching in large literature databases such as MEDLINE does not yield all the studies pertinent to a review. Joyce et al. (1998) found that for 89 review articles on Chronic Fatigue Syndrome, only three (3.4%) described the search strategy. Furthermore, authors from specific fields (in this case laboratory medicine and psychiatry) preferentially cited references from their own disciplines. Likewise, authors had a higher likelihood of citing from their own countries (i.e., the United States and the United Kingdom). Another limitation of secondary literature is its potential to be incorrect. As noted earlier, Antman et al. (1992) found that medical textbooks, review articles, and practice recommendations often lag behind the edge of accumulated knowledge. Sometimes secondary references do not provide the most accurate information. For example, Cohen (2001) has found that the Physicians’ Desk Reference (PDR, Medical Economics, www.pdr.net), a venerable information resource for clinicians and their patients, does not contain the full range of dosages for commonly prescribed drugs. Noting that the PDR contains the information in drug package inserts, he found that lower doses of 48 drugs discovered effective in clinical trials were not included in the recommendations for use. In recent years there have been efforts to improve the secondary literature. As noted in the discussion on primary literature, most journals have adopted the use of strudtured abstracts. An increasing number of journals have also been requiring structured abstracts for review articles, which describe the methodology used to collect and analyze primary data for the article. Another innovation has been the publication of Best Evidence (American College of Physicians, www. acpon-
52
I. Basic Concepts
line.org, and BMJ Publishing, www.bmj.com), which provides extended, structured abstracts for the most important articles recently published in the clinical medicine primary literature. These reviews provide a concise summary of the original study, along with a brief expert commentary. A more recent approach to summarizing evidence even more concisely comes from Clinical Evidence (BMJ Publishing). Another type of secondary literature gaining increased interest as efforts to standardize health care evolve consists of clinical practice guidelines. These are, “systematically developed statements to assist practitioners and patient decisions about appropriate health care for specific clinical circumstances” (Field and Lohr, 1990). These guidelines are used as more than just documents for reading for they also serve to guide documentation and provide decision logic for the provision of medical care (Shiffman, Brandt et al., 1999). One concern with practice guidelines is their sheer number. A study of British general medical practices found 855 different guidelines had been sent to physicians in 65 different practices, which could be stacked nearly 70 cm tall (Hibble, Kanka et al., 1998). Another concern with guidelines is that clinicians do not use them. Cabana et al. (1999) performed a systematic review assessing reasons for lack of adherence to guidelines, finding a variety of factors often dependent on the unique characteristics of the locations where they were studied. The major categories of barriers include awareness of guidelines, agreement with them, altered expectations of patient outcomes, inability to overcome the inertia of previous practice, and barriers to carry them out. As with other literature, guidelines become outdated quickly. A recent analysis which found that a group of guidelines from the Agency for Healthcare Research and Quality (AHRQ, www.ahrq.gov) had a half-life of 5.8 years, recommended that all guidelines be updated at least every 3 years (Shekelle, Ortiz et al., 2001).
2.6 Electronic Publishing In the first edition of this book, IR systems were still viewed as a means to get to paper-based scientific journals. While it was noted that online full-text journals had been in existence since the 1980s, high-cost, low-resolution displays, and bandwidth-limited networks precluded their routine access. In the early twenty-first century, however, the picture is entirely changed. Most scientific journals are published in some sort of electronic form, even if they are not readily accessible to most users. The elimination of technical barriers has uncovered a series of new barriers, as will be explained in this section. Another phenomenon of electronic publishing has been the growth of the Web as means to disseminate not only traditionally published health information, but all sorts of other health information. The Web has been used to publish new forms of health information not only for healthcare professionals but for their patients as well. This has given rise to a new concern about the quality of health information on the Web, especially as it related to consumer health information, which will also be discussed on this section. The ease of replicating, deleting, and altering information on the Web has led the American Association for the Advancement of Science (AAAS) to define a “definitive publication” in science, especially in the context of the electronic en-
2. Health and Biomedical Information
53
vironment (Frankel, Elliot et al., 2000). Such a publication should be peerreviewed; the following statements apply, as well: • • • •
The publication must be publicly available. The relevant community must be made aware of its existence. A system for long-term access and retrieval must be in place. The publication must not be changed (technical protection and/or certification are desirable). • It must not be removed (unless legally unavoidable). • It must be unambiguously identified (e.g., by some sort of identifier). • It must have a bibliographic record (metadata) containing certain minimal information. • There must be a plan for archiving and long-term preservation.
2.6.1 Benefits and Challenges of Electronic Scholarly Publication Most scientific journals are published electronically in some form already. Journals that do not publish electronically likely could do so easily, since most of the publishing process has already been converted to the electronic mode. The technical impediments to electronic publishing of journals have largely been solved. A modern Internet connection is sufficient to deliver most of the content of journals. Indeed, a near turnkey solution is already offered through Highwire Press (www.highwire.org), a spin-off from Stanford University, which has an infrastructure that supports journal publishing from content preparation to searching and archiving. Some journals publish exclusively electronically, such as Medscape General Medicine (www.medscape.com/Medscape/GeneralMedicine/journal/public/mg) and Journal of Medical Internal Research (www.jmir.org), and the journals of Biomed Central (BMC, www.biomedcentral.com). There is great enthusiasm for electronic availability of journals, as evidenced by the growing number of titles to which libraries provide access. When available in electronic form, journal content is easier and more convenient to access. Indeed, a faculty member at Oregon Health & Science University noted that she did not consider the library to subscribe to a journal unless the publication was available in electronic form. Furthermore, authors have incentive for electronic availability of their papers. Lawrence (2001) has found that papers from computer science that are mounted on the Web have a higher likelihood of being cited than those that are not. And since citations are important to authors for academic promotion and grant funding, authors have incentive to maximize the accessibility of their published work. The technical challenges to electronic scholarly publication have been replaced by economic and political ones (Hersh and Rindfleisch, 2000; Anonymous, 2001f). Printing and mailing, tasks no longer needed in electronic publishing, comprised a significant part of the “added value” from publishers of journals. Since the intellectual portion of the peer review process is carried out by scientists, the role of the publisher can be reduced or perhaps eliminated with electronic submission of manuscripts and their electronic distribution to peer reviewers. There is, however, still some value added by publishers, such as copy editing and production. Even if
54
I. Basic Concepts
publishing companies as they are known were to vanish, there would still be some cost to the production of journals. Thus, while the cost of producing journals electronically is likely to be less, it is not zero, and even if journal content is distributed “free,” someone has to pay the production costs. The economic issue in electronic publishing, then, is who is going to pay for the production of journals. This introduces some political issues as well. One of them centers around the concern that much research is publicly funded through grants from federal agencies such as the National Institutes of Health (NIH) and the National Science Foundation (NSF). These agencies, as well as scientists themselves, want to see the research published in high-quality journals. But in the current system, especially in the biomedical sciences (and to a lesser extent in nonbiomedical sciences), researchers turn over the copyright of their publications to journal publishers. Indeed, some even have to pay to have the papers published. The political concern is that the public funds the research and the universities carry it out, but individuals and libraries then must buy it back from the publishers to whom they willingly cede the copyright. This problem is exacerbated by the general decline in funding for libraries that has occurred over the last couple decades (Boyd and Herkovic, 1999; Meek, 2001). The current situation persists partly because journals, especially the prestigious ones, have a monopoly of sorts on the best research. Academic researchers in particular are promoted via promotion and tenure committees, which judge them based on their publication record in prestigious journals. A number of solutions have been proposed. A nearly universal theme in these proposals has been for academic institutions to retain copyright of scholarly material and grant limited licenses to publishers (Boyd and Herkovic, 1999). Harnad (1998) published a vision for a model of electronic publishing that eliminated traditional publishers. He argued that authors and their institutions could pay the cost of production of manuscripts up front after they were accepted through a peer review process. He suggests this cost could even be included in the budgets of grant proposals submitted for funding agencies. After the paper was published, the manuscript would thereafter be freely available on the Web. One model for freely distributed publishing already exists in the genomics community, where all data from genome research, including the Human Genome Project, has been available for over a decade (Wheeler, Chappey et al., 2000). There are, of course, some significant differences between this community and the clinical research community. In the genomics community, the data in the various genome databases (to be described in Chapter 4) is primary, with published reports (which are copyrighted by journals) of secondary value. In clinical research, the published article is still the primary vehicle of dissemination of research at this time. In fact, the raw data for most clinical research is not available, partially out of concern for patient confidentiality but also because the scientific culture of clinical research does not value sharing it. A solution that has emerged for access to publicly funded research is PubMed Central (PMC, pubmedcentral.gov). Proposed by former NIH director Harold Varmus, PubMed Central began as an initiative called “E-Biomed” (Varmus,
2. Health and Biomedical Information
55
1999). Run by the NIH, this resource would consist of two parts. The first would contain preprint manuscripts that had not yet been subject to peer review, while the second would be an archive of research reports already published in peerreviewed journals. The normal process of peer review and journal publishing would continue, but PMC would make prepublished or postpublished scientific reports freely available. The Varmus proposal set off vigorous debate. There was considerable opposition to the idea of a preprint server. It would clearly violate the Ingelfinger rule, especially as the International Committee of Medical Journal Editors has declared that, “electronic publishing (which includes the Internet), is publishing” (Anonymous, 1997a). While some fields, most notably physics, already encourage widespread dissemination of manuscripts not yet published in the peer-reviewed literature, there was great concern that the preprint archive would invite abuse, blurring the distinction between peer-reviewed and non-peer-reviewed research, especially for those less savvy about publishing practices (Relman, 1999). The preprint server portion of PMC has been abandoned for these and other reasons (E. Marshall, 1999). There was also great resistance from publishers. This led to alteration of the proposal to allow publishers to maintain copyright as well as optionally keep the papers on their own servers. A lag time of up to 6 months is allowed so that journals can reap the revenue that comes with initial publication. The newer approach to PMC has been accepted by many leading journals. Indeed, some have argued it will not adversely affect, and may enhance, the revenue streams of journals (R. Roberts, 2001). Another concern with electronic publishing is archiving. With paper journals, readers, and libraries subscribe to the journal and own their paper copies. Given the printing process, many individual copies of the content have been produced and widely disseminated, making it unlikely that all copies will ever disappear. This has changed somewhat with electronic journals. Readers and libraries generally obtain a license to access the database, but there are fewer “copies” of the entire database. While some individuals may download parts of the database for their own electronic archives, the only complete copies of the database are those held on the servers and backup storage media of the publisher. It is unclear where the data would go if the publisher went out of business. It is conceivable that papers could be lost from the archive of science. These concerns have led to the formation of initiatives to ensure archiving of scientific information, which include the National Digital Information Infrastructure Preservation Program (NDIPP) of the U.S. Library of Congress, as well as the Digital Preservation Coalition in the UK (Beagrie, 2002). The debate over electronic publishing and associated issues has continued (e.g., see www.nature.com/nature/debates/e-access/index.html). Another initiative to make research articles more widely and freely available is the Public Library of Science (PLS, publiclibraryofscience.org). Scientists who sign the PLS open letter pledge to publish in and subscribe to only journals that agree to make their research reports freely available within 6 months of publication on the Web via the PMC
56
I. Basic Concepts
or any other equivalent model. Over 28,000 scientists in 172 countries have signed the letter, agreeing to adhere to the pledge starting in September, 2001. Some have argued that what is really needed is a new model and approach to scientific publishing that is completely electronic and incorporates the goals of PMC and PLS. Such an approach has been launched by BMC, which aims to provide “peer reviewed research across all areas of biology and medicine, with immediate, barrier-free access for all.” On its Web site, BMC states that its goal is to provide a means for the dissemination without charge to the reader of rapidly peer-reviewed and published science. Of course, such a goal requires a business model, and BMC is currently supported via advertising on its site. BMC charges authors modest processing and editing fees, which are waived for those who cannot afford them. Ultimately the BMC model views the costs of publishing, which are greatly reduced in the electronic model, as a cost of research to be borne by those who fund research, such as governmental agencies and commercial concerns. BMC currently houses 18 biology and 39 medicine journals, with each names and devoted to specific subject areas (e.g., BMC Bioinformatics, BMC Physiology). The site also contains four affiliated journals as well as abstracts from a small number of scientific meetings. Articles are published in native HTML Web page format as well as a more visually pleasing and better-printing PDF format. A relatively simple search engine allows searching of individual journals of all titles on the site. The search engine also allows queries to be passed to PubMed or PMC. All publications in BMC have been part of PMC since BMC’s inception and meet the criteria of the PLS. The BMC publishing process provides many appealing aspects to scientists. First, the peer review process is rapid. The Web site claims that the time from submission to publication averages 35 days. This short period is made possible by the use of a completely electronic process, including online submission and peer review. In addition, because all publishing is electronic, articles can be published as soon as they are accepted. The BMC Web site also boasts other advantages to its process, including the lack of space constraints, which means that papers worthy of publication do not need to compete with each other for part of a finite space as in print journals. Authors also maintain copyright of their articles, although they agree to grant BMC an exclusive license to republish the article, including in print form. All BMC articles are indexed in MEDLINE, and thus research published in BMC is no more difficult to find than that published elsewhere. BMC also provides a standard means for citing articles, which should ensure that they are easy to cite as well as access. Finally, BMC is working with a number of archiving efforts to ensure that content will be accessible in perpetuity. There is concern, of course, over whether articles published in BMC will have the “prestige” associated with scientific publishing. Certainly there is no inherent reason why high-quality science cannot be published under this model. But it will succeed only if scientists submit their best research to BMC and efforts like it. In summary, there continue to be challenges for electronic publication of health
2. Health and Biomedical Information
57
and biomedical journals. There is perceived unfairness that public monies are used to fund biomedical research whose results are then copyrighted by publishers. There is also concern with the evolution from many paper copies of journals to fewer electronic ones, whether from the standpoint of subscribers who lose access to their “copies” once their subscriptions end or archivists who want to insure the long-term preservation of the archive of science. There are also concerns over the cost of accessing electronic archives. While most acknowledge such information cannot truly be free, it is unclear whether the emerging models will inhibit the access to scientific information. 2.6.2 Quality of Health Information on the Web With the growth of the Web, a new form of publishing has emerged, mostly removed from traditional scientific publishing. A great deal of information, virtually all of it secondary literature, is now published directly on the Web. A large fraction of this information is aimed at nonprofessional audiences (i.e., the patient or consumer). Many laud this development as empowering to those most directly affected by health care—those who consume it (Eysenbach, Su et al., 1999). Others express concern about patients misunderstanding or being purposely misled by incorrect or inappropriately interpreted information (Jadad, 1999). Some clinicians also lament the time required to go through stacks of printouts patients bring to the office. Whether this phenomenon is good or bad, it is too late to change it, especially as the baby boomer generation, the most high educated and affluent generation ever, begins to become elderly. The Web is inherently democratic, allowing virtually anyone to post information. This is no doubt an asset in a democratic society like the United States. However, it is potentially at odds with the operation of a professional field, particularly one like healthcare, where practitioners are ethically bound and legally required to adhere to the highest standard of care. To the extent that misleading or incorrect information may be posted on the Web, this standard is challenged. A major concern with health information on the Web is the presence of inaccurate or out-of-date information. A recent systematic review of studies assessing the quality of health information found that 55 of 79 studies came to the conclusion that quality of information was a problem (Eysenbach, Powell et al., 2002). Among these studies were: • Impiccatore et al. (1997) found that of 41 Web pages dealing with childhood fever, only four adhered to all the guidelines listed in an authoritative reference, with two pages giving advice to use aspirin, which is associated with Reye’s syndrome and has not been recommended as therapy for childhood fever in decades. • Davison (1997) identified 167 Web pages providing dietary advice and found that less than half provided information consistent with Canadian government guidelines for nutrition. • McClung et al. (1998) found 28 pages with information on the treatment of
58
I. Basic Concepts
childhood diarrhea,only 12 of which met the criteria in a position paper on the topic from the American Academy of Pediatrics. • Hatfield et al. (1999) found a major consumer drug information Web site that had inaccurate content. • Biermann et al. (1999) evaluated 170 pages on Ewing’s Sarcoma and found gross errors on many, including inaccurate statistics on survival rate from this rare form of cancer. • Tamm et al. (2000) found that 5 of 38 sites did not reflect guidelines on mammography. • Miles et al. (2000) found that 40 of 45 sites gave unsound advice about weight loss and diet. • Pandolfini et al. (2000) found that 10 of 19 sites with information on childhood cough gave more incorrect than correct advice. • Latthe et al. (2000a) identified nine pages on the topic of menorrhagia, all but one of which lacked information deemed essential for treatment from a practice guideline on the condition published by the Royal College of Obstetricians and Gynaecologists. • Latthe et al. (2000b) found all 32 pages covering the topic of emergency contraception did not provide information deemed essential for treatment from a practice guideline. • Jiang (2000) found a high variability in quality of pages on orthodontics, with those oriented to professionals of higher quality than those aimed at the general public. • Lissman and Boehnlein (2001) assessed sites on treatment of depression, finding that only half mentioned any symptoms or criteria for depression and less than half made any mention of medications, psychotherapy, or professional consultation as suggested treatments for depression. Other studies have found sites that give advice contradicting what is known from the medical literature. The Federal Trade Commission sponsored a Health Surf Day in 1998 that identified over 400 pages promoting unproven treatments for cancer, AIDS, heart disease, arthritis, and multiple sclerosis (Anonymous, 1997c). Wright et al. (1999) found that most Web sites on Chronic Fatigue Syndrome in children gave unsupported etiological explanations and failed to follow established treatment advice. Gillois et al. (1999) found four of eight sites providing cardiovascular risk prediction used invalid information. Soot et al. (1999) examined 50 Web pages on three topics in vascular surgery and found that 16 had information that was misleading or unconventional. Suarez-Almazor et al. (2000) found that 44% of pages about rheumatoid arthritis promoted unproven alternative therapies, with those sites more likely to have financial interests in the products listed than general pages. Berland et al. (2001) assessed the completeness and accuracy of coverage for clinical elements deemed important for consumers in four health topics: breast cancer, childhood asthma, obesity, and depression. The proportion of sites hav-
2. Health and Biomedical Information
59
ing more than minimal coverage of elements varied from 40% in obesity to 67% in breast cancer. For the sites having adequate coverage, the proportion having completely correct information varied from 75% in depression to 91% in breast cancer. The study also assessed Spanish-language sites and found lower rates of adequate coverage and correctness. In many of these studies, the sites evaluated were from academic medical centers or other prestigious medical institutions. Since academic sites are usually managed in a decentralized manner (i.e., individual departments and often individual faculty maintain their own pages), one cannot assume that quality checking persists down to each department and faculty member with a Web page like it tends to do on highly controlled corporate Web sites. In addition, resources one might normally consider to be of high repute have been found to have misinformation as well. The study by Biermann et al. (1999), for example, found inaccurate information on the Encyclopedia Britannica Web site (www.britannica.com). As noted earlier, the PDR has also been found to have incomplete information on drug dosages (J. Cohen, 2001b). A related concern is that over half of authors of clinical practice guidelines endorsed by major specialty societies have received financial support from the pharmaceutical industry, with interactions affecting over 80% of such guidelines (Choudhry, Stelfox et al., 2002). A majority of these guidelines have no mechanism to disclose such support, providing at least a potential conflict of interest. Of course, misleading and inaccurate health information is not limited to the Web. Shuchman and Wilkes (1997) assessed the problems of health news reporting in the general press, noting four problem areas: 1. Sensationalism—there are a variety of reasons for sensationalism, from the desire of media executives to sell newspapers or increase television viewing to the efforts of scientists or their institutions to garner publicity for prestige and/or funding. 2. Biases and conflicts of interest—reporters may be misled by incomplete presentation of information or undisclosed conflict of interest by scientists, institutions, or the pharmaceutical industry. 3. Lack of follow-up—the press often has a short attention span and does not continue its coverage of stories that are initially sensationalized. 4. Stories that are not covered—health-related stories compete with other stories for coverage in the press. There are also instances of scientists who have, sometimes unwittingly, signed agreements forbidding publication of research not approved by the sponsor. One well-known case involved a study showing that generic versions of a thyroid medication were comparable to a brand-name drug (Rennie, 1997). More recently, Schwartz et al. (2002) documented that of 149 scientific meeting research presentations receiving substantial attention in the news media, 76%
60
I. Basic Concepts
were nonrandomized, 25% had fewer than 30 subjects, and 15% were nonhuman studies. Furthermore, half were not subsequently published in MEDLINE-indexed journals. Another concern about health information for consumers is readability. It has been found that most patients (Overland, Hoskins et al., 1993; Foltz and Sullivan, 1996; Williams, Counselman et al., 1996) and parents of child patients (Murphy, 1994) read at an average of a fifth to sixth grade level. Reading ability also declines with age (Gazmararian, Baker et al., 1999). Those who deliver consumer health information must therefore take readability into account. The standard measure of assessing readability is the Flesch–Kinkaid score (Flesch, 1948). This measure calculates reading grade level (RGL) based on average sentence length (ASL, the number of words divided by the number of sentences) and average number of syllables per word (ASW, the number of syllables divided by the number of words): RGL (0.39 * ASL) (11.8 * ASW) 15.59
(7)
Graber et al. (1999) found that a sample of patient education material from the Web was written at a tenth-grade level. O’Mahony (1999) reviewed a sample of Web sites in Ireland and also found that the average reading level was about the tenth grade. Berland et al. (2001), who used the Fry Readability Graph (Fry, 1977), which is also validated in Spanish, determined that no English-language site they evaluated had readability below the tenth grade level, while over half were written at a college level and 11% at a graduate school level. Over 86% of Spanish sites were also written at the high school level or above. Eysenbach and Diepgen (1998) note that the problem of poor-quality and hardto-read information on the Web is exacerbated by a “context deficit” that makes poor-quality information more difficult to distinguish. These authors note there are fewer clear “markers” of the type of document (e.g., professional textbook vs patient handout) and that the reader of a specific page may not be aware of the “context” of a Web site that includes disclaimers, warnings, and so forth. Furthermore, information may be correct in one context but incorrect in another, and this difference may not be detectable on a random page within a Web site (e.g., the differences in treatment in children vs adults or across different ethnic groups). The impact of this poor quality information is unclear. A recent systematic review of whether harm has resulted from information obtained on the Internet found 15 case reports (Crocco, Villasis-Keever et al., 2002). The review noted that larger studies of whether Internet information has caused general harm to patients have not been done. Two well-known case reports of substances ordered over the Internet causing harm have been identified (Weisbord, Soule et al., 1996; Hainer, Tsai et al., 2000), and no doubt others exist, but the aggregate harm caused by misinformation has not been documented. A dissenting view has been provided by Ferguson (2002), who argues that patients and consumers actually are savvy enough to understand the limits of quality of information on the Web and that they should be trusted to discern quality using their own abilities to con-
2. Health and Biomedical Information
61
sult different sources of information and communicate with health care practitioners and others who share their condition(s). Indeed, the ideal situation may be a partnership among patients and their health care practitioners, as it has been shown that patients desire that their practitioners be the primary source of recommendations for on-line information (Tang, Newcomb et al., 1997). This lack of quality information has led a number of individuals and organizations to develop guidelines for assessing the quality of health information. These guidelines usually have explicit criteria for a Web page that a reader can apply to determine whether a potential source of information has attributes consistent with high quality. One of the earliest and most widely quoted sets of criteria was published in JAMA (Silberg, Lundberg et al., 1997). These criteria stated that Web pages should contain the following: 1. The name, affiliation, and credentials of the author—readers may differ on the value of an individual’s credentials, but the information should be listed to be assessed by all. 2. References to the claims made—if health claims are made, they should contain references to legitimate scientific research documenting the claim. 3. Explicit listing of any perceived or real conflict of interest—a conflict of interest does not disqualify someone from posting information, but all perceived or real conflicts of interests must be disclosed, as is required of those who teach continuing education courses. 4. Date of most recent uptake—even though the Web is relatively new, health information becomes outdated quickly, and the date that a page was most recently updated should be listed. Another early set of criteria was the Health on the Net (HON) codes (www.hon.ch), a set of voluntary codes of conduct for health-related Web sites. Sites that adhere to the HON codes can display the HON logo. These codes state: 1. Any medical/health advice provided and hosted on this site will only be given by medically/health trained and qualified professionals unless a clear statement is made that a piece of advice offered is from a non–medically/health qualified individual/organization. 2. The information provided on this site is designed to support, not replace, the relationship that exists between a patient/site visitor and his/her existing physician. 3. Confidentiality of data relating to individual patients and visitors to a medical/health Web site, including their identity, is respected by this Web site. The Web site owners undertake to honor or exceed the legal requirements of medical/health information privacy that apply in the country and state where the Web site and mirror sites are located. 4. Where appropriate, information contained on this site will be supported by clear references to source data and, where possible, have specific HTML links to the data. The date when a clinical page was last modified will be clearly displayed (e.g., at the bottom of the page).
62
I. Basic Concepts
5. Any claims relating to the benefits/performance of a specific treatment, commercial product, or service will be supported by appropriate, balanced evidence in the manner outlined above in Principle 4. 6. The designers of this Web site will seek to provide information in the clearest possible manner and provide contact addresses for visitors that seek further information or support. The Webmaster will display his/her E-mail address clearly throughout the Web site. 7. Support for this Web site will be clearly identified, including the identities of commercial and non-commercial organizations that have contributed funding, services or material for the site. 8. If advertising is a source of funding it will be clearly stated. A brief description of the advertising policy adopted by the Web site owners will be displayed on the site. Advertising and other promotional material will be presented to viewers in a manner and context that facilitates differentiation between it and the original material created by the institution operating the site. A number of other criteria for Web page quality have been put forth, often associated with checklists or other instruments that can be used to rate actual sites and their pages. These were reviewed by Kim et al. (1999), who found that there was fairly high agreement on the different criteria. These authors determined that most of the criteria could be grouped under 12 specific categories dealing with site content, design, disclosure, currency, authority, ease of use, accessibility, links, attribution, intended audience, contact or feedback, and user support. In another review of quality criteria, Jadad and Gagliardi (1998) took a less favorable view. These authors noted that most individual instruments were incomplete or inconsistent. They also pointed out that few such instruments have been validated. A case in point comes from the study on the quality of information concerning childhood cough described earlier (Pandolfini, Impiccatore et al., 2000); this study found no relationship between adherence to common indicators of quality and the amount of correct or incorrect information on a page. Do Web sites actually adhere to quality standards? A study by Hersh et al. (Hersh, Gorman et al., 1998) attempted to look at pages retrieved by a medical librarian attempting to answer questions generated in the course of clinical practice. They found that only 30% having a listed author, 12% having sources for claims made, and 18% showing the date of most recent update. Virtually no sites indicated any conflict of interest, regardless of whether one was in fact present. Shon and Musen (1999) found a similar low rate of quality indicators on pages about breast cancer. Other studies have shown similar rates of not having author names on pages (Doupi and van der Lei, 1999; Hatfield, May et al., 1999; O’Mahony, 1999; Latthe, Latthe et al., 2000b; Pandolfini, Impiccatore et al., 2000; Tamm, Raval et al., 2000), not providing references for claims made (Biermann, Golladay et al., 1999; Doupi and van der Lei, 1999; Gillois, Colombet et al., 1999; Hatfield, May et al., 1999; Hernandez-Borges, Macias-Cervi et al., 1999;
2. Health and Biomedical Information
63
O’Mahony, 1999; Soot, Moneta et al., 1999; Bykowski, Alora et al., 2000; Latthe, Latthe et al., 2000b; Pandolfini, Impiccatore et al., 2000; Tamm, Raval et al., 2000), not showing date of most recent update (Doupi and van der Lei, 1999; Hernandez-Borges, Macias-Cervi et al., 1999; O’Mahony, 1999; Latthe, Latthe et al., 2000a; Pandolfini, Impiccatore et al., 2000; Tamm, Raval et al., 2000), and not indicating the presence or absence of conflict of interest (Doupi and van der Lei, 1999; Gillois, Colombet et al., 1999; Hatfield, May et al., 1999; O’Mahony, 1999; Latthe, Latthe et al., 2000a). Might it be possible for Web “robots” to automatically detect these criteria? Price and Hersh (1999) have looked into this possibility. While the results were too preliminary to be definitive, early indications were that some of these criteria can be detected and the output from a search can be reordered to give more prominent ranking of higher quality pages. One observation from the small data set used to evaluate the system was that the quality criteria listed earlier may not truly be associated with the actual quality of pages. These investigators noted, for example, the low-quality pages were more easily identifiable from their exclamation points and “1-800” telephone prefixes to call to order products. Fallis and Fricke (2002) investigated the predictive nature of Web page attributes more comprehensively, attempting to ascertain which ones might be associated with quality. They assessed pages providing advice on treating childhood fever in the home, an area in which wide consensus exists among experts. Pages were rated for accuracy by two independent observers. Those above the median accuracy scores were deemed “more accurate” and those below were deemed “less accurate.” The authors found the following indicators to be most predictive of more accurate pages: • Organizational domain (i.e., a .org domain as opposed to a .com or .edu domain) • Display of the HON code logo • Claim of copyright Among the indicators not predictive of quality were those related to the criteria of Silberg et al. such as listing of authorship, currency, or references. Other nonpredictive indicators were the presence of spelling errors, exclamation points, advertising, and a high number of in-links to the page. A similar study by Kunst et al. (2002) similarly showed that source, currency, and evidence hierarchy had only moderate correlation with accuracy of information for five common health topics on well-known health-related Web sites. These authors concluded that apparently credible Web sites may not provide more accurate health information. Of course, one downside to publishing data like these and using them to judge quality of output algorithmically is that those who seek to deceive may be motivated to “game” the system to make their pages to appear to be of higher quality. If these criteria do not completely discriminate quality of Web pages, then what is a healthcare provider or patient to do? For the instances of pages that promote clearly unproven treatments, the task is fairly straightforward. The Fed-
64
I. Basic Concepts
eral Trade Commission (FTC) has provided a set of guidelines to detect “virtual” treatments with a higher likelihood of being unproven (Anonymous, 2001w). Such pages are likely to contain: • Phrases like “scientific breakthrough,” “miraculous cure,” “exclusive product,” “secret formula,” and “ancient ingredient.” • Use of “medicalese”—impressive-sounding terminology to disguise a lack of good science. • Case histories from “cured” consumers claiming amazing results. Such testimonials also imply, that the experience recounted is typical for all consumers using the product or service. A Web page visitor who sees such a testimonial is well advised to ask for proof that the term “typical” has been properly used. • A laundry list of symptoms the product cures or treats. • The latest trendy ingredient touted in the headlines. • A claim that the product is available from only one source, for a limited time. • Testimonials from “famous” medical experts. • A claim that the government, the medical profession, or research scientists have conspired to suppress the product. When the information is less flagrantly invalid, the options are less clear. One might surmise that the prestige of the institution could be an indicator of quality. This is unfortunately not the case: several of the studies documenting incorrect information found some of their poor-quality information on the sites of prestigious medical institutions or highly reputable publishers. Could governments play a role in regulating the quality of health information? Most believe that their role is likely to be limited. Hodge et al. (1999), for example, note that while the FTC has a mandate to regulate false or deceptive commercial information and the Food and Drug Administration is charged with regulating information about drugs and medical products, neither is likely to be able to comprehensively monitor all commercial health information on the Internet. And of course, an increasing fraction of that information is likely to arise outside the country, where U.S. governmental entities have no jurisdiction at all. These authors conclude that self-regulation and education is likely to be the best approach. Such regulation may take the form of principles like those maintained by the American Medical Association for its Web site (Winker, Flanagin et al., 2000). These guidelines cover not only issues related to quality, but also advertising, sponsorship, privacy, and electronic commerce. Another approach to insuring Web site quality may be accreditation by a third party. The American Accreditation HealthCare Commission (called URAC), recently announced a process for such accreditation (www.accreditation.urac.org). The URAC standards manual provides 53 standards to support accreditation. They are based on a group of consumer protection principles developed by the HiEthics initiative (www.hiethics.com) supported by many major Internet health sites and content providers. The URAC standards cover six general issues: health content editorial process, disclosure of financial relationships, linking to other Web sites, privacy and security, consumer complaint mechanisms, and internal
2. Health and Biomedical Information
65
processes required to maintain quality over time. To receive accreditation, sites will have to meet the requirements listed in the standards manual. Thirteen commercial Web sites were in the first group awarded accreditation (Fox, 2001).
2.7 Use of Knowledge-Based Health Information Information sources, print or computer, are approached for two reasons: the need to locate a particular item of information, such as a document or book, or the need to obtain information on a particular subject. Subject needs, according to Lancaster and Warner (1993), fall into three categories: • The need for help in solving a certain problem or making a decision • The need for background information on a topic • The need to keep up with information in a given subject area The first two are called retrospective information needs, in that documents already published are sought, while the latter need is called a current awareness need, which is met by filtering new documents to identify those on a certain topic. Retrospective needs may also be classified by the amount of information needed (Lancaster and Warner, 1993): • A single fact • One or more documents but less than the entire literature on the topic • A comprehensive search of the literature It will be seen later that the interaction with an information system varies based on these different needs. Wilkinson and Fuller (1996) describe four types of information needs for document collections: • • • •
Fact-finding—locating a specific item of information Learning—developing an understanding of a topic Gathering—finding material relevant to a new problem not explicitly stated Exploring—browsing material with a partially specified information need that can be modified as the content is viewed
Another perspective on the use of information classifies the kinds of information needs characteristic of users of health information. Gorman (1995) defines four states of information need: • Unrecognized need—clinician unaware of information need or knowledge deficit • Recognized need—clinician aware of need but may or may not pursue it • Pursued need—information seeking occurs but may or may not be successful • Satisfied need—information seeking successful This section focuses on the information needs and uses of healthcare practitioners. A recent review summarized all studies of clinicians’ information seeking
66
I. Basic Concepts
(McKnight and Peet, 2000). The discussion begins with an overview of models of physician thinking, followed by descriptions of the information they need and the sources they actually use, as well as comments on the usage of information by nurses and other practitioners.
2.7.1 Models of Physician Thinking Most work assessing the mental process of health care has focused on physicians. The traditional view of physician thinking is based upon the hypotheticodeductive model (Elstein, Shulman et al., 1978b). In this model, the physician begins forming hypotheses based on the initial information obtained, usually the patient’s chief complaint. The skilled physician already begins to focus on datadriven hypotheses, which subsequently lead to hypothesis-driven selection of the next data to be collected. The process is iterated until one or more diagnoses can account for all the observations (or at least the observations deemed necessary to explain). An alternative model, which is not necessarily at odds with the hypotheticodeductive view, has been proposed by Schmidt et al. (1990). These authors note that one implication of the hypotheticodeductive model is that diagnostic failures arise from taking shortcuts or giving insufficient attention to details. However, they have observed that experienced clinicians actually gather smaller amounts of data and are able to arrive at correct diagnoses with fewer hypotheses. Schmidt et al. theorize that medical knowledge is contained in “illness scripts,” which are based not only on learned medical knowledge, but also past (and especially recent) experience. These scripts are based on causal networks that represent objects and their relationships in the world. These networks tell physicians, for example, that fluid in the lungs causes shortness of breath and that one of the causes of fluid in the lungs is heart failure. Medical education consists of building these causal networks in the mind. As the student progresses through medical education and attains clinical experience, the networks become compiled into higher level, simplified models that explain patients signs and symptoms under diagnostic labels. There is considerable evidence for this model. First, the hypotheticodeductive model might imply that those with the best problem-solving skills should consistently be the best diagnosticians. Yet studies in which physicians and medical students were given patient management problems (PMPs), which simulate the clinical setting, find that there is wide variation in performance on different problems by the same practitioners (Elstein, Shulman et al., 1978b). Additional supporting evidence is that experienced physicians are much better able than students to recall details of patient encounters when the findings are randomly reordered (Schmidt, Norman et al., 1990). This is because more experienced practitioners attach specific patients to instances of the scripts. Another finding in support of this model is from Patel et al. (1989), who noted that experienced physicians tend to make minimal use of basic science in their diagnostic efforts;
2. Health and Biomedical Information
67
rather, they match patients to patterns of clinical presentations for various diseases. This is consistent with advanced training leading to building high-level scripts based on clinical findings. Florance (1992) also looked at aspects of clinical decision making and noted that it involves both declarative knowledge (i.e., facts) and procedural knowledge (i.e., how to apply those facts). She noted that the latter tends to be more useful for diagnosis, while the former is usually more beneficial for therapy. Since there tends to be more declarative knowledge in the literature, she calls for more procedural knowledge to be added to the literature. Of course computer applications, such as decision support systems, may be able to fill this void. Some researchers express concern that clinicians rely too much on personal knowledge and experience and not enough on aggregated experience and/or published literature. Sox et al. (1988) note an availability heuristic, where clinicians inflate the diagnostic probability of a disease based on recent or otherwise wellremembered cases. Tanenbaum (1994) carried out an ethnographic study of physicians being exposed to outcomes and research data, finding that they often continued to rely on personal experience despite contrary data from outcomes research or the medical literature. Some medical editors have lamented that the journal literature is underused (Huth, 1989; Kassirer, 1992), yet others point out that this resource is too fragmented and time-consuming to use (Shaughnessy, Slawson et al., 1994; Hersh, 1999). McDonald (1996) has noted that there is often not enough evidence to inform clinical decisions and that clinicians rely on heuristics to guide their decision-making, advocating that such heuristics be improved when the robustness of evidence is sparse.
2.7.2 Physician Information Needs As noted by Gorman (1995), physicians may have information needs that they do not immediately recognize. Some of these needs may become recognized, pursued, or satisfied. Understanding these needs is important to building healthcare IR systems. 2.7.2.1 Unrecognized Needs One of the difficulties in characterizing unrecognized information needs derives from the physicians’ lack of direct awareness of their existence. As such, these needs can be identified only indirectly through the measurement of knowledge dissemination, knowledge stores, and outcomes of clinical practice that reflect application of knowledge. Several older studies have demonstrated that medical knowledge is disseminated only slowly to the practitioner. Stross and Harlan (1979) looked at the dissemination of information on the use of photocoagulation in diabetic retinopathy, an important advance against the blindness that can complicate diabetes. Over 2 years after initial publication of the benefit of this therapy, less than half
68
I. Basic Concepts
of primary care physicians were aware of the results. A similar finding was reported when physicians were asked about their knowledge of the Hypertension Detection and Follow-Up Study, which demonstrated the benefit of antihypertensive therapy: only half the physicians queried were aware of the findings 2 to 6 months after publication (Stross and Harlan, 1981). Williamson et al. (1989) performed a similar study, looking at six important then-recent medical advances and finding that anywhere from 20 to 50% of physicians were unaware of them. It is likely that the general increase of mass media coverage to medial topics has increased the dissemination of medical advances, especially in the Internet age, but no recent studies have ascertained quantitatively whether this is the case. Another line of evidence demonstrating lack of information comes from physicians who take recertification examinations. The scores of family practitioners, who are required to recertify every 6 years, tend to decline with each recertification examination (Leigh, Young et al., 1993). Likewise, when internists with various levels of experience were given an 81-question subset of the American Board of Internal Medicine examination, a direct correlation was found between score and years out of residency training (Ramsey, Carline et al., 1991). Both of these studies were limited in two ways. First, it is unknown how examinations of these types correlate with practice skill. It has already seen that experience is an important variable in addition to knowledge for physicians (recall the model of Schmidt, Norman et al., 1990). Second, physicians do use during their practice information resources that were not available during the test-taking situations. Thus some physicians who performed poorly at regurgitating knowledge might be quite effective at finding the required information and applying it. Does this lack of information have a significant impact on patient care? This is a complex question, which is difficult to answer due to the many variables present in a clinical encounter. There is evidence, however, that clinicians could be making better decisions. A report by the President’s Advisory Commission on Consumer Protection and Quality in the Health Care Industry noted that significant problems in health care include medical error, overuse of services, underuse of services, and unexplained variation in use of services (Anonymous, 1998). A widely cited report by the Institute of Medicine has noted that twenty-first century health care must be quality-focused, patient-centered, and evidence-based (Anonymous, 2001d). Specific studies have shown, for example, the antibiotics continue to be prescribed inappropriately 25 to 50% of the time, according to infectious disease experts (Kunin, Tupasi et al., 1973; Bernstein, Barriere et al., 1982; Gonzales, Bartlett et al., 2001). Likewise, adherence to published practice guidelines continues to be low: only 45 to 84% of recommended routine screening exams are performed for diabetic patients (Weiner, Parente et al., 1995) and only one-quarter of elderly patients are treated for hypertension in a way consistent with guidelines (Knight, Glynn et al., 2000). Similar lack of use of proven effective therapies for acute myocardial infarction has been demonstrated as well (Ellerbeck, Jencks et al.,
2. Health and Biomedical Information
69
1995). Of course the problems concerning the quality of health care, patient safety, and medical errors go well beyond access to IR systems, though the use of such systems will play a solution. 2.7.2.2 Recognized Needs Physicians and other healthcare practitioners do recognize they have unmet information needs. A number of studies described in this section have attempted to gain insight into the quantity and nature of questions asked in clinical practice. Many investigators have attempted to measure such needs, although their results differ as a result of variations in practice settings, types of physician studied, and how information need itself was defined (Gorman, 1995). Even though the results are not entirely consistent, they do reveal that physicians have significant unmet information needs. The first study of this type, performed by Covell et al. (1985), found that while physicians thought they had an unmet information need for about one out of every 77 patients, they actually had an average of two unmet needs for every three patients (0.62 unanswered questions per patient). Using a similar methodology with different physicians, Gorman and Helfand (1995) found a nearly identical frequency (0.60 per patient) of unmet information needs. The former study assessed urban internists and specialists in Los Angeles, while the latter focused on urban and rural primary care physicians in Oregon. Other studies using different methodologies have obtained varying results, though they all demonstrate that physicians have significant unmet information needs in practice. Timpka and Arborelius (1990) studied “dilemmas” in a simulated environment with general practitioners in Sweden. While the most common dilemma type was social and organizational, there was an average of 1.84 medical knowledge dilemmas per patient. Osheroff et al. (1991), who used ethnographic techniques to assess information needs in teaching hospital rounds, found an average of 1.4 questions per patient (excluding those asked in the course of teaching). Dee and Blazek (1993) measured unmet needs in after hours interviews and found an average of 0.33 questions per patient. Ely et al. (1999) observed family physicians in Iowa who were observed to have 0.32 questions per patient. Most of the foregoing studies attempted to identify the nature of the information needs observed. All found that they were highly specific to patient problems. Ely et al. (1999) developed a taxonomy of generic questions, finding 69 different types, the top 10 of which are listed in Table 2.6. This table also gives the percentage of asked questions that were pursued as well as pursued questions that were answered, showing in general that treatment questions were more likely to be pursued and answered than diagnostic ones. This taxonomy was refined and validated with questions from Oregon primary care practitioners, and question types were found to be assignable with moderate reliability (kappa 0.53) (Ely, Osheroff et al., 2000).
70
I. Basic Concepts
TABLE 2.6. Questions Most Commonly Asked, Pursued, and Answered
Generic question What is the cause of symptom X? What is the dose of drug X? How should I manage disease or finding X? How should I treat finding or disease X? What is the cause of physical finding X? What is the cause of text finding X? Could this patient have disease or condition X? Is test X indicated in situation Y? What is the drug of choice for condition X? Is drug X indicated in situation Y?
How many asked? (%)
How many asked were pursued? (%)
How many pursued were answered? (%)
9 8 7
9 85 29
50 97 83
7 7 4 4
33 18 40 14
72 46 72 67
4 3
29 47
83 76
3
25
78
Source: Ely, J., Osheroff, J., et al. (1999). Analysis of questions asked by family doctors regarding patient care. British Medical Journal, 319:358–361. Courtesy of the British Medical Journal Publishing group.
2.7.2.3 Pursued Needs One consistent finding in the studies of unmet information needs was that physicians decided against pursuing answers for a majority of the questions. Covell et al. (1985), Gorman and Helfand (1995), and Ely et al. (1999) found that answers were pursued only 30 to 36% of the time, indicating that 64 to 70% of information needs remained unmet. These studies also consistently found that when information was pursued, the most common sources were other humans, followed closely by textbooks and drug compendia. Use of journal articles as well as computer sources was low. Gorman and Helfand (1995) attempted to define the factors that were most likely to be associated with the decision to pursue an answer to a question. They defined 11 attributes of clinical questions and used multiple logistic regression in an attempt to identify those most likely to correlate with an answer being sought, as shown in Table 2.7. The most likely factors to cause answer seeking were as follows: the question required an urgent answer, it was likely to be answerable, and it would help manage other patients besides the one who had generated the question. The potential for a question to benefit a physician’s general knowledge or to reduce his or her liability risk was not associated with pursuit of an answer. Covell et al. (1985), who also attempted to identify the impediments to answer seeking, found that physicians either were too busy or did not have immediate access to an answer. Another significant impediment to information seeking was the disarray of the typical practitioner’s library, consisting of out-of-date textbooks and inadequately indexed journal collections. A number of survey studies have attempted to delineate the information resources used by physicians more precisely. One limitation in all these studies is that the information milieu has changed substantially in recent years. Indeed, as
2. Health and Biomedical Information
71
TABLE 2.7. Factors Influencing a Physician’s Decision to Seek an Answer to a Question Factors most associated with pursuit of an answer Urgency—The question had to be answered soon. Answerability—The physician felt an answer was likely to exist. Generalizability—An answer would help manage other patients. Factors not associated with answer seeking Knowledge—How much was previously known about the problem? Uneasiness—How uneasy did the physician feel about the problem? Potential help—An answer could help the patient. Potential harm—Not having an answer could hurt the patient. Edification—An answer would benefit the practitioner’s general knowledge. Liability—The problem involved liability risk. Knowledge of peers—Peers of the practitioner know the answer. Difficulty—How difficult would it be to find the answer? Source: Gorman and Helfand (1995).
noted in the survey of physician computer use by the American Medical Association described in Chapter 1, the number of physicians using the Internet nearly doubled between 1999 and 2000 (Anonymous, 2001a). As such, the use of knowledge resource use by physicians has likely changed substantially since many of the studies described throughout this section were performed. Nonetheless, some insight can be gained from looking at the resources used, which indicate the types of information physicians are wishing to pursue. Ely et al. (1999) looked at information resources available in the offices of 103 family physicians. Among the resources owned were books (100%), reprint files (68%), posted (i.e., on wall or door) information (76%), and computers (26% in the office, but 45% having one at home), and “peripheral brains” (i.e., personal notebooks of clinical information) (29%). All physicians owned a drug prescribing reference, and all but one owned a general medical textbook. Other books likely to be owned covered adult infectious disease (89%), general pediatrics (83%), orthopedics (82%), dermatology (79%), and adult cardiology (77%). Most studies of information needs and use have focused on primary care physicians. Shelstad and Clevenger (1996), however, assessed general surgeons in New Mexico and found that their needs were for the most part met with traditional paper resources. Significant impediments to the use of newer electronic resources existed for this group of surgeons. The most common reasons for seeking information among the New Mexico surgeons were patient care (98%) and continuing education (83%). The most common sources of information were professional meetings (97%), medical literature (96%), and colleagues (93%). Those who found it difficult to access information resources most often listed time demands of practice (71%), isolation from medical schools (30%), and computer illiteracy (28%). As noted earlier, physicians have historically relied on other physicians to meet a significant fraction of their information needs. Perhaps another ap-
72
I. Basic Concepts
proach to meeting information needs through technology might be to use the latter more in a communications mode. Bergus et al. (1998) set up an e-mail consultation service for primary care clinicians to ask questions of specialists at the University of Iowa. Forty clinicians used the service 237 times in 18 months, with high satisfaction among both those asking and answering the questions. 2.7.2.4 Satisfied Needs What information resources are most likely to satisfy an information need? In a study of knowledge resource preferences of family physicians, Curley et al. (1990) found that “cost” variables (e.g., availability, searchability, understandability, clinical applicability) were more closely associated with the decision to use a resource than “benefit” variables, such as its extensiveness and credibility. As with the studies described earlier, the family physicians in this study were more likely to use textbooks, compendia, and colleagues for information seeking over journal literature and bibliographic resources (Connelly, Rich et al., 1990). These observations have led Shaughnessy et al. (1994) to propose a formula for the usefulness of information: Usefulness
Relevance Validity Work
(8)
That is, the value of information is proportional to its relevance and validity to the clinical situation and inversely proportional to how difficult it is to find and apply. Some studies have also attempted to measure the use of computerized information sources. A number of these will be described in greater detail in Chapter 7, where the focus will be on how often and how successfully they are used. Not surprisingly, most of the early studies of pursued information needs found very little use of computers for seeking information. Covell et al. (1985), Williamson et al. (1989), Curley et al. (1990), and Gorman and Helfand (1995) found that fewer than 10% of physicians regularly use online literature searching. Even when use of online searching is prevalent, the frequency of its use pales with the frequency of information needs. In both inpatient (Haynes, McKibbon et al., 1990) and outpatient (Hersh and Hickam, 1994) settings, observed usage is never more than a few times per month among medical residents and faculty, which is far below the unmet information need frequency in the studies described. Gorman et al. (1994) addressed the issue of whether physicians can meet their information needs through computer-based information sources. They chose a random sample of 50 questions that arose in clinical care where physicians chose not to pursue an answer. These questions were then given to librarians, who performed a literature search and returned three articles that might be relevant to the physician. In 28 instances (56%), the physicians indicated that at least one
2. Health and Biomedical Information
73
of the articles was relevant to the question. They also stated that for 22 questions (46%), a clear answer to the question was provided, and for 19 questions (40%), the information would have had an impact on care of the patient. The limitation of this approach to obtaining information for physicians, they noted, was the time (averaging 43 minutes per search per librarian) and cost of searching (averaging $27 per search). Gorman also found in a subsequent study that only a third of the studies contained “high quality” evidence, such as randomized controlled trials of therapeutic interventions (Gorman, 1993). A similar study was performed by Giuse et al. (1994), who looked at the answerability of questions about AIDS. Even though the study was simulated (i.e., questions were generated via chart review of patients already seen), the general categories of questions and their likelihood of being answered were comparable to those from field studies. About 92% of the questions were answerable: fourfifths during a first phase that utilized information resources commonly available to general internists (paper-based general textbooks and MEDLINE) and the remainder during a more intensive second phase undertaken in an academic medical library. The most useful resources for the latter phase were specialty textbooks. This study showed that a variety of resources were needed to answer clinical questions in this domain. Although few studies have addressed these questions in light of the massive growth of physician use of computers and the Internet, it is likely that with the growth of the Web, along with increasing use of handheld and other portable devices, physicians are using information resources during practice more frequently. There are still, however, impediments, which are due to a variety of factors, such as the need to find a computer use, and the amount of time required to log on to the appropriate resource and perform searching. Chueh and Barnett (1997) have argued that a new model is needed for delivery of information to clinicians based on “just in time” principles of inventory management from the manufacturing industry. The model requires not only that information be readily (i.e., quickly) available, but also that it be specific to the context of the patient. One approach to doing this involves attempting to link from the context of the electronic medical record (e.g., diagnoses, medications, and conditions detected, such as rapid decrease in hemoglobin level) directly to appropriate information (Cimino, 1996). Studies remain to be done to determine whether direct linking based on information in the record or providing a pop-up query box is faster.
2.7.3 Information Needs of Other Healthcare Professionals The information needs and usage of nonphysician healthcare providers has been much less studied. In many ways, however, the data on nurses, the next-most studied group, show a picture similar to that for physicians. One limitation of these studies, however, is that almost all data are derived from surveys, without direct observation. While this does not necessarily invalidate the results, one of the studies cited earlier showed considerable divergence between reported and observed needs (Covell, Uman et al., 1985), while another research group found
74
I. Basic Concepts
a wide disparity when they measured pursued (Ely, Bruch et al., 1992) versus unmet needs (Ely, Osheroff et al., 1999). The survey studies across different nursing professionals do show, however, fairly consistent results. Like physicians, nurses tend to rely on their own collections of textbooks and journals as well as on colleagues. A British study found the most common sources of information used for nursing problems were books (22%), followed by nursing colleagues (21%), journals (14%), and medical colleagues (11%) (Williamson, 1990). An American study found an even larger percentage of nurses consulting other people for information needs as well as a lack of awareness of library resources (Bunyan and Lutz, 1991). On the latter point, it was noted that nurses comprised 34% of the hospital staff, yet only 6% of the hospital’s library patrons. Spath and Buttlar (1996) studied acute-care nurses in Ohio, most of whom were staff nurses with an RN degree. The most common information resources used were professional journals (79.4%), other nurses (64.7%), and the library card catalog (51.0%). Use of online or CD-ROM materials was minimal. More than half the respondents used the library for personal interest in a subject (68.6%) or to obtain information about a diagnosis (59.8%). Half reported that they read no or one professional journal regularly and another 42% reported reading 2 or 3. A total of 9% spent zero hours per month reading professional literature, while 45% spent 1 to 3 hours, 26% spent 4 to 6 hours, and the remainder spent more. The one study of observed needs was an ethnographic study of general medical and oncology nurses in Canada (Blythe and Royle, 1993). It was found that nurses sought information in two general areas, individual patient care and broader nursing issues. Most information seeking was related to patient care. Information seeking not related to patient care was generally done during quiet periods on a shift, while professional developmental reading tended to be done at home. Urquhart and Crane (1994) attempted to assess nurses’ information skills by using a simulated vignette in the library. They predefined “evidence of an information-seeking strategy” as behavior tending to support a belief that certain activity would yield a particular type of information or indicate that a particular sequence of steps was followed in gaining information. Slightly under half of all subjects displayed an information-seeking strategy, with those displaying such a strategy much more likely to consult more than two information sources. Both groups were equally likely to consult colleagues for information, and while those with an information-seeking strategy were more likely to use the library, they were no more likely to use literature searching. Corcoran-Perry and Graves (1990) performed a study of information seeking by cardiovascular nurses that went beyond the use of knowledge-based resources and included seeking of patient-specific information as well. They found that the most common reason for using “supplemental” (not available in personal memory) information was for patient-specific data (49%), such as general information about the patient, medications, or laboratory reports. The next most common type of information sought was institution-specific information (27%), such as tracking equipment, medications, reports, or people. The seeking of domain knowledge represented only 21% of supplemental information sought, being most commonly related to medications or cardiovascular conditions. The most com-
2. Health and Biomedical Information
75
mon sources of supplemental information were other people (nurses 26% of the time and other personnel 19% of the time), patient records (25%), references (15%), and a computer terminal (10%). Beyond nursing, there are few information needs studies of other healthcare professionals. Dental hygienists (Gravois, Fisher et al., 1995) and physical therapists (Bohannon, 1990), like physicians and nurses, have been found to rely on professional journals, colleagues, and continuing education, with little use of bibliographic databases. Among rural healthcare practitioners in Hawaii, allied health professionals, social workers, and administrators tended to use journal articles less and newsletters, videos, and resource directories more commonly than physicians used them (Lundeen, Tenopir et al., 1994). They consulted colleagues at a rate comparable to that of physicians and rarely made use of the university library’s online databases. Another group in the healthcare field whose information needs have been minimally assessed is biological researchers. This is particularly pertinent in light of the recent growth of bioinformatics tools that they increasingly rely on to perform their work. Stevens et al. (2001) recently classified the tasks of such researchers to determine which bioinformatics tools would be most valuable to them.
2.8 Evidence-Based Medicine As noted already in this chapter, there has been considerable effort devoted to organizing scientific literature and teaching clinicians to use it to find the best evidence for making clinical decisions. The philosophy of this approach, EBM, is that clinical care should be guided by the best scientific evidence. As was seen in Figure 1.4, while factors in addition to evidence influence medical decision making, when evidence is used in a decision, it should be of the highest quality. In fact, EBM is really just a set of tools to inform clinical decision making. It allows clinical experience (art) to be integrated with best clinical science. Also, EBM makes the medical literature more clinically applicable and relevant. In addition, it requires the user to be facile with computers and IR systems. There are many well-known resources for EBM. The original textbook by Sackett et al. (2000) is now in a second edition. A series of articles published in JAMA is now available in handbook (Guyatt and Rennie, 2001) and reference book format (Guyatt, Rennie et al., 2001). There are a number of comprehensive Web sites, including the Oxford University Centre for Evidence-Based Medicine (cebm.jr2.ox.ac.uk), the Canadian Centres for Health Evidence (www.cche.net), and the Mount Sinai Hospital Centre for Evidence-Based Medicine (www.library.utoronto.ca/medicine/ebm). The process of EBM involves three general steps: • Phrasing a clinical question that is pertinent and answerable • Identifying evidence (studies in articles) that address the question • Critically appraising the evidence to determine whether it applies to the patient The phrasing of the clinical question is an often-overlooked portion of the EBM process. There are two general types of clinical question: background questions
76
I. Basic Concepts
and foreground questions (Sackett, Richardson et al., 2000). Background questions ask for general knowledge about a disorder, whereas foreground questions ask for knowledge about managing patients with a disorder. Background questions are generally best answered with textbooks and classical review articles, whereas foreground questions are answered using EBM techniques. Background questions contain two essential components: a question root with a verb (e.g., what, when, how) and a disorder or aspect of a disorder. Examples of background questions include What causes pneumonia? and When do complications of diabetes usually occur? Foreground questions have three or four essential components: the patient and/or problem, the intervention, the comparison intervention (if relevant), and the clinical outcome(s). There are four major foreground question categories: • • • •
Therapy (or intervention)—benefit of treatment or prevention Diagnosis—test diagnosing disease Harm—etiology of disease Prognosis—outcome of disease course
The original approach to EBM, called “first-generation” EBM by Hersh (1999), focused on finding original studies in the primary literature and applying critical appraisal. As already seen, accessing the primary literature is challenging and time-consuming for clinicians for a variety of reasons. This has led to what Hersh (1999) calls “next-generation” EBM and focuses on the use of “synthesized” resources, where the literature searching, critical appraisal, and extraction of statistics operations are performed ahead of time. This approach puts EBM resources in the context of more usable information resources as advocated in the InfoMastery concept of Shaughnessy et al. (1994) and the JIT information model of Chueh and Barnett (1997).
2.8.1 First-Generation EBM When first developed, EBM focused on identifying the appropriate foreground question, finding evidence in the primary literature (usually individual studies), and critically appraising it. Table 2.8 gives an overview of the major question types, the best studies for evidence, and the most important questions to ask for study design. Discussed later are the best sources of information for these studies (Chapter 4) and how well various strategies work in finding them (Chapters 6 and 7). After studies with evidence have been identified, they need to be critically appraised by three questions: • Are the results valid? • What are the results? • Will they help in caring for the patient? RCTs are considered to be the best evidence for therapy or interventions because their basic design minimizes bias by ideally keeping everything the same except for the intervention being studied, which is then randomized. For example, while many individuals tout the value of vitamin C for the prevention of the
2. Health and Biomedical Information
77
TABLE 2.8. Overview of Major Question Types, Best Studies for Evidence, and Most Important Questions to Ask for Study Design in Evidence-Based Medicine Category of question
Best type of study for evidence
Therapy
Randomized controlled trial
Diagnosis
Blinded comparison with gold standard
Harm
Randomized controlled trial (if possible) or case–control study
Prognosis
Follow-up of representative and well-defined group with a disease or condition
Most important questions to ask of study design Was the assignment of patients to treatments randomized? Were all the patients who entered the trial properly accounted for and attributed at its conclusion? Was there an independent, blind comparison with a reference standard? Did the patient sample choice include an appropriate spectrum of patients similar to that to which the test would be applied in clinical practice? Where there clearly identified comparison groups that were similar with respect to important determinants of outcome (other than the one of interest)? Were outcomes and exposures measured in the same way in groups being compared? Was there a representative patient sample at a well-defined point in the course of the disease? Was follow-up sufficiently long and complete?
common cold, a meta-analysis of numerous RCTs has shown the vitamin to be ineffective for this purpose (Hemila, 1997). Also, RCTs tend to emphasize clinical end points and patient-oriented outcomes. It has been observed that minor cardiac arrhythmias, such as premature ventricular contractions after an acute myocardial infarction, are associated with more serious arrhythmias, such as ventricular fibrillation, which is fatal without defibrillation. As a number of new drugs were discovered that reduced the frequency of these minor arrhythmias, it was assumed this would result in a corresponding decrease of more serious arrhythmias. However, RCTs (strengthened by follow-up meta-analyses) have shown that while the drugs do suppress arrhythmias, they are associated with a higher mortality because of other problems arising from their use (Sadowski, Alexander et al., 1999). All kinds of health intervention, from surgery to alternative medicine, can be assessed by means of RCTs. There is nothing inherently “biomedical” about them. There are other study designs that can be used to assess interventions, and it may be that their evidence yields similar results to RCTs. Both prospective cohort studies and retrospective case-control series have been found to lead to similar results in one analysis of five clinical topics (Concato, Shah et al., 2000) and in another analysis of 19 clinical topics (Benson and Hartz, 2000). It has also been argued that papers presenting evidence of lower quality, namely, case reports and case series, still have value in the medical literature (Vandenbroucke, 2001). They potentially allow recognition of new diseases, new manifestations of old diseases, and detection of adverse effects of treatment. Others have noted that RCTs have their limitations as well, such as the modification of the intended treatment (due
78
I. Basic Concepts
to protocol error or lack of compliance) and the lack of outcome determination (due to difficulty in ascertaining outcomes or because the patient was not studied for long enough) (Rabenback, Viscoli et al., 1992). Once a therapy or intervention study has been found to be valid and applicable to a patient, the next step is to determine its treatment effects, that is, how well the treatment works. Treatment effect can be assessed with two measures, one determining how much the risk of an adverse outcome is relatively reduced by the treatment (relative risk reduction, or RRR) and the other stating how many patients must be treated to reduce the adverse outcome by one patient (number needed to treat, or NNT). These are calculated by determining the control event rate (CER, the rate of adverse outcomes for the control group) and the experimental event rate (EER, the rate of adverse outcomes for the experimental group). The RRR is then calculated by: CER EER RRR CER
(9)
To calculate the NNT, one must first know the absolute risk reduction (ARR) of the therapy: ARR CER EER
(10)
1 NNT ARR
(11)
NNT is then calculated by:
The precision of these values can be calculated by the 95% CI. In addition, NNT can be calculated from the OR that is commonly report in meta-analyses by using the patient-expected event rate (PEER): 1 [PEER (1 OR)] NNT (1 PEER) PEER (1 OR)
(12)
Some therapies, or course, lead to harm. Others may provide some benefit and some harm. A similar number needed to harm (NNH) can be calculated similar to NNT, and a likelihood of help versus harm (LHH) can be calculated, which balances the two values to determine whether the treatment is suited for the patient (Sackett, Richardson et al., 2000). Although these measures are considered to be essential components of EBM, less than 3% of RCTs in the literature actually report them, leading to calls for the inclusion of NNT and NNH data as standard parts of the RCT reporting process (Cheng, Melnikow et al., 2001). The evidence for diagnostic tests uses different measures and studies, since the goal is to determine whether a disease is or is not present. The measures used to assess the accuracy of a diagnostic are sensitivity and specificity. To calculate them, a 2 2 contingency table is set up (Table 2.9). Sensitivity is the proportion of patients with the disease who have a positive test: a Sensitivity ac
(13)
2. Health and Biomedical Information
79
TABLE 2.9. Contingency Table for Assessing a Diagnostic Test Disease Diagnostic test result Positive Negative
Present
Absent
a c
b d
Specificity is the proportion of patients without the disease who have a negative test: d Specificity bd
(14)
There is an inverse relationship between the two measures. If one sets the “cutoff” for disease more stringently with a diagnostic test, then the specificity will increase while the sensitivity will decline. From these measures, Bayes’ theorem can be used to determine the probability of a disease being present before and after a diagnostic test (Sackett, Richardson et al., 2000). Another measure sometimes calculated for a test is its positive predictive value (PPV), or the proportion of patients with a positive test who actually have the disease: a PPV ab
(15)
Questions of harm can be answered with RCTs, but in most instances it would be unethical to deliberately expose a person to harm (i.e., a researcher could not in good conscience “randomize” someone to an agent solely to determine its harmfulness). Harm can be measured in the instance of an RCT when it is assessed as adverse effects from a therapy that is being tested for its benefits. But usually harm must be measured from observational study designs, ideally where two groups of otherwise similar individuals differ only in their exposure to the potentially harmful agent. The best of these designs are the cohort study, where individuals are followed forward prospectively, and the case–control study, where individuals are matched as identically as possible, differing only with respect to exposure to agent. To quantify the effect of the harmful agent, the NNH can be calculated by the reciprocal of the absolute rate increase (ARI) of the event caused by exposure to it: 1 NNH ARI
(16)
A number of limitations of first-generation EBM have been noted (Hersh, 1999): 1. Time—as mentioned earlier and described in more detail in Chapter 7, the process outlined in this section takes a great deal of time, which a busy clinicians is unlikely to have. 2. Expertise—although this is changing, very few clinicians are skilled in applying EBM as described here. 3. Comprehensiveness—relying on single studies for EBM calculations is prob-
80
I. Basic Concepts
lematic because there are often numerous studies on a given topic. 4. Publication bias—as seen earlier, publication bias may result in the exclusion of some results from the literature. Another concern about EBM is whether its skill set can be learned by clinicians. Most studies of instruction in EBM have focused on medical students and residents. A systematic review of earlier studies found that gains in knowledge were more likely to occur when the subjects instructed in its use were students rather than residents (Norman and Shannon, 1998). A more recent study of first-year internal medicine residents found that skills in question formulation, literature searching, and understanding quantitative outcomes were significant and durable over time, although skills in critical appraisal were not (Smith, Ganschow et al., 2000).
2.8.2 Next-Generation EBM The limitations of first-generation EBM have led to next-generation EBM, where the premise is that not every clinician should need to do critical appraisal on each problem de novo. The goal of next-generation EBM is to create “synthesized” or “predigested” EBM content. This approach will likely assist in overcoming the first-generation problems of time (evidence is found more quickly), expertise (critical appraisal is done by experts), and comprehensiveness (techniques like meta-analysis are a staple of this approach). Of course, the problem of publication bias will not be ameliorated with this approach and may actually be exacerbated, given the greater reliance on meta-analysis. There are a number of tools of “next-generation” EBM (Hersh, 1999). The first of these is the more informative abstract (e.g., Best Evidence, which provides extended, structured abstracts for the most important articles recently published in the clinical medicine primary literature). Such abstracts provide a concise but structured summary of the original study, along with a brief commentary. By trying to distill many pages down to bare essential information, however, these publications do sometimes omit important information (Devereaux, Manns et al., 2001). Probably the most important tool of next-generation EBM has been the systematic review (Mulrow and Cook, 1998). A systematic review is a disciplined, focused assessment of the evidence on a topic. Meta-analysis is often used, but sometimes it cannot be done because there are insufficient studies or because the studies available are too heterogeneous to quantitatively analyze. Cook et al. (1997) distinguish a systematic review from a narrative (traditional) review by noting that the systematic kind usually assesses a focused clinical question by using comprehensive information sources, explicit searching strategies, criterionbased selection of studies, rigorous critical appraisal, and a quantitative summary. The process is usually evidence based. A narrative review, on the other hand, is likely to be broader in scope, with sources and their selection not usually specified (leading to potential bias) and a qualitative summary. Narrative reviews do have their role in meeting foreground information needs, but systematic reviews better serve making evidence-based decisions.
2. Health and Biomedical Information
81
The best-known producer of systematic reviews is the Cochrane Collaboration (Levin, 2001). This collaboration is named after Archie Cochrane, who noted, “It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials” (Cochrane, 1972). The Cochrane Collaboration is an international collaboration with the aim of preparing and maintaining systematic reviews of the effects of health care interventions. The Cochrane Database of Systematic Reviews has over a thousand reviews across medicine and will be described in greater detail in Chapter 4. Traditional journals are increasingly publishing systematic reviews. The Evidence-Based Practice Centers of AHRQ are also a source of systematic reviews (which the centers call evidence reports). A final tool of next-generation EBM is the critically appraised topics (CATS). These were originally defined to be “quick and dirty” systematic reviews of one or a few articles on a given topic, but their definition will be expanded for the purposes of this discussion to include any collection that summarizes the evidence less formally than systematic reviews. In fact, many CATS are summaries of systematic reviews. The original CATS adhered to a template that followed the critical appraisal approach of EBM and would be collected in databases. A number of Web sites provide access to a wide variety of CATS, such as the previously mentioned Oxford University Centre for Evidence-Based Medicine. Additional CATS-like information resources have been developed. These resources are characterized by their relevance to clinical practice, use of exclusively patient-oriented outcomes (symptoms, mortality, cost, or quality of life as opposed to test results), and their ability to allow users to change their clinical practice. One approach has been Patient-Oriented Evidence That Matters (POEMS) (Shaughnessy, Slawson et al., 1994). POEMS are published by the Journal of Family Practice as well as on a Web site (www.infopoems.com). Another publication that provides highly distilled evidence is Clinical Evidence. To the extent that they are built using evidence-based techniques, clinical practice guidelines can fall under this category of EBM information as well. Haynes (2001) notes that there is actually a spectrum of means by which evidence is disseminated. Dubbed the “4S model” and shown in Figure 2.5, it describes a hierarchy of evidence, with information seekers advised to begin their search at the highest level for the problem that prompted their search. The lower levels provide more details for the evidence, with the studies at the bottom providing the original data. But the upper levels provide the evidence in a briefer and more accessible form, with the ultimate level being its incorporation into the logic of computerized decision support systems with electronic medical records systems.
2.8.3 Limitations of EBM The entire medical community has not unequivocally embraced EBM. Feinstein, as already noted, expresses concern about the use of meta-analysis (Feinstein, 1995) and the lack of judgment that may emanate from a strictly “by the numbers” approach (Feinstein and Horwitz, 1997). Miles et al. (1997) lament that
82
I. Basic Concepts
FIGURE 2.5. The “4S” model of hierarchy of evidence. (Haynes, R. (2001). Of studies, syntheses, synopses, and systems: the “45” evolution of services for finding current best evidence. ACP Journal Club, 134:A11–A13. Reprinted with permission.)
EBM is so narrowly focused on sound methodology that it may lose the bigger clinical picture. Tonelli (1998) notes that individual differences are important. Larson, in the keynote address at a conference on translating research into practice in Washington, DC, in 1998, noted other problems that arise when clinical decisions are based solely on evidence: • • • •
Costs often not considered Individual preference and variability ignored Regional differences in practice Marketing and other factors driving patient preferences, including desire for alternative medicine • Selective use of evidence by clinicians, patients, payors, and others Norman acknowledges these criticisms and argues that proponents of EBM must conduct research on its effectiveness, incorporate more holistic perspectives on evidence, move from teaching appraisal skills to practitioners toward expert reviews (as described in Section 2.8.2), develop better strategies for teaching its use in clinical practice, and acknowledge and address the concerns of its critics (E. Norman, 1999).
Chapter 3 System Evaluation
This chapter, which introduces the methods used to evaluate IR systems, is placed early in the book to emphasize the central importance of evaluation. It does not discuss many results of evaluation per se, but rather defines a perspective of evaluation that will be discernible in the views of various approaches and systems in IR presented in ensuing chapters. Evaluation of IR systems is important for many reasons. Like any other technology, it is expensive, if not to individuals then the institutions in which they work. And as with many areas of computer use, there is a good deal of hype, marketing and otherwise, about the benefits of the technology. Thus the main reason to evaluate is to determine whether a system helps the users for whom it is intended. However, after this benefit has been demonstrated, it must also be shown that the system justifies its cost. And ultimately, the system’s real-world results should indicate some measure of improvement on the tasks in which it is used. This notion has an analogy in the medical research world to outcomes research, where an intervention must demonstrate not only efficacy (benefit in controlled research) but also effectiveness (benefit in the world outside the controlled research setting). Evaluations of IR systems are often classified as macroevaluations or microevaluations. Macroevaluations look at the whole IR system and/or the user’s interaction with it. This type of evaluation can take place in either a laboratory or a real-world setting. At times, however, one wishes to evaluate individual components of IR systems. Such evaluations are called microevaluations, and the motivations for doing them are to assess individual components of the system, to solve problems that arise with its use, and to determine how changes in the system might impact performance. They are typically performed in a laboratory or other controlled setting. This chapter first provides an overview of research methodology, with an eye to understanding the problems of error and validity that affect experimentation. Next is an overview of evaluation classifications used to assess IR systems. This is followed by a discussion on the types of IR evaluation experiments in general, with detailed emphasis on the most prevalent kinds of study, namely those using relevance-based measures. Finally, the limitations of relevance-based meas83
84
I. Basic Concepts
ures and alternatives to them for better evaluating the use of IR systems, especially in health-care settings are described.
3.1 Overview of Research There are many ways to classify research. This section begins with a broad perspective of research but then focuses in on comparative research, since that is the most common type of research both in healthcare and IR. Friedman et al. (2000) classify evaluation research into two broad categories. Objectivist research, often called quantitative research, assumes there is merit to the object under study (e.g., it cures a disease or retrieves relevant documents); it also assumes agreement among rational individuals about the benefit of the object, numerical measurement that allows statistical analysis of performance, and the existence of a setting (laboratory or real world) to demonstrate superiority to a competing object. Subjectivist research, sometimes called qualitative research, assumes that observations about the research object vary based on the observer, recognizes that its merit and worth vary by context of people and/or places, and focuses on verbal and descriptive data. The values of these types of research are not mutually exclusive. While an IR system can be improved by doing research on how effectively it retrieves relevant documents for the user and helps him or her answer questions better, it can also be bettered by listening to users describe their interactions with the system and how it is or is not helpful in finding information. The most common type of objectivist research is the comparison-based study that employs experiments. Other types include the objectives-based approach, which assesses whether a resource or system is meeting the goals of its designers, and the decision facilitation approach, where data is collected to help developers and designers make decisions on the further course of development of a resource or system.
3.1.1 Comparative Research Comparative research is the most common type of research of both medical interventions and IR systems. After all, the goal of research is usually to determine whether one approach to something works better than another. Comparative research uses experiments whose purpose is to learn truth about the world by means of the most objective, disinterested process possible. Science, of course, is a human endeavor, and thus has human failings, but nonetheless is the most objective means to observe worldly phenomena. In comparative research there is usually some sort of hypothesis being tested. In medical research, the hypothesis might be that a certain diagnostic test is more accurate than another or that one treatment for a disease is better than another. In IR, the hypothesis might be that a certain approach to retrieval yields more documents relevant to the topics the user is searching on or that a particular system helps a user answer questions more frequently than another. Experiments do
3. System Evaluation
85
not always need to yield something “better.” They may also aim to determine the cause of an event or to identify factors associated with particular outcomes. While science is often thought to take place in laboratory, it need not. Much biomedical research, for example, takes place in the clinician’s office and/or local community. Such research, of course, must be as controlled as possible, to be certain that the observations are valid and reproducible. IR system research is difficult (though not impossible) to do in the field, since the interference required to monitor users or capture their actions may interfere with the very task the researcher is attempting to assess. Scientific research is likely to ask questions about a population. In biomedical research, that population may be a group of individuals at risk for a disease, perhaps because they have a set of symptoms that suggest its diagnosis. Or, the population might be a group of individuals who already have a disease, in which case the research is likely to focus on interventions to reduce the symptoms or outright cure the condition. Research questions in IR are similar. For example, researchers might want to know how well a system provides relevant information to users. Since information always has intended users, the population of study will be a group specified by one or more characteristics, which may be demographic, professional, educational, or other attributes. Different users (e.g., clinicians, consumers, genomics researchers) are different populations with different needs and expectations for searching. In scientific studies, whether in medicine or IR, it is impractical to observe entire populations. That is, all individuals who might be at risk for diabetes nor all physicians who would use a new EBM resource can be observed. Instead, researchers study samples that represent populations. Ideally, the samples are chosen to accurately represent the populations and interventions being studied. The samples should be selected randomly within the population to ensure that members have an equal chance of being included in the sample. If an intervention is going to be applied, it needs to be certain that any member of the population being randomized can end up in any study group. In a comparative study, once a sample has been designated, the next step is to apply an intervention (such as beginning a medical treatment of a disease or establishing an IR system to meet information needs). There is an experimental group that is given the “new” intervention (i.e., the new diagnostic test, medical treatment, or IR system) and a control group that receives the “old” intervention, which could be one that is known to be the most effective existing intervention or, in some instances, a sham (or placebo) intervention. The selection of the control group is important in both medical and IR research because the new experimental approach being tested must be compared against the approach already known to be most effective. This can be particularly problematic in IR research when a new approach shows improvement over a baseline control that does not represent the most effective approach already known. The characteristics observed in comparative studies are variables. Those that capture the outcomes of interest are dependent or outcome variables. Other variables are often included in studies to explain the measured values of dependent variables.
86
I. Basic Concepts
These are called independent or predictor variables. Some research focuses on building models with independent variables to explain their effect on dependent variables (e.g., determining the factors that are associated with a user’s successful operation of an IR system to answer questions, such as age, computer experience, etc.). After the intervention has been performed and a result obtained, the next step is to determine whether the result is correct. Researchers hope that their results reflect the truth; however, there are other possibilities. There are two general categories of error in extrapolating the observations of a sample to the entire population, bias and chance. Bias is the systematic error introduced into research studies from sampling, measurements, or other problems. Fletcher et al. (1988) describe three general types of bias for biomedical research, which are also pertinent to IR research: 1. Selection bias—the subjects or phenomenon studied in the sample do not represent those of the population. 2. Measurement bias—the measurement of the phenomenon in the sample is systematically different from that used for the population. 3. Confounding bias—another factor unknown to the observer is making the sample or measurements unrepresentative of the population. Friedman et al. (2000) note other types of bias and effects that compromise research: 1. Assessment bias—subjects in the experiment allow their own feelings or beliefs about the system—positive or negative—to affect how they interact with the intervention and thus bias the results. 2. Allocation bias—researchers or others may “cheat” the randomization process to make sure “easier” cases get assigned to the intervention they prefer. 3. Hawthorne effect—humans tend to improve their performance if they know they are being studied. His was first demonstrated in the Hawthorne Factory in Chicago, where productivity increased when room illumination was raised but also increased when it was accidentally reduced (Roethlisberger and Dickson, 1939). It was determined that the subjects improved their productivity because they knew they were being observed. 4. Checklist effect—when humans are given checklists for a cognitive task, such as one that includes interaction with an IR system, their decision making tends to be more complete and better structured. The other category of error is chance. In this type of error, an observation occurs because the randomization process has inadvertently produced a sample that does not accurately represent the population. One method that helps to minimize chance error is the use of statistical analysis. The appropriate statistical test helps determine how unlikely the results are due to chance. In fact, the major purpose of inferential statistics is to determine how well observations of the sample can be extrapolated to the whole population. These statistics identify two types of error, commonly called alpha and beta. Alpha error determines how likely it is that
3. System Evaluation
87
a difference in treatment groups is actually due to chance. This is the famous “p value,” with a level of p .05 indicating there is only a one in 20 chance that an observation is due to random error. Note that this does not say the result is not due to random error, only that such error is highly unlikely. Beta error states how likely it is that a nondifference in groups truly represents a nondifference. This type of error is often overlooked in scientific research, in that results found to be “not statistically significant” may very well have a low likelihood of being able to detect a difference in the first place. The measure of a study to avoid beta error is called its power. Another important concept in research design is validity. Bias and chance reflect the internal validity of experiments, that is, how well the study design is free of systematic and/or random error. Studies also, however, must exhibit external validity, which reflects how well their results apply to the real world. In the biomedical realm, if there is a disease that is treated successfully in laboratory animals, the results do not necessarily generalize to humans. In the case of IR systems, an approach found to work in simulated testing may not work with real users because they might not apply it in the same way it was applied in the simulation. Ensuring external validity, also called generalizability, is actually a bigger problem in IR evaluation research than in medicine, since IR studies are often done in laboratory settings. The measures most commonly used to evaluate IR systems are recall and precision, but later in the chapter it is shown that the usual ways of measuring them may compromise their external validity.
3.1.2 Other Types of Research The most common form of subjectivist research is the responsive–illuminative approach, in which the viewpoints of the users and other who are impacted by the system are assessed (Friedman, Owens et al., 2000). Many of these studies employ ethnography; that is, the investigators immerse themselves in the environment in which the system is being used. Other research involves focus groups, selected panels who participate in a structured interview process and attempt to obtain a consensus on something, such as an IR system. Significant effort is devoted to analyzing the narrative of participants to identify patterns and insights. Another example of this type of research is usability testing, in which users are provided tasks to perform with the system and their actions are captured and analyzed (Nielsen, 1993). Usability testing may involve measurement of ability to complete tasks, but the results are usually analyzed more for sake of ascertaining users’ interactions and satisfaction with the system. One method of assessing usability is to monitor users’ interactions with a system, from recording the interactions on videotape to tracing their actions and actual keystrokes (Shneiderman, 1992). Another approach is to capture the users’ thinking, usually via protocol analysis, a process in which users are questioned about what they are doing and encouraged to “think aloud” (Ericsson and Simon, 1993). This technique has been used to capture the thinking of physician diagnosis (Elstein, Shulman et al., 1978a) and MEDLINE indexing (Crain, 1987).
88
I. Basic Concepts
3.2 Classifications of Evaluation Now that the basic concepts of research have been introduced, the area of IR research can be addressed specifically. In the first chapter, several different models, or views, or the IR world were presented. The differing models were not necessarily at odds with each other; rather, they looked at the problems from different perspectives. This section presents four different classifications that do not conflict but rather view the process from different perspectives. The first classification, by Lancaster and Warner, focuses on the effectiveness of using the system at different levels, related to success of the user and the cost associated with that success. This is followed by the classification of Fidel and Soergel, a more granular classification that attempts to define all the variables that can influence the outcome of using an IR system. These are followed by two additional views, one focusing on the benefit of the system in the context in which it is used and the other pertaining to degrees of simulation in IR evaluation.
3.2.1 Lancaster and Warner Lancaster and Warner (1993) define three levels of evaluation, as shown in Table 3.1. The first level is the evaluation of the effectiveness of the system and the user interacting with it. (Health services researchers would actually state that efficacy and not effectiveness is being studied here, but Lancaster and Warner’s own language shall be used.) At this level, the authors identify three general criteria for effectiveness: cost, time, and quality. While issues of cost and time are straightforward, those of quality are considerably more subjective. In fact, what constitutes quality in a retrieval system may be one of the most controversial questions in the IR field. This category also contains the relevance-based measured of recall and precision, which are the most frequently used evaluation measures in IR. The second level of retrieval evaluation in Lancaster and Warner’s schema is cost-effectiveness. This level measures the unit costs of various aspects of the retrieval system, such as cost per relevant citation or cost per relevant document. As IR systems become increasingly inexpensive, or outright subsidized by institutions, the individual searcher is less concerned with these issues, although administrators and others who make purchasing decisions must increasingly grapple with them. Examples of cost-effectiveness studies include the comparison of online versus CD-ROM access to databases (Welsh, 1989) and the relationship between the cost of increased indexing detail versus the likelihood that the added value will enable users to find additional information (Mandel, 1988). The final level of evaluation in the schema is cost–benefit, which compares the costs of different approaches directly. A cost–benefit study might attempt to determine whether it is worth the money to license a commercial version of MEDLINE that has additional features as opposed to utilizing the free system
3. System Evaluation
89
TABLE 3.1. Lancaster and Warner’s Classification of IR Evaluation I. Evaluation of Effectiveness A. Cost 1. Monetary cost to user (per search, document, or subscription) 2. Other, less tangible, cost considerations a. Effort involved in learning how to use the system b. Effort involved in actual use c. Effort involved in retrieving documents d. Form of output provided by system B. Time 1. Time from submission of request to retrieval of references 2. Time from submission of request to retrieval of documents 3. Other time considerations, such as waiting to use system C. Quality 1. Coverage of database 2. Completeness of output (recall) 3. Relevance of output (precision) 4. Novelty of output 5. Completeness and accuracy of data II. Evaluation of cost-effectiveness A. Unit cost per relevant citation retrieved B. Unit cost per new (previously unknown) relevant citation retrieved C. Unit cost per relevant document retrieved III. Cost–benefit evaluation—value of systems balanced against costs of operating or using them Source: Reprinted with permission of Information Resources Press, a division of Herner and Company, from Lancaster FW & Warner AJ, Information Retrieval Today, 1993, p. 161.
provided by the NLM. Some examples include measuring the savings due to better information in an airline company (Jensen, Asbury et al., 1980) and attempting to measure the value of medical libraries (King, 1987). In the latter study, when healthcare professionals sought information in hospital libraries (facilities that are lacking in many hospitals), almost two-thirds responded that the information they obtained would change their practice patterns.
3.2.2 Fidel and Soergel Fidel and Soergel (1983) devised a classification to catalog for researchers all the factors that need to be controlled in IR evaluations. This classification can also, however, be used to review the components that should be studied (or at least considered) in effective IR evaluation. 3.2.2.1 Setting The setting of IR system use is an important variable in the evaluation process because users in different settings have different needs and motivations. The traditional site for retrieval systems was the library. Users of IR systems in libraries are typically doing research of a more comprehensive nature, more likely to be seeking a variety of documents (background information and current research)
90
I. Basic Concepts
and viewpoints. Increasingly, however, IR systems are being used in nonlibrary settings, such as medical work settings or in the home. Users in these settings are likely to have different needs. Health-related questions, whether posed by clinicians or patients, tend to be much more focused, with users tending to seek direct answers rather than comprehensive treatises on the topics. Furthermore, user needs are likely to have more acuity, and prompt decisions may be mandatory (Gorman and Helfand, 1995). 3.2.2.2 User Fidel and Soergel distinguish the user from the searcher. The former is the person who is seeking information, while the latter actually interacts with the IR system. In the past, before direct end-user access to IR systems was common, the searcher was typically a skilled intermediary, usually a librarian. In the current era of widespread end-user searching, however, the user is likely to also be the searcher. Different users can have divergent needs. For example, Wallingford et al. (1990) have shown that medical researchers are likely to have information needs different from those of clinicians. Researchers tend to want to retrieve more information on a given topic, whereas clinicians are often satisfied with just the amount necessary to answer their clinical question. As a result, researchers tend to do comprehensive searches that seek to obtain a broad segment of relevant information, whereas clinicians are more likely to carry out focused searches. Additional aspects about the user that have attracted research interest include cognitive factors likely to predict successful searching. In a large study of mediated on line searching. Saracevic et al. (1988) looked at a number of attributes of users and how they predicted obtaining a successfully mediated search, such as word association ability, deductive inference skill, and preferred styles of learning. Hersh et al. (2002) have employed a similar approach studying clinical searchers. 3.2.2.3 Request Another important factor in evaluation is the type of request. It was seen in the last chapter that seekers of health information have a variety of questions. Even the same user can come back to a system with different information needs. Fidel and Soergel (1983) identify several categories of user search requests: 1. Background—the user needs background information on a topic (e.g., a researcher moving into a new area of study). 2. Comprehensive—the user needs a complete exposition of the topic (e.g., someone writing a book delving into previously unfamiliar areas). 3. Discussion—the user needs to know what the prevailing wisdom of a topic area is (e.g., the controversy over screening for and treating prostate cancer).
3. System Evaluation
91
4. Fact—the user needs to know a specific fact (e.g., the optimal dosage of a drug). 5. Updating—the user is already knowledgeable about a field but needs to learn whether there have been new developments. 3.2.2.4 Database The evaluation of IR entails several important aspects concerning databases (or content). The most important of these is coverage. Databases vary in how they cover information, even on the same topic. For example, a primary literature database such as MEDLINE has different kinds of information from a secondary database, such as a medical textbook. Not only will users get different types of information from each, but they will approach the search process differently, since they are likely to have varying information needs. Some IR systems may subdivide portions of a larger database in different ways, which could also result in different searching outcomes. The best example of this occurs with the MEDLINE. Virtually every MEDLINE vendor segments the database by years, although in different ways. The NLM’s PubMed system, on the other hand, does not segment it at all. This can affect the outcomes of searching and thus evaluations. Another issue of increasing importance in electronic databases is timeliness. As databases proliferate, users increasingly rely on them and expect them to be up to date. Several years ago, the NLM issued a policy with MEDLINE for standalone (i.e., CD-ROM) products, declaring that disks have an “expiration date,” after which they are no longer valid. Some vendors have adopted that policy for all their databases, essentially selling their products as “subscriptions.” Online versions of many databases, including MEDLINE, are now updated weekly, although a vendor that licenses MEDLINE might update less frequently. An additional variable that must be considered in a database is cost. As the “dot-com” boom of the late 1990s fades, users are increasingly going to have to pay to access information, especially specialized information without a large market base to allow distribution of the costs. While the cost to end users is often subsidized by the institutions they work for, someone will still be footing the bill. A related issue is the cost of production. If the database is expensive to build or index, then unless it offers some sort of unique information or functionality, it may not be available to a large number of users. The final aspect of the database is indexing. There are actually two aspects of indexing that must be addressed: the indexing terms and attributes and the indexing process. For the former, variables of interest include the type and coverage of the terms and attributes. They may consist of a controlled vocabulary assigned by human indexers and/or all the words that occur in the text fields of documents. For controlled vocabulary, one wants to know the coverage, especially with respect to the terms that searchers may employ. For the indexing process, variables of interest include the exhaustivity, specificity, and consistency, which will be discussed in Chapter 5.
92
I. Basic Concepts
3.2.2.5 Search System There are many facets to the search system. First, how is the system accessed? The telecommunications mode by which it is accessed may influence how successfully it is used. For example, a multimedia database with high-resolution images might be more difficult to interact with over a low-bandwidth (e.g., telephone modem) connection. Cost has already been discussed in relation to specific databases, but the cost of a search system itself must also be considered. Even though most databases are accessed over the Internet, a higher bandwidth (and more expensive) connection may be necessary for certain databases. In general, while costs may not affect the individual searcher, especially in the academic setting where many databases are licensed, this factor could have an impact in the evaluation of systems on the institutional level. An additional cost factor is related to searching assistance. Some systems offer elaborate searching aids, such as index term lookup and online help, which have implementation costs. Sometimes users need the expertise of another human to help them use the system. This assistance requires planning ahead of time and may also add to the cost. Many systems also require user training, which also incurs cost. Another consideration for the search system is the formatting of document representations and their text. Systems differ, for example, in how they display document representations. Some display just the title of documents that match the query, whereas others display a synopsis that includes a portion of the title, authors, and indexing terms. On the Web, some systems store their documents in the native HTML, while others use PDF, which provides more formatting control. 3.2.2.6 Searcher As noted earlier, Fidel and Soergel differentiate the user from the searcher. Of course, with users increasingly doing their own searches, the distinction is becoming blurred. Nonetheless, the skill of the person operating the IR system is a factor that cannot be ignored. Some studies have shown significant differences between searchers of similar background but different levels of training or experience. For example, Haynes et al. (1990) showed that expert searchers (physicians or librarians) received on average twice as many relevant documents per search on the same topic as novice searchers. Hersh et al. (2002) found that the use by experienced searchers of specific features of MEDLINE was highly associated with the individuals’ ability to use the system to answer clinical questions. 3.2.2.7 Search Process There are many aspects of the user’s interaction with the system that must be described. Many of them revolve around time. How long does it take to formulate a search? What is the system response time for finding search terms, identifying matching documents, or displaying individual documents? As will be discussed,
3. System Evaluation
93
a large number of IR evaluations are run in “batch” mode, where there is no interaction with a user and the system, only direct entry of “canned” searches into the system. Some have seriously questioned the value of these evaluation results in light of the absence of user interaction with the system. 3.2.2.8 Search Outcome The final aspect of Fidel and Soergel’s classification is the search outcome. As noted in connection with Lancaster and Warner’s classification, there are many ways to measure the outcome of using an IR system. The most common methods used by IR researchers are the twin measures of recall and precision, which measure the quantities of relevant documents retrieved. But there are many other measures that assess outcome, including virtually all those described for Lancaster and Warner’s levels of cost-effectiveness and cost–benefit. Another issue is whether the search outcome measured should be an actual outcome or an intermediate one. An actual outcome might be how well a system is used to answer clinical questions or whether it results in improved patient care. An intermediate outcome is some measure of the system’s efficacy while being used for a task, such as recall and precision. (Users do not utilize retrieval systems to obtain good recall and precision per se; rather, they aim to retrieve relevant documents that will allow them to carry out the task that motivated use of the system in the first place.)
3.2.3 Hersh and Hickam While both Lancaster and Warner and Fidel and Soergel have “outcome” in their classifications, it may be desirable to expand these schemas. In particular, there are (at least) six questions one can ask related to an IR resource (system or information collection) in a particular setting. These questions were developed for a systematic review assessing how well physicians use IR systems (Hersh and Hickam, 1998), but they could be applied to other users and settings. They will serve as the focus for classifying the evaluation studies presented in Chapter 7. 3.2.3.1 Was the System Used? An important question to ask about an IR resource is whether it was actually used by those for whom it was provided. Measurement of system or collection use can be gleaned from user questionnaires, preferably, however, it is done directly by system logging software. It is important to know how frequently people used a resource, since to be installed in the first place, someone had to have thought it would be beneficial to users. The nonuse of a system or collection is a telling evaluation of its (non)value to users. 3.2.3.2 For What Was the System Used? A related concern is knowing the tasks for which the system was being used. One might want to know what information collections were used (if there were
94
I. Basic Concepts
more than one) and what types of question were posed. In a clinical setting, there might be interest in what kind(s) of clinical problem led to use of which resource. Likewise, it may be important to know whether the system was used as a primary information resource or to obtain references for library lookup. 3.2.3.3 Were the Users Satisfied? The next question to ask is whether users were satisfied with the IR system. User satisfaction is an important question both for administrators who make decisions to install and maintain systems and for researchers trying to determine the role of systems for users. It is also relatively straightforward to assess, with the use of instruments such as questionnaires, direct observation, and focus groups. A well-known instrument for assessing computer software is the Questionnaire for User Interface Satisfaction (QUIS) (Chin, Diehl et al., 1998). 3.2.3.4 How Well Did People Use the System? Once it has been determined that systems were used and liked, the next issue is how effectively they were actually used. Whereas frequency of use and user satisfaction are relatively simple concepts, the notion of “how well” someone uses a system is more complex. Does one operate at the level of counting the number of relevant documents obtained, perhaps over a given time period? Or are larger issues assessed, such as whether use of the system results in better patient care outcomes? An example of the latter would be showing that the system had led a practitioner to make better decisions or had resulted in better patient outcomes. This issue will be addressed further shortly. 3.2.3.5 What Factors Were Associated with Successful and Unsuccessful Use of the System? Whether an IR system works well or does not work well, there are likely explanations for the result. A variety of factors (e.g., demographic, cognitive, experiential) can be measured and correlated with the outcome of system use. Furthermore, if the system did not perform well, a researcher might wish to ask why. The assessment of system failure, called failure analysis, typically involves retrospectively determining the problems and ascertaining whether they were due to indexing, retrieval, user error, or some combination of these. 3.2.3.6 Did the System Have an Impact? The final, and obviously most important question, is whether the system had an impact. In the case of clinical users, this might be measured by some type of improved healthcare delivery outcome, such as care of better quality or reduced cost. This item, which is addressed in the schemas of both Lancaster and Warner and Fidel and Soergel, takes on increased pertinence in health care, given the emphasis on quality of care and the desire to control costs. Of course, demonstrating that a computer system has an impact in actual patient outcome is diffi-
3. System Evaluation
95
cult because this effect is often indirect (Wyatt and Spiegelhalter, 1991). This is particularly true of IR systems, where not only is use for a given patient optional, but also each new patient on whose behalf the system is used is likely to require a different kind of use. As such, there have been few studies of patient outcomes as related to IR systems. Such studies are easier to do for computer-based decision support systems, where the same function (e.g., recommended drug dose, alert for an abnormal laboratory test)is used for the same situation each time (Johnston, Langton et al., 1994).
3.2.4 Simulation in Evaluation Experiments While the outcome of the use of IR systems in real-world settings is the most desirable goal of evaluation, simulation of those settings sometimes can provide a great deal of knowledge about those systems without the expense and distraction of real-world use. The classifications of evaluation discussed earlier looked at the IR system in real settings. While field research is the best method for making observations about IR systems in the real world, simulated settings provide valuable information about IR systems. The use of simulated users, settings, and queries has often been criticized for producing results that have insufficient external validity to permit generalization to real-world searching. When done properly, however, simulated searching situations may provide valuable insights in a controlled and unobtrusive manner into the workings of IR systems on everyday tasks. One can classify the simulation in experiments by three dimensions of the evaluation setting: 1. User—Does the user interact with the system (real), or are previous interactions reentered into the system (simulated)? 2. Setting—If the user is interacting with the system, is the interaction occurring in an operational (real) or controlled (simulated) setting? (Simulated interactions would obviously occur in only controlled settings.) 3. Queries—If the user is interacting with the system, are the queries generated by the user (real) or had they been generated earlier (simulated)? (Simulated queries would obviously occur only with simulated interactions.)
3.3 Relevance-Based Evaluation As shown in the preceding section, there are many ways to measure the quality of searching with IR systems. While many studies have focused on a wide variety of performance measures, the most widely used are still the relevance-based measures of recall and precision. These were first defined decades ago by Kent et al. (1955) and achieved prominence by their use in the Cranfield studies of the 1960s (Cleverdon and Keen, 1966). Indeed, many consider them to be the “gold standard” of retrieval evaluation. Yet as will be seen in this and other chap-
96
I. Basic Concepts
ters, their use entails some serious problems, especially when they are the sole measurements in an evaluation. It is not that they are unimportant conceptually, but rather that they are difficult to measure in operational settings and do not necessarily correlate with the success of using an IR system. In acknowledgment of their prevalence, however, this section will define them, explain their use, and discuss their limitations.
3.3.1 Recall and Precision The relevance-based measures of recall and precision quantify the number of relevant documents retrieved by the user from the database and in his or her search. Ignoring for a moment the subjective nature of relevance, it can be seen in Figure 3.1 that for a given user query on a topic, there are a number of relevant documents (Rel), retrieved documents (Ret), and retrieved documents that are also relevant (Retrel). Recall is the proportion of relevant documents retrieved from the database: Retrel Recall Rel
(1)
In other words, recall answers the question, For a given search, what fraction of all the relevant documents have been obtained from the database? One problem with equation (1) is that the denominator implies that the total number of relevant documents for a query is known. For all but the smallest of databases, however, it is unlikely, perhaps even impossible, for me to succeed in identifying all relevant documents in a database. Thus most studies use the measure of relative recall, where the denominator is redefined to represent the number of relevant documents identified by multiple searches on the query topic.
FIGURE 3.1. Graphical depiction of the elements necessary to calculate recall and precision.
3. System Evaluation
97
TABLE 3.2. Table of Retrieved and/or Relevant Documents for a Query to Calculate Recall and Precision Retrieved Not retrieved Total
Relevant
Not relevant
Total
30 20 50
70 999,880 999,950
100 999,900 1,000,000
Precision is the proportion of relevant documents retrieved in the search: Retrel Precision Ret
(2)
This measure answers the question, For a given search, what fraction of the retrieved documents are relevant? A sample recall and precision matrix is shown in Table 3.2. The database contains a total of one million documents. For this particular query, there are 50 known relevant documents. The searcher has retrieved 100 documents, of which 30 are relevant to the query. The proportion of all relevant documents obtained, or recall, is 30/50, or 60%. The fraction of relevant documents from the set retrieved is 30/100, or 30%. 3.3.1.1 Similarities to Medical Diagnostic Test Evaluation Table 3.2 is very similar to the matrix used to calculate the diagnostic test performance measures of sensitivity and specificity. In fact, if “relevance” is changed to “presence of disease” and “number retrieved” is changed to “number with positive test result,” then recall is identical to sensitivity, while precision is the same as positive predictive value. (Specificity would be a much less useful number in IR research, since the numbers of both relevant and retrieved articles for a given query tend to be small. With large databases, therefore, specificity would almost always approach 100%). It is known in medical testing that there is a trade-off between sensitivity and specificity. That is, if the threshold is changed for a positive test, it will change not only the proportion of people correctly diagnosed, but also the proportion incorrectly diagnosed. If the threshold for diagnosis is lowered, then the test will usually identify more true positive cases of the disease (and thus raise sensitivity) but also will identify more false positive instances. The relationship between sensitivity (recall) and positive predictive value (precision) is not quite so direct, but it usually occurs in IR systems. The trade-off can be demonstrated qualitatively by comparing searchers of different types, such as researchers and clinicians. A researcher would more likely want to retrieve everything on a given topic. This searcher (or an intermediary) would thus make the query statement broad to be able to retrieve as many relevant documents as possible. As a result, however, this searcher would also tend to retrieve a high number of nonrelevant documents as well. Conversely, a clinician searching for a small number of good
98
I. Basic Concepts
articles on a topic is much less concerned with complete recall. He or she would be more likely to phrase the search narrowly, aiming to obtain just a few relevant documents without having to wade through a large number of nonrelevant ones. 3.3.1.2 Practical Issues in Measuring Recall and Precision The values of recall and precision were defined in an era when searches were done in batches and users did not directly interact with systems. In the modern era of interactive end-user searching, however, numerous practical problems arise when one is attempting to measure recall and precision. For example, what is a query? Is it a single search against the system, or is it all the searches on a given topic? Does it occur in a single setting, or can the user come back to the system after utilizing the documents, perhaps hours or days later? How is a search assessed that starts out poorly, perhaps with errors due to accidental entry of wrong commands (such as using an AND instead of an OR or vice versa) or improper search terms (perhaps accidentally misspelled), but later improves to retrieve many relevant and few nonrelevant documents? Should the documents retrieved during the bad part of the search, accidental or otherwise, be excluded? Swanson (1977) has stated that IR is an interactive “trial-and-error” process, in which single point interactions with the IR system are inadequate for evaluation of the entire information-seeking process. There are also instances of different types of retrieval systems having different retrieval processes. In the traditional set-oriented Boolean retrieval systems, the user typically builds sets of documents that contain terms, which are made successively smaller by use of AND and OR until a final manageable set of documents is obtained. But in systems using automated indexing and retrieval (most notably Web search engines), there is use of relevance ranking; that is, a complete search is entered and a list of documents is returned, ranked for “relevance” (typically measured in number of terms common to the search statement and document). In this instance, the user may enter only one search and focus on the documents there. Comparing searches done by these two approaches can be problematic (Hersh, Buckley et al., 1994). Another problem is deciding what constitutes a retrieved document. Should it be the document “surrogate” (i.e., the title with or without other brief information) presented after a search has been entered, or it should it only be the documents examined in more detail by the searcher (i.e., the whole MEDLINE reference or Web page viewed after the surrogate has been chosen)? And again, should documents retrieved by a poor, perhaps erroneous, original search be counted as “retrieved”? Another problem lies in establishing a protocol for measuring recall and precision across a set of queries. Most investigators simply take the mean. But is the mean the best measure? Hersh and Hickam (1995) showed that choosing to use the median instead of the mean for recall and precision sometimes led to different conclusions from summary measures. A related issue is whether all queries
3. System Evaluation
99
should be counted equally. Some queries might have many more relevant documents, and one could argue that such queries should carry more weight. An additional problem involves handling the indirect relationship between recall and precision. It has been noted that different types of queries can affect both measures. For example, a broadly worded query will generally achieve high recall and low precision, whereas a narrowly worded query will result in low recall and high precision. How are two such queries compared? Some investigators have proposed measures to combine recall and precision, but most require stating numerically the relative importance of one measure to the other. For example, van Rijsbergen (1999) proposed the single evaluation measure E: 1 E 1 a/P (1 a)/R
(3)
where a is a value between zero and one, indicating the relative value of recall, and E varies inversely with recall and precision from zero to one (i.e., higher levels of recall and precision will move E toward one and lower levels will push it toward zero). Another method of using a single measure to combine recall and precision is the recall–precision table, which can be used only for ranked output, as explained in the next section. 3.3.1.3 The Special Case of Ranked Output Many IR systems use relevance ranking, whereby the output is sorted by means of measures that attempt to rank the importance of documents, usually based on factors related to frequency of occurrence of terms in both the query and the document. In general, systems that feature Boolean searching do not have relevance ranking, while those featuring natural language searching tend to incorporate it. Systems that use relevance ranking tend to have larger but sorted retrieval outputs, and users can decide how far down to look. Since the more relevant documents tend to be ranked higher, this approach gives users a chance to determine whether they wants lower recall and higher precision (just look at the top of the list) or higher recall and lower precision (keep looking further down the list). One problem that arises when one is comparing systems that use ranking versus those that do not is that nonranking systems, typically using Boolean searching, tend to retrieve a fixed set of documents and as a result have fixed points of recall and precision. Systems with relevance ranking, on the other hand, have different values of recall and precision depending on the size of the retrieval set the system (or the user) has chosen to show. For this reason, many evaluators of systems featuring relevance ranking will create a recall–precision table (or graph) that identifies precision at various levels of recall. The “standard” approach to this was defined by Salton (1983), who pioneered both relevance ranking and this method of evaluating such systems. To generate a recall–precision table for a single query, one first must determine the intervals of recall that will be used. A typical approach is to use inter-
100
I. Basic Concepts
vals of 0.1 (or 10%), with a total of 11 intervals from a recall of 0.0 to 1.0. The table is built by determining the highest level of overall precision at any point in the output for a given interval of recall. Thus, for the recall interval 0.0, one would use the highest level of precision at which the recall is anywhere greater than or equal to zero and less than 0.1. Since the ranked output list is scanned from the top, the number of relevant documents is always increasing. Thus, each time a new relevant document in the list is identified, it must first be determined whether it is in the current interval or the next one (representing higher recall). For the appropriate interval, the new overall precision is compared with the existing value. If it is higher, then the existing value is replaced. When there are fewer relevant documents than there are intervals (e.g., 10 intervals but fewer documents), one must interpolate back from the higher interval. For example, if there are only two relevant documents, then the first relevant one would fall at a recall level of 0.5 and would require interpolation of the current overall precision value back to the preceding levels of recall (i.e., 0.4, 0.3, 0.2, 0.1, 0.0). Conversely, when there are more relevant documents than intervals, one must compare each level of precision within the recall interval to all the others to determine the highest one. An example should make this clearer. Table 3.3 contains the ranked output from a query of 20 documents retrieved, and 7 are known to be relevant.
TABLE 3.3. Example Ranked Output of 20 Documents with 7 Known to Be Relevant Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 aRel,
Relevancea Rel NRel Rel NRel Rel Rel NRel NRel Rel NRel NRel NRel NRel Rel NRel NRel NRel NRel NRel Rel
Recall and precisionb R 1/7, P 1/1 R 2/7, P 2/3 R 3/7, P 3/5 R 4/7, P 4/6 R 5/7, P 5/9
R 6/7, P 6/14
R 7/7, P 7/20
relevant document; NRel, nonrelevant one. time a relevant document is encountered, recall (R) and precision (P) are calculated to be entered into the recall–precision table (Table 3.4). bEach
3. System Evaluation
101
TABLE 3.4. The Recall–Precision Table Resulting from the Data in Table 3.3 Recall
Precision
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
1.00 1.00 0.67 0.60 0.60 0.67 0.56 0.56 0.43 0.35 0.35
Table 3.4 is a recall–precision table for the documents in Table 3.3 with recall intervals of 0.1. Note that there are fewer intervals than documents, so interpolation is needed. The first document in Table 3.3 is relevant. Because there are seven relevant documents, the recall is 1/7 or 0.14. The overall precision at this point is 1/1 or 1.0, and its value is entered into the table for the recall level of 0.1. Since there are fewer than 10 relevant documents, there will be separate precision for the recall level of 0.0, so the value from the recall level of 0.1 is interpolated back to the 0.0 level. The second document is not relevant, but the third document is. The overall level of recall is now 2/7 or 0.28, so the new level of precision, 2/3 or 0.67, is entered into the recall level of 0.2 in Table 3.4. The following document is not relevant, but the fifth document is, moving the overall recall level up to 3/7 or 0.42. The new precision is 3/5 or 0.60, and it is entered into Table 3.4 at the recall level of 0.4. Notice that there was no value to enter into the recall level of 0.3, so the value at the 0.4 level is interpolated back to the 0.3 level. The rest of the results are shown in Table 3.4. For a whole set of queries, the values at each recall level are averaged. In general, the values for precision over a set of queries will fall with increasing level of recall. To compare different systems, or changes made in a single system, three or more of the precision levels are typically averaged. When the recall interval is 0.1, one might average all 11 intervals or just average a few of them, such as 0.2, 0.5, and 0.8. An approach that has been used more frequently in recent times has been the mean average precision (MAP), which is similar to precision at points of recall but does not use fixed recall intervals or interpolation (Voorhees, 1998). Instead, precision is measured at every point at which a relevant document is obtained, and the MAP measure is found by averaging these points for the whole query. There are other approaches to comparing ranked output. One is to fix the level of documents retrieved (e.g., 30), which gives recall and precision points that can then be compared by means of ranked output against those of other systems, including systems that do not use weighting at all. An additional approach fur-
102
I. Basic Concepts
ther develops the medical diagnostic testing analogy described earlier. In this case, precision (or positive predictive value) is converted to specificity, generating a receiver operating characteristic (ROC) curve for each query (Hanley and McNeil, 1983). The “performance” of a diagnostic test is determined by the area under such a curve, and the performance of a query can be found similarly. This also has the benefit of giving a single value of performance for the query. The areas for each curve can be averaged and compared statistically (Hersh, Hickam et al., 1992). 3.3.1.4 Enhancements to Recall and Precision A number of variations to recall and precision have been proposed over the years. One approach has been to “score” the output of a search with more than just relevant documents. Verhoeven et al. (1999), for example, have proposed a search quality score, with one point each for an article that is 1. Topically relevant 2. From a quality journal (i.e., one having an impact factor above the median for its subject group) 3. A good study design (e.g., a clinical trial or follow-up study) 4. A review article (since such articles often summarize research) Another approach is to attempt to give value for documents judged to be more highly relevant and ranking higher in the output list. Many studies (e.g., Haynes, McKibbon et al., 1990; Hersh and Hickam, 1994) have measured relevance as more than a simple binary value. The preceding formulas for recall and precision, however, assume relevance to be binary. Jarvelin and Kekalainen (2000) have proposed two measures related to the value to the degree of relevance and rank of the document in the output list. The measure to add value based on relevance is called cumulative gain (CG). A cumulative score is kept, with additional score is added based on the degree of relevance of the document: CG(i) CG(i 1) G(i)
(4)
where i is the document’s rank in the output list (e.g., the top ranking document has i 1) and G(i) is the relevance value. The measure based on document rank is called the discounted cumulative gain (DCG). G(i) DCG(i) DCG(i 1) log(i)
(5)
When i 1, CG(1) is set to 1 if there is no relevant document and DCG(1) is set to 1 to avoid dividing by zero. Jarvelin and Kekalainen (2000) advocate that CG be plotted versus DCG, with the performance of systems assessed based on the value of CG relative to DCG. An example of these measures is shown in Table 3.5. Relevance has been assigned on a scale of 0 to 3, with CG and DCG calculated accordingly.
3. System Evaluation
103
TABLE 3.5. Example of Cumulative Gain and Discounted Cumulative Gain for 10 Ranked Documents Rated for Relevance on a Scale of 0 to 3 Rank 1 2 3 4 5 6 7 8 9 10
Relevance
Log (rank)
CG(i)
DCG(i)
0 3 2 3 0 0 1 2 2 3
0 0.30 0.48 0.60 0.70 0.78 0.85 0.90 0.95 1
1 3 6 6 6 7 9 11 14 14
1 7.64 13.93 13.93 13.93 15.22 17.58 19.80 22.94 22.94
Source: Jarvelin and Kekalainen (2000).
It should be noted that neither of these methods has been validated with real users. A particular limitation of the approach of Verhoeven et al. is that the criteria used are not completely consistent with study quality as defined by advocates of evidence-based medicine (Sackett, Richardson et al., 2000). Nonetheless, variations mentioned in this section do suggest approaches that can be explored further in research.
3.3.2 What Is Relevance? To this point, relevance has merely been defined as a document meeting an information need that prompted a query. This fixed view of relevance makes recall and precision very straightforward to calculate. But as it turns out, relevance is not quite so objective. For example, relevance as judged by physicians has a moderately high degree of variation, as evidenced by Table 3.6, which shows the overlap between judges in assigning relevance of MEDLINE references to queries generated in a clinical setting (Hersh, Buckley et al., 1994). This level of disagreement has been verified in similar assessments (Haynes, McKibbon et al., 1990; Hersh and Hickam, 1993; Hersh, Hickam et al., 1994; Hersh and Hickam, 1995). In each of these studies, the kappa statistic has been used to measure interrater reliability. This statistic, described in greater detail in Section 3.5, is comTABLE 3.6. Overlap of Judges on Assigning Relevance of Documents Retrieved by Clinical Questions Using the MEDLINE Databasea Judge 2 Judge 1 Definitely relevant Probably relevant Not relevant aJudgments
Definitely relevant
Probablyrelevant
Not relevant
127
112 97
96 224 779
were rated on a three-point scale: definitely relevant, possibly relevant, and not relevant. Source: Hersh, Buckley, et al. (1994).
104
I. Basic Concepts
monly used to assess agreement in diagnostic evaluations, such as X-ray or pathology specimen reading (Kramer and Feinstein, 1981). Interest in relevance has waxed and waned over the years. There was a great deal of theoretical thinking and research into relevance in the 1960s, culminating in Saracevic’s seminal review paper (Saracevic, 1975). That paper summarized all the classifications and research data to that time. Two basic problems, Saracevic noted, were the lack of agreement on the definition of relevance (he identified seven different views of it) and the paucity of experimental data supporting either those definitions or how relevance was being applied in evaluation studies. There has been a rekindling of interest in relevance in recent years, most likely owing to the increasing prevalence of IR systems, with a resultant increase in claims and counterclaims about their performance. Schamber et al. (1990) attempted to resurrect debate over the theoretical notions of relevance. Their approach pared Saracevic’s categories of relevance down to two: a system-oriented topical view and a user-oriented situational view. These two views are not at odds with Saracevic’s classification, since the situational category encompasses several of Saracevic’s views that were conceptually similar. The categories of Schamber et al. will be used for the following discussion. 3.3.2.1 Topical Relevance The original view of relevance is that of topical relevance, which Saracevic called the system’s view of relevance. In this view, a document is relevant because part or all of its topical coverage overlaps with the topic of the user’s information need. There is a central but questionable assumption that underlies this view of relevance, noticed by Meadow (1985), which is that the relevance relationship between query and document is fixed. But just because a document is “about” an information need does not mean that it is relevant. A clinician with a patient care problem incorporating the treatment of hypertension with a certain drug most likely will not want to retrieve an article dealing with the use of that drug to treat hypertension in rats. Likewise, a research pharmacologist studying the molecular mechanisms of blood pressure reductions with that drug probably will not want articles about clinical trials with the drug. Furthermore, a medical student or a patient, who knows far less than the clinician or researcher, may not want to retrieve this article at all because its language is too technical. The topical view of relevance persists for several reasons, however. First, it is associated with a perception of objectivity, hence reproducibility. Another reason is that quantitative methods to assess IR systems with situational relevance are difficult to perform and interpret. But perhaps the main reason for the survival of topical relevance is that this view has led to relatively easy measures for quantifying performance in IR systems. The notion of a fixed relevance between query and document greatly simplifies the task of IR evaluation, since if the relevance of a document with respect to a document is fixed, then evaluation can
3. System Evaluation
105
be simulated (without human users) quite easily once relevance judgments have been made. This approach to evaluation has been the modus operandi of a large segment of the IR research world, particularly among those who advocate automated approaches to IR. This approach makes the task of evaluation quite easy in that system users are unnecessary. All that is needed is a test collection consisting of queries, documents, and relevance judgments. When a new system is implemented, or an existing one is modified, evaluation is a simple matter of running the existing queries into the new system and measuring recall and precision. There is reason, however, to question the external validity of the results obtained with this sort of evaluation, which will be explored in greater detail later in this chapter and in Chapter 7. 3.3.2.2 Situational Relevance The second category of relevance attempts to incorporate the user’s situation into the judgment. Saracevic called this view the destination’s view, while others have termed variations of it “situational” (Schamber, Eisenberg et al., 1990), “logical” (Cooper, 1973), or “psychological” (Harter, 1992) relevance. The majority underlying assumption in this view is that the user’s situation and needs cannot be separated from the relevance judgment. Rees (1966) said: “There is no such thing as the relevance of a document to an information requirement, but rather the relevance judgment of an individual in a specific judging situation . . . at a certain point in time.” Cooper (1973) defined the difference between (topical) relevance and utility, arguing that the latter could be measured to assess what value information actually provided to the user. The case for situational relevance was stated more recently by Schamber et al. (1990), who noted the prevalence of the topical view of relevance but highlighted its problems, especially its use in the making of assertions about the nature and performance of IR systems. These researchers argued that the situational approach, based on the dynamics of human–system interactions, could be used to make systematic and reliable measurements. Situational relevance can be challenged from two perspectives. The first is in fact very pertinent to IR in the healthcare domain, namely, that the user may be distinctly unqualified to judge relevance. It was noted in the last chapter, for example, that many physicians lack the skills to critically appraise the medical literature (Anonymous, 1992). Thus, a user may deem an article relevant to a given issue, yet be unable to recognize that it is flawed or that the results described do not justify the conclusions published. The second challenge is whether the variance of the situational picture has an impact on retrieval performance measurements. Lesk and Salton (1968) carried out a study in which users originated a query and judged their retrieved documents for relevance. Relevance judgments were also made by another subject expert. Additional sets of relevance judgments were created by taking the intersection and union of both judges’ relevance assessments. Recall and precision
106
I. Basic Concepts
were then measured based on the original retrieval results, showing that the different judgment sets had little effect on overall results. In other words, algorithms that performed well under one set of relevance judgments performed well under all of them, with the poorly performing algorithms faring poorly under all sets as well. Voorhees (1998) has noted this constancy with data from the Text REtrieval Conference (TREC), as well.
3.3.3 Research About Relevance Judgments Despite all the disagreement about the nature of relevance, few studies have actually attempted to investigate the factors that influence relevance judgment. Most data on relevance judgments come from two large studies done in the 1960s (Cuadra and Katter, 1967; Rees and Schultz, 1967). Cuadra and Katter developed a five-category classification scheme of the factors that could effect relevance judgments, which Saracevic later used in his review paper to summarize the results of research from these studies and others (Saracevic, 1975). The first category was type of document, such as its subject matter and the quantity of it available to the relevance judge. It was found that subject content was the most important factor influencing relevance judgments, indicating that topical relevance does have importance. It was also discovered that specific subject content in a document led to higher agreement among judges. Regarding the amount of document representation available to the relevance judge, it was clear that the title alone led to poor agreement, but there were conflicting results with respect to whether abstract text or increasing amount of full text was better. The second category was the query or information needs statement. In general, the more judges knew about a user’s information need, the more agreement they had. However, the less they knew about the query, the more likely they were to classify documents as relevant. It was also found that statements in documents that resembled the query statement increased the likelihood of a positive relevance judgment. The third category was the relevance judge. Increased subject knowledge of the judge and his or her familiarity with subject terminology correlated with consistency of agreement but varied inversely with number of documents judged relevant. Professional or occupational involvement with users’ information problem also led to higher rates of agreement, regardless of specific subject knowledge. Differences in intended use of documents (i.e., use for background, updating, etc.) also produced differences in relevance judgments. Level of agreement of the relevance judgment was found to be greater for nonrelevant than for relevant documents. The fourth category was judgment conditions, such as different definitions of relevance or varied pressures on the judges. Changing the definition of relevance did not appear to lead to different relevance judgments. However, greater time or stringency pressures did have an effect, causing more positive relevance judgments. In the last category, judgment mode, it was found that judges tend to prefer (i.e., to feel more “comfortable” or “at ease” with) more categories in a rating
3. System Evaluation
107
scale. It was also noted that the end points of scales (i.e., very relevant or very nonrelevant) tended to be used most heavily, although ratings were not normally distributed but rather skewed in one direction. Another finding was that relative scores for a group of document judgments were more consistent than absolute scores. That is, users tended to rank documents for relevance in the same order, even if they chose different categories or scores of relevance. Research in relevance judgments did not pick up again until the mid-1980s, when Eisenberg (1988) began to investigate methods for estimating relevance. Concerned that fixed, named categories of relevance were problematic, he adapted the technique of magnitude estimation, where subjects made their judgments on analog scales without named points. In particular, he used a 100 mm line, with the categories of relevant and nonrelevant as the end points. This approach was found to lessen the problem of relevance judges spreading out their judgments across the fixed, labeled categories of a traditional relevance scale. This technique has also been used to assess how the order of presentation of documents influences relevance judgments. Eisenberg and Barry (1988) gave subjects a set of 15 documents and an information needs statement. Based on earlier work, the relative relevance of each document was known. The documents were presented in either random, high to low, or low to high order. A “hedging phenomenon” was observed, wherein judges tended to overestimate the relevance of initial documents in the set ordered “low to high” and to underestimate relevance for the initial documents in the other set. Subsequent studies of relevance judgments have used more traditional judgment methods. Parker and Johnson (1990), using 47 queries into a database of computer science journal references, found that no difference in relevance judgments occurred with retrieval sets less than or equal to 15 documents (which was the size of Eisenberg’s set). But for larger sets, relevant articles ranked beyond the fifteenth document were slightly less likely to be judged relevant than if they had occurred in the first 15. Another study measured the effect of relevance judgments on a set of 16 documents and 4 queries on alcohol studies made by judges of different types (researcher vs student), levels (senior vs junior), specialties (biomedicine vs social science), and types of evaluation context (relevance or utility) (Regazzi, 1988). The author found in general that there were no significant differences among the different attributes of the judges and their rankings, although it was possible that differences were not detected owing to small sample size. (No power calculations were reported.) Florance and Marchionini (1995) provided additional insight into relevance by assessing how three physicians processed the information in a group of retrieved articles on six clinical topics. The order of the presentation not only had a dramatic effect on relevance, but also showed that the information in the articles was complementary and interrelated. The authors identified two strategies these physicians used to process the information. In the additive strategy, information from each successive paper reinforced what was present in ones. In the recursive strategy, on the other hand, new information led to reinterpretation of pre-
108
I. Basic Concepts
viously seen information and reconsideration of the data in the light of new evidence. This work demonstrated that simple topical relevance oversimplies the value of retrieved documents to users. Barry (1994) assessed the factors that lead a searcher to pursue a document after its retrieval by an IR system. Looking at users in academic libraries who were asked to evaluate the output of their search during a protocol analysis, Barry found seven general criteria for pursuing a document and measured their frequency. 1. Information content of document (e.g., depth/scope of document, objective accuracy/validity, recency: 35.1%) 2. User’s background/experience (e.g., content novelty to user, background experience, user’s ability to understand: 21.6%) 3. User’s belief and preferences (e.g., subjective accuracy/validity, user’s emotional response: 15.8%) 4. Other information sources within the environment (e.g., consensus within field, external verification by other sources, availability via other sources: 14.6%) 5. Sources of the document (e.g., source quality, source reputation/visibility: 7.2%) 6. The document as a physical entity (e.g., obtainability, cost: 2.7%) 7. The user’s situation (e.g., time constraints, relationship with author: 2.9%) This study indicates that the topical content does play an important role in determining relevance to the user, but there are many situational factors, such as novelty to the user and subjective assessment of accuracy and/or validity. Wang (1994) found similar results in research attempting to model the decision-making process applied to pursuing documents after retrieval. Her model linked document information elements (i.e., title, author, abstract, journal, date of publication, language, media) with criteria (i.e., topicality, novelty, quality, availability, authority) that would lead a user to decide whether to seek a retrieved article. Like Barry, Wang found that while topicality was the reason most likely to lead to pursual, other factors had significant influence, such as the recency of the article, its availability, and the reputation or authority of the author(s). Su (1994) assessed how well recall and precision were associated with a user’s satisfaction of their search. Assessing end users who had mediated searches in an academic setting, Su found that absolute recall was much more important to them than precision. While the users’ satisfaction with the precision of their results was associated with overall satisfaction, the actual value of precision in the searches done for them was not. This study was done in a research setting and gives credence to the assertion that searchers in research settings are likely to value recall over precision.
3.3.4 Limitations of Relevance-Based Measures If relevance judgments are situational and inherently variable across judges, then what does this say about the use of recall and precision? One of the harshest critics of these measures has been Swanson (1988), who has argued that, “An information need cannot be fully expressed as a search request that is independent
3. System Evaluation
109
of innumerable presuppositions of context—context that itself is impossible to describe fully, for it includes among other things the requester’s own background of knowledge.” Harter (1992) likewise has argued that fixed relevance judgments cannot capture the dynamic nature of the user’s interaction with an IR system. Even if relevance were a relative concept that existed between certain bounds so that measures based on it, such as recall or precision, could be made, there are still a number of limitations associated with the use of these measures to assess user–system interaction. Hersh (1994) has noted that the magnitude of a significant difference (such as between systems) is not known. The beginning of this chapter discussed the notion of statistical significance, which when present ensures that the difference between two values is not merely due to chance. But just because statistical significance exists does not mean that a difference is meaningful. Using a medical example, consider a new drug being used to treat diabetes, and suppose it lowers blood sugar by an average of 5 mg/dL. Readers with a medical background will note that this value is insignificant in terms of treatment of diabetes or its long-term outcome. Yet one could design a study with a very large sample size that could show statistical significance for the results obtained with this clinically meaningless difference. Yet clinical significance between different levels of recall and precision has never really been defined for IR. In other words, it is unknown whether the difference between, say, 50 and 60% recall has any significance to a real user whatsoever. Related to this issue is the assumption that more relevant and fewer nonrelevant articles are better. For example, in some instances complete precision is not totally desired. Belkin and Vickery (1985) have noted that users often discover new knowledge by “serendipity.” A famous newspaper columnist, the late Sydney Harris, used to periodically devote columns to “things I learned while looking up something else.” Sometimes there is value in learning something new that is peripheral to what one is seeking at the moment. Similarly, at times when complete recall is not desired and may even be a distraction—for example, to the busy clinician who seeks a quick answer to a question. A final problem with recall and precision is that they are often applied in a context different from that of the original experiments, or more problematically, in no context at all. The latter case may be akin to testing a medical therapy without a disease. While simulation can be used to achieve meaningful results in IR evaluation, it must go beyond simple batching of queries into a retrieval system. There must be some interaction with the user, even if that user is acting within a simulated setting. Salton et al. (1990), for example, were highly critical of approaches to IR utilizing natural language processing (NLP) techniques (which will be described in detail in Chapter 9). Their criticism was based mostly on studies in which systems utilizing NLP are given batched queries and output matching documents. Some of the value of NLP, however, may be in helping the user to better formulate the initial query or to reformulate subsequent queries. The techniques of NLP may also assist the user in finding better search terms. These types of benefit cannot be measured in batch-style recall–precision experiments.
110
I. Basic Concepts
Relevance-based retrieval evaluation has clearly had many benefits, allowing assessment, at least superficially, of the value of different approaches to indexing and retrieval. It is clearly useful in the early stages of system development when one is trying to choose from among different indexing and retrieval approaches. Hersh (1994) has stated that conventional recall–precision studies with topical relevance should be done early in the evaluation of a system. The problem comes when investigators attempt to draw conclusions about indexing and retrieval approaches based solely on recall and precision results.
3.3.5 Alternatives to Relevance-Based Measures What alternatives to relevance-based measures can be used for determining performance of individual searches? Harter admits that if measures using a more situational view of relevance cannot be developed for assessing user interaction, then recall and precision may be the only alternatives. Thus Egan et al. (1989) evaluated the effectiveness of Superbook by assessing how well users could find and apply specific information. Mynatt et al. (1992) used a similar approach in comparing paper and electronic versions of an online encyclopedia, while Wildemuth et al. (1995) assessed the ability of students to answer testlike questions by using a medical curricular database. Hersh and his colleagues have adapted these methods to searching for answers to medical questions in an electronic textbook (Hersh, Elliot et al., 1994) and MEDLINE (Hersh, Pentecost et al., 1996). The Interactive Track at the Text REtrieval Conference has also demonstrated the feasibility of this approach (Hersh, 2001). Another approach is to employ subjectivist or usability principles. Although at the risk of inducing the Hawthorne effect, observation and monitoring of users can provide insights. As mentioned earlier, one method is to monitor users’ interaction with a system, recording it on videotape and/or actually tracing people’s actions and keystrokes (Shneiderman, 1998). Researchers can also attempt to capture their thinking via protocol analysis (Ericsson and Simon, 1993) or to observe system use via ethnographic means (e.g., Fidel, Davies et al., 1999).
3.4 The Text REtrieval Conference No overview of IR evaluation can ignore TREC, the Text REtrieval Conference (trec.nist.gov) organized by the U.S. National Institute for Standards and Technology (NIST: www.nist.gov) (Voorhees and Harman, 2000). Started in 1992, TREC has provided a testbed for evaluation and a forum for presentation of results. TREC is organized as an annual event at which tasks are specified and queries and documents are provided to participants. Participating groups submit “runs” of their systems to NIST, which calculates the appropriate performance measure, usually recall and precision. One of the motivations for starting TREC was the observation that much IR evaluation research (prior to the early 1990s) was done on small test collections that
3. System Evaluation
111
were not representative of real-world databases. Furthermore, some companies had developed their own large databases for evaluation but were unwilling to share them with other researchers. TREC was therefore designed to serve as a means to increase communication among academic, industrial, and governmental IR researchers. Although the results were presented in a way that allowed comparison of different systems, conference organizers advocated that the forum not be a “competition” but instead a means to share ideas and techniques for successful IR. In fact, participants are required to sign an agreement not to use results of the conference in advertisements and other public materials (Voorhees and Harman, 2000). The original TREC conference featured two common tasks for all participants. An ad hoc task simulated an IR system, where a static set of documents was searched using new topics, similar to the way a user might search a database or Web search engine for the first time. A routing task, on the other hand, simulated a standing query against an oncoming new stream of documents, similar to a topic expert’s attempt to extract new information about his or her area of interest. The original tasks used newswire and government documents, with queries created by U.S. government information analysts. System performance has been measured by a variety of measures similar to those described in Section 3.3. Relevance judgments have been performed by the same information analysts who create the queries (Voorhees, 1998). By the third TREC conference (TREC-3), interest was developing in other IR areas besides ad hoc tasks and routing. At that time, the conference began to introduce tracks geared to specific interests. In an overview of TREC, Voorhees recently categorized the tasks and tracks associated with them (Voorhees and Harman, 2002): • • • • • • •
Static text—Ad Hoc Streamed text—Routing, Filtering Human in the loop—Interactive Beyond English (cross-lingual)—Spanish, Chinese, and others Beyond text—OCR, Speech, Video Web searching—Very Large Corpus, Web Answers, not documents—Question-Answering
A number of other tracks have appeared as well: the Query Track compared queries within systems, and the Database Merging Track assessed documents retrieved from multiple discrete databases and merged into a single output for a query. So much interest has developed in the specific tracks that the routing and ad hoc tracks were discontinued after TREC-6 and TREC-8, respectively. The results from a number of these tracks will be discussed in Chapter 8 and beyond. However, different evaluation measures have been necessary for some of the tracks because the notion of a relevant document no longer is solely how performance is appropriately measured. In the Question-Answering Track, for example, the goal of the task is to provide an answer within a 50-byte span of the document (Voorhees and Tice, 2000). Furthermore, only the top-ranking document that contains the answer is important, since any further answers would be
112
I. Basic Concepts
TABLE 3.7. Table to Calculate Kappa Statistic for Two Observers Judging Whether an Event is X or Y Observer 2 Observer 1 X Y Total
X
Y
Total
a c ac
b d bd
ab cd abcd
redundant. This track therefore uses the mean reciprocal rank (MRR), which is calculated by: 1 MRR Rank of document with answer
(6)
The Filtering Track is an outgrowth of the routing task, with the difference that the user is truly interested in a stream of documents as opposed to a ranked list, which was how the original routing task was evaluated (Lewis, 1996). The information filtering setting is typically in a busy work environment, where the user does not want to spend time on nonrelevant documents. This track’s evaluation measure therefore is based on a utility score that includes a penalty for nonrelevant documents that are retrieved: Utility (ur relevant documents retrieved) (unr nonrelevant documents retrieved)
(7)
where ur and unr are relative utilities of the value of retrieving relevant and nonrelevant documents, respectively. In recent iterations of the track, the values of ur and unr have been set at 2 and –1 (Robertson and Soboroff, 2001). The TREC experiments have also led to research about evaluation itself. Voorhees (1998), for example, has assessed the impact of different relevance judgments on results in the TREC ad hoc task. In the TREC-6 ad hoc task, over 13,000 documents among the 50 queries had duplicate judgments. Substituting one set of judgments for the other was found to cause minor changes in the MAP for different systems but not their order relative to other systems. In other words, different judgments changed the MAP number but not the relative performance among different systems. Zobel (1998) has demonstrated that the number of relevant documents in the ad hoc track is likely underestimated, hence recall may be overstated. TABLE 3.8. Sample Data to Calculate Kappa Observer 2 Observer 1
Relevant
Nonrelevant
Total
Relevant Nonrelevant Total
75 2 77
3 20 23
78 22 100
3. System Evaluation
113
The TREC environment also has its share of critics. Those who have argued that system assessments based solely on topical relevance assessments and not employing real users are implicitly criticizing the TREC model (Swanson, 1977; Harter, 1992; Hersh, 1994). Blair (2002) notes the problems in calculating recall that have been put forth by others and further argues that the TREC ad hoc experiments overemphasize the importance of recall in the operational searching environment. He does not, however, acknowledge the TREC Interactive Track which addresses some of his concerns.
3.5 Measures of Agreement Much IR evaluation involves human judgments. Such judgments may consist of determining whether documents are relevant or indexing terms are appropriately assigned by a person or computer. There are a variety of measures for assessing how well humans agree on these judgments. Probably the best-known and most widely used among these in the kappa statistic (J. Cohen, 1960). There are other measures of agreement and reliability for judgments, which are described in the textbook by Friedman and Wyatt (1997). The kappa statistic measures the difference between observed concordance (OC) and expected concordance (EC). Although the kappa statistic can be calculated for more than dichotomous variables, the example here will use a variable that can only have to values (such as relevant or nonrelevant). Table 3.7 defines the variables for the following formulas: ad OC abcd (a c)(a b) (b d)(c d) abcd abcd EC abcd
(8)
(9)
OC EC Kappa (10) 1 EC In general, the following kappa values indicate the stated amount of agreement (J. Cohen, 1960): • Poor 0.4 • Fair 0.4–0.6 • Good 0.6–0.8 • Excellent 0.8 The kappa value for the relevance judgments presented in Table 3.5 was 0.41 (Hersh and Hickam, 1994), with comparable results obtained in other studies (Haynes, McKibbon et al., 1990; Hersh and Hickam, 1992, 1993, 1995). Table 3.8 presents sample data to calculate kappa for relevance judgments. The OC is 95/100 0.95. The EC is [(77 78)/100 (23 22)/100]/100 0.65. The kappa is therefore (0.95–0.65)/(1 0.65) 0.86.
This page intentionally left blank
Part II State of the Art
This page intentionally left blank
Chapter 4 Content
In the first edition of this book, describing the content available in IR systems was relatively simple. The Web and hypermedia in general were in their infancy, so most content could be classified as either bibliographic or full text. The former consisted of databases that cataloged books or periodicals, while the latter contained the full text of a relatively small number of books, periodicals, and other resources available in electronic form. Most available full-text content was distributed via CD-ROM, with online content limited mainly to text-only displays of periodicals. The world has changed considerably in the last half-decade. Not only has the quantity of content exploded, but so has its diversity. The “mode of access” described in the first edition (e.g., CD-ROM, dial-up terminal) is largely irrelevant, since virtually all content is accessed over the Internet via Web-based interfaces. The massive amount of online content makes devising a schema that classifies it for presentation in this chapter all the more challenging. Furthermore, the lack of boundaries on the Web hinders explicit classification of some resources into one category or another. Nonetheless, the chapter will begin with a discussion of the classification followed by a tour through its four major categories.
4.1 Classification of Health Information The classification schema is shown in Table 4.1. It is not a pure schema, for some of the subcategories represent type of content (e.g., literature reference databases) while others describe its subject (e.g., genomics databases). Nonetheless, the schema does cover the major types of health and biomedical information available electronically, even if the boundaries between the categories have some areas of fuzziness. The classification will be revisited again in Chapter 6, when the approaches to searching the various resources are covered. The first category consists of bibliographic content. It includes what was for decades the mainstay of IR systems: literature reference databases. Also called bibliographic databases, this content is still a key online health information resource. Even if the entire scientific publishing enterprising were to move online, 117
118
II. State of the Art
TABLE 4.1. Classification of Health Information Content 1. Bibliographic a. Literature Reference Databases b. Web catalogs c. Specialized registries 2. Full text a. Periodicals b. Books c. Web sites
3. Databases and collections a. Images b. Genomics c. Citations d. EBM resources 4. Aggregations a. Genomics b. Clinical c. Consumer
literature references databases will likely still maintain widespread use as an entry point into the scientific literature. A second, more modern type of bibliographic content is the Web catalog. There are increasing numbers of such catalogs, which consist of Web pages that contain mainly links to other Web pages and sites. It should be noted that there is a blurry distinction between Web catalogs and aggregations (the fourth category). In general, the former contain only links to other pages and sites, while the latter include actual content that is highly integrated with other resources. The final type of bibliographic content is the specialized registry. This resource is very close to a literature reference database except that it indexes more diverse content than scientific literature. The second category is full-text content. A large component of this content consists of the online versions of books and periodicals. As already noted, a wide variety of the traditional paper-based medical literature, from textbooks to journals, is now available electronically. The electronic versions may be enhanced by measures ranging from the provision of supplemental data in a journal article to linkages and multimedia content in a textbook. The final component of this category is the Web site. Admittedly the diversity of information on Web sites is enormous, and sites may include every other type of content described in this chapter. However, in the context of this category, “Web site” refers to the vast number of static and dynamic Web pages that sit out on a discrete Web location. The third category consists of databases and other specific collections of information. These resources are usually not stored as freestanding Web pages but instead are often housed in database management systems. The first resource in this category consists of image databases, such as collections of images from radiology, pathology, or other areas. The second resource comprises the various genomics databases that have arisen through gene sequencing, protein characterization, and other genomic research. The third resource contains citation databases, which provide bibliographic linkages of scientific literature. The final resource in this category consists of the many databases designed to maintain information concerning evidence-based medicine (EBM). The final category consists of aggregations of the first three categories. A number of Web sites actually consist of collections of other types of content, aggregated to form a coherent resource. Examples of this will be described in Section 4.5.
4. Content
119
4.2 Bibliographic Bibliographic content consists of references or citations to complete resources. It tends to be richer than other content in metadata. This is not surprising, since bibliographic databases in essence consist of metadata. The original IR databases from the 1960s were bibliographic databases that typically contained references to literature on library shelves. Library card catalogs are also a type of bibliographic database, with the electronic version usually called an online public access catalog (OPAC). Bibliographic databases were designed to steer the searcher to printed resources, not to provide the information itself. Most have fields not only for the subject matter, such as the title, abstract, and indexing terms, but also other attributes, such as author name(s), publication date, publication type, and grant identification number. In this section, discussion begins with literature reference databases, followed by descriptions of Web catalogs and specialized registries. Although not typically thought of as bibliographic databases, many Web catalogs can be viewed as bibliographic in that they provide links to other information sources. Indeed, some modern bibliographic databases offer direct linkage to the literature they are referencing, hence are becoming similar to Web catalogs.
4.2.1 Literature Reference Databases As already noted, the literature reference database MEDLINE is probably the best-known IR application in healthcare. Produced by the NLM, MEDLINE was virtually synonymous with online searching for healthcare topics for many years. There are actually a number of additional bibliographic databases, which are produced by the NLM and other information providers. 4.2.1.1 MEDLINE MEDLINE contains bibliographic references to all the biomedical articles, editorials, and letters to the editor in approximately 4500 scientific journals. The journals are chosen for inclusion by an advisory committee of subject experts convened by the National Institutes of Health. At present, about 300,000 references are added to MEDLINE yearly. Dating back to its inception in 1966, it contains over 11 million references. The MEDLINE database is the electronic version of Index Medicus, a print publication that was formerly the most common way to access medical literature. This publication was founded in the nineteenth century by Dr. John Shaw Billings, who from 1865 to 1895 headed the forerunner of the NLM, the Library of the Surgeon General’s Office (DeBakey, 1991). Billings was the first to diligently catalog the literature of medicine, culminating in the first volume of Index Medicus published in 1879. Some medical libraries still subscribe to Index Medicus, though MEDLINE has largely superseded it. One enduring value of Index Medicus is that it allows searching of pre-1966 literature (although NLM also maintains a database OLDMEDLINE, which has references to some 1958 to 1965 literature).
120
II. State of the Art
TABLE 4.2. Fields in the MEDLINE Database Tag
Name
AB AD
Abstract Affiliation
AID
Article Identifier
AU CI CIN CN
Author Name Copyright Information Comment In Collective Name
CON CY DA DCOM DEP DP EDAT EIN
Comment On Country Date Created Date Completed Date of Electronic Publication Publication Date Entrez Date Erratum In
GS ID
Gene Symbol Identification Number
IP
Issue
IS JC JID
ISSN Journal Title Code MLM Unique ID
LA LR
Language Last Revision Date
MH MHDA
MeSH Terms MeSH Date
PG PHST PMID PS PST PT RF RIN RN
Page Number Publication History Status Date PubMed Unique Identifier Personal name as Subject Publication Status Publication Type Number of References Retraction In EC/RN Number
ROF RPF RPI SB
Retraction Of Republished From Republished In Journal Subset
Description Abstract Institutional affiliation and address of the first author, and grant numbers Article ID values may include the pii (controlled publisher identifier) or doi (digital object identifier) Authors’ names Copyright statement Reference containing a comment about the article Corporate author or group names with authorship responsibility Reference on which the article comments Place of publication of the journal Used for internal processing at NLM Used for internal processing at NLM Electronic publication date Date the article was published Date the citation was added to PubMed Reference containing a published erratum to the article Abbreviated gene names (used 1991–1996) Research grant numbers, contract numbers, or both that designate financial support by any agency of the U.S. Public Health Service Number of the issue, part, or supplement of the journal in which the article was published International Standard Serial Number of the journal MEDLINE unique three-character code for the journal Unique journal ID in NLM’s catalog of books, journals, and audiovisuals Language in which the article was published Data a change was made to the record during a maintenance procedure NLM’s controlled vocabulary Date MeSH terms were added to the citation; the MeSH date is the same as the Entrez date until MeSH are added Full pagination of the article History status date Unique number assigned to each PubMed citation Individual is the subject of the article Publication status Type of material the article represents Number of bibliographic references for review articles Retraction of the article Number assigned by the Enzyme Commission to designate a particular enzyme or by the Chemical Abstracts Service for Registry Numbers Article being retracted Original article Corrected and republished article Code for a specific set of journals
4. Content
121
TABLE 4.2. Fields in the MEDLINE Database (Continued) Tag
Name
SI
Secondary Source Identifier
SO TA TI TT UI UIN UOF URLF
Source Journal Title Abbreviation Title Words Transliterated/Vernacular Title MEDLINE Unique Identifier Upate In Update Of URL Full-Text
URLS
URL Summary
VI
Volume
Description Identifies a secondary source that supplies information (e.g., other data sources, databanks, and accession numbers of molecular sequences discussed in articles) Composite field containing bibliographic information Standard journal title abbreviation The title of the article Non-Roman alphabet language titles are transliterated Unique number assigned to each MEDLINE citation Update to the article Article being updated Link to the full-text of article at provider’s Web site; may be incomplete Link to the article summary at provider’s Web site; may be incomplete Journal volume
Source: National Library of Medicine.
The MEDLINE database has evolved over the years. Beginning in 1975, the NLM began adding abstracts for about 75% of all references. The MeSH indexing vocabulary, to be covered in the next chapter, has expanded to over 19,000 terms. Additional attributes have been added to MEDLINE, such as the secondary source identifier (SI), which provides a link to records in other databases, such as the GenBank database of gene sequences, and the URL full text (URLF), which provides a link to the Web location of the full text (usually on the publisher’s Web site). Other attributes have been enhanced, such as publication type (PT), which lists, for example, whether the article is a meta-analysis, practice guideline, review article, or randomized controlled trial. The current MEDLINE record contains up to 49 fields, all of which are listed in Table 4.2. A clinician may be interested in just a handful of these fields, such as the title, abstract, and indexing terms. But other fields contain specific information that may be of great importance to a smaller audience. For example, a genome researcher might be highly interested in the SI field to link to genomic databases. Even the clinician may, however, derive benefit from some of the other fields. For example, the PT field can help in the application of EBM when one is searching for a practice guideline or a randomized controlled trial. A sample MEDLINE record is shown in Figure 4.1. MEDLINE is accessible by many means and available without charge via the PubMed system at the NLM (pubmed.gov), which provides access to other databases as well. MEDLINE is the database, and PubMed is the computer system used to access it and the other databases (Anonymous, 2001m). A number of other Web sites provide MEDLINE for free. Some information vendors, such as Ovid Technologies (www.ovid.com) and Aries Systems (www.ariessys.com), license the content and provide value-added services that can be accessed for a
122
II. State of the Art
UI—88050247 PMID—3675956 DA—19871231 DCOM—19871231 LR—20001218 IS—0889-7190 VI—33 IP—3 DP—1987 Jul–Sep TI—Effects of 25(OH)-vitamin D3 in hypocalcemic patients on chronic hemodialysis. PG—289-92 AD—Department of Medicine, College of Medicine, University of Illinois, Chicago. AU—Kronfol NO AU—Hersh WR AU—Barakat MM LA—eng PT—Journal Article CY—UNITED STATES TA—ASAIO Trans JC—ASA JID—8611947 RN—0 (Parathyroid Hormones) RN—32222-06-3 (Calcitriol) RN—7440-70-2 (Calcium) RN—7723-14-0 (Phosphorus) RN—EC 3.1.3.1 (Alkaline Phosphatase) SB—IM MH—Alkaline Phosphatase/blood MH—Calcitriol/*therapeutic use MH—Calcium/blood MH—Comparative Study MH—Human MH—Hypocalcemia/*drug therapy/etiology MH—Parathyroid Hormones/blood MH—Phosphorus/blood MH—Renal Dialysis/*adverse effects EDAT—1987/07/01 MHDA—1987/07/01 PST—ppublish SO—ASAIO Trans 1987 Jul–Sep;33(3):289–92. FIGURE 4.1. Sample MEDLINE record. (Courtesy of the National Library of Medicine.)
4. Content
123
fee by individuals and institutions. All non-NLM vendors of MEDLINE pay a fee to license the content from the NLM. It is important to remember that providers of the MEDLINE service do not change any content of the database. They may provide a subset based on certain years or eliminate some of the more infrequently used fields, but the basic MEDLINE reference obtained from one MEDLINE product should have identical content for the fields that are present. This does not mean that a search with one system will yield the same results as another. Not only do vendors segment the database differently (see next paragraph), but as will be seen in Chapter 6, many search systems work differently and yield different results. Some providers segment the very large MEDLINE database into smaller components. This is done for a variety of reasons, not the least of which is to provide users with the ability to limit the size of their retrieval sets. It also allows users to focus on a particular time period, usually the literature of the last few years. Another approach to segmentation that was more popular with CD-ROM products (limited by the size of CD-ROM disks) was to create subsets of the database based on medical specialty, so-called designer MEDLINE. The first product to do this was the Primary Care Subset of the Knowledge Finder product (Aries Systems), which provided 4 to 5 years of coverage for a subset of 270 journals of interest to primary care physicians. Aries and other vendors created similar subsets for pathologists, orthopedists, and others. But most search services now offer only the complete database. 4.2.1.2 Other NLM Bibliographic Resources MEDLINE is only one of many databases produced by the NLM. Not only are a number of more specialized databases also available, but they are also accessed from a variety of interfaces. While most of these databases are bibliographic, some provide full text (described in Section 4.3). In general, the NLM’s other databases have fields defined in ways similar or identical to MEDLINE. In response to the growing number of databases and users’ desires to mix and match them differently, the NLM recently unveiled a plan to reorganize its databases (Anonymous, 2000f). Bibliographic databases are now organized into three general categories: 1. Citations to journal and other periodical articles from 1966 to the present— these consist of MEDLINE, MEDLINE in-process citations (added into MEDLINE but not yet indexed), and publisher-supplied citations (added into PubMed but not planned for indexing in MEDLINE) 2. Citations to books, monographs, whole serials, and audiovisual (AV) material 3. Citations to selected journal articles published prior to 1966 and scientific meeting abstracts The primary interface for access of the first category in PubMed, which provides access to the MEDLINE and HealthSTAR databases, the latter of which contains
124
II. State of the Art
TABLE 4.3. PubMed Subsets (Courtesy of the National Library of Medicine.) Subset AIDS Bioethics Complementary and alternative medicine Core clinical journals Dental journals History of medicine MEDLINE Nursing journals In process Publisher PubMed Central Space life sciences Toxicology
Description Citations about AIDS Citations about bioethics Citations about complementary and alternative medicine Citations from several hundred major clinical journals Citations from dental journals Citations about the history of medicine MEDLINE citations Citations from nursing journals MEDLINE entries not yet indexed Entries from publishers not intended for MEDLINE indexing Citations present in PubMed Central Citations about space life sciences Citations about toxicology
Source: National Library of Medicine.
citations from journals covering health services research and nonclinical aspects of health care (e.g., economics, politics). The PubMed subset (SB) field allows searches to be limited to MEDLINE, in-process citations, publisher-supplied citations, and several subject subsets, all of which are listed in Table 4.3. The main interface for access to books, serials, and AV materials is provided by LOCATORplus (locatorplus.gov). The LOCATORplus system is essentially the NLM’s OPAC, though it also contains some materials which do not reside at the NLM. The latter consist of materials owned by the regional libraries of the National Network of Libraries of Medicine or other organizations that have agreements with NLM to provide records in a variety of areas, such as health services research, space life sciences, bioethics, and family planning/population research. Historically, the NLM did not index scientific meeting abstracts except in a few specific subject areas and continues to not do so generally. This is because in general, conference proceedings publications are thought to be less critically peer-reviewed than journal publications as well as of less interest to searchers outside the specialty from which they were generated. The first clinical area for which scientific meeting abstracts were added to an NLM bibliographic database was cancer. These abstracts were added to the CANCERLIT database, which was jointly developed by the NLM and the National Cancer Institute (NCI) and is now maintained by the latter. The second area in which scientific meeting abstracts were added to a bibliographic database was AIDS. The abstracts were added to AIDS-related references in the AIDSLINE database. This was done because it was determined that much critical research was presented at the annual international AIDS meeting. Furthermore, with the rapid discovery of new information about the disease, there was impetus to make this information searchable as soon as possible, even before more fully developed and peer-reviewed journal articles appeared. Conference
4. Content
125
proceedings abstracts that do not appear in MEDLINE are now accessed via the NLM Gateway described in Section 4.5. 4.2.1.3 Non-NLM Bibliographic Databases The NLM is not the sole producer of bibliographic databases. A number of other entities, public and private, produce a wide variety of databases. Many of these databases used to be available from the two largest online information vendors, BRS (now owned by Ovid) and Dialog (Dialog Corp., www.dialog.com). Now, however, many information producers provide access to their bibliographic databases directly on their own Web sites. Information resources are produced by a number of other NIH institutes besides the NLM. The NCI, as noted, offers CANCERLIT as part of its CancerNet system, which provides a great deal of full-text information on cancer as well (cancernet.nci.nih.gov). Other U.S. government agencies provide bibliographic databases, too. The Department of Education produces the Educational Resources Information Center (ERIC) database (www.ed.gov), which has over 775,000 citations from education-related literature. There are a variety of non-NLM bibliographic databases that tend to be more focused on subjects or resource types. The major non-NLM database for the nursing field is CINAHL (Cumulative Index to Nursing and Allied Health Literature, CINAHL Information Systems, www.cinahl.com), which covers nursing and allied health literature, including physical therapy, occupational therapy, laboratory technology, health education, physician assistants, and medical records. Another prominent subject-specific database is PsycINFO (www.apa.org/psycinfo), which is produced by the American Psychological Association. PsycINFO is a bibliographic database with over 1.5 million references from more than 1600 journals dating back to 1887. There are also private companies that produce databases as broad in scope as MEDLINE, but with other features. Two well-known databases that provide alternate access to the medical literature are Current Contents (Institute for Scientific Information, www.isinet.com) and BIOSIS Previews (BIOSIS, www.biosis.org). The Current Contents series provides bibliographic indexes with abstracts for a variety of scientific fields, including biomedicine. The databases are available not only online, but also as a monthly diskette subscription series. BIOSIS Previews offers access to a number of resources that are not available in the databases already mentioned, including citations to research and technical reports, conference proceedings, symposia, and other sources. Another well-known series of large databases is part of Excerpta Medica (Elsevier Science Publishers, www.excerptamedica.com). EMBASE, the electronic version of Excerpta Medica, is referred to by some as the “European MEDLINE” (www.embase.com). It contains over 8 million records dating back to 1974. EMBASE covers many of the same medical journals as MEDLINE but with a more international focus, including more non-English-language journals. These journals are often important for those carrying out meta-
126
II. State of the Art
analyses and systematic reviews, who need access to all the studies done across the world. Other large bibliographic databases of which healthcare is only a part are of importance to many healthcare researchers. A database of all available books is Books in Print (R. R. Bowker, www.booksinprint.com). Of course, the major online booksellers such as Amazon.com (www.amazon.com) and Barnes & Noble (www.bn.com) also can be considered to maintain bibliographic databases of books. Another important database in Dissertation Abstracts (ProQuest Information and Learning, www.umi.com:8090/proquest/), which provides a citation for virtually every dissertation submitted to North American universities since 1861. Abstracts of dissertations were added in 1980, and in 1988 citations from a number of European universities began to be included.
4.2.2 Web Catalogs While some may not consider Web catalogs to be bibliographic content, they share many features with traditional bibliographic databases. This is especially true for Web catalogs that provide other content on their sites or more exhaustive descriptions of Web sites than a traditional bibliographic database might.
FIGURE 4.2. Medical Matrix topic page for Cardiology. (Courtesy of Medical Matrix.)
4. Content
127
One of the original Web catalogs was Medical Matrix (www.medmatrix.org), which existed before the Web as a text file of medical resources on the Internet. It has since developed an exhaustive database of sites that are rated by its editorial board consisting of physicians and other healthcare professionals. For each topic area, a variety of links are provided for different types of resources. Figure 4.2 shows the top of the page for Cardiology topics. A variety of other health-oriented Web catalogs exist, each with unique features. CliniWeb is a catalog of clinical Web pages written at the level of healthcare students and providers (www.ohsu.edu/cliniweb) (Hersh, Brown et al., 1996). Unlike most catalogs, CliniWeb provides links to individual topics within sites. Other Web catalogs tend to maintain pointers to whole Web sites. The pages in the CliniWeb catalog are organized so that each represents a level of the MeSH hierarchy, allowing the user to browse through the hierarchy (Fig. 4.3). From each page, the user can advance to a higher level (by clicking on the text that says Back to the previous level) or a lower level (by clicking on the tern name). Users can select the Web pages associated with each MeSH term and link directly to them as well as send the term to PubMed for a query. PubMed searches can be limited to review articles or focused on therapy or diagnosis. The PubMed results are presented in the standard PubMed page so the search can be further refined.
FIGURE 4.3. CliniWeb page for Cardiovascular Diseases. (Courtesy of Oregon Health & Science University.)
128
II. State of the Art
Other well-known Web catalogs include the following: • HealthWeb (healthweb.org)—topics maintained by a consortium of 12 midwestern universities (Redman, Kelly et al., 1997) • HealthFinder (healthfinder.gov)—consumer-oriented health information maintained by the Office of Disease Prevention and Health Promotion of the U.S. Department of Health and Human Services • OMNI (Organising Medical Networked Information, omni.ac.uk)—a United Kingdom–based Web catalog (Norman, 1996) that includes a recent subset devoted to nursing, midwifery, and allied health professions (NMAP, nmap.ac.uk) • HON Select (www.hon.ch/HONselect)—a European catalog of clinicianoriented Web content from the Health on the Net Foundation • MedWeb Plus (www.medwebplus.com)—a commercial Web catalog of clinicianoriented content Two large general (i.e., not limited to health) Web catalog projects, Yahoo (www.yahoo.com) and Open Directory (dmoz.org), have significant health components.
4.2.3 Specialized Registries A specialized registry has overlap with literature reference databases and Web catalogs but can be considered to be distinct in that it points to more diverse information resources. An example of a specialized registry is the Catalog of U.S. Government Publications, which provides bibliographic links to all governmental publications whether in print, online, or both (www.access.gpo.gov/su_docs/ locators/cgp/index.html). Another specialized registry of great importance for health care is the National Guidelines Clearinghouse (NGC, www.guideline.gov). Produced by the Agency for Healthcare Research and Quality (AHRQ), it contains exhaustive information about clinical practice guidelines. Some of the guidelines produced are freely available, published electronically and/or on paper. Others are proprietary, in which case a link is provided to a location at which the guideline can be ordered or purchased. The overall goal of the NGC is to make evidence-based clinical practice guidelines and related abstract, summary, and comparison materials widely available to health-care and other professionals. The fields provided in the NGC are listed in Table 4.4. The criteria for a guideline being included in the NGC are as follows: • Contains systematically developed statements that include recommendations, strategies, or information that assists physicians and/or other healthcare practitioners and patients make decisions about appropriate health care for specific clinical circumstances. • Produced under the auspices of medical specialty associations; relevant professional societies, public or private organizations, government agencies at the federal, state, or local level; or healthcare organizations or plans.
4. Content
129
TABLE 4.4. Fields in National Guideline Clearinghouse Guideline Title Bibliographic Source(s) Number of References Guideline Availability Availability of Companion Documents Availability of Related Patient Resources Guideline Status/Update Information Guideline Length Issuing Organization(s) Guideline Developer(s) Guideline Developer Comment Guideline Endorser(s) Adaptation Organization Type Source(s) of Funding Funding Source ID Guideline Committee Composition of Group That Authored the Guideline Date Released Guideline Category Clinical Specialty Disease/Condition(s) Guideline Objective(s) Method of Review of the Guideline Recommendations Description of Method of Review of the Guideline Recommendations Implementation Plan Developed? (Yes/No) Description of Implementation Strategy
Intended Users Target Population Age of Target Population Sex of Target Population Interventions and Practices Considered Major Outcomes Considered Cost Analysis Performed? (Yes/No) Cost Analysis Methods Used to Collect the Evidence Description of Methods Used to Collect the Evidence Number of Source Documents Methods Used to Assess the Quality and Strength of the Evidence Rating Scheme for the Strength of the Evidence Methods Used to Analyze the Evidence Description of Methods Used to Analyze the Evidence Qualifying Statements Major Recommendations Clinical Algorithm(s) Type of Evidence Supporting Recommendations Potential Benefits Subgroup(s) of Patients Most Likely to Benefit Potential Harms Subgroup(s) of Patients Most Likely to Experience These Harms
Source: Agency for Heathcare Research & Quality.
• Corroborating documentation can be produced and verified that a systematic literature search and review of existing scientific evidence published in peerreviewed journals was performed during the guideline development. • Written in the English language, current, and the most recent version produced (i.e., documented evidence can be produced or verified that the guideline was developed, reviewed, or revised within the last 5 years).
4.3 Full Text Full-text content contains the complete text of a resource as well as associated tables, figures, images, and other graphics. If a database has a corresponding print version, then the text portions of the electronic and print versions should be nearly identical. The original full-text databases were online versions of journals and thus tended to be either primary literature or mixtures of primary and secondary literature. As the price of computers and CD-ROM drives fell in the early 1990s, adaptation of nonjournal secondary sources such as textbooks increased. This trend has not only continued with the growth of the In-
130
II. State of the Art
ternet but led to the development of vast Web sites with information aimed at a variety of audiences. Full-text products usually do not have associated human-assigned indexing terms. Instead, the indexing terms are typically the words that appear in the text, as will be described in Chapter 5. Appendix 1 represents a full-text database, containing a title and body of text for each of 10 sample documents.
4.3.1 Periodicals The technical impediments to electronic publishing of journals have long passed, and as discussed in Chapter 2, the challenges now are mostly political and economic (Hersh and Rindfleisch, 2000). Most scientific journals are published electronically in one form or another. Some are published by the company that produces the print version of the journal, such as Academic Press (www.idealibrary.com) and Elsevier (www.elsevier.nl). Over 300 others are published by Highwire Press (www.highwire.org), an outgrowth of the Stanford University Library, which provides a Web site, searching and browsing interfaces, and development tools for journals whose publishers have not moved directly into electronic publishing. Some well-known journals that utilize Highwire Press include the British Medical Journal (www.bmj.com), the New England Journal of Medicine (www.nejm.org), and Journal of the American Medical Informatics Association (www.jamia.org). A handful of journals, such as Journal of the American Medical Association (www.jama.com), Annals of Internal Medicine (annals.org), and Nature (nature.com) have created their own Web sites for their journals. Some journals have been developed exclusively electronically, such as Medscape General Medicine (www.medscape.com/ Medscape/GeneralMedicine/journal/public/mg) and Journal of Medical Internet Research (www.jmir.org). A number of government entities provide periodical information in full-text form. Among the best known of these are • Morbidity and Mortality Weekly Report (MMWR, www.cdc.gov/mmwr) from the Centers for Disease Control and Prevention (CDC) • Journal of the National Cancer Institute (JNCI, published through Oxford University Press and available via Highwire, jnci.oupjournals.org) • Evidence Reports of AHRQ (www.ahrq.gov/clinic/epcix.htm) Electronic publication of journals allows additional features not possible in the print world. Journal editors often clash with authors over the length of published papers (editors want them short for readability, whereas authors want them long to be able to present all ideas and results). To address this situation, the BMJ initiated an electronic-long, paper-short (ELPS) system that provides on the Web site supplemental material that did not appear in the print version of the journal. Journal Web sites can provide additional description of experiments, results, images, and even raw data. A journal Web site also allows more dialogue about ar-
4. Content
131
ticles than could be published in a Letters to the Editor section of a print journal. Electronic publication also allows true bibliographic linkages, both to other full-text articles and to the MEDLINE record, typically from PubMed. These features are also facilitated by the Highwire software. Many journals provide access to PDF versions of articles that print in a more readable format than a Web page, usually in a layout identical to the printed version. The Web also allows linkage directly from bibliographic databases to full text. Again, the impediments here are more economic than technical. As noted, PubMed maintains a field for the Web address of the full-text paper. This linkage is active when the PubMed record is displayed, but users are met by a password screen if the article is not available for free. There is often an option to enter a credit card number of gain access, but the typical price for electronic access ($10–$15 per article) is an impediment to many. Other publishers, such as Ovid and MDConsult (www.mdconsult.com), provide access within their own interfaces to articles from journals that they have licensed for use in their systems. Some additional collections of full-text journal articles were described in Chapter 2 in the discussion of electronic publishing and its impediments. One of these is PubMedCentral (pubmedcentral.nih.gov), which aims to be a freely available archive of scientific research reports. The other is Biomed Central (www.biomedcentral.com), an effort to create online scientific journals that can be accessed without cost.
4.3.2 Textbooks The most common secondary literature source is the traditional textbook, an increasing number of which are available in computer form. One of the first textbooks available electronically was Scientific American Medicine (www. samed.com). Subsequently, the Physician’s Desk Reference (Medical Economics, Inc., www.pdr.net) and the Merck Manual (Merck & Co., www.merck.com/ pubs/mmanual/) because available. The latter is one of the few traditional medical textbooks available for free on the Web. A common approach with textbooks is bundling, sometimes with linkages across the bundled texts. An early bundler of textbooks was Stat!-Ref (Teton Data Systems, www.statref.com), which like many began as a CD-ROM product and then moved to the Web. Stat!-Ref offers over 30 textbooks. A product that implemented linking early was Harrison’s Plus (McGraw-Hill, www.harrisonsonline.com), which contains the full text of Harrison’s Principles of Internal Medicine and the drug reference, U.S. Pharmacopeia. As readers of a general medical textbook (Harrison’s) are unlikely to want to see encyclopedic drug information this text, Harrison’s Plus links every mention of a drug to the exhaustive information about it in the U.S. Pharmacopeia. Other well-known providers of multiple online textbooks are MDConsult (www.mdconsult.com) and LWW Medicine (www.lwwmedicine.com). One textbook site designed from the beginning to be electronic is eMedicine (www.emedicine.com).
132
II. State of the Art
Electronic textbooks offer additional features beyond text from the print version. While many print textbooks do feature high-quality images, electronic versions offer the ability to have more pictures and illustrations. They also have the ability to use sound and video, although few do this now. As with full-text journals, electronic textbooks can link to other resources, including journal references and the full articles. Many Web-based textbook sites also provide access to continuing education self-assessment questions and medical news. And finally, electronic textbooks let authors and publishers provide more frequent updates of the information than is allowed by the usual cycle of print editions, where new versions come out only every 2 to 5 years. Making a textbook or other tertiary literature source usable as an electronic database requires some reorganization of the text. The approach used by most vendors is to break books down into “documents” along their hierarchical structure. Since the text of most books is divided into chapters, sections, subsections, and so forth, a typical approach will be to reduce the text to the lowest level in a subsection. Figure 4.4 demonstrates how this is done in the Merck Manual. The indexing and retrieval based on this approach is described in the next two chapters.
FIGURE 4.4. A section from the Merck Manual. (Courtesy of Merck.)
4. Content
133
Another type of publication long of interest to clinicians in print form is the collected summary of journal articles. The Yearbook Series (e.g., Yearbook of Medicine) from Mosby (www.mosby.com) has been in existence in print form for over half a century. More recent efforts using this approach have been the Massachusetts Medical Society’s Journal Watch (www. jwatch.org) and, from the American College of Physicians–American Society of Internal Medicine (ACP-ASIM, www.acponline.org), Journal Club. The latter is a supplement to ACP-ASIM’s journal Annals of Internal Medicine and uses a highly structured format designed to provide the reader all the important details of the study, including pertinent EBM statistics such as patient population, intervention, and number needed to treat (McKibbon, Wilczynski et al., 1995). There is also an increasing amount of full-text drug information. In addition to the already mentioned U.S. Pharmacopeia, other drug references are available as well. One drug information resource long viewed as an unbiased source of drug information by physicians is the Medical Letter on Drugs and Therapeutics (Medical Letter, www.medletter.com), which is available directly on the publisher’s Web site as well as through other online database providers. Another popular source of drug information is the Physician’s Desk Reference (Medical Economics, www.pdr.net). Other sources of drug information include GenRx (Mosby, www.genrx.com) and Clinical Pharmacology (Gold Standard Multimedia, cp.gsm.com). A growing trend is to redesign full-text information for use on handheld computers. The advantage of these devices for IR databases is their portability, though they are limited by constraints in screen size and memory capacity. They are also usually not connected to networks, though their synchronization capability allows information to be updated frequently, including over the Internet. Available resources include: • Griffith’s Five-Minute Consult (Lippincott Williams & Wilkins, www.5mcc.com) • The Merck Manual • Harrison’s Principles of Internal Medicine Companion Handbook—the pocketbook version of Harrison’s • A number of drug references, such as ePocrates (www.epocrates.com), Tarascon ePharmacopeia (www.medscape.com), and Adverse Drug Interactions (www.medletter.com) Another important full-text resource is Online Mendelian Inheritance in Man (OMIM), available through PubMed. A key feature of this reference is its linkage to references in MEDLINE as well as genome databases. The latter will be described in more detail in Section 4.4 Consumer health information also has been an area of rapid growth in fulltext information (Jimison and Sher, 1995; Eysenbach, 2000a). While traditional consumer-oriented books on health topics are still plentiful in bookstores, and some have migrated online (e.g., The Merck Manual Home Edition, www.
134
II. State of the Art
merckhomeedition.com/home/html), the real growth has occurred with consumeroriented Web sites, which are described next.
4.3.3 Web Sites Bearing in mind that the definition of “Web site” for the classification in this chapter is a discrete collection of Web pages providing full-text information, this section distinguishes Web sites from bibliographic databases, repurposing of books and other printed materials, specialized databases and collections, and aggregations of all of these. Even with this truncated definition, the number of Web sites is huge. As stated in Chapter 1, there are likely over 100,000 sites on the Web that contain health information, a large proportion of which probably have various amounts of full-text information described in this section. Health-oriented Web sites are produced by everyone from individuals to nonprofit entities to companies to governments. The Web has fundamentally altered the publishing of health information. To begin with, the bar of entry has been significantly lowered. Virtually anyone can have access to a Web server, and with that access, he or she can become a “publisher” of health or any other type of information. The ease of producing and disseminating has had ramifications: for example, the ease of copying threatens protection of intellectual property, and the ease of pasting copy can lead to plagiarism. The Internet, through Web sites, news groups, e-mail lists, and chat rooms also rapidly speeds the dissemination of information and misinformation. Nonetheless, there are a great many Web sites that empower the healthcare provider and consumer alike. Probably the most effective user of the Web to provide health information is the U.S. government. The bibliographic databases of the NLM, NCI, AHRQ, and others have been described. These agencies have also been innovative in providing comprehensive full-text information for healthcare providers and consumers as well. Some of these will be described later as aggregations, since they provide resources of many different types. Smaller yet still comprehensive Web sites include the following: • Health Topics (www.cdc.gov/health/diseases.htm) and Traveler’s Health (www. cdc.gov/travel/) Web sites of the CDC • Health Information from the National Institute of Diabetes and Digestive and Kidney Diseases (www.niddk.nih.gov/health/health.htm) • Health Information from the National Heart, Lung, and Blood Institute (www.nhlbi.nih.gov/health/index.htm) • Drug use and regulatory information from the Food and Drug Administration (FDA, www.fda.gov) A large number of private consumer health Web sites have emerged in recent years. Of course they include more than just collections of text, but also interaction with experts, online stores, and catalogs of links to other sites. Sites with the largest amounts of full-text consumer health information include the following:
4. Content
135
• CBS Healthwatch (cbshealthwatch.medscape.com)—a large collection of consumer health information • Intelihealth (www.intelihealth.com)—a large number of consumer health topics developed by the faculty of Harvard Medical School • Netwellness (www.netwellness.com)—not only a vast library of health information (some of which is restricted access) but also has a database of questions posed to its panel of experts in a wide variety of topics • WebMD (my.webmd.com)—a large collection of consumer health information Among the many entities that provide information for patients are the following: • American Academy of Dermatology—www.aad.org/pamphlets • American Academy of Family Physicians—www.familydoctor.org • Canadian Mental Health Association—www.cmha.ca/english/info_centre/publications/mh_pamphlet_index.htm • Federal Consumer Health Information Center—www.pueblo.gsa.gov/health.htm • Oregon Health & Science University (OHSU)—www.ohsuhealth.com • RxMed—www.rxmed.com/handouts.html Some sites provide handouts in low-literacy formats and/or other languages, such as Spanish. These include the following: • British Columbia Persons with AIDS Society—low-literacy handouts for people with AIDS (www.bcpwa.org/Lowlit/lowlit.htm) • FDA—easily read handouts on drug topics (http://www.fda.gov/opacom/lowlit/ 7lowlit.html) • OHSU/Hood River Community Health Outreach Project—handouts in English and Spanish (www.ohsu.edu/library/hoodriver/pamphlets/pamphletindex.shtml) There are also Web sites that provide information geared toward healthcare providers, typically overviews of diseases, their diagnoses, and their treatments, medical news and other resources for providers often are offered as well. These sites include the following: • eMedicine—began as an attempt to create online textbooks in a variety of subjects and evolved into a large collection of review articles (www.emedicine.com) • Medscape—contains reports from recent conferences and journal publications as well as clinical overviews (www.medscape.com) • Praxis—contains overviews of clinical topics (praxis.md) • WebMD—a large collection of clinician-oriented information (www.webmd.com) A number of organizations have used the Web to publish the full text of their clinical practice guidelines, including the following: • American College of Cardiology—www.acc.org/clinical/statements.htm • Institute for Clinical Systems Improvement—www.icsi.org/guidelst.htm • International Diabetes Federation—www.diabetesguidelines.com/health/dwk/ pro/guidelines/ • University of California San Francisco—dom.ucsf.edu/resources/guidelines/
136
II. State of the Art
4.4 Databases/Collections The Web has also proven to be an excellent means for making available a variety of databases. Some of these databases are constantly changing, which it difficult to maintain versions stored on static HTML pages. As a result, they are stored in databases and delivered on dynamic Web pages that are created when the content is requested. Some content is maintained in databases to provide more protection of the content than would be possible if it sat out on the Web as HTML pages.
4.4.1 Images Image collections have always been an important part of healthcare practice, education, and research, and a variety have been made available on the Web. These collections tend to come and go, and often their Web addresses change over time. Table 4.5 provides a sampling of current image databases. Two image collections in Table 4.5 merit special mention. The Visual Human Project of the NLM is a collection of three-dimensional representations of normal male and female bodies (Spitzer, Ackerman et al., 1996). It consists of crosssectional slices of cadavers, with sections of 1 mm in the male and 0.3 mm in the female. Also available from each cadaver are transverse computerized tomography (CT) and magnetic resonance (MR) images. The raw images themselves are very large: each of the 1871 cross-sectional anatomic images is 2048 1216 pixels at 24-bit color, for a size of 7.5 megabytes per image. Compressed (e.g., JPEG) versions of the images have also been made available, which are more feasible for use in Web-based applications. A variety of searching and browsing interfaces have been created which can be accessed via the project Web site (www.nlm.nih.gov/research/visible/visible_human.html). Another image collection of note is the Slice of Life videodisc (Stensaas, 1994), which consists of over 44,000 images and about 65 video sequences from all fields of medicine that can be (and are) used with a number of text-based applications. The latest edition of this cooperative effort contains contributions from 63 institutions, two professional societies, a pharmaceutical company, and 240 individuals across the world. The videodisc frames contain a variety of image types, including those from cytology, electrocardiography, electron microscopy, endoscopy, gross anatomy, microscopic anatomy, people and patients, and angiography, as well as x-rays and other radiologic images. Subject areas include cardiology, cytology, embryology, gross anatomy, hematology, histology, microbiology, neuroanatomy, parasitology, pathology, radiology, gastrointestinal endoscopy, colonoscopy, dermatology, and ophthalmology. A companion Slice of Brain videodisc has recently been created. Each image is associated with a title and description, which can be used by developers to identify appropriate images for their applications. Slice of Life is used not only for IR systems but for other types of applications as well, such as computer-assisted instruction.
4. Content
137
TABLE 4.5. A Sampling of Image Databases on the Web Name General purpose Virtual Human Images from the History of Medicine Slice of Life Mascagni HON Media Clinical Skills Database Dermatology Atlas of Dermatology Dermatology Imaging Bank Integumentary System Dermatologic Image Database Dermatologic On-Line Image Atlas
Dermatopathology Pathology WebPath TumorBoard Case Database Search Engine Urbana Atlas of Pathology Radiology BrighamRad
MedPix Medical Image Database Cardiothoracic Imaging
Organization
Web address
National Library of Medicine National Library of Medicine
www.nlm.nih.gov/research/visible/visible_human.html (adopted for use in many other databases) www.ihm.nlm.nih.gov
University of Utah University of Iowa Health on the Net Foundation Harvard Medical School
medstat.med.utah.edu/sol www.lib.uiowa.edu/hardin-www/mascagni www.hon.ch/HONmedia institute.bidmc.harvard.edu/clinicalskills/default.asp
Loyola University Medical Center University of Utah University of California Davis University of Iowa FriedrichAlexanderUniversity of ErlangenNürnberg, Germany Indiana University
www.meddean.luc.edu/lumen/MedEd/ medicine/dermatology/melton/atlas.htm www.medlib.med.utah.edu/kw/derm
University of Utah TumorBoard.com University of Pittsburgh Medical Center University of Illinois
www.medlib.med.utah.edu/WebPath/webpath.html
Brigham and Women’s Hospital Uniformed Services University Yale University
medocs.ucdavis.edu/DER/420/course.htm tray.dermatology.uiowa.edu/DermImag.htm www.dermis.net/doia
www.pathology.iupui.edu/drhood.html
www.tumorboard.com path.upmc.edu/cases/engine.html
www.med.uiuc.edu/PathAtlasf/titlepage.html
brighamrad.harvard.edu
rad.usuhs.mil/synapse/
info.med.yale.edu/intmed/cardio/imaging/
138
II. State of the Art
4.4.2 Genomics Genomics is the study of genetic material in living organisms. A milestone was reached in 2001 with the publication of a “working draft” of the human genome published simultaneously by the publicly sponsored Human Genome Project (Anonymous, 2001e) and the private company Celera Genomics (Venter, Adams et al., 2001). The final sequence of the 3 billion nucleotides that comprise the human genome is expected to be completed in 2003. Some have argued that the knowledge gained from the Human Genome Project will revolutionize the diagnosis, treatment, and prevention of disease (Collins and McKusick, 2001). One unique aspect of the genomics community (certainly in comparison to other biomedical sciences) has been the sharing of data among researchers. Some of this sharing has been made possible by the development of public databases from the National Center for Biotechnology Information (NCBI, www.ncbi. nlm.nih.gov). However, scientists themselves as well as those developing databases with genome-related content have in general made their information widely available. The myriad of genomics databases are reviewed annually in the first issue of the journal Nucleic Acids Research (nar.oupjournals.org). A cataloging of these databases was recently provided by Baxevanis (2002). The NCBI organizes genomic databases of four general types (Wheeler, Church et al., 2002): 1. Nucleotide sequences—sequences of base pairs from deoxyribonucleic acid (DNA) and ribonucleic acid (RNA) 2. Protein sequences—sequences of amino acids that comprise proteins 3. Protein structures—the three-dimensional structure of proteins 4. Genomes and maps—the locations of genes on chromosomes All these databases are linked among themselves, along with PubMed and OMIM, in the NCBI’s Entrez system (www.ncbi.nlm.nih.gov/Entrez). The prototype nucleotide sequence database is GenBank (Benson, KarschMizrachi et al., 2002). This resource contains over 12.8 million sequences and 13.5 billion base pairs for 75,000 different living species. GenBank is continually updated as researchers add more data and as linkages to other databases become available. A new full release occurs every 2 months. The database can be searched at the NLM Web site or downloaded for loading into local databases. GenBank also contains sequences of proteins that are encoded by the nucleotide sequences. These sequences are parsed out to form the GenPept database that consists only of protein sequences. Another protein sequence database is SWISSPROT (www.expasy.ch/sprot/sport-top.html). Since the function of proteins is highly dependent upon their three-dimensional structure, protein structure databases are of increasing importance. A public database of known proteins and their structures is the Protein Data Bank (PDB, www.rcsb.org/pdb/) (Berman, Westbrook et al., 2000). Actual structures are maintained in the Molecular Modeling Database (MMDB, www.ncbi.nlm.nih.gov/Structure/MMDB/mmdb.shtml) (Wang, Addess et al., 2000).
4. Content
139
FIGURE 4.5. NCBI map viewer for the BRCA1 gene. (Courtesy of the National Library of Medicine.)
Some NCBI databases are aggregations of other databases. For example, the Unigene database clusters GenBank data by genes (Wheeler, Church et al., 2002). Likewise, LocusLink brings together all the known information about a gene (GenBank), the proteins it produces (GenPept and PDB), its structure is available (MMDB), the diseases it causes (the OMIM textbook), and references of scientific literature written about all these (the MEDLINE database) (Pruitt and Maglott, 2002). These data permit aggregated structures to be formed, in particular maps of chromosomes and the genes they contain. The Entrez Map Viewer gives a graphical depiction of the location on a given chromosome of a specific gene, as well as links to each gene’s LocusLink record (see Figure 4.5). Another effort at integrating genomic information is the HOWDY project from Japan (Hirakawa, 2002). The next direction in genomics research involves the use of gene expression microarrays. Also called DNA chips, microarrays measure the gene expression or functional activity of DNA from a biological sample (Duggan, Bittner et al., 1999). Abnormalities in gene expression can be associated with diseases or different metabolic responses to drugs. The linkage of genes with associated MEDLINE records and their MeSH terms makes possible the potential for discovery of other disease processes and physiological phenomena (Masys, Welsh et al., 2001).
140
II. State of the Art
The real innovation of the genomics databases is their integration. Indeed, the linkage of information and the way data are shared differ distinctly from conventions in the clinical world, where the databases, including many described in this chapter, exist as information islands or silos on the Web. Although most clinical databases are easy to reach and to navigate, there is no simple way to seamlessly move across them (e.g., link from a database of systematic reviews to the original studies comprising a review or a textbook description of the disease or treatment being reviewed).
4.4.3 Citations Chapter 2 described bibliometrics, the field concerned with linkage of the scientific literature. Bibliometric databases can be very useful in IR; that is, searchers may wish to find articles by tracing their references. The Science Citation Index (SCI) and Social Science Citation Index (SSCI) are databases of citations in the scientific literature. As will be noted in Part III, citations allow the retrieval of additional relevant references that cannot be accessed by means of conventional searching. A recent development from ISI is the Web of Science, a Web-based
FIGURE 4.6. Citations and links to them in Web of Science. (Courtesy of the Institute for Scientific Information.)
4. Content
141
interface to the SCI and SSCI databases. Figure 4.6 shows a screen with citations to some of the author’s works, including the first edition of this book; the citations are not highlighted because the SCI and SSCI databases only contain journal literature.
4.4.4 EBM Databases There has also been a great deal of effort in recent years to develop resources that adhere to the principles of EBM but avoid the scattering and fragmentation of the primary literature (Hersh, 1999). One type of approach is the structured abstract and was pioneered by ACP Journal Club and is now part of an electronic product called Best Evidence, which adds Evidence-Based Medicine (a print journal that adopts the ACP Journal Club approach beyond internal medicine) and the textbook Diagnostic Strategies for Common Medical Problems (ACP-ASIM). Another approach has been to develop databases of systematic reviews. As noted in Chapter 2, the most prominent among these efforts is the CDSR, which contains over 1,000 systematic reviews on a variety of clinical topics. The CDSR is part of The Cochrane Library, which also contains: • The Database of Abstracts of Reviews of Effectiveness—structured abstracts of systematic reviews in other publications • The Cochrane Controlled Trials Register—a bibliography of controlled trials identified by the Cochrane Collaboration and others, including reports published in conference proceedings and in many other sources not currently listed in MEDLINE or other bibliographic databases • The Cochrane Methodology Register—a bibliography of articles and books on the science of research synthesis • The NHS Economic Evaluation Database—a register of published economic evaluations of healthcare interventions from the British National Health Service • Health Technology Assessment Database—information on healthcare technology assessments • Cochrane Database of Methodology Reviews (CDMR)—full text of systematic reviews of empirical methodological studies • A handbook on critical appraisal and the science of reviewing research • A glossary of methodological terms • Contact details for Collaborative Review Groups and other entities in the Cochrane Collaboration Systematic reviews are complex to read. As noted in Chapter 2, Haynes (Haynes, 2001) has asserted that clinicians prefer highly synthesized synopses of evidencebased information. Some content which adheres to this approach includes: • • • •
Clinical Evidence—an “evidence formulary” (www.clinicalevidence.com) Up-to-Date—content centered around clinical questions (www.uptodate.com) InfoPOEMS—“patient-oriented evidence that matters” (www.infopoems.com) Physicians’ Information and Education Resource (PIER)—“practice guidance
142
II. State of the Art
statements” for which every test and treatment has associated ratings of the evidence to support them (pier.acponline.org) • EBM Solutions—evidence-based disease management guidelines from a consortium of six academic medical centers (www.ebmsolutions.com)
4.4.5 Other Databases There are many other databases on the Web, many of which fall outside the scope of IR databases. Some, nonetheless, are relevant to health care and warrant mention: • ClinicalTrials.gov—a database of all currently active clinical trials funded by the NIH • Computer Retrieval of Information on Scientific Projects (CRISP, crisp.cit. nih.gov)—a database of all of grants, contracts, and other projects conducted or funded by the NIH
4.5 Aggregations The real value of the Web, of course, is its ability to aggregate completely disparate information resources. This chapter so far has focused for the most part on individual resources. This section provides some examples of highly aggregated resources. Probably the largest aggregated consumer information resource is MEDLINEplus (medlineplus.gov) from the NLM (N. Miller, Lacroix et al., 2000). MEDLINEplus includes representatives of the types of resources already described, aggregated for easy access to a given topic. At the top level, MEDLINEplus contains: • Health topics • Drug information • Dictionaries • Directories • Other resources MEDLINEplus currently contains over 400 health topics. The selection of topics is based on analysis of those used by consumers to search for health information on the NLM Web site (N. Miller, Lacroix et al., 2000). Each topic contains links to health information from the NIH and other sources deemed credible by its editorial staff. There are also links to current health news, a medical encyclopedia, drug references, and directories, along with a preformed PubMed search, related to the topic. Figure 4.7 shows the top of the MEDLINEplus page for cholesterol. Similar to MEDLINEplus, but focused on cancer information, is CancerNet from the NCI (cancernet.nci.nih.gov). Information is provided on all facets of the disease. Within CancerNet is a resource called the Physician Data Query (PDQ), which consists of several databases, including the following:
4. Content
143
FIGURE 4.7. MEDLINEplus topic Cholesterol. (Courtesy of the National Library of Medicine.)
1. Cancer treatment file—list of over 80 cancers and their treatments 2. Directory file—list of over 19,000 physicians who treat cancer 3. Patient information file—list of over 75 cancers and their treatments for patients and other laypersons 4. Protocol file—list of over 1500 active or approved cancer treatment trials 5. Protocol backfile—list of cancer treatment trials no longer accepting patients 6. Supportive care—information of supportive care for cancer treatment Consumers are not the only group for whom aggregated content has been developed. Indeed, one might classify some of the larger clinician Web sites listed in Section 4.3 or the NCBI Entrez system as such. One aggregated resource for clinicians that combines several applications is MedWeaver (Unbound Medicine, www.unboundmedicine.com) (Detmer, Barnett et al., 1997). MedWeaver combines three applications: 1. MEDLINE 2. DxPlain—a diagnostic decision support that maintains profiles of findings for over 2000 diseases and generates differential diagnoses when cases of findings are entered (Barnett, Cimino et al., 1987) 3. A Web catalog
144
II. State of the Art
FIGURE 4.8. Links to resources for a middle-aged male with chest pain and shortness of breath in MedWeaver. (Courtesy of Unbound Medicine.)
Figure 4.8 shows MedWeaver links elicited by a case of a middle-aged man with chest pain and shortness of breath. A differential diagnosis is generated, with links to the DxPlain disease profile, MEDLINE, and entries in a Web catalog. The Explain option lists the diagnostic reasoning of DxPlain. TABLE 4.6. Categories of Content in NLM Gateway Category Journal citations Books, serials, and audiovisuals Consumer health resources Meeting abstracts Other collections Source: National Library of Medicine.
Collections MEDLINE (PubMed), OLDMEDLINE LOCATORplus MEDLINEplus Health Topics, MEDLINEplus Drug Information, DIRLINE AIDS Meetings, Health Services Research Meetings HSRProj
4. Content
145
The NLM provides other aggregations. A practitioner-oriented aggregation is the NLM Gateway (gateway.nlm.nih.gov), which aims to provide access to all NLM databases within via a single searching interface. Table 4.6 lists the categories of content and their collections in the NLM Gateway. A more focused but still comprehensive aggregation is ToxNet (toxnet.nlm.nih.gov), which includes bibliographic and full-text resources on toxicology and related areas. Some commercial efforts have also attempted to aggregate broad amounts of clinical content along with content about practice management, information technology, and other topics. These include the following: • MDConsult—developed by several leading medical publishers • Merck Medicus (www.merckmedicus.com)—developed by the well-known publisher and pharmaceutical house, available to all licensed U.S. physicians, and including such well-known resources as Harrison’s Plus, MDConsult and DxPlain. • Medem (www.medem.com)—sponsored by a number of medical societies, including the American Medical Association
Chapter 5 Indexing
In the first chapter, indexing was defined as the process of assigning metadata, consisting of terms and attributes, to documents. This process is also called tagging. There are two reasons to index document collections, one cognitive and one mechanical. The cognitive reason for indexing is to represent the content of individual documents so that searchers may retrieve them accurately. The mechanical reason for indexing is to enable computer programs to rapidly determine which documents contain content described by specific terms and attributes. This chapter will explore the indexing process in more detail. After some introductory discussion, the two broad approaches to indexing, manual and automated, will be described. For manual indexing, approaches applied to bibliographic, full-text, and Web-based content will be presented. This will be followed by a description of automated approaches to indexing, with discussion limited to those used in large-scale systems. (Research approaches will be discussed in Part III.) The problems associated with each type of indexing will also be explored. The final section describes computer data structures used to maintain indexing information for efficient retrieval.
5.1 Types of Indexing The indexing of documents for content long preceded the computer age. The most famous early cataloger of medical documents, John Shaw Billings, avidly pursued and cataloged medical reference works at the Library of the Surgeon General’s Office (W. Miles, 1982). In 1879 he produced the first index to the medical literature, Index Medicus, which classified journal articles by topic. For over a century, Index Medicus was the predominant method for accessing the medical literature. By the middle of the twentieth century, however, the chore of manually cataloging and indexing of the expanding base of medical literature was becoming overwhelming, but fortunately the beginning of the computer age was at hand. While initial efforts at automation were geared toward improving the efficiency of the indexing and publishing process, the potential value of using computers 146
5. Indexing
147
for actual retrieval became apparent as well, with the birth of MEDLINE in the early 1960s. By the 1990s, MEDLINE, the electronic version of Index Medicus, had made the paper version nearly obsolete (except for those needing literature prior to 1966). Even though the medium has changed, the human side of indexing the medical literature has not. The main difference in the computer age is that a second type of indexing, automated indexing, has become available. The most modern commercial content is indexed in two ways: 1. Manual indexing—where human indexers, usually using standardized terminology, assign index terms and attributes to documents, often following a specific protocol 2. Automated indexing—where computers make the indexing assignments, usually limited to breaking out each word in the document (or part of the document) as an indexing term Manual indexing has mostly been done with bibliographic databases. In the age of proliferating electronic databases, such as online textbooks, practice guidelines, and multimedia collections, manual indexing has become either too expensive or outright infeasible for the quantity and diversity of content now available. Thus there are increasing numbers of databases that are indexed only by automated means. Recall from Chapter 1 that the indexing process uses one or more indexing languages to represent the content of documents and queries for retrieval of documents. In the human indexing process, the main indexing language is usually a controlled vocabulary of terminology from a field. When relationships among different terms are specified, this vocabulary is called a thesaurus. The indexing language for word indexing, however, consists of all the words that are used for indexing (often minus a small number of common function words, called a stop list or negative dictionary), with no control imposed. Some authors classify indexing differently, distinguishing it as either precoordinate or postcoordinate. These distinctions are usually but not necessarily applied to human indexing, since they refer to whether the indexing terms are coordinated at indexing (precoordinate) or retrieval (postcoordinate) time. In precoordinate systems, the indexing terms are searchable only as a unit; thus they are “precoordinated.” While many early retrieval systems required precoordinated searching on full terms only, most modern systems allow searching on the individual words of indexing terms, hence are “postcoordinated.”
5.2 Factors Influencing Indexing A variety of factors influence indexing. Usually careful consideration must be given to selecting appropriate terms that lead to the most effective retrieval by users. Two measures reflect the depth and breadth of indexing, specificity and exhaustivity, respectively. These measures can also be used as criteria for eval-
148
II. State of the Art
uating the quality of indexing for any specific purpose. Another concern with the quality of indexing is inconsistency. While not an issue with automated systems whose computer algorithms produce the same results every time, manual indexing must be consistent for users who anticipate terms being assigned to documents they expect to retrieve. Of course, the ultimate measure of indexing quality is how well users can use the indexing to access the documents they need, which will be covered in Chapter 7. The first measure of indexing, specificity, refers to the detail or precision of the indexing process and indicates its depth. The desired level of specificity is dependent on both users and databases. Users with much knowledge of a subject area will likely want the highest level of specificity. Researchers, for example, may recognize distinct genes or clinical variations associated with a disease that are less known to clinicians. Thus a researcher might find indexing geared to the clinicians to be insufficiently specific, resulting in loss of precision when searching. Likewise, a clinician who found indexing geared to the researcher too specific might experience loss of recall owing to improper use of highly specific indexing terms. In general, more indexing specificity translates into better retrieval precision, assuming that searchers understand and apply the terms in their queries properly. Exhaustivity indicates the completeness of indexing or its breadth. In the human indexing process, terms are generally assigned to documents when they are one of the focal subjects of a document. Increasing exhaustivity of indexing will tend to increase recall, since more possible indexing terms will increase the chance of retrieving relevant documents. On the other hand, excessive exhaustivity will result in diminished precision, especially if search terms are only loosely related to documents retrieved by the searcher. The final measure of indexing quality is consistency. It has been shown that indexing consistency leads to improved retrieval effectiveness (Leonard, 1975). Hooper’s measure has been used to indicate the percentage consistency of indexing (Funk and Reid, 1983): i Consistency(A,B) ijk
(1)
where A and B are th two indexers, i is number of terms A and B assign in agreement, j is the number of terms assigned by A but not B, and k is the number of terms assigned by B but not A. For example, if two indexers assigned 15 and 18 terms, respectively, 11 of which were in agreement, their consistency would be 11/[11 (15 11) (18 11)] 0.5 or 50%. Duval et al. (2002) have elucidated a set of principles they believe are necessary for Web-based metadata. These include: • Modularity—Components of systems must be built in a modular fashion, with new schemas based on previous work rather than starting anew, including the use of namespaces, which are formal collections of terms managed by explicit policies or algorithms
5. Indexing
149
• Extensibility—Allowing extensions to accommodate specific needs (e.g., the ability to utilize controlled vocabularies in health-related applications) • Refinement—Allowing greater levels of detail for specific domains (e.g., publication types) • Multilingualism—Providing access that reflects cultural and linguistic differences
5.3 Controlled Vocabularies Before discussing indexing processes in detail, it is important to describe controlled vocabularies. While these vocabularies are most often used in manual indexing, numerous research projects have attempted to employ them for automated indexing, as described in later chapters. This section will first discuss some general principles in thesaurus construction, followed by a description of the controlled vocabulary used most often in medical IR systems, the Medical Subject Headings (or MeSH) vocabulary. This will be followed by a discussion of other controlled vocabularies and the Unified Medical Language System (UMLS) project.
5.3.1 General Principles of Controlled Vocabularies Before discussing specific vocabularies, it is useful to define some terms, since different writers attach different definitions to the various components of thesauri. A concept is an idea or object that occurs in the world, such as the condition under which human blood pressure is elevated. A term is the actual string of one or more words that represent a concept, such as Hypertension or High Blood Pressure. One of these string forms is the preferred or canonical form, such as Hypertension in the present example. When one or more terms can represent a concept, the different terms are called synonyms. A controlled vocabulary usually contains a list of certified terms that are the canonical representations of the concepts. Most thesauri also contain relationships between terms, which typically fall into three categories: 1. Hierarchical—terms that are broader or narrower. The hierarchical organization not only provides an overview of the structure of a thesaurus but also can be used to enhance searching (e.g., MeSH tree explosions described in Chapter 6). 2. Synonymous—terms that are synonyms, allowing the indexer or searcher to express a concept in different words. 3. Related—terms that are not synonymous or hierarchical but are somehow otherwise related. These usually remind the searcher of different but related terms that may enhance a search.
5.3.2 The Medical Subject Headings (MeSH) Vocabulary Created by the NLM for indexing Index Medicus, the MeSH vocabulary is now used to index most of the databases produced by the NLM (Coletti and Bleich,
150
II. State of the Art
2001). Other organizations use MeSH to index their bibliographic content as well, such as the Web catalog CliniWeb (Hersh, Brown et al., 1996) and National Guidelines Clearinghouse (www.guideline.gov). The latest version of MeSH contains over 19,000 headings (the word MeSH uses for the canonical representation of its concepts). It also contains over 100,000 supplementary concept records in a separate chemical thesaurus. In addition, MeSH contains the three types of relationships described at the end of Section 5.3.1. 1. Hierarchical—MeSH is organized hierarchically into 15 trees, which are listed in Table 5.1. 2. Synonymous—MeSH contains a vast number of entry terms, which are synonyms of the headings. The majority of these entry terms are “nonprint entry terms,” which are mostly variations of the headings and entry terms in plurality, word order, hyphenation, and apostrophes. There are a smaller number of “print entry terms,” which appear in the printed MeSH catalog. These are often called see references because they point the indexer or searcher back to the canonical form of the term. 3. Related—terms that may be useful for searchers to add to their searches when appropriate are suggested for many headings. The MeSH vocabulary files, their associated data, and their supporting documentation are available on the NLM’s MeSH Web site (www.nlm.nih.gov/mesh/). There is also a browser that facilitates exploration of the vocabulary (www.nlm. nih.gov/mesh/MBrowser.html). There are also paper-based catalogs that present: 1. MeSH terms and their associated data (e.g., each heading’s name, entry terms, tree addresses, related terms, and scope notes) in alphabetic order (Anonymous, 2001f) 2. Tree structures that show terms listed hierarchically by tree (Anonymous, 2001g) 3. A permuted index of words in MeSH, followed by all the headings and entry terms in which the words occur (Anonymous, 2001h) Figure 5.1 shows the screen image from the MeSH browser containing all the data in the vocabulary for the term Hypertension. The page displayed by the
TABLE 5.1. The 15 Trees in MeSH Under Which All Headings Are Classified A. Anatomy B. Organisms C. Diseases D. Chemicals and Drugs E. Analytical, Diagnostic, and Therapeutic Techniques and Equipment F. Psychiatry and Psychology G. Biological Sciences H. Physical Sciences Source: National Library of Medicine.
I. Anthropology, Education, Sociology, and Social Phenomena J. Technology and Food and Beverages K. Humanities L. Information Science M. Persons N. Health Care Z. Geographical Locations
5. Indexing
151
FIGURE 5.1. The MeSH browser page for the heading Hypertension. The components in the note include the tree number, the text summarizing the term usage from the indexing manual, the scope note from the MeSH manual for searchers, an entry term (Blood Pressure, High), two related terms (Antihypertensive Agents and Vascular Resistance), the allowable subheadings, and the unique identifier. (Courtesy of the National Library of Medicine.)
browser also displays the location of the term in the MeSH hierarchy. Figure 5.2 shows a partially pruned version of some of the terms in hierarchical proximity to Hypertension. Table 5.2 lists all the MeSH terms that contain the word high. There are additional features of MeSH designed to assist indexers in making documents more retrievable (Anonymous, 2000d). One of these is subheadings, which are qualifiers to headings that can be attached to narrow the focus of a term. In the Hypertension, for example, the focus of an article may be on the diagnosis, epidemiology, or treatment of the condition. Assigning the appropriate subheading will designate the restricted focus of the article, potentially enhancing precision for the searcher. Table 5.3 lists the subheadings of MeSH and their hierarchical organization. There are also rules for each tree restricting the attachment of certain subheadings. For example, the subheading drug therapy cannot be attached to an anatomic site, such as the femur. The allowed
152
II. State of the Art
FIGURE 5.2. The MeSH hierarchy for the heading Hypertension, which is denoted by the heavy box. Other (but not all) terms at each level are shown. (Courtesy of the National Library of Medicine.)
subheadings for a given term are shown as the Allowable Qualifiers in the browser (Fig. 5.1). Another feature of MeSH that helps retrieval is check tags. These are MeSH terms that represent certain facets of medical studies, such as age, gender, human or nonhuman, and type of grant support. They are called check tags because the indexer is required to use them when they describe an attribute of the study. For example, all studies with human subjects must have the check tag Human assigned. The list of check tags is shown in Table 5.4. Related to check tags are the geographical locations in the Z tree. Indexers must also include these, like check tags, since the location of a study (e.g., Oregon) must be indicated. A feature gaining increasing importance for EBM and other purposes is the publication type, which describes the type of publication or the type of study. A searcher who wants a review of a topic will choose the publication type Review or Review Literature. Or, to find studies that provide the best evidence
5. Indexing
153
TABLE 5.2. Permuted List of All MeSH Headings Containing the Word high in the main heading (shown) or Entry Term (only shown if the main heading does not contain the word high) Antiretroviral Therapy, Highly Active Cardiac Output, High Chromatography, High Pressure Liquid Computing Methodologies (entry term: High Performance Computing) Dental High-Speed Equipment Dental High-Speed Technique Euthyroid Sick Syndromes (entry term: High T4 Syndrome) Hearing Loss, High-Frequency High Mobility Group Proteins High Pressure Neurological Syndrome High-Energy Shock Waves High-Frequency Jet Ventilation High-Frequency Ventilation Higher Nervous Activity Hypergravity (entry term: High Gravity) Hypertension (entry term: Blood Pressure, High) Infant Equipment (entry term: High Chairs) Interleukin-14 (entry term: High Molecular Weight B-Cell Growth Factor) Kininogen, High-Molecular-Weight Lipoproteins, HDL (entry term: High-Density Lipoproteins) Lipoproteins, HDL Cholesterol (entry term: High Density Lipoprotein Cholesterol) Lymphoma, High-Grade Maxillary Sinus (entry term: Antrum of Highmore) Microwaves (entry term: Extremely High Frequency Radio Waves) Polyethylene (entry term: High-Density Polyethylene) Population Growth (entry term: High Fertility Population) Pregnancy, High-Risk Radio Waves (entry term: High Frequency Waves) Radiotherapy, High-Energy Socioeconomic Factors (entry term: High-Income Population) Tangier Disease (entry term: Familial High-Density Lipoprotein Deficiency Disease) Technology, High-Cost Ubiquitin (entry term: High Mobility Protein 20) Women (entry term: High Risk Women) Source: National Library of Medicine.
for a therapy, the publication type Meta-Analysis, Randomized Controlled Trial, or Controlled Clinical Trial would be used. Table 5.5 shows a subset of publication types that are listed in the PubMed help file (Anonymous, 2001i). While not necessarily helpful to a searcher using MeSH, the tree address is an important component of the MeSH record. The tree address shows the position of a MeSH term relative to others. At each level, a term is given a unique number that becomes part of the tree address. All children terms of a higher level term will have the same tree address up to the address of the parent. As seen in Figure 5.2, the tree addresses for children terms for Hypertension have the
154
II. State of the Art
TABLE 5.3. MeSH Subheadings: Indented Terms Are Children Terms Hierarchically adverse effects poisoning toxicity analysis blood cerebrospinal fluid isolation & purification urine anatomy & histology blood supply cytology pathology ultrastructure embryology abnormalities innervation chemistry agonists analogs & derivatives antagonists & inhibitors chemical synthesis complications secondary cytology pathology ultrastructure diagnosis pathology radiography radionuclide imaging ultrasonography embryology abnormalities epidemiology ethnology mortality etiology chemically induced
complications secondary congenital embryology genetics immunology microbiology virology parasitology transmission metabolism biosynthesis blood cerebrospinal fluid deficiency enzymology pharmacokinetics urine microbiology virology organization & administration economics legislation & jurisprudence manpower standards supply & distribution trends utilization pharmacology administration & dosage adverse effects poisoning toxicity agonists antagonists & inhibitors contraindications diagnostic use pharmacokinetics
physiology genetics growth & development immunology metabolism biosynthesis blood cerebrospinal fluid deficiency enzymology pharmacokinetics urine physiopathology secretion statistics & numerical data epidemiology ethnology mortality supply & distribution utilization surgery transplantation therapeutic use administration & dosage adverse effects contraindications poisoning therapy diet therapy drug therapy nursing prevention & control radiotherapy rehabilitation surgery transplantation
Source: National Library of Medicine.
TABLE 5.4. MeSH Check Tags Animal Case report Comparative study English abstract Source: National Library of Medicine.
Female Human In vitro Male
Pregnancy Support, Non-US Gov’t Support, US Gov’t, Non-PHS Support, US Gov’t, PHS
5. Indexing TABLE 5.5. Most Common MeSH Publication Types from the PubMed Help File Addresses Bibliography Biography Classical Article (for republished seminal articles) Clinical Conference (for reports of clinical case conferences only) Clinical Trial (includes all types and phases of clinical trials) Clinical Trial, Phase I Clinical Trial, Phase II Clinical Trial, Phase III Clinical Trial, Phase IV Comment (for comment on previously published article) Congresses Controlled Clinical Trial Consensus Development Conference Consensus Development Conference, NIH Corrected and Republished Article (consider Published Erratum) Dictionary Directory Duplicate Publication (duplication of material published elsewhere) Editorial Evaluation Studies Festschrift (for commemorative articles) Guideline (for administrative, procedural guidelines in general) Historical Article (for articles about past events) Interview Journal Article (excludes Letter, Editorial, News, etc.) Lectures Legal Cases (includes law review, legal case study) Letter (includes letters to editor) Meeting Abstract Meta-Analysis (quantitative summary combining results of independent studies) Multicenter Study News (for medical or scientific news) Newspaper Article Overall (collection of articles; consider Meeting Report) Periodical Index (for cumulated indexes to journals) Practice Guideline (for specific healthcare guidelines) Published Erratum (consider Corrected and Republished Article) Randomized Controlled Trial Retracted Publication (article later retracted by author) Retraction of Publication (author’s statement of retraction) Review (includes all reviews; consider specific types) Review, Academic (comprehensive, critical, or analytical review) Review, Multicase (review with epidemiological applications) Review of Reported Cases (review of known cases of a disease) Review of Literature (general review article; consider other reviews) Review, Tutorial (broad review for nonspecialist or student) Scientific Integrity Review (U.S. Office of Scientific Integrity reports) Technical Report Twin Study (for studies of twins) Validation Studies Source: National Library of Medicine.
155
156
II. State of the Art
same tree address up to the last number. It should be noted that a MeSH term can have more than one tree address. Pneumonia, for example, is a child term of both Lung Diseases (C08.381) and Respiratory Tract Infections (C08.730). It thus has two tree addresses, C08.381.677 and C08.730.610. Another feature of MeSH is related concepts. Most well-designed thesauri used for IR have related terms, and MeSH is no exception. Related concepts are grouped into three types. The first is the see related references. The MeSH documentation provides examples of how concepts are related (Anonymous, 2000d): • Between topics related conceptually (e.g., Naval Medicine and Diving) • Between an organ and a procedure (e.g., Bile Ducts and Cholangiography) • Between an organ and a physiological process (e.g., Bone and Bones and Osteogenesis) • Between a physiological process and a related disease (e.g., Blood Pressure and Hypertension) • Between an organ and a drug acting on it (e.g., Bronchi and Bronchoconstrictor Agents) • Between a physiological process and a drug acting on it (e.g., Blood Coagulation and Anticoagulants) An additional type of related concept is the consider also reference, which is usually used for anatomical terms and indicates terms that are related linguistically (e.g., by having a common word stem). For example, the record for the term Brain suggests considering terms Cerebr- and Encephal-. A final category of related concepts consists of main heading/subheading combination notations. In these instances, unallowed heading/subheading combinations are referred to a preferred precoordinated heading. For example, instead of the combination Accidents/Prevention & Control, the heading Accident Prevention is suggested. Figure 5.1 demonstrates two other features of MeSH terms. The first is the Annotation, which provides tips on the use of the term for searchers. For example, under Congestive Heart Failure, the searcher is instructed not to confuse the term with Congestive Cardiomyopathy, a related but distinctly different clinical syndrome. Likewise, under Cryptococcus, the searcher is reminded that this term represents the fungal organism, while the term Cryptococcosis should be used to designate diseases caused by Crptococcus. The second feature is the Scope Note, which gives a definition for the term.
5.3.3 Other Indexing Vocabularies MeSH is not the only thesaurus used for indexing biomedical documents. A number of other thesauri are used to index non-NLM databases. CINAHL, for example, uses the CINAHL Subject Headings, which are based on MeSH but have additional domain-specific terms added (Brenner and McKinin, 1989). EMBASE, the so-called “European MEDLINE” that is part of Excerpta Medica, has a vo-
5. Indexing
157
cabulary called EMTREE, which has many features similar to those of MeSH (www.elsevier.nl/homepage/sah/spd/site/locate_embase.html). EMTREE is also hierarchically related, with all terms organized under 16 facets, which are similar but not identical to MeSH trees. Concepts can also be qualified by link terms, which are similar to MeSH subheadings. EMTREE includes synonyms for terms, which include the corresponding MeSH term. As noted, a number of other entities use MeSH as part of the indexing process but add other attributes as well. For example, the NGC has a classification scheme that contains controlled terminology for attributes about guidelines in the following categories (www.guideline.gov/FRAMESETS/static_fs.asp?viewabout. overview.classify): • • • • • • • • • • •
Clinical Specialty Disease/Condition (uses other vocabularies besides MeSH) Guideline Category Intended Users Method of Review of Guideline Recommendations Methods Used to Analyze the Evidence Methods Used to Assess the Quality and Strength of the Evidence Methods Used to Collect the Evidence Organization Type Target Population Treatment/Intervention
The PsycINFO (www.apa.org/psycinfo) database uses two indexing vocabularies. The first is a thesaurus of psychology terms, containing over 7000 terms and constructed like a typical thesaurus (Anonymous, 2001j). The second is a set of Classification Codes and Categories, a set of around 150 codes that classify references into broad categories of experimental psychology, treatment, education, and others (www.apa.org/psycinof/about/classcodes.html). The genomics community has also begun to develop vocabularies to describe the roles of genes and proteins in cell. Called the Gene Ontology (GO, www.geneontology.org), its goal is to provide controlled vocabularies that describe aspects of molecular biology (Anonymous, 2000e; Anonymous, 2001c). The vocabulary covers three general areas: • Molecular functions—the function of the gene product at the biochemical level • Biological processes—the biological role of the gene product • Cellular components—the part of the cell where a gene product is found
5.3.4 The Unified Medical Language System One problem for the medical informatics field in general is the proliferation of different controlled vocabularies. Many of these vocabularies were developed for specific applications, such as epidemiological studies, coding for billing, and medical expert systems. It was recognized by the NLM and others as early as the 1980s
158
II. State of the Art
that a significant impediment to the development of integrated and easy-to-use applications was the proliferation of disparate vocabularies, none of which was compatible with any other. Not only did this hamper individual applications, in that the user had to learn a new vocabulary for each application, but the integration of these applications was obstructed as well. The vision of a clinician seamlessly moving among an electronic medical record, literature databases, and decision support systems could not be met if those applications could not communicate with each other by means of a common underlying vocabulary. This is not necessarily surprising, since many vocabularies were created for different purposes. For example, MeSH is used for literature indexing, while ICD9 is used to code diagnoses for billing, SNOMED is used to represent patientspecific information, and CPT-4 is used to code procedures, and so on. Many medical record systems (e.g., Costar and HELP) as well as specialized decision support programs (QMR, Iliad, DxPlain) have their own vocabularies and cannot take data directly from sources other than user input. Applications designed to integrate or interact with other applications, however, cannot communicate because a common language is lacking. A number of analyses have shown that many vocabularies used in medicine for a variety of purposes do not provide comprehensive coverage of concepts (Cimino, Hripcsak et al., 1989). For example, Hersh et al. (1994) have shown that up to 25% of the noun phrases occurring in user information need statements to an IR system were not represented in a combination of vocabularies. Another problem with many existing vocabularies is that terms are expressed by different string forms (see Table 5.6). Furthermore, many of these terms are expressed in forms not common to the end user. The UMLS Project was undertaken with the goal of providing a mechanism for linking diverse medical vocabularies as well as sources of information (Lindberg, Humphreys et al., 1993). When the project began, it was unclear what form the final products would take, and several years of work went into defining and TABLE 5.6. Synonymous Terms from the MeSH and SNOMED Vocabularies Used in Medicine MeSH Acute yellow atrophy Arenaviridae Baritosis Facial hemiatrophy Facial nerve Homosexuality Islands of Langerhans Leukemia, myelocytic Ross River virus Round window Scleroderma, systemic Subacute sclerosing panencephalitis Sturge–Weber syndrome Source: Hersh and Greenes (1990).
SNOMED Atrophy, acute yellow LCM group virus Barium lung disease Romberg’s syndrome Seventh cranial nerve Homosexual state Islets of Langerhans Myeloid leukemias Epidemic Australian polyarthritis Cochlear window Generalized scleroderma Dawson’s encephalitis Encephalotrigeminal angiomatosis
5. Indexing
159
building experimental versions of the UMLS resources (Barr, Komorowski et al., 1988; D. Evans, 1988; Masarie, Miller et al., 1991). There are now three components of the UMLS Knowledge Sources: the Metathesaurus, the Semantic Network, and the Specialist Lexicon (Humphreys, Lindberg et al., 1998). This section will focus on the Metathesaurus, while the other components are described in connection with the research applications they are part of in later chapters. A major focus of the UMLS Metathesaurus has been to create linkages among these disparate vocabularies, not only assisting interprogram communication but also providing a richer vocabulary for IR and other applications. The Metathesaurus component of the UMLS links parts or all of over two dozen vocabularies, including portions of those already listed. The Metathesaurus does not create a new, unified vocabulary, which some early workers (Barr, Komorowski et al., 1988; D. Evans, 1988; Masarie, Miller et al., 1991) had actually called for. Rather, it designates conceptual linkages across existing vocabularies. In the Metathesaurus, all terms that are conceptually the same, such as those listed in the rows of Table 5.6, are linked together as a concept. Each concept may have one or more terms, each of which represents an expression of the concept from a source vocabulary that is not just a simple lexical variant (i.e., differs only in word ending or order). Each term may consist of one or more strings, which represent all the lexical variants that are represented for that term in the source vocabularies. One of each term’s strings is designated as the preferred form, and the preferred string of the preferred term is known as the canonical form of the concept. There are rules of precedence for the canonical form, the main one being that the MeSH heading is used if one of the source vocabularies for the concept is MeSH. Each Metathesaurus concept has a single concept unique identifier (CUI). Each term has one term unique identifer (LUI), all of which are linked to the one (or more) CUIs with which they are associated. Likewise, each string has one string unique identifier (SUI), which in turn are linked to the LUIs in which they occur. Table 5.7 list the concepts, terms, and strings for the concept atrial fibrillation. The English-language components are displayed graphically in Figure 5.3. The canonical form of the concept and one of its terms is atrial fibrillation, with the other term being auricular fibrillation. Within both terms are several strings, which vary in word order and plurality. When two concepts in the Metathesaurus have the same string form (e.g., the disease Cold and temperature description Cold), each is given an identifier to make the canonical forms unique: Cold and Cold . The Metathesaurus also contains a wealth of additional information, samples of which are listed in Table 5.8. In addition to the synonym relationships between concepts, terms, and strings described earlier, there are also nonsynonym relationships between concepts. There are also a great many attributes for the concepts, terms, and strings, such as definitions, lexical types, and occurrence in various data sources. Also provided with the Metathesaurus is a word index that connects each word to all the strings it occurs in, along with its concept, term, and string identifiers. The 2002 edition of the Metathesaurus includes over 800,000 concepts and 1.9 million concept names from more than 60 different biomedical source vocabularies (Anonymous, 2002e). The Metathesaurus is organized into ASCII files that
160
II. State of the Art
TABLE 5.7. Concepts, Terms, and Strings for the Concept atrial fibrillation C0004238ENGPL0004238PFS0016668Atrial Fibrillation0 C0004238ENGPL0004238VCS0353486ATRIAL FIBRILLATION0 C0004238ENDPL0004238VCS0356997Atrial fibrillation0 C0004238ENGPL0004238VCS0415228atrial fibrillation0 C0004238ENGPL0004238VPS0016669Atrial Fibrillations0 C0004238ENGPL0004238VWPS0041388Fibrillations, Atrial0 C0004238ENGPL0004238VWS0220832Fibrillation, atrial3 C0004238ENGPL0004238VWS0288232FIBRILLATION ATRIAL0 C0004238ENGPL0004238VWS0372707Fibrillation, Atrial0 C0004238ENGPL0004238VWS1912105Fibrillation; atrial3 C0004238ENGSL0004327PFS0016899Auricular Fibrillation0 C0004238ENGSL0004327VCS0353545AURICULAR FIBRILLATION0 C0004238ENGSL0004327VCS0357130Auricular fibrillation0 C0004238ENGSL0004327VPS0016900Auricular Fibrillations0 C0004238ENGSL0004327VWPS0041389Fibrillations, Auricular0 C0004238ENGSL0004327VWS0372708Fibrillation, Auricular0 C0004238ENGSL0495689PFS0580463AF-Atrial fibrillation3 C0004238ENGSL1217801PFS1459008afib2 C0004238ENGSL1221970PFS1459006a fib2 C0004238ENGSL1224181PFS1459007af2 C0004238FINPL1553135PFS1849044eteisvaerinae3 C0004238FREPL0163285PFS0227766FIBRILLATION AURICULAIRE2 C0004238GERPL1310841PFS1552787Vorhofflimmern3 C0004238GERSL0424046PFS0548328VORHOFLIMMERN2 C0004238GERSL1234913PFS1476859Aurikulaeres Flimmern3 C0004238GERSL1258828PFS1500774HERZVORHOFFLIMMERN2 C0004238ITAPL1253260PFS1495206Fibrillazione atriale3 C0004238PORPL1785065PFS2082206FIBRILAOCO ATRIAL3 C0004238PORSL0429848PFS0554130FIBRILACAO AURICULAR2 C0004238PORSL1252199PFS1494145FIBRILHACAO AURICULAR2 C0004238PORSL1785066PFS2082207FIBRILAOCO AURICULAR3 C0004238RUSPL0902558PFS1106375PREDSERDII FIBRILLIATSIIA3 C0004238SPAPL0343987PFS0452292FIBRILACION ATRIAL3 C0004238SPASL0442909PFS0567191FIBRILACION AURICULAR2 C0004238SPASL1228799PFS1470745AURICULAR, FIBRILACION2 Source: National Library of Medicine.
follow a relational database organization. In recent years, terms for non-English languages have been added to the Metathesaurus, initially from among the two dozen translations of MeSH but later from other vocabularies as well. A number of IR applications have made use of the Metathesaurus, most of which are research applications that will be described in later chapters (Anonymous 2002f).
5.4 Manual Indexing As mentioned, manual indexing was the only type of indexing possible prior to the computer age. This circumstance may have influenced much of the early work in IR systems that focused only on this aspect of indexing (along with the fact
5. Indexing
161
Atrial Fibrillation
Concept
af afib
Atrial Fibrillation Terms
a fib Auricular Fibrillation AF - Atrial Fibrillation
Strings Auricular Fibrillation Atrial Fibrillation
Fibrillation, Atrial
Fibrillation, Auricular
Auricular Fibrillations
Atrial Fibrillations
FIGURE 5.3. Graphical depiction of atrial fibrillation in the Metathesaurus. (Courtesy of the National Library of Medicine.) TABLE 5.8. Some Metathesaurus Data Elements (Courtesy of the National Library of Medicine.) Concept names CUI—unique identifier for concept LUI—unique identifier for term TS—term status, whether term is a preferred name or synonym LAT—language of term, currently restricted to French and English SUI—unique identifier for string STT—string type, whether string is preferred form, a particular type of varient (i.e., changed word order, singular of preferred form, plural of preferred form), or other type of variant STR—string, alphanumeric string Relationships between concepts REL—related concepts and their type of relationship (i.e., hierarchical or horizontal) COC—co-occurring concepts (concepts and the source in which they co-occur) Concept attributes ST—concept attributes status, whether term is reviewed or unreviweed SCT—syntactic category (part of speech for term) STY—semantic type (designated semantic type from Semantic Network) DEF—definition from MeSH or Dorland’s Illustrated Medical Dictionary CXT—context; hierarchical context in each source vocabulary concept appears LO—locator; occurrence of termin selected sources M##—MEDLINE postings; occurrence of term (if MeSH) in MEDLINE and bacl files SOS—scope statement, MeSH scope note Term attributes LT—lexical tag (whether term is abbreviation, acronym, eponym, trade name, etc.) String attribute SO—source, vocabulary in which string originated DC—descriptor class, type of MeSH descriptor for MeSH terms TH—thesaurus ID, unique identifier from source vocabulary Source: National Library of Medicine.
162
II. State of the Art
that these early machines probably also lacked the power to build large indexes of words in databases). Virtually all human indexing systems utilize a thesaurus. This section will describe the components of thesauri and then look at one example in detail, the MeSH thesaurus (Coletti and Bleich, 2001).
5.4.1 Bibliographic Manual Indexing Manual indexing of bibliographic content is the most common and developed use of such indexing. Bibliographic manual indexing is usually done by means of a controlled vocabulary of terms and attributes, often called a thesaurus. This function has been particularly developed by the NLM through MeSH, which will be the focus of this section. Most databases utilizing human indexing usually have a detailed protocol for assignment of indexing terms from the thesaurus. The MEDLINE database is no exception. The principles of MEDLINE indexing were laid out in the two-volume MEDLARS Indexing Manual (Charen, 1976, 1983). Subsequent modifications have occurred with changes to MEDLINE, other databases, and MeSH over the years (Anonymous, 2000a). This section will present an overview of the MEDLINE indexing process. With the large volume of references constantly being added to MEDLINE, it would be impossible for indexers to read the entirety of every article they index. Rather, they follow the “read/scan” method outlined by Bachrach and Charen (1978): 1. Read and understand the title. 2. Read the introduction to the point at which the author states the purpose, and correlate it with the title. 3. Read chapter, section, and paragraph headings, noting italic and boldface copy; read charts, plates, and tables, laboratory methods, and case reports. 4. Read the summary or conclusions. 5. Scan the bibliographic references. 6. Scan the abstract for hints about items missed in the text but confirm the existence of such items in the text. 7. Scan the author’s own indexing if present. After this process, the indexer assigns from five to twelve headings, depending upon the complexity and length of the article (Bachrach and Charen, 1978). Terms are assigned if the concept is discussed and if any of the following conditions is met: • • • • •
Occurs in the title, purpose, or summary. Is significant in research generally or the results of this paper specifically. Is a check tag. Is covered by several sections or paragraphs. Is in a table or figure.
The major concepts of the article, usually from two to five headings, are designed as central concept headings, and designated in the MEDLINE record by
5. Indexing
163
an asterisk. (Noncentral concepts used to be called non–Index Medicus terms, since they were not represented in Index Medicus.) The indexer is also encouraged to assign the appropriate subheadings. Finally, the indexer must also assign check tags, geographical locations, and publication types. The NLM also edits some of the other fields of the MEDLINE record. For example, author names are formatted with the last name followed by a space and then the initials of the first and middle (if present) name. The NLM policy on the number of authors included in the MEDLINE record has varied over the years. The current policy includes all author names, though in past years it was limited to 10 (1984–1995) or 25 (1996–1999). When the policy changes, it applies only to new records added to the database (i.e., existing records are not changed). Author and institutional names are entered as they appear in the journal, which leads to much variation in authors’ names and affiliations (e.g., some of this author’s articles in MEDLINE have his name listed Hersh WR while others have Hersh W).
5.4.2 Full-Text Manual Indexing Few full-text resources are manually indexed. One type of indexing that commonly takes place with full-text resources, especially in the print world, is that performed for the index at the back of the book. However, this information is rarely used in IR systems; instead, most online textbooks rely on automated indexing (see later). One exception to this is MDConsult (www.mdconsult. com), which uses back-of-book indexes to point to specific sections in its online books.
5.4.3 Web Manual Indexing The Web both is and is not a good place for manual indexing. On one hand, with over a billion pages, manual indexing of more than a fraction of it is not feasible. On the other hand, the lack of a coherent index makes searching much more difficult, especially when specific resource types are being sought. A simple form of manual indexing of the Web takes place in the development of the Web catalogs and aggregations described in Chapter 4. These catalogs make not only explicit indexing about subjects and other attributes, but also implicit indexing about the quality of a given resource by the decision of whether to include it in the catalog. Web catalogs use a variety of indexing classification schemes, from MeSH in CliniWeb to ad hoc ones in Medical Matrix and HealthWeb. Some classifications are derived from well-formulated principles. The health topics selected for MEDLINEplus, for example, were developed from analysis of consumers’ searches on the NLM site (N. Miller, Lacroix et al., 2000). This section focuses on more formal approaches to manual indexing of Web content. Two major approaches to manual indexing have emerged on the Web, which are not mutually incompatible. The first approach, that of applying metadata to Web pages and sites, is exemplified by the Dublin Core Metadata Ini-
164
II. State of the Art
tiative (DCMI, www.dublincore.org). The second is to build directories of content, popularized initially by the Yahoo search engine (www.yahoo.com). A more public approach to building directories has been the Open Directory Project (dmoz.org), which carries on the structuring of the directory and entry of content by volunteers across the world. 5.4.3.1 Dublin Core Metadata Initiative One of the first frameworks for metadata on the Web was the DCMI (Weibel, 1996). The goal of the DCMI has been to develop a set of standard data elements that creators of Web resources can use to apply metadata to their content. The specification has defined 15 elements, as shown in Table 5.9 (Anonymous, 1999). Each element is an attribute–value pair: for example, the attribute DC.Title contains the value of the name of the resource and the attribute DC.Subject has values that list its subject domain. The elements in the DCMI do not differ greatly from metadata elements in conventional paper-based resources, such as the Dewey decimal system for library catalogs or the MEDLINE database for TABLE 5.9. The Dublin Core Metadata Element Set Dublin core element
Definition
DC.title DC.creator
The name given to the resource The person or organization primarily responsible for creating the intellectual content of the resource The topic of the resource A textual description of the content of the resource The entity responsible for making the resource available in its present form A date associated with the creation or availability of the resource A person or organization not specified in a creator element who has made a significant intellectual contribution to the resource but whose contribution is secondary to any person or organization specified in a creator element The category of the resource The data format of the resource, used to identify the software and possibly hardware that might be needed to display or operate the resource A string or number used to uniquely identify the resource Information about a second resource from which the present resource is derived The language of the intellectual content of the resource An identifier of a second resource and its relationship to the present resource The spatial or temporal characteristics of the intellectual content of the resource A rights management statement, an identifier that links to a rights management statement, or an identifier that links to a service providing information about rights management for the resource
DC.subject DC.description DC.publisher DC.date DC.contributor
DC.type DC.format
DC.identifier DC.source DC.language DC.relation DC.coverage DC.rights
Source: Anonymous (1999).
5. Indexing
165
FIGURE 5.4. Metadata for book Web site in DCMI.
medical literature. A large number of projects have used the DCMI in a wide variety of topical areas (www.dublincore.org/projects). The DCMI was recently anointed a standard by the National Information Standards Organization (NISO) with the designation Z39.85 (Anonymous, 2001d). The DCMI is more a semantic conceptualization than a definable syntax, and as such it does not completely identify how one is to represent the metadata. One simple approach, adopted by many organizations, is to put the metadata elements right in the Web page, using the HTML tag META. Figure 5.4 shows what metadata might be associated with this book if it were available on a Web site. The original DCMI specification had a number of limitations. The most obvious was the lack of a standardized syntax, that is, a standard method for expressing the values of attributes. Dates comprise a well-known example. For example, the date 2003-2-5 is generally interpreted as February 5th in the United States and May 2nd in European countries. As any user of MEDLINE who is searching for articles by a specific person or institution knows, of course, the lack of a standardized syntax is not unique to the DCMI. (Names and locations in MEDLINE are complicated by inconsistent usage in source articles.) The standardized syntax problem has been partially rectified with the development of DCMI Qualifiers (Anonymous, 2000c). This document proposes to standardize the use of DC.Date with the ISO 8601 standard (which defines dates as YYYY-MM-DD) (Wolf and Wicksteed, 1997). The DCMI Qualifiers document specifies other qualifiers. For DC.Format, it specifies two refinements, extent (size of duration) and medium, expressed as the appropriate Internet Media Type (IMT, www.isi.edu/in-notes/iana/assignments/media-types/media-types), such as text/html or application/pdf. Likewise, for DC.Language, the use of Internet Engineering Task Force (IETF) Request for Comments (RFC) 1766 is mandated; that is, a two-letter language code must be followed by an optional two-letter country code derived from standards 649 and 3166, respectively, of the International Organization for Standardization (ISO).
166
II. State of the Art
But even with recommendations for specific qualifiers, DCMI does not have an explicit mechanism for enumerating the values of other elements. This is crucial for areas like healthcare, where controlled vocabularies as well as resource types are important. For example, the new DCMI Qualifiers specify the use of five vocabularies, one of which is MeSH. However, no syntax is provided to show how MeSH should be expressed (e.g., by the string form of the term, the tree address, or the MeSH unique identifier). Since string form and tree address are not persistent, their use could cause problems when MeSH is changed. There is also no specified syntax for other aspects of MeSH, such as subheadings and central concept designations. For resource types, the DCMI does provide a Type Vocabulary consisting of nine elements: collection, dataset, event, image, interactive resource, service, software, sound, and text (Anonymous, 2000b). This group, of course, is incomplete for the healthcare professional who is seeking, for example, a review article, case report, radiological image, practice guideline, or randomized controlled trial. There have been two medical adaptations of the DCMI. Malet et al. (1999) proposed the Medical Core Metadata (MCM) project, which would extent the DCMI to allow more health-specific tagging. The MCM project would allow the development of a “globally distributed biomedical knowledge base” that would encode all such knowledge on the Web. The DCMI elements receiving the most attention in the MCM proposed were Subject and Type. The main decisions with regard to Subject were the requirement for a controlled vocabulary and the determination that it be MeSH. A specific syntax was developed for expressing MeSH headings, subheadings, and central concept designations using the SCHEME modifier within the META tag. If the MeSH term in the Subject field of the sample metadata from Figure 5.3 were to be expressed in MCM’s syntax, it would be formatted as follows: A second project applying the DCMI to healthcare resources is the Catalogue et Index des Sites Médicaux Francophones (CISMeF) (Darmoni, Leroy, et al., 2000). A catalog of French-language health resources on the Web, CISMeF has used DCMI to catalog over 13,000 Web pages, including information resources (e.g., practice guidelines, consensus development conferences), organizations (e.g., hospitals, medical schools, pharmaceutical companies), and databases. The Subject field used the French translation of MeSH (dicdoc.kb.inserm.fr: 2010/basismesh/mesh.html) but also includes the English translations. For Type, a list of common Web resources has been enumerated, as given in Table 5.10. In addition to the problem of an underdeveloped syntax, early DCMI proposals suffered, perhaps unfairly, by being expressed in HTML. This tended to imply that the metadata would reside in Web pages. Metadata should not reside within a resource, however, particularly within Web pages. First, the practice encourages the author of the page to perform the indexing. However, the page author is not necessarily the best person to provide the metadata. He or she may
5. Indexing
167
TABLE 5.10. DC.Type enumeration from CISMeFa Advertisements (PT) Architectural drawings (PT) Commercial company Community networks Database (PT) Database, bibliographic Directory Annual directory Catalogs (PT) Registry Resource guides (PT) Education Teaching material Educational courses Instruction (PT) Problems and exercises (PT) Tutorial Teaching structure School University Training Establishment, institution, organization Foundation Hospital Hospital department Image database Library Museum]
Newsgroup and discussion list Patient information Periodicals (PT) Publisher Research structure Scientific society Search tools Society Software Text Bibliography (PT) Congresses (PT) Consensus development conference (PT) Dictionary (PT) Dissertation, memoir Educational courses Encyclopedias (PT) Guide Guidelines Practice guidelines Journal article (PT) Legislation (PT) Medical thesis Monograph (PT) Problems and exercises (PT) Technical report (PT) Trade association, trade society
aPT
indicates MEDLINE publication type. Source: Darmont, Leroy et al. (2000).
be unskilled in indexing, may have an ulterior motive (such as using excess indexing terms in an attempt to increase page hits), or may not comply with the proper format of a given standard. Just as the NLM employs trained indexers to assign MEDLINE metadata, high-quality Web catalogs should employ standards of quality control and indexing expertise. Another problem with the implication that DCMI should reside in Web pages is the assumption that all indexed resources should be at the granularity of the individual page. Like many information resources, print or electronic, many Web sites are not mere collections of HTML pages. Rather, they have organization and structure. A simple example is the online textbook in which the content is organized hierarchically. A more complex example is an aggregation Web site with pages providing not only information but also linkages across databases and applications. The actual usage of DCMI tags in Web pages is quite low. The study of Web size by Lawrence and Giles (1999) estimated that only 0.3% of all Web pages use DCMI. This study also found that about 34% of Web pages use the META tag for keywords or description.
168
II. State of the Art
The limitations in the DCMI and HTML-based metadata have been recognized, and solutions have been proposed. One emerging standard for cataloging metadata is the Resource Description Framework (RDF) (E. Miller, 1998). A framework for describing and interchanging metadata, RDF is usually expressed in Extensible Mark-Up Language (XML), a standard for data interchange on the Web. Key features of XML are its ability to express complex data, its readability, and the growing array of tools to parse and extract data from encoded documents. Increasingly XML is being used to interchange data between databases, and it has been designated the preferred interchange format in the Clinical Document Architecture of the Health Level-7 (HL7, www.hl7.org) standard (Dolin, Alschuler et al., 2001). RDF consists of the following entities: • A resource is anything that can have a Unique Resource Identifier (URI). A URI can be a Web page (identified by a URL) or individual elements of an XML document. • A property is an attribute of a resource, such as an author or subject. • A statement is the combination of a resource, a property, and a value for the property. RDF is expressed in a subject-predicate-object format. An example of an RDF statement is a book (resource) authored (property) by William Hersh (value). The object can be a literal (string) or a resource. In this example, the author can be a name (literal) or structured resource, such as an XML structure with the author’s name, address, phone, email, and so on. RDF properties can be represented in XML. This requires an XML Document Type Definition (DTD) or Schema, both of which specifiy the meaning of the tags in XML documents. In the case of a DTD defining the use of DCMI in RDF, this would define the meaning of the tags and format of the document (Beckett, Miller et al., 2000). As the document provides a link back to the DTD and definitions therein, a machine can read the XML document and process it according to the DTD. Figure 5.5 shows the metadata of Figure 5.4 reformulated in RDF. Using RDF to represent DCMI moves the metadata outside the Web page, thus decoupling metadata and content. As a result of this advantage, different metadata providers can maintain different sets of metadata. Much as the metadata of MEDLINE and EMBASE cover the same content (journal articles) but with varying overlap (higher representation of non-English-language journals in the latter) and different metadata schemas (e.g., MeSH vs EMTREE), RDF allows different entities to maintain their own collections of metadata. This permits different “brands” of indexing, which can compete with each other to provide the best metadata for their intended audiences. RDF allows individuals or groups to define a common semantics expressed in a standardized syntax. For example, the MCM Project may decide that for a metadata framework applied to health-related documents, the subject field (i.e., DC.Subject) must use the MeSH vocabulary. This would be explicitly defined in RDF using the XML namespace mechanism. XML namespaces provide a
5. Indexing
169
Information Retrieval: A Health and Biomedical Perspective William Hersh, M.D.[oan/dc:creator> Information storage and retrieval/dc:subject> Health care A book describing the use of information retrieval systems in health care. Springer-Verlag 2003-1-1 Book en-US FIGURE 5.5. Metadata for book Web site in RDF.
method for unambiguously identifying the semantics and conventions governing the particular use of property types. This is done by uniquely identifying the governing authority of the vocabulary. It should be noted that DCMI is not the only metadata standard that has been proposed for Web content. Another platform is the Platform for Internet Content Selection (PICS, www.w3.org/PICS). This standard was initially proposed as a means for allowing the rating of content on the Web, such as the identification of “adult” Web content to which families may wish to limit their children’s access. This standard has actually been adopted as a means to represent various metadata elements of health-related Web sites, with a particular focus on designating the quality of Web sites (Eysenbach and Diepgen, 1998; Eysenbach, Yihune et al., 2000). 5.4.3.2 Open Directory Another approach to cataloging content on the Web has been to create directories of content. The first major effort to create these was the Yahoo! search engine, which created a subject hierarchy and assigned Web sites to elements within it (www.yahoo.com). When concern began to emerge that the Yahoo directory was proprietary and not necessarily representative of the Web community at large (Caruso, 2000), an alternative movement sprung up, the Open Directory Project. Table 5.11 lists the top-level entries of the Open Directory.
170
II. State of the Art
TABLE 5.11. Top Level of the Open Directory ‹RDF xmlns:r=”http://www.w3.org/TR/RDF/” xmlns:d=”http://purl.org/dc/elements/1.0/” xmlns=”http://directory.mozilla.org/rdf”› ‹Topic r:id=”Top”› ‹tag catid=”1”/› ‹d:Title>Top›/d:Title› ‹narrow r:resource=”Top/Arts”/› ‹narrow r:resource=”Top/Business”/› ‹narrow r:resource=”Top/Computers”/› ‹narrow r:resource=”Top/Games”/› ‹narrow r:resource=”Top/Health”/› ‹narrow r:resource=”Top/Home”/› ‹narrow r:resource=”Top/News”/› ‹narrow r:resource=”Top/Recreation”/› ‹narrow r:resource=”Top/Reference”/› ‹narrow r:resource=”Top/Regional”/› ‹narrow r:resource=”Top/Science”/› ‹narrow r:resource=”Top/Shopping”/› ‹narrow r:resource=”Top/Society”/› ‹narrow r:resource=”Top/Sports”/› ‹narrow r:resource=”Top/Test”/› ‹narrow r:resource=”Top/World”/› ‹narrow r:resource=”Top/Private”/› ‹narrow r:resource=”Top/Bookmarks”/› ‹/Topic› . . . Health
5. Indexing
171
TABLE 5.11. Top Level of the Open Directory (Continued) . . . Source: dmez.org
5.4.4 Limitations of Human Indexing The human indexing process is imperfect. Some of its limitations stem from the use of a thesaurus, which may not contain all the important terminology in a field or may not word the terms in a way that allows nonexpert users to readily identify and apply them. One study of 75 MEDLINE queries generated in a clinical setting contained terms that could not be found in the UMLS Metathesaurus, which is a superset of the MeSH vocabulary (Hersh, Hickam et al., 1994). A thesaurus also may not be up to date. In the mid-1980s, for example, knowledge and terminology related to AIDS expanded and changed, with MeSH lagging several years behind. Another problem with human indexing, described earlier, is inconsistency. Funk and Reid (1983) evaluated indexing inconsistency in MEDLINE by identifying 760 articles that had been indexed twice by the NLM. The most common reasons for duplicate indexing were the accidental resending of an already-indexed article to another indexer and instances of a paper being published in more than one journal. Using Hooper’s equation, Funk and Reid generated the consistency percentages for each category of MeSH term shown in Table 5.12. As can be seen, the most consistent indexing occurred with check tags and central concept headings, although even these only ranged in the level of 61 to 75%. The least consistent indexing occurred with subheadings, especially those assigned to non–central concept headings, which had a consistency of less than 35%. Crain (1987) used protocol analysis in an attempt to determine the reasons for interindexer inconsistency. She observed indexers during the actual indexing process, prompting them to think aloud and explain their rationales
172
II. State of the Art
TABLE 5.12. Consistency of MEDLINE Indexing by Category of MeSH Category Check tags Central concept headings Geographics Central concept subheadings Subheadings Headings Central concept heading/subheading combination Heading/subheading combination
Consistency (%) 74.7 61.1 56.6 54.9 48.7 48.2 43.1 33.8
Source: Funk and Reid (1983).
for MeSH term assignment. Three reasons were identified as leading to inconsistencies: 1. Prior experience in assigning a concept to a MeSH term—some indexers were so used to terms appearing in articles of certain types that they automatically assigned them without a great deal of cognitive reflection. 2. Idiosyncratic rules for assigning concept importance—indexers were more likely to assign terms unfamiliar to them. 3. Differing interpretation of the instructions provided by NLM for indexers, resulting in their being applied differently among indexers.
5.5 Automated Indexing In automated indexing, the second type of indexing that occurs in most commercial retrieval systems, the work is done by a computer. Although the mechanical running of the automated indexing process lacks cognitive input, considerable intellectual effort may have gone into development of the process, so this form of indexing still qualifies as an intellectual process. This section will focus on the automated indexing used in operational IR systems, namely the indexing of documents by the words they contain.
5.5.1 Word Indexing People tend not to think of extracting all the words in a document as “indexing,” but from the standpoint of an IR system, words are descriptors of documents, just like human-assigned indexing terms. Most retrieval systems actually use a hybrid of human and word indexing, in that the human-assigned indexing terms become part of the document, which can then be searched by using the whole controlled vocabulary term or individual words within it. As will be seen in the next chapter, most MEDLINE implementations have always allowed the combination of searching on human indexing terms and on words in the title and abstract of the reference. With the development of full-text resources in the 1980s
5. Indexing
173
and 1990s, systems that allowed word indexing only began to emerge. This trend increased with the advent of the Web. Word indexing is typically done by taking all consecutive alphanumeric sequences between “white space,” which consists of spaces, punctuation, carriage returns, and other nonalphanumeric characters. Systems must take particular care to apply the same process to documents and the user’s queries, especially with characters such as hyphens and apostrophes. The process usually generates an inverted file, as described in Section 5.7. These files can store the part of the document in which the word occurs. They may also store the word’s position in the document, which can use proximity searching as described in the next chapter.
5.5.2 Limitations of Word Indexing Simple word indexing has a number of obvious limitations, as is well known by anyone who has tried to search for information on the programming language Java and ended up with articles about coffee. The potential pitfalls include the following: • Synonymy—different words may have the same meaning, such as high and elevated. This problem may extend to the level of phrases with no words in common, such as the synonyms hypertension and high blood pressure. • Polysemy—the same word may have different meanings or senses. For example, the word lead can refer to an element or to a part of an electrocardiogram machine. • Content—words in a document may not reflect its focus. For example, an article describing hypertension may make mention in passing to other concepts, such as congestive heart failure, that are not the focus of the article. • Context—words take on meaning based on other words around them. For example, the relatively common words high, blood, and pressure, take on added meaning when occurring together in the phrase high blood pressure. • Morphology—words can have suffixes that do not change the underlying meaning, such as indicators of plurals, various participles, adjectival forms of nouns, and nominalized forms of adjectives. • Granularity—queries and documents may describe concepts at different levels of a hierarchy. For example, a user might query for antibiotics in the treatment of a specific infection, but the documents might describe specific antibiotics themselves, such as penicillin. A number of approaches to these problems have been proposed, implemented, and evaluated. For example, natural language processing techniques have been tried for recognizing synonyms, eliminating the ambiguity from polysems, recognizing the context of phrases, and overcoming morphological variation. The limited successes with these approaches have been difficult to generalize and are research problems that will be described in Part III. While the MeSH vocabulary
174
II. State of the Art
and associated features in MEDLINE (e.g., the explosion function described in the next chapter) have handled granularity well in the manual indexing approach, automated approaches to recognizing hierarchical relationships have not lent themselves to generalization.
5.5.3 Word Weighting One limitation of word indexing that has been addressed with some success is content, or the ability to give higher weight to more important words in a document that improve retrieval output. Based on an approach developed by Salton in the 1960s (Salton, 1991), this approach has proven effective particularly for inexperienced searchers, who of course comprise the majority of those using Web search engines. Sadly, Salton, a true pioneer in the IR field, passed away in 1995 just as the approach he created was starting to achieve use in commercial systems. Salton’s approach goes by a variety of names, such as automated indexing, natural language retrieval, statistical retrieval, and the vector–space model. A key element, no matter what the name, has been the use of techniques that do not require manual activities. Despite widespread adoption of the weighting of indexing terms and their use in natural language retrieval with relevance ranking, other techniques innovated by Salton remain research lines of investigation and will be covered in Part III. The remainder of this section will focus on Salton’s basic approach, sometimes called the TF*IDF approach. Salton’s work in the 1960s was influenced by work done in the 1950s by Luhn (1957), an IBM researcher who asserted that the content of documents themselves could be used for indexing. The majority of researchers until that time had assumed that human selection of indexing terms was the most appropriate method for indexing. Luhn noted that words in English followed the law of Zipf, where frequency of the word in a collection of text times rank of the word by frequency is a constant. He proposed, therefore, that words in a collection could be used to rate their importance as indexing terms. He asserted that words of medium frequency had the best “resolving power,” that is, were best able to distinguish relevant from nonrelevant documents, and advocated that words with high and low frequency be removed as indexing terms. The most well-known data to support the Zipfian distribution of the English language came from the Brown Corpus, a collection of word frequencies based on a variety of English language texts totaling a million words (Kucera and Francis, 1967). Table 5.13 shows the 10 most common words in English, along with the Zipfian constant. The Brown Corpus also showed that 20% of words in English account for 70% of usage. Salton (1983) extended Luhn’s ideas and was the first to implement them in a functioning system. Salton asserted that Luhn’s proposals were probably too simplistic. One would not want to eliminate, for example, high-frequency words like diagnosis and treatment, which might be necessary to distinguish documents about these subtopics of a disease. Likewise, one would not necessarily want to eliminate very-low-frequency words like glucagonoma, since
5. Indexing
175
TABLE 5.13. The 10 Most Common Words in the Million-Word Brown Corpus with Rank and Frequency Term the of and to a in that is was he
Rank
Frequency
(Rank frequency)/1000
1 2 3 4 5 6 7 8 9 10
69,971 36,411 28,852 26,149 23,237 21,341 10,595 10,099 9,816 9,543
70.0 72.8 86.6 104.6 116.2 128.0 74.2 80.8 88.3 95.4
Source: Salton (1983).
there might be few documents about this rare type of tumor in any medical database. Salton introduced the notion of an indexing term’s discrimination value, which is its ability to distinguish relevant from nonrelevant documents on a given topic. In practice, a term with a high discrimination value is one that occurs frequently in a small number of documents but infrequently elsewhere. The value of this approach can be shown with a hypothetical example. Consider two databases, one focused on the topic of AIDS and another covering general medicine. In the former, a word like AIDS would be unlikely to be useful as an indexing term because it would occur in almost every document and, when it did, would be nonspecific. The words more likely to be useful in an AIDS database would be those associated with specific aspects of the disease, such as Pneumocystis, carinii, and zidovudine. In a general medicine database, on the other hand, only a small portion of documents would cover the topic of AIDS and thus it would probably be a good indexing term. The first step in word-weighted indexing is similar to all other word-based indexing approaches, which is to identify the appropriate portions of a research amenable to such indexing (e.g., the title and text of an article or its MEDLINE reference) and break out all individual words. These words are filtered to remove stop words, which are common words (e.g., those at the top of the Brown Corpus list) that always occur with high frequency, hence are always of low discrimination value. The stop word list, also called a negative dictionary, varies in size from the seven words of the original MEDLARS stop list (and, an, by, from, of, the, with) to the list of 200 to 250 words more typically used. Examples of the latter are the 250-word list of van Rijsbergen (1979) and the 471-word list of Fox (1992). The PubMed stop list is enumerated in Table 5.14. It should be noted, of course, that stop words can sometimes be detrimental. For example, most stop word lists contain the word a, whose elimination would be problematic in the case of documents discussing Vitamin A or Hepati-
176
II. State of the Art
TABLE 5.14. The PubMed Stop List a
during
latter
primarily
throughout
about
each
latterly
promptly
thru
above
ec
lb
pt
thus
abs
ed
ld
quickly
to
accordingly
effected
letter
quite
together
across
eg
like
rather
too
after
either
ltd
readily
toward
afterwards
else
made
really
towards
again
elsewhere
mainly
recently
try
against
enough
make
refs
type
all
especially
many
regarding
ug
almost
et
may
relate
under
alone
etc
me
said
unless
along
ever
meanwhile
same
until
already
every
mg
seem
up
also
everyone
might
seemed
upon
although
everything
ml
seeming
us
always
everywhere
mm
seems
use
am
except
mo
seen
used
among
find
more
seriously
usefully
amongst
for
moreover
several
usefulness
an
found
most
shown
using
analyze
from
mostly
shall
usually
and
further
mr
she
various
another
gave
much
should
very
any
get
mug
show
via
anyhow
give
must
showed
was
anyone
go
my
shown
we
anything
gone
myself
shows
were
anywhere
got
namely
significantly
what
applicable
gov
nearly
since
whatever
apply
had
necessarily
slightly
when
are
has
neither
so
whence
arise
have
never
some
whenever
around
having
nevertheless
somehow
where
as
he
next
someone
whereafter
assume
hence
no
something
whereas
at
her
nobody
sometime
whereby
5. Indexing
177
TABLE 5.14. The PubMed Stop List (Continued) be
here
noone
sometimes
wherein
became
hereafter
nor
somewhat
whereupon
because
hereby
normally
somewhere
wherever
become
herein
nos
soon
whether
becomes
hereupon
noted
specifically
which
becoming
hers
nothing
still
while
been
herself
now
strongly
whither
before
him
nowhere
studied
who
beforehand
himself
obtained
sub
whoever
being
his
of
substantially
whom
below
how
off
such
whose
beside
however
often
sufficiently
why
besides
hr
on
take
will
between
i
only
tell
with
beyond
ie
onto
th
within
both
if
other
than
without
but
ii
others
that
wk
by
iii
otherwise
the
would
came
immediately
ought
their
wt
can
importance
our
theirs
yet
cannot
important
ours
them
you
cc
in
ourselves
themselves
your
cm
inc
out
then
yours
come
incl
over
thence
yourself
compare
indeed
overall
there
yourselves
could
into
owing
thereafter
yr
de
investigate
own
thereby
dealing
is
oz
therefore
department
it
particularly
therein
depend
its
per
thereupon
did
itself
perhaps
these
discover
just
pm
they
dl
keep
precede
this
do
kept
predominantly
thorough
does
kg
present
those
done
km
presently
though
due
last
previously
through
Source: National Library of Medicine.
178
II. State of the Art
TABLE 5.15. A Simple Stemming Algorithm 1. If word ends in “ies” but not “eies” or “aies” then replace “ies” with “y” 2. If word ends in “es” but not “aes” or “ees” or “oes” then replace “es” with “e” 3. If word ends in “s” but not “us” or “ss” then delete “s” Source: Harman (1991).
tis A. In general, however, the elimination of stop words is beneficial not only for term discrimination purposes, but also for making indexing and retrieval more computationally efficient. For example, their removal leads to the reduction in size of the inverted disk files that store indexing information, since stop words tend to have a large number of postings and thus consume disk space. Eliminating these words also allows faster query processing, since stop words tend to occur in many documents, adding to the computational requirement of building and ranking retrieval sets. In the next step, words not on the stop list undergo stemming to reduce them to their root form. The purpose of stemming is to ensure words with plurals and common suffixes (e.g., -ed, -ing, -er, -al) are always indexed by their stem form (Frakes, 1992). The benefit of stemming, however, is less clear (Harman, 1991). Not only are actual experimental results mixed, but simple algorithmic rules for stemming can be shown to lead to erroneous results (e.g., stemming aids to aid). Stemming does, however, tend to reduce the size of indexing files and also leads to more efficient query processing. Stemming will be discussed more thoroughly in Chapter 8, but a simple stemming algorithm to remove plurals is listed in Table 5.15. The final step is to assign weights to document terms based on discrimination ability. A commonly used measure that typically achieves good results is TF*IDF weighting, which combines the inverse document frequency (IDF) and term frequency (TF). The IDF is the logarithm of the ratio of the total number of documents to the number of documents in which the term occurs. It is assigned once for each term in the database, and it correlates inversely with the frequency of the term in the entire database. The usual formula used is: number of documents in database IDF(term) log 1 number of documents with term
(2)
The TF is a measure of the frequency with which a term occurs in a given document and is assigned to each term in each document, with the usual formula: TF(term,document) frequency of term in document
(3)
In TF*IDF weighting, the two terms are combined to form the indexing weight, WEIGHT: WEIGHT(term,document) TF(term,document) IDF(term)
(4)
With this weighting approach, the highest weight is accorded to terms that occur frequently in a document but infrequently elsewhere, which corresponds to
5. Indexing
179
Salton’s notion of discrimination value. Appendix 2 contains the inverted file generated by indexing the documents in Appendix 1. Stop word filtering based on the MEDLARS list (and, an, by, from, of, the, with) and stemming based on the algorithm in Table 5.15 have been used.
5.5.4 Link-Based Indexing Another automated indexing approach generating increased interest is the use of link-based methods, fueled no doubt by the success of the Google (www.google. com) search engine. These methods have a lineage back to bibliometrics, introduced in Chapter 2, where the notion of citation gives some idea of the quality of a publication. Extended to the Web, the “Google approach” gives weight to pages based on how often they are cited by other pages. The full description of the Google retrieval engine will be presented in the next chapter, but the PageRank (PR) calculation that values pages based on linkages will be presented here. In a simple description, PR can be viewed as giving more weight to a Web page based on the number of other pages that link to it (www.google.com/technology/index.html). Thus, the home page of the NLM or JAMA is likely to have a very high PR, whereas a more obscure page will have a lower PR. The PR algorithm was developed by Brin and Page (1998). To calculate it for a given page A, it is assumed that there is a series of pages T1, . . . , Tn having links to A. There is another function C(A) that is the count of the number of links going out of page A. There is also a “damping factor” d that is set between 0 and 1, by default at 0.85. Then PR is calculated for A as:
PR(T1) PR(Tn) PR(A) (1 d) d ... C(T1) C(Tn)
(5)
When implemented efficiently on a moderately powered workstation, PR can be calculated for a large collection of Web pages.
5.5.5 Web Crawling The Web presents additional challenges for indexing. Unlike most fixed resources such as MEDLINE, online textbooks, or image collections, the Web has no central catalog that tracks all its pages and other content. The fluid and dynamic nature of the Web makes identifying pages to be indexed a challenge for search engines. While some Web site developers submit their site URLs, the search engines themselves must still identify all the pages in the sites. The usual approach to finding Web pages for indexing is to use “crawling” or “spidering.” Essentially, the search engine finds a page, indexes all the words on the page, and then follows all links to additional pages. The process is repeated for all pages not already indexed. This process is important to search engines, as they compete with each other to be able to boast the largest size. It was estimated in 2001 that there were over 2 billion pages on the public Web, with the largest search engine (Google) having found just over half (Glossbrenner and Gloss-
180
II. State of the Art
brenner, 2001). The use of Web crawling is not limited to automated indexing. Hersh et al. (1999) have used Web crawling within high-quality sites to identify individual pages for indexing in CliniWeb.
5.6 Index Imaging As noted in Chapter 4, a growing amount of online content consists of images. To be retrievable, images must be indexed. Indexing images involves processing either the image itself or some surrogate, such as a textual description of it, to identify the content. Analyzing the image itself for indexable features is a process that has been investigated for many years without generalizable success (Yasnoff and Bacus, 1984; Robinson, Tagare et al., 1996; Chu, Johnson et al., 2000). More success has been achieved with indexing the text associated with the image. What is the optimal approach to indexing images? This area is relatively new, so there are no studies that identify the needs of users or indicate how likely it is that one will retrieve images from collections. Most approaches use either the text already associated with the image (such as the clinical report) or other descriptions provided. Greenes et al. (1992) note that most clinical observations are represented along a findings–diagnosis continuum, where they may be expressed differently based on how much diagnostic interpretation the clinician is adding. For example, an abnormality in a chest X-ray may be described as an increased density by one interpreter and a nodule by another. The latter contains more interpretation of the finding than the former. Thus the authors advocate a semantic network-based approach to representing findings that makes the differences along the findings–diagnosis continuum explicit. In addition to the expected slots for anatomic site, procedure type, evaluation technique, and organism observation, there are slots for the following: 1. Elemental findings—the simplest description of a finding (e.g., increased density) 2. Composite findings—a description with some deductive information added (e.g., nodule) 3. Etiological diagnosis—a diagnosis based on inference from the finding, such as the apical infiltrations seen on a classic chest X-ray in tuberculosis 4. Inference procedure—the procedure used to infer composite findings or etiological diagnoses from elemental findings How do large-scale image collections represent their images? The Slice of Life videodisc (University of Utah, Salt Lake City) contains an index with free text fields for the image title, comments, author, image type, stain, magnification, orientation, and X-ray type (Stensaas, 1994). Most of these fields contain free text that can be searched by a retrieval application. The Visible Human project has
5. Indexing
181
created a database of metadata for its image called Anatline (anatline.nlm. nih.gov). Each image is associated with several data elements including the type of image (e.g., digital color, CT, MRI) and the anatomical location from the UMLS Metathesaurus.
5.7 Data Structures for Efficient Retrieval As mentioned at the beginning of the chapter, the second purpose of indexing is to build structures so that computer programs can rapidly ascertain which documents use which indexing terms. Whether indexing is by terms in a thesaurus or words, IR systems are feasible only if they can rapidly process a user’s query. A timely sequential search over an indexed text database is infeasible if not impossible for any large document collection. A computer index usually consists of a group of inverted files, where the terms are “inverted” to point to all the documents in which they occur. The algorithms for building and maintaining these structures are described well by Frakes and Baeza-Yates (1992). Appendices 2 and 4 contain inverted files based on the sample database of Appendix 1. Appendix 2 lists all the words that occur in the data-
Term . . BETA BLOCKER BLOOD . .
#Docs . . 4 4 7 . .
Ptr
Document 5 6 7 8 5 6 7 8 1 2 3 5 7 9 10
#Words 1 1 1 1 2 2 1 1 1 1 1 1 2 1 1
Ptr
Pos 27 16 45 22 28 31 17 31 46 32 10 7 16 5 35 52 19 26
FIGURE 5.6. Part of the inverted file group for documents from Appendix 1, based on the indexing in Appendix 2, for the indexing terms BETA, BLOCKER, and BLOOD. The dictionary file contains the indexing terms, the number of documents in which they occur, and a pointer to the list of documents containing each term in the postings file. The postings file contains the document number for each indexing term, the number of words in the document, and a pointer to the list of word positions in the position file. The position file contains the word positions for the document. The pointers represent file addresses on the disk and are given by arrows rather than numbers to enhance readability
182
II. State of the Art
base (after passing through a stop list and stemming filter, the latter to be discussed in Chapter 8), while Appendix 4 contains a list of MeSH terms assigned to the documents. The inverted file group for Appendix 2 as it would be stored on a computer disk is shown in Figure 5.6. The first file is the dictionary file, which contains each indexing term along with a number representing how many documents contain the term and a pointer to the postings file. The postings file consists of a sequential list of all the documents that contain the indexing term. If it is desired to keep positional information for the indexing term (to allow proximity searching), then the postings file will also contain a pointer to the position file, which sequentially lists the positions of each indexing term in the document. The structure of the position file depends on what positional information is actually kept. The simplest position file contains just the word position within the document, while more complex files may contain the not only the word number, but also the sentence and paragraph number within the document. Position files may seem inappropriate for MeSH terms, whose position in the field is meaningless. Recall, however, that some systems use the hybrid method of word indexing on MeSH terms, which allows proximity searching in the MeSH fixed. The final component of inverted files is a mechanism for rapid lookup of terms in the dictionary file. This is typically done with a B-tree, which is a disk-based method for minimizing the number of disk accesses required to find a term in an index, resulting in fast lookup. The B-tree is very commonly used for keys in a DBMS. Another method for fast term lookup is hashing (Wartik, Fox et al., 1992).
Chapter 6 Retrieval
The last two chapters discussed the content and organization of textual databases. Chapter 4 covered the different types of databases available, while Chapter 5 showed how they are indexed for optimal retrieval. This chapter, which explores the interaction of the IR system with the user, the person whom it is intended to serve, covers the entire retrieval process, from search formulation to system interaction to document delivery. The relationship between the IR system and its users has changed considerably over the years. In the 1960s, users had to undergo formal training at the NLM before being allowed to access the databases. Searching was done by filling out a form that had to be mailed to the NLM, with a “turnaround” time of 2 to 3 weeks for the results to be mailed back. In the 1970s, NLM databases could be directly accessed by trained searchers over time-sharing networks, though those wanting searches done still had to go through trained intermediaries, typically librarians. Because the user had to make an appointment with the intermediary and wait for him or her to do the search and return the results, the turnaround time became as low as 2 to 3 days. In the 1980s, online databases first became available to “early adopter” end users. Connecting to networks, then information providers, and then databases was still somewhat laborious. The 1990s saw the explosion of end-user searching on the Web. The ease of use provided by powerful servers and graphical user interfaces as well as the general ubiquity of the Internet made searching a mainstream task performed by millions. At the beginning of the twenty-first century, it is unclear what advances will improve searching next. This chapter begins by discussing the search process. Attention is then turned to general principles of searching, including description of exact-match and partial-match approaches. This is followed by the description of specific searching interfaces. The chapter closes with discussion of document delivery and a specific type of retrieval known as information filtering.
6.1 Search Process Chapter 2 discussed the three general reasons for consulting IR systems (Lancaster and Warner, 1993): a need for help in solving a certain problem, a need for background information, and a need to keep up with a subject. Furthermore, 183
184
II. State of the Art
for each information need there was a spectrum of possible amounts of information needed, from a single fact to a few documents to an exhaustive collection of literature. The variation in these needs results in different strategies for interacting with the IR system. Pao describes four stages a searcher might go through before actually sitting down to an IR system (Pao, 1989). In the first stage, the information problem stage, the user determines that an information deficiency exists. With the problem identified, the user formulates an information need, determining what must be known to solve the information problem. Next the user formulate a question that will motivate the actual interaction with the IR system. This question is then converted to a request, which is the search statement actually submitted to the system. Based on the results of any stage, the user may return to earlier stages and modify them. While any type of user can have any type of information need, the needs of certain groups of users are likely to differ from those of other groups. In the healthcare field, one can readily discern the different needs of clinicians and researchers (Wallingford, Humphreys et al., 1990). Clinicians, including physicians, nurses, dentists, and other allied healthcare providers, are likely to have specific needs in solving problems (Gorman and Helfand, 1995). In general, they want a search to be more precise and to include the most relevant documents to their specific need. Researchers, on the other hand, are more likely to have broader needs on a given topic. For example, someone writing a paper will want a definitive overview of the topic, whereas a researcher exploring a new topic will want a great deal of background information. Researchers are likely to be more tolerant of retrieving nonrelevant references to make sure they find all the relevant ones and in fact may benefit from the “serendipity” of off-focus retrievals (Belkin and Vickery, 1985).
6.2 General Principles of Searching Whereas each of the four stages identified by Pao (1989) is important in helping users to meet their information needs, the step of going from question to request is most important for IR system designers. After all, this is the step that will allow users to actually find documents that will meet their needs and ultimately solve their information problems. This section describes the general principles for retrieval in most currently available IR systems. It initially compares the two most common approaches to searching, exact-match (or Boolean or setbased) searching and partial-match (or natural language, ranked, or automated) searching. Then the selection of search terms and attributes, to prepare for the description of specific searching interfaces, is discussed in Section 6.3.
6.2.1 Exact-Match Searching In exact-match searching, the IR system gives the user all documents that exactly match the criteria specified in the search statement(s). Since the Boolean operators AND, OR, and NOT are usually required to create a manageable set
6. Retrieval
185
of documents, this type of searching is often called Boolean searching. Furthermore, since the user typically builds sets of documents that are manipulated with the Boolean operators, this approach is also called set-based searching. Most of the early operational IR systems in the 1950s through 1970s used the exact-match approach, even though Salton was developing the partial-match approach in research systems during that time (Salton and Lesk, 1965). In modern times, exact-match searching tends to be associated with retrieval from bibliographic databases, while the partial-match approach tends to be used with full-text searching. Typically the first step in exact-match retrieval is to select terms to build sets. Other attributes, such as the author name, publication type, or gene identifier (in the secondary source identifier field of MEDLINE), may be selected to build sets as well. Since the user typically has an information need less broad than “all documents on a particular disease or treatment,” the Boolean operators are used to focus the search output on all the elements in the information need. These operators also serve to create a document set that can be realistically analyzed. Blair (1980) speaks of the “futility point” of search results, the number of documents beyond which a searcher will continue to look at the results. He speculates that the values of this point is 50 documents. Once the search term(s) and attribute(s) have been selected, they are combined with the Boolean operators. The use of Boolean operators derives from Boolean algebra, which is based on set theory, the branch of mathematics dealing with sets and their algebraic manipulation. In set theory, a set is defined as a collection of elements. Examples of sets include all documents with the indexing term Hypertension or all students in a class. There are three common operations that are performed on sets, which correspond to the three common Boolean operators used in IR systems: intersection, union, and complement. These operations are depicted by Venn diagrams in Figure 6.1. The intersection of two sets is the set that contains only the elements that are common to both sets. This is equivalent to the Boolean AND operator. This operator is typically used to narrow a retrieval set to contain only documents about two or more concepts. For example, if one desired documents on the use of the drug propanolol in the disease Hypertension, a typical search statement might be propanolol AND Hypertension. Using the AND would most likely eliminate articles, for example, on the use of propanolol in migraine headaches and the diagnosis of hypertension. The union of two sets is the set that contains all the elements that occur in either set, equivalent to the Boolean OR operator. This operator is usually used when there is more than one way to express a concept. For example, the name of the virus that causes AIDS has carried a number of names and acronyms over the years. When simultaneously discovered by French and American scientists, it was assigned the names Lymphadenopathy Associated Virus (LAV) and Human T-Cell Leukemia Virus 3 (HTLV-3), respectively. It was later renamed Human Immunodeficiency Virus (HIV). Likewise, one of the initial treatments for AIDS was originally called azidothymidine or AZT but is now called zidovudine.
186
II. State of the Art
FIGURE 6.1. The Boolean operators AND, OR, and NOT.
The complement of a set is the set with all the elements of some universal set that are not contained in the complemented set, equivalent to the Boolean NOT operator. However, for practical reasons, most IR systems use the NOT operator as a complement. In a large database such as MEDLINE, with its many millions of documents, the set of all documents that do not, for example, contain the term Hypertension, could be very large. As a result, most IR systems use
6. Retrieval
187
TABLE 6.1. Five-Document Five-Term Database Documents Terms
1
2
3
4
5
Documents with term
IDF
A B C D E
1 0 1 0 1
0 2 0 1 0
0 0 0 1 0
3 0 2 0 0
1 1 3 0 0
3 2 3 2 1
1.22 1.40 1.22 1.40 1.70
NOT as a subtraction operator that must be applied to another set. Some systems more accurately call this the ANDNOT operator. Boolean operations can be demonstrated by looking at Table 6.1, which shows a small five-document, five-term database. The query A AND B will retrieve only document 5, since that is the only term common to both documents. The query A OR B, on the other hand, will retrieve documents 1, 2, 4, and 5, since one of the query terms is present in these four documents. The query A NOT B will retrieve documents 1 and 4, since they contain term A but not B.
6.2.2 Partial-Match Searching Although partial-match searching was conceptualized very early, it did not see widespread use in IR systems until the advent of Web search engines in the 1990s. This is most likely because exact-match searching tends to be preferred by “power users,” whereas partial-match searching is preferred by novice searchers. Whereas exact-match searching requires an understanding of Boolean operators and (often) the underlying structure of databases (e.g., the many fields in MEDLINE), partial-match searching allows a user to simply enter a few terms and start retrieving documents. Despite the surface simplicity of partial-match searching, however, its effectiveness can be comparable to that of exact-match searching (see Chapter 7), and new research approaches using it (see Chapter 8) can be quite complex. The development of partial-match searching is usually attributed to Salton (1991), who pioneered the approach in the 1960s. Although partial-match searching does not exclude the use of nonterm attributes of documents, and for that matter does not even exclude the use of Boolean operators (e.g., see Salton, Fox et al., 1983), the most common use of this type of searching is with a query of a small number of words, also known as a natural language query. Because Salton’s approach was based on vector mathematics, it is also referred to as the vector–space model of IR. In the partial-match approach, documents are typically ranked by their closeness of fit to the query. That is, documents containing more query terms will likely be ranked higher, since those with more query terms will in general be more likely to be relevant to the user. As a result this process is called relevance ranking. The entire approach has also been called statistical retrieval, word–statistical retrieval, and ranked retrieval.
188
II. State of the Art
The most common approach to document ranking in partial-match searching is to give each a score based on the sum of the weights of terms common to the document and query. Terms in documents typically derive their weight from the TF*IDF calculation described in Chapter 5. Terms in queries are typically given a weight of one if the term is present and zero if it is absent. The following formula can then be used to calculate the document weight across all query terms: Document weight
Weight of term in query
all query terms
Weight of term in document (1)
This may be thought of as a giant OR of all query terms, with sorting of the matching documents by weight. The usual approach is for the system to then perform the same stop word removal and stemming of the query that was done in the indexing process. (The equivalent stemming operations must be performed on documents and queries so that complementary word stems will match.) The sample database in Table 6.1 can demonstrate partial-match searching. To simplify calculations, Table 6.1 shows the IDF for each term and Table 6.2 shows the TF*IDF weighting for each term in each document. A query using the terms “A B C” will retrieve four documents with the following ranking: 1. 2. 3. 4.
Document Document Document Document
5 4 2 1
(1.22 1.40 3.67 6.29) (3.67 2.44 6.11) (2.80) (1.22 1.22 2.44)
In general, documents that contain more of the query terms will be ranked higher. As seen in the sample query, document 5 contains three terms in the query while document 4 contains two. However, document 2 only has one query term while document 1 has two. But it achieves its higher ranking because the term present in document 2 has a higher weight than the two terms in document 1 combined. This results often occurs when a term has a higher IDF, which is consistent with the rationale of IDF in that terms which occur with less frequency across the entire database are more “discriminating” than those which are more common. One problem with TF*IDF weighting is that longer documents accumulate more weight in queries simply because they have more words. As such, some
TABLE 6.2. TF*IDF Weighting for the Terms and Documents in Table 6.1
1
2
3
4
5
Relevance feedback query
1.22 0.00 1.22 0.00 1.70
0.00 2.80 0.00 1.40 0.00
0.00 0.00 0.00 1.40 0.00
3.67 0.00 2.44 0.00 0.00
1.22 1.40 3.67 0.00 0.00
1.61 1.70 0.22 0.70 0.00
Documents Terms A B C D E
6. Retrieval
189
approaches “normalize” the weight of a document. The most common approach is cosine normalization: Document weight
Weight of term in query Weight of term in document all query terms
Weight of term in query Weight of term in document all query terms all query terms 2
2
(2) A variety of other variations to the basic partial-matching retrieval approach have been developed. All are considered to be research approaches and are covered in more detail in Chapter 8. Some apply different weighting than the simple TF*IDF, while others add more weight to certain parts of documents, such as the title or the anchor text in a Web link. Some apply different approaches to normalization. Relevance feedback, a feature allowed by the partial-match approach, permits new documents to be added to the output based on their similarity to those deemed relevant by the user. This approach also allows reweighting of relevant documents already retrieved to higher positions on the output list. The most common approach is the modified Rocchio equation employed by Buckley et al. (1994a). In this equation, each term in the query is reweighted by adding value for the term occurring in relevant documents and subtracting value for the term occurring in nonrelevant documents. There are three parameters, , , and , which add relative value to the original weight, the added weight from relevant documents, and the subtracted weight from nonrelevant documents, respectively. In this approach, the query is usually expanded by adding a specified number of query terms (from none to several thousand) from relevant documents to the query. Each query term takes on a new value based on the following formula: New query weight ( Original query weight) 1 number of relevant documents
weight in document all relevant
(3)
documents
1 number of nonrelevant documents
weight in document
all nonrelevant documents
When the parameters, , , and are set to one, this formula simplifies to: New query weight Original query weight Average term weight in relevant documents Average term weight in nonrelevant documents
(4)
In its rightmost column, Table 6.2 shows the new weight for each query term when documents 2 and 4 are relevant, documents 1 and 5 are not relevant, and term D is added to the query (since it occurs in document 2, which is relevant.
190
II. State of the Art
TABLE 6.3. Weighting of Terms in Documents After Rocchio Relevance Feedback Process Documents Terms
1
2
3
4
5
A B C D E
1.97 0.00 0.27 0.00 0.00
0.00 4.75 0.00 0.98 0.00
0.00 0.00 0.00 0.98 0.00
5.90 0.00 0.54 0.00 0.00
1.97 2.38 0.81 0.00 0.00
As can be seen, none of the query terms maintains its original weight of 1.0. The weights of terms in query are now: • A 1 (3.67/2) (2.44/2) 1.61 • B 1 (2.8/2) (1.4/2) 1.70 • C 1 (2.44/2) (4.89/2) 0.22 • D 0 (1.7/2) 0 0.70 • E 0 (no relevant docs so none added) By applying these new query weights to equation 1 above, the weight of terms in document can be recalculated, as shown in Table 6.3. These weights can then be used with equation 1 above to re-weight the documents after the relevance feedback process: 1. 2. 3. 4. 5.
Document Document Document Document Document
2 4.75 0.98 7.72 4 5.90 0.54 5.36 5 1.97 2.38 0.81 3.53 1 1.97 0.27 1.70 3 0.98
As can be seen, document 2 has moved to the top of the list, while document 5 has fallen and document 3 has been added. These phenomena typically occur with relevance feedback, in general (but not always) achieving the goal of moving existing relevant documents higher in the output list, adding new relevant ones, and moving existing nonrelevant documents lower. A number of IR systems offer a variant of relevance feedback that finds similar documents to a specified one. PubMed allows the user to obtain “related articles” from any given one in an approach similar to relevance feedback but which uses a different algorithm, which will be described in Chapter 8 (Wilbur and Yang, 1996). A number of Web search engines allow users to similarly obtain related articles from a specified Web page.
6.2.3 Term Selection As noted in Chapter 5, search terms are generally either terms from a controlled vocabulary assigned by a manual indexing process or words extracted from the documents themselves by means of an automated process. While either type of term can be used in either type of searching approach, controlled vocabulary
6. Retrieval
191
terms are most commonly employed in exact-match systems. Typically when appropriate controlled terms are not available, document words are applied. Partial-match systems almost always use document words as indexing terms, though they may be qualified by where they occur, such as in the title or anchor text. 6.2.3.1 Term Lookup Whereas early IR systems required users to enter search terms with minimal assistance (i.e., terms had to be found in telephone-book-sized catalogs and typed in exactly), most modern systems provide assistance with term lookup. PubMed (pubmed.gov), for example, provides a MeSH browser that allows users to type in one or more words from a term, select the appropriate term, and add it to the search. In Figure 6.2, for example, the user entered angiotensin converting and received a list of suggested MeSH terms. The Ovid system (www.ovid.com) also allows user lookup of MeSH terms but uses a somewhat different approach. This system displays a ranked list of MeSH terms that most commonly appear in MEDLINE references when the word(s) entered also occurs. For example, typing in troglitazone, a drug that is not in MeSH, will lead to suggestion of the terms Thiazoles, which is the drug category to which troglitazone belongs, and Diabetes Mellitus, Non-InsulinDependent, which is a disease treated by the drug.
FIGURE 6.2. PubMed MeSH Browser. (Courtesy of the National Library of Medicine.)
192
II. State of the Art
6.2.3.2 Term Expansion Some systems allow terms in searches to be expanded by using the wild-card character, which adds all words to the search that begin with the letters up until the wildcard character. This approach is also called truncation. Unfortunately there is no standard approach to using wild-card characters, so the syntax for them varies from system to system. PubMed, for example, allows a single asterisk at the end of a word to signify a wild-card character. Thus the query word can* will lead to the words cancer and Candida, among others, being added to the search. Ovid allows the use of four different wild-card characters. In its basic mode interface, the asterisk character can be used, and it functions similarly to the asterisk in PubMed. In the advanced mode, however, there are three different wild-card characters: • Limited truncation allows all terms with the root and the specified number of digits to be added to the query. It is specified by the dollar sign followed by a digit. For example, dog$1 retrieves documents containing dog and dogs but not dogma. • Mandated wild card allows single characters to be substituted within or at the end of a word. It is specified by the pound sign (#). For example, wom#n will lead to retrieval of documents containing woman or women. There must be a character present; that is, dog# will not retrieve dog. • Optional wild card allows zero or more characters to be substituted within or at the end of a word. It is specified by the question mark. For example, colo?r will lead to retrieval of documents containing color or colour. This character cannot be used after the first letter of a word. The AltaVista search engine (www.altavista.com) takes a different approach still. The asterisk can be used as a wild-card character within or at the end of a word but only after its first three letters. For example, col*r will retrieve documents containing color, colour, and colder. 6.2.3.3 Other Word-Related Operators Some IR systems have operators that require more than just the term being present. The proximity operator, for example, specifies not only that two words be present but also that they occur within a certain distance of each other in the document. It is essentially an AND with the additional restriction that the words be within a specified proximity. These operators can help control for the context of words in a concept. For example, a user looking for documents on colon cancer might specify that the words colon and cancer appear within five words of each other. This would capture documents with phrases like colon cancer and cancer of the colon but would avoid documents that discuss cancer in one part and mention colon in another (e.g., a description of the effect of a type of cancer that somehow affects colon motility). As with other features, different systems implement and have varying syntax for proximity operators. PubMed does not provide proximity searching although it does, as will be described shortly, allow searching against a list of common multiword phrases, such as health planning. Ovid, on the other hand, does provide this
6. Retrieval
193
capability. Its ADJ operator requires words to be adjacent and in the order specified. For example, the query blood ADJ pressure will retrieve only documents that have the phrase blood pressure. The operator ADJn, where n is a digit, requires words to be within n characters of each other. For example, colon ADJ5 cancer will retrieve documents with both colon cancer and cancer of the colon. The Web search engine AltaVista has a proximity operator NEAR, which requires target words to be within 10 words of each other. Another word-related term specification feature in Ovid is the FREQ command, which will not retrieve a document unless a word or phrase is present a minimum number of times. One word-related feature no longer found in most systems is synonym specification. The now-retired STAIRS system from IBM (www.ibm.com) has a SYN operator that allows a user to designate two words as synonymous. For example, entering cancer SYN carcinoma will cause the two words to be used interchangeably (i.e., connected to the other with OR whenever either is used). 6.2.3.4 Subheadings Recall from Chapter 5 that some thesauri, including MeSH, contain subheadings, which are modifiers that can be attached to terms. For example, one can restrict the retrieval of documents on the subject of Hypertension to just those covering diagnosis or treatment. Many MEDLINE systems also allow searching with so-called floating subheadings, which allow the searcher to specify just the subheading itself, as if it were a solitary, unattached indexing term. Subheadings have the effect of increasing the precision of a search, since documents on other aspects of the search term should be eliminated from the retrieval set. They should have no effect on recall, since the documents not retrieved should be focused on other aspects of the term. However, it is possible that the indexers may not assign a subheading where one is warranted; thus the use of the subheading will preclude that document from being retrieved. Since subheading assignment is the least consistent area of indexing term assignment (see Chapter 5), most expert searchers advise care in the use of subheadings. 6.2.3.5 Explosions Like the subheading, the explosion operation requires a controlled vocabulary. It further requires that the vocabulary have a hierarchical organization. The explosion operation is an OR of the term exploded with all the narrower terms below it in the hierarchy. It is typically used when a user wants to retrieve documents about a whole class of topics. For example, a user might want to obtain documents on all types of anemia. There are many types of anemia, due to deficiencies of nutrients (iron, vitamin B12, folate), disease processes (hemolytic anemia, anemia of chronic disease), and genetic disorders (sickle cell anemia and other hemoglobinopathies). If a searcher wanted to obtain information on all those anemias in the MEDLINE database, the general MeSH term Anemia, would be exploded, which would have the effect of combining the general and specific terms together with the OR operator. Another common reason for using explosions is to search on a category of drugs. For example, most drugs in the ACE In-
194
II. State of the Art
hibitors category (e.g., captopril, enalapril, lisinopril) function similarly, so the user interested in the category would want to explore the general MeSH term ACE Inhibitors. Most MEDLINE systems, including PubMed, automatically explode all terms that are not terminal terms or leaf nodes in the hierarchy (i.e., do not have a more specific term below them). While this approach is in general effective (especially for the examples just given), it can be detrimental when the MeSH hierarchy is not a true “is-a” or “part-of” hierarchy—that is, if the more general term is not a generalization of the more specific terms. This occurs with the MeSH term Hypertension, which is autoexploded. The terms underneath it in the hierarchy represent elevated blood pressure in specific body locations, such as the portal vein of the liver (Portal Hypertension) or the arteries of the lung (Pulmonary Hypertension). These terms are included in the autoexplosion even though most searches on essential hypertension (designated by the MesH term Hypertension) would not want to search on them. As with subheadings, explosions require quality, consistent indexing. One of the principles taught to MEDLINE indexers to make explosions work well is that documents should be indexed to the “deepest” level of a hierarchy. Thus if a document deals with a specific type of anemia, such as iron-deficiency anemia, it will be indexed on that specific term. This allows searchers to home in on documents specifically about the disease with the MeSH term Iron-deficiency anemia, or still capture it more broadly by exploding the MeSH term, Anemia. Explosions have the effect of increasing recall, since additional documents indexed on related but more specific terms are brought into the retrieval set. The effect on precision is variable, since the exploded topics may be relevant to the more general search term, or they may be too specific for the needs of the searcher.
6.2.4 Other Attribute Selection As noted already, indexing not only consists of terms, but also entails other attributes about documents against which the user might desire to search. In MEDLINE, for example, the user might wish to search for papers by a specific author, in a specific journal, or representing a certain type of publication, such as a review article or a randomized controlled trial. On the Web, one might wish to search against only certain parts of Web pages, such as the text in a title or the anchor text in links. One might also wish to restrict Web searches to specific domains or hosts. The syntax for specifying these attributes is explained in the discussion of the different searching interfaces in the next section.
6.3 Searching Interfaces In describing interfaces to various IR systems, the goal is not to exhaustively cover all features of all systems, but rather to demonstrate how features are implemented within various systems. Precedence is given to displaying systems of
6. Retrieval
195
historical or market importance. The organization of this section follows the classification of content that was used described in Chapter 4.
6.3.1 Bibliographic As noted in Chapter 4, bibliographic content includes literature reference databases, Web catalogs, and specialized registries. 6.3.1.1 Literature Reference Databases Literature reference databases, or bibliographic databases, continue to be the flagship IR application in health care. MEDLINE, in particular, is the most common IR database used by physicians, according to a recent survey by the AMA (Anonymous, 2001a). MEDLINE is most commonly accessed through the PubMed system at the NLM. While not the original or most feature-laden means to access MEDLINE, PubMed is important because it is maintained by the producer of MEDLINE, the NLM, and is free to anyone in the world. Until the mid-1990s, MEDLINE was most commonly accessed using a command-line interface on a system called ELHILL over time-sharing networks accessed either by a dedicated line or telephone modem connection. One early innovative system was PaperChase, which was the first MEDLINE system geared to clinicians (Horowitz, Jackson et al., 1983). It provided features taken for granted in modern systems, such as input of word fragments (instead of requiring the whole MeSH term), assistance with MeSH term look up, intuitive description of the Boolean operators, and ability to handle American and British spelling variants. Another innovative approach to MEDLINE came from the NLM in the 1980s. Grateful Med was the first MEDLINE system to run on a personal computer and feature a “full-screen” interface that provided text boxes for specific fields (such as author, title, and subject), check boxes for tags such as English language and review articles, and expression of subject terms both as MeSH terms and text words (Haynes and McKibbon, 1987). One particularly valuable feature of Grateful Med was the ability to compose a search offline and then connect online only to log in and obtain results of the search. During an era of costly online charges, this enabled users to save money, unless (as happened not too infrequently) they needed to reconnect to run a modified version of their search. Grateful Med eventually moved to the Web in Internet Grateful Med, sporting an expert assistant called COACH that helped users diagnose problematic searches (Kingsland, Syed et al., 1992), but was discontinued by NLM in favor of PubMed and the NLM Gateway. The initial searching screen for PubMed is shown in Figure 6.3. Although presenting the user with a simple text box, PubMed does a great deal of processing of the user’s input to identify MeSH terms, author names, common phrases, and journal names (Anonymous, 2001i). In this automatic term mapping, the following steps are taken after removal of stop words. 1. MeSH translation: PubMed first tries to map input text to MeSH headings, MeSH entry terms, subheadings, and terms from Supplementary Concept (Sub-
196
II. State of the Art
FIGURE 6.3. PubMed search window. (Courtesy of the National Library of Medicine.)
stance) Names. When a term is found, it is searched as both the MeSH term and its text words. For example, the query colon cancer treatment with radiation is translated to the search (((“colonic neoplasms” [MeSH Terms] OR colon cancer [Text Word]) AND ((“therapy” [Subheading] OR “therapeutics” [MeSH Terms]) OR treatment [Text Word])) AND (“radiation” [MeSH Terms] OR radiation [Text Word])). 2. Journal name translation: For all remaining words that do not map to MeSH (which could be all words if no MeSH terms are found), an attempt is made to map them to the journal abbreviation used in PubMed. For example, the string ann intern med maps to the journal Annals of Internal Medicine. 3. Phrase mapping: All remaining words are next compared against a list of common phrases. 4. Author mapping: If the remaining words contain at least two words, one of which has only one or two letters, word pairs are matched against the MEDLINE author field. For example, the string hersh wr maps to an author search by this author. Remaining text that PubMed cannot map is searched as text words (i.e., words that occur in any of the MEDLINE fields). For example, the word string does not occur in any MeSH term and will be searched as a text word across the en-
6. Retrieval
197
tire MEDLINE record. As noted already, MEDLINE allows the use of wild-card characters. It also allows phrase searching in that two or more words can be enclosed in quotation marks to indicate they must occur adjacent to each other. If the specified phrase is in PubMed’s phrase index, then it will be searched as a phrase. Otherwise the individual words will be searched. Both wild-card characters and specification of phrases turn off the automatic term mapping. PubMed allows specification of other indexing attributes by two means. First, the user may type the attribute and its value directly into the text box. (This can also be done to directly enter MeSH terms.) For example, a search of asthma/therapy [mh] AND review [pt] will find review articles indexed on the MeSH term Asthma and its subheading Therapy. The use of field tags turns off automatic term mapping. Many attributes can also be specified via the PubMed “Limits” screen (see Figure 6.4). These include publication types, subsets, age ranges, and publication date ranges. As in most bibliographic systems, users search PubMed by building search sets and then combining them with Boolean operators to tailor the search. Consider a user searching for studies assessing the reduction of mortality in patients with congestive heart failure (CHF) through the use of medications from the angiotensin-converting (ACE) inhibitors class of drugs. A simple approach to such a search might be to combine the terms ACE
FIGURE 6.4. PubMed limits screen. (Courtesy of the National Library of Medicine.)
198
II. State of the Art
Inhibitors and CHF with an AND. The easiest way to do this is to enter the search string ace inhibitors AND CHF. (The operator AND must be capitalized because PubMed treats the lowercase and as a text word, since some MeSH terms, such as Bites and Strings, have the word and in them.) A more advanced searcher might put ACE Inhibitors and CHF into separate sets for later manipulation in the searching process. Figure 6.5 shows the PubMed History screen such a searcher might develop. This searcher has seen, for example, that there are 3475 references about ACE inhibitors and CHF. If he or she is knowledgeable about evidence-based medicine (EBM), the next step is to try to limit the search to the best evidence. Since this question is about treatment of disease, it would be logical to use the Publication Type tag to limit retrieval to RCTs. This still yields 501 references. A knowledge of EBM, however, would lead one to believe that if over 500 clinical trials have been done on this topic, someone has probably done a meta-analysis. Employing that Publication Type tag reduces the output to 20 references, which is very manageable. The user may also be interested in practice guidelines; this Publication Type tag can be used in a limit as well. PubMed also provides a Preview/Index screen that lets the user build search sets directly without having to view intermediate results. PubMed has another approach to finding best evidence that is simpler though
FIGURE 6.5. PubMed history screen. (Courtesy of the National Library of Medicine.)
6. Retrieval
199
not as flexible. PubMed allows the user to enter “clinical queries,” where the subject terms are limited by search statements designed to retrieve the best evidence based on principles of EBM. There are two different approaches. The first uses strategies for retrieving the best evidence for the four major types of clinical question. These strategies arise from research assessing the ability of MEDLINE search statements to identify the best studies for therapy, diagnosis, harm, and prognosis (Haynes, Wilczynski et al., 1994). For each of the four question types, there are two strategies (see Table 6.4). The strategy emphasizing sensitivity (leading to higher recall) aims to include as many relevant articles as possible, while the strategy emphasizing specificity (leading to higher precision) aims to exclude as many nonrelevant articles as possible. The second approach to retrieving the best evidence aims to retrieve evidence-based resources that are syntheses and synopses, in particular meta-analyses, systematic reviews, and practice guidelines. When the clinical queries interface is used, the search statement is processed by the usual automatic term mapping and the resulting output is limited (via AND) with the appropriate statement (see Figure 6.6). Another feature allowed by PubMed is the ability to embed search statements in the uniform resource locator (URL), enabling other applications or Web pages to send queries to it (Anonymous, 2001e). This feature is used by both CliniWeb TABLE 6.4. PubMed Clinical Queries (sensitivity and specificity values from (Haynes, Wilczynski et al., 1994)) Category
Optimized for
Sensitivity/ specificity (%/%)
Therapy
Sensitivity
99/74
Specificity
57/97
Sensitivity
92/73
Specificity
55/98
Sensitivity
82/70
Specificity
40/98
Sensitivity
92/73
Specificity
49/97
Diagnosis
Etiology
Prognosis
Source: National Library of Medicine.
PubMed search statement randomized controlled trial [PTYP] OR drug therapy [SH] OR therapeutic use [SH:NOEXP] OR random* [WORD] (double [WORD] AND blind* [WORD]) OR placebo [WORD] sensitivity and specificity [MESH] OR sensitivity [WORD] OR diagnosis [SH] OR diagnostic use [SH] OR specificity [WORD] sensitivity and specificity [MESH] OR (predictive [WORD] AND value* [WORD]) cohort studies [MESH] OR risk [MESH] OR (odds [WORD] AND ratio* [WORD]) OR (relative [WORD] AND risk [WORD]) OR case control* [WORD] OR case-control studies [MESH] case-control studies [MH:NOEXP] OR cohort studies [MH:NOEXP] incidence [MESH] OR mortality [MESH] OR follow-up studies [MESH] OR mortality [SH] OR prognos* [WORD] OR predict* [WORD] OR course [WORD] prognosis [MH:NOEXP] OR survival analysis [MH:NOEXP]
200
II. State of the Art
FIGURE 6.6. PubMed clinical queries screen. (Courtesy of the National Library of Medicine.)
(Hersh, Ball et al., 1999) and MEDLINEplus (N. Miller, Lacroix et al., 2000). A comprehensive set of commands is available, including the ability to do a general search, use the linkages in PubMed to link out from specific items, and retrieve just the text of results (not the entire Web page). PubMed has additional features available to the user once a MEDLINE record has been selected for viewing. As described earlier, a “related articles” command finds references similar to the current one by means of a relevance feedback process. For some journals, the user can also click on a button to be linked to the journal’s Web site where the full text is available. The full text will appear if the article can be accessed without charge; if a subscription is required, the Web site can determine whether the user is eligible to view the article. From the MEDLINE record page, the user can also click on LinkOut, which will provide a page of links to other publishers that offer the article as well as general consumer information about the article’s topic from MEDLINEplus. PubMed is, of course, not the only interface for searching MEDLINE. Two well-known commercial interfaces used at many medical centers are produced by Ovid and Aries Systems (Knowledge Finder, www.kfinder.com). The most notable features of Ovid are its different approach to mapping to MeSH, the availability of virtually all features via its “Limits” interface, and the direct linkage
6. Retrieval
201
to full text of the articles also licensed by Ovid. Particularly useful is the direct linkage from MEDLINE references to the structured, expanded abstracts (when available) in Best Evidence as well as reviews from the Cochrane Database of Systematic Reviews (CDSR) when the reference is cited by the review. Knowledge Finder is the only major MEDLINE interface to offer a partial-match approach to retrieval (though many exact-match features are available as well). It also provides linkages to EBM resources along with automated synonym mapping (e.g., Advil to Ibuprofen) and British/American spelling equivalents. There are other ways to access MEDLINE without charge on the Web. WebMEDLINE (www.medweaver.com/webmedline/new.html) provides a simpler user interface to PubMed. Searches created by using its interface are sent to PubMed, with results redisplayed in another simple view. Another site that puts a different interface in front of PubMed is Infotrieve (www.infotrieve.com). BioMedNet (www.biomednet.com), owned by Reed-Elsevier, provides free MEDLINE to those who register on the site. Access to full-text and other resources linked from MEDLINE is provided at a cost. Medscape (www.medscape.com) provides access to MEDLINE through a licensed version of the Knowledge Finder system. Studies in the past have shown considerable variation in the search results of different systems searching the same underlying MEDLINE database (Haynes, McKibbon et al., 1985; Haynes, Walker et al., 1994). There is no reason to believe the same situation does not still exist. 6.3.1.2 Web Catalogs Web catalogs take a variety of approaches to searching. A number of them allow searching directly over their catalog and then provide links to the items referenced. Medical Matrix, for example, has a title, resource type, and description of all resources in its catalog. Users can search over the catalog and click on links to sites that are retrieved (Figure 6.7). An advanced searching interface allows searching by resource type (e.g., full text, images, practice guidelines, etc.). HealthFinder and HealthWeb offer similar approaches, with a simple search interface that allows retrieval over titles and resource descriptions in the catalogs. All these systems also allow browsing of their subject classifications. The Open Directory project allows searching over the category names in its vast classification hierarchy. As seen in Figure 6.8, topics might appear in a number of hierarchies, each with a different number of Web sites. CliniWeb allows searching to find MeSH terms that link the user into the MeSH hierarchy by which the pages of the catalog are organized. All these Web catalogs allow browsing through their subject hierarchies. (See Section 4.2.2 for a description of how this is done in CliniWeb.) 6.3.1.3 Specialized Registries The National Guidelines Clearinghouse (NGC) offers a simple text box searching interface but also provides a “detailed search” interface that allows all the attributes in the clearinghouse to be searched. These include the type of guideline,
202
II. State of the Art
FIGURE 6.7. Medical Matrix advanced search window. (Courtesy of Medical Matrix.)
the type of organization that produces such guides, the intended users, and the clinical specialty (see Figure 6.9), as well as limits on the content, target population, gender, and date of publication. The “results” interface allows not only viewing of the attributes for one guideline but also side-by-side comparison of the attributes of two or more guidelines. When the full text of the guideline is available on the Web, a link is provided directly to it. The NGC site also provides an increasing number of summaries of guidelines on selected topics (www.guideline.gov/FRAMESETS/guideline_fs.asp?viewsynthesis).
6.3.2 Full Text In general, full-text searching interfaces offer fewer features than their bibliographic counterparts. This is in part because the amount of metadata available for full-text resources is smaller. Unlike the rich metadata provided in bibliographic databases such as MEDLINE and the NGC, most full-text “documents” consist of just a title and body of text. One advantage of full text is its complete body of text, which provides more words to search against; but when those words do not represent the focus of the content of the document, this feature can be a disadvantage.
FIGURE 6.8. Open Directory results of search on acupuncture. (Courtesy of Open Directory.)
FIGURE 6.9. National Guidelines Clearinghouse detailed search window. (Courtesy of the Agency for Healthcare Research & Quality.)
204
II. State of the Art
6.3.2.1 Periodicals As noted in Chapter 4, many journals have become available in full text on the Web. A number of them use the facilities of Highwire Press. An early highprofile journal that made its complete contents available on the Web was the British Medical Journal (BMJ), which continues to be available without charge. The Highwire system provides a retrieval interface that searches over the complete online contents for a given journal. users can search for authors, words limited to the title and abstract, words in the entire article, and within a date range (Figure 6.10). The interface also allows searching by citation by entering volume number and page as well as searching over the entire collection of journals that use Highwire. Users can also browse through specific issues as well as collected resources. Once an article has been found, a wealth of additional features are available (Figure 6.11). First, the article is presented both in HTML and PDF form, with the latter providing a more readable and printable version. Links are also pro-
FIGURE 6.10. British Medical Journal search window. (Courtesy of British Medical Journal Publishing group.)
6. Retrieval
205
FIGURE 6.11. British Medical Journal additional features. (Courtesy of British Medical Journal Publishing group.)
vided to related articles from the journal as well as the PubMed reference and its related articles. Also linked are all articles in the journal that cited this one, and the site can be configured to set up a notification e-mail when new articles cite the item selected. Finally, the Highwire software provides for “Rapid Responses,” which are online letters to the editor. The online format allows a much larger number of responses than could be printed in the paper version of the journal. 6.3.2.2 Textbooks Most electronic textbook searching interfaces offer variations on the basic theme of entering words to search against sections of the textbook organized along the book’s organizational structure. One searching interface that offers somewhat more functionality is Stat!-Ref (www.statref.com), which contains over 30 medical textbooks, from simple pocket guides to encyclopedic subspecialty tomes (Figure 6.12). Its searching interface allows the user to select any number of textbooks. The user can also set the “precision” of the search, which allows the words
206
II. State of the Art
FIGURE 6.12. Stat!-Ref search window. (Courtesy of Teton Data Systems.)
entered to be combined with the Boolean OR or AND, or to be limited by proximity within a certain number of words (NEAR) or adjacent (ADJ). The interface also allows stemming or word synonym expansion to be optionally used. The results screen provides links that take the user right to the appropriate section of the textbook, from where he or she can navigate to other portions of the book. 6.3.2.3 Web Search Engines Entire books have been written on Web search engines (Cooper, 2000; Glossbrenner and Glossbrenner, 2001), and their number and features are too numerous to document in a book devoted specifically to healthcare IR. Thus this section will cover the most important features of such tools and any aspects of them that are relevant to health care. Since these features change frequently, the reader is referred to the help files of the specific engines for the most up-to-date information on their capabilities. Most search engines use a basic variant of partial-match searching. The pages have generally been discovered by “crawling” the Web, and their words are typically indexed by means of a word-based approach. The general approach to re-
6. Retrieval
207
trieval is that users enter a natural language query statement and the search engine responds with a ranked list of matching Web pages. Most search engines show 10 matching Web pages per screen of output, but usually it is possible to configure the page to allow a different number to be displayed. Because most search engines have indexed upward of a billion pages, some constraints must be placed on the size of the output. As with other retrieval systems, this is typically done with Boolean operators. Most search engines, such as Google (www.google.com), Lycos (www.lycos.com), and HotBot (www.hotbot.com), use the AND operator between words by default. That is, when a group of words is entered as a query to these systems, all the words must be present in the Web page for it to appear in the list of retrieved pages. Two notable exception to using AND as a default are the basic search modes of AltaVista (www.altavista.com) and Excite (www.excite.com), which use a default of OR, similar to the original partial-match approach developed by Salton (1991). Google and Hotbot allow use of the OR operator between words, while AltaVista and Excite allow the use of AND. All these search engines allow the use of the plus sign () in front of words to indicate that word must be present in the document, although in Google it applies only to stop words that would otherwise be ignored. All the search engines just named except Google and Lycos allow the use of parentheses to nest searches so that parts of the search statement can be isolated. For example, the search A OR (B AND C) will first find all pages with B and C and then combine them via OR with all pages containing A. The search A OR B AND C, on the other hand, will first perform the OR of documents containing A or B and then combine these via AND with those containing C. The foregoing search engines also have some form of NOT operator. All allow use of the minus sign () to indicate a word must be absent in the page for it to be retrieved. AltaVista, HotBot, and Excite also allow the use of the NOT operator between words, which functions identically to the minus sign. Additional means for controlling the output size are proximity operators. All the foregoing search engines allow phrase searching by enclosing phrases in quotation marks. This is helpful in all search engines, but particularly in those that use a default OR for combining words. In AltaVista, for example, the search back pain will match any documents that include either word. If the form “back pain” is used, only pages having that phrase will be retrieved. The advanced search interface of AltaVista also allows the NEAR operator, which specifies that terms must be within 10 words of each other. For example, in colon near cancer, the words colon and cancer must be within 10 words of each other. This allows for intervening words (e.g., cancer of the colon). Search output size can also be controlled by applying various limits. All the search engines discussed in this section allow results to be limited to a specified language. All but Lycos and Excite allow searches to be limited to pages created or updated during a specified time period. All but Excite allow the search terms to be limited to specific fields within the Web page. Table 6.5 lists some of the field restrictions in Google. HotBot has a variety of limits denoting different types of Web page elements, such as images, video, or Java applets.
208
II. State of the Art
TABLE 6.5. Some of the Field Restrictions That Can Be Applied in Google Tag intitle: allintitle: inurl: allinURL:
site: link:
Restriction
Example
At least one word required to be inpage title All words required to be in page title At least one word required to be in page URL All words required to be in page URL
intitle: HIV AIDS requires HIV or AIDS to appear in title allintitle: HIV AIDS requires both HIV and AIDS to appear in title inurl: PDF Medical requires PDF or Medical to appear in URL allinurl: PDF Medical requires both PDF and Medical to appear in URL site: nlm.nih.gov limits search to pages at the NLM Web site link: pubmed.gov limits search to pages that link to PubMed
Page required to be in specified domain Page required to have specified link
A variety of other features are present in different Web search engines to facilitate searching. Stop words are used in Google, HotBot, and Excite but not in Lycos or AltaVista. HotBot and AltaVista use stemming, but the others do not. HotBot and AltaVista also allow case-sensitive searching, in which one or more capitalized letters will force the word to occur exactly as it is entered. For example, if mesh is entered, pages containing any form of the word (e.g., mesh, MeSH) will be matched. However, if MeSH is entered into either of these search engines, only pages containing MeSH with that exact capitalization will be retrieved. These two search engines also allow wild-card searching. In AltaVista, a wild-card character can appear after the first three characters of a word within it or at the end. Thus as noted earlier, col*r will retrieve pages with the words color, colour, or colder in AltaVista. All the search engines use some form of relevance ranking for sorting their output. Their exact formulas are “trade secrets,” but it is most likely that they use some variant of TF*IDF weighting. AltaVista allows sorting of the output by specifying words or phrases that should be given the highest weight. Google employs the PageRank approach, which gives added weight to pages that have a higher number of links to them (Brin and Page, 1998). This results in higher ranking of Web pages that are more “popular” as denoted by large numbers of links to them. There are some Web search engines that have more unique features. Northern Light (www.northernlight.com), for example, has many of the features already discussed, such as Boolean operators, phrase searching, and field searching. But it also uniquely sorts its results into Custom Search Folders that group output by topic or document type. Northern Light also has a Special Collection of content that includes the full text of 25 million documents from over 7000 journals, books, magazines, and newswires. The user searches over document summaries in the Special Collection and must pay a fee to access most of the documents. Another unique search engine is AskJeeves (www.askjeeves.com), which requires its in-
6. Retrieval
209
put to be in question form. The engine then tries to provide an “answer,” which is a Web page that someone performing the same search found to be relevant. Some search engines are “meta” engines, taking the user’s input, searching a variety of different search engines, and presenting the results. The first meta–search engine was MetaCrawler (www.metacrawler.com), which ranks matching pages by their presence and rank in the individual engines. Another meta–search engine is DogPile (www.dogpile.com), which presents results sequentially by individual engine in an order that can be controlled by the user. The Apple Macintosh operating system (www.apple.com) has a built-in meta–search engine called Sherlock. There is also a Windows-based application called Copernic (www.copernic.com), which is a stand-alone program that provides metasearching. Meta–search engines have less functionality than the underlying engines they search, their output is smaller, and they time out when results are not received in adequate time (Glossbrenner and Glossbrenner, 2001). A Web searching site that is often listed among search engines is Yahoo (www.yahoo.com). However, Yahoo is actually a Web catalog, with whole Web sites manually indexed with a classification hierarchy. Searches in Yahoo retrieve specific terms the classification hierarchy. The output lists both matching terms and whole Web sites that have been classified to matching terms. When the number of categories matched is inadequate, Yahoo passes the search string to Google and presents that search engine’s results. A number of the Web search engines just described also provide a catalog listing for each page retrieved (when that page is in the catalog). Google, HotBot, and Lycos use the Open Directory catalog, while AltaVista uses LookSmart (www.looksmart.com). One concern with Web search engines is their ability to be “gamed,” in one way by the Web site author and in another way by the search engine itself. There are high economic stakes in search engine output. One issue is that Web site developers wish to have their content ranked as high as possible. Another is that search engine owners need a revenue stream to maintain their product, which users are able to access without charge. With a knowledge of how search engines work, Web developers can attempt to increase the ranking of their pages by adding words that users are likely to employ in their searches (Glossbrenner and Glossbrenner, 2001). If they do not want the user to see these words in the text, they can be hidden in the page’s META tag or made invisible by using either an ultrasmall font size or a font color the same as the background. While developers early in the Web’s existence used the word sex, a more sophisticated approach might employ the name of a company’s competitor or one of its products. Search engines fight a constant battle to keep their engines from being gamed in this way. They also find ways to identify content as “safe” (i.e., nonpornographic), but this can be difficult when similar words are quite important when used in a medical context (e.g., breast, sex, and the XX chromosome pair). The other approach to gaining comes from search engine developers, who have incentive to sell ranking positions (Anonymous, 2001b). While none of the foregoing search engines actually alters its output in this manner, some (AltaVista, HotBot, and Lycos) have been accused of blurring the distinction between search
210
II. State of the Art
results and advertising. All search engines display advertising that is selected on the basis of the terms entered. A consumer group, Commercial Alert (www.commercialalert.com), has argued that these sites do not make advertising distinct enough to clearly separate it from search results. The search engine GoTo (www.goto.com) touts such “paid placement” as a feature (Cohen, 2001). Noting the way that gaming of the system arises from hidden words and that many mediocre-quality pages appear in search engine output, GoTo ranks pages based on how much sites have paid for placement. Commercial Alert argues that this practice is deceptive to unknowing consumers, but GoTo responds that the feature is clearly explained on the site, and the amount each of the other sites has paid is listed. How can Web search engines be used optimally in the search for health information? Their massive breadth is both a boon and a bane. These search engines are more likely to be most up to date because they find new Web pages by crawling instead of manually entering them. However, entering pages via crawling precludes quality control. All pages found by the crawling process are entered into the search engine. Although this is precisely what they are designed to do, recall the problem discussed in Chapter 2 of the variable quality and accuracy of health information on the Web. Moreover, search engines also do not discriminate between content aimed at users of different types, such as patients, primary care physicians, or specialists. A clear advantage of Web catalogs is that they can be limited by their inclusion criteria to content of a certain type or presumed quality.
6.3.3 Databases and Collections As noted in Chapter 4, a number of health resources on the Web consist of collections of content, sometimes stored in databases and delivered dynamically. This section describes the searching approaches to these resources. 6.3.3.1 Images The first group of content described was image databases. As Chapter 5 explained, image collections are generally indexed by textual descriptions of individual images. When the collection is small, a searching interface might even be present, with a browsing interface providing links to the images. When searching is allowed, the capabilities are generally minimal, usually consisting of simple word matching between query and image description terms. Figure 6.13 shows the search interface to the BrighamRad radiology image collection, which allows key words as well as certain constraints to be specified. Most searching interfaces to the Visible Human Project provide a human body that can be clicked on to show corresponding cross sections. 6.3.3.2 Genomics A second group of resources in the databases and collections content category was genomics databases. Searching in these databases may involve text strings
6. Retrieval
211
FIGURE 6.13. BrighamRad search window. (Courtesy of the Department of Radiology at Brigham and Women’s Hospital, http://www.brighamrad.harvard.edu.)
such as disease and drug names. But it may also involve codes such as gene identifiers. All these can be entered into the Entrez interface, which is very similar to PubMed. (In fact, PubMed is part of Entrez.) Both text strings and identifiers are treated similarly by Entrez, and they can be combined via Boolean sets and linked directly to them from the list of results. There is one additional type of searching done in genomics databases, which is different from most of the IR described in this book, which aims to retrieve documents. This is sequence searching: here the goal is to take a sequence of data, such as nucleotides in DNA or amino acids in a protein, and find comparable though not necessarily identical sequences in the database. This type of searching is done by using the Basic Local Alignment Search Tool (BLAST) family of computer programs (Wheeler, Chappey et al., 2000). Such searches are done, for example, to find genes that code for a known protein or comparable genes across species.
212
II. State of the Art
6.3.3.3 Citations While the main use of citation databases is to identify linkages in the scientific literature, there is sometimes the need to search them in a conventional manner. If nothing else, the user needs to identify the specific works of an author or a particular citation. Or, a user who wished to browse the database in a more exploratory manner might want to have the ability to search by author or title words. The Web of Science database has a relatively simple interface that allows such searching. 6.3.3.4 Other Databases Chapter 4 listed a variety of databases in the “other” category. Most employ simple text-word searching. The ClinicalTrials.gov site has a simple text-box interface that searches over the text in the clinical trials records, as well as a focused search interface that allows searching over specific fields (Figure 6.14).
FIGURE 6.14. ClinicalTrials.gov focused search window. (Courtesy of the National Library of Medicine.)
6. Retrieval
213
6.3.4 Aggregations The final group of information resources described in Chapter 4 consisted of aggregations of content. The MEDLINEplus system of aggregated consumer health resources provides a simple text box but also features a more advanced interface (Figure 6.15) that allows exact vs close (using stemming) match of words as well as the requirement for matching all words (AND) or just some (OR). The user can also look up terms in a medical dictionary to check their spelling and have the search run against the entire site or just for information in certain areas. The practitioner-oriented MedWeaver system provides a variety of searching options. The user can perform traditional searches of MEDLINE by using an interface similar to that of Ovid and of Web pages using an approach similar to
FIGURE 6.15. MEDLINEplus advanced search window. (Courtesy of the National Library of Medicine.)
214
II. State of the Art
that of CliniWeb. But one unique MedWeaver capability allows the user to enter a case into the DxPlain decision support system and retrieve information based on the disease profile generate by case data, as shown in Figure 4.8. The other practitioner-oriented aggregations described in Chapter 4 provide relatively simple text-word searching with the use of Boolean operators. Like many of the aggregated resources, the NLM Gateway sports a simple interface (similar to that of PubMed), which belies a great deal of processing that goes on in the background. Entered query terms are sent to the underlying databases, with the aggregated results presented in a table (Figure 6.16). The portion of the search sent to PubMed is processed as if the search were entered directly into PubMed.
6.4 Document Delivery Although there is much enthusiasm for access to online full-text journals, there is still a demand for paper copies of journal articles to be delivered to individuals. Either the journal may not be available online or the user may not have a
FIGURE 6.16. NLM Gateway search results. (Courtesy of the National Library of Medicine.)
6. Retrieval
215
subscription permitting access to it. In these situations, paper documents must be delivered by some means. In the traditional library searching setting, the user searches a bibliographic database, finds the appropriate references, and goes to the stacks to find the desired article. Because the particular reference may not in the library’s collection, most libraries have an interlibrary loan department, which can obtain the reference or a copy from another library. With the increasing prevalence of IR system usage outside libraries, document delivery has become an important function of libraries. Many libraries have document delivery services, and some of the online services allow the user to order a copy of the document while viewing its reference online. NLM allows documents to be ordered through PubMed by means of its DOCLINE service (http://www.nlm.nih.gov/docline/ newdocline/html). Requests are routed to regional libraries around the world, which provide the documents to users for a charge. Most commercial MEDLINE systems also provide the capability for users to order paper documents from institutions at which the systems are licensed. The cost of document delivery is still rather high, averaging about $8 to $10 per article. In addition, there is a delay between the time a document is ordered and its delivery by surface mail. Some systems feature fax transmission of documents, but this is pricier still. Another option used increasingly is electronic mailing of a scanned version of the document.
6.5 Notification or Information Filtering Another way of accessing information is via notification, information filtering, or selective dissemination of information. Information filtering is the retrieval of new documents from an incoming stream, a sort of electronic clipping system. It is suited for user who is perpetually interested in a topic, wanting to gather all new documents that are generated. The filtering system is used to set up a profile, which is a search statement run against all new documents added to one or more databases. Information filtering is very similar to IR, though Belkin and Croft note some distinct differences (Belkin and Croft, 1992). For example, IR systems are oriented toward one-time use for distinct information needs, while filtering systems imply the repeated use of the same system for the same query. In addition, whereas IR systems deal with relatively static databases, filtering systems are concerned with selection of information from dynamic streams of data. Finally, timeliness is more likely to be an important aspect of the functionality of filtering systems. Most of the major MEDLINE search systems offer some variant of information filtering. PubMed allows users to register to set up a cubby, where search strategies can be saved and run again in the future retrieving only documents that have been added to PubMed since the last time the search was run. Ovid offers functionality truer to an information filtering task. Users can save search strategies and then have them run against new documents added to the database, with the results sent to the user via e-mail. The interface to set up the process is shown in Figure 6.17.
216
II. State of the Art
FIGURE 6.17. Ovid information filtering setup screen. (Courtesy of Ovid Publishing.)
The information filtering task has been studied extensively at TREC, but much less in the healthcare domain. One analysis found that the commercial products vary widely in their operation and usefulness (Bandemer and Tannery, 1998). As a result, it is unclear what the major problems might be for users. Clearly there is a recall–precision trade-off problem, with the user needing to achieve a balance between detecting relevant materials and ignoring nonrelevant materials. The changing of users’ interests over time and the selection of appropriate databases are other issues that must be addressed.
6. Retrieval
217
6.6 Searching with the Sample Database Some of the challenges that arise in searching can be demonstrated with the sample database in Appendix 1. This final section of the chapter demonstrates the results of Boolean operations; it does not show optimal searching strategies. To explore some of the issues related to exact-match searching, consider the query of the MeSH terms Hypertension AND Adrenergic Beta Receptor Blockaders. This query will retrieve only document 5. A strategy to broaden the query might be the use of a term related to Hypertension, which is Antihypertensive Agents. Since the searcher would want documents with either of those two indexing terms, an OR would be used. The search statement (Hypertension OR Antihypertensive Agents) AND Adrenergic Beta Receptor Blockaders would retrieve documents 5 and 7. Another strategy to broaden the search might be to explode the term Adrenergic Beta Receptor Blockaders to ensure that actual drugs in this category are added to the search statement. Thus the search statement (Hypertension OR Antihypertensive Agents) AND explode Adrenergic Beta Receptor Blockaders would retrieve documents 5, 6, and 7. A searcher might also want to try using text words. Recall from Chapter 5 that most systems allow text-word searching on the title, body text, and indexing terms. A text-word search might consist of the words hypertension AND beta AND blocker, which would retrieve documents 5, 7, and 8. Document 6 would not be retrieved because it does not contain the word hypertension in any of its fields. To demonstrate some of the issues associated with partial-match searching, consider a user interested in the drug treatment of hypertension. A query might be: drug treatments of hypertension. Tables 6.6 and 6.7 illustrate the retrieval process. As indicated by the listing in Table 6.6, the word of is a stop word and is thus discarded, while the word treatments is stemmed to treatment. The word stems are then used to match documents. Table 6.6 lists the nonstop query words, their IDFs, and the TFs for each document in which they occur. The word treatment occurs in documents 3, 5, and 8. Since the word occurs just once in each document, it contributes a score
TABLE 6.6. Results of Search Using TF*IDF Weighting for Words in Query drug treatments of hypertension to Appendix 1 Documents treatment - 1.52* (doc 3)1.0 1.52 (doc 5)1.0 1.52 (doc 8)1.0 1.52
hypertension - 1.22* (doc 1)1.30 1.59 (doc 2)1.0 1.22 (doc 3)1.30 1.59 (doc 4)1.30 1.59 (doc 5)1.0 1.22 (doc 8)1.0 1.22
drug (doc (doc (doc
- 1.52* 5)1.30 1.98 6)1.0 1.52 7)1.0 1.52
218
II. State of the Art TABLE 6.7. Scores for the Documents in the Appendix 1 Database with the Query from Table 6.6 Document 1 2 3 4 5 6 7 8 9 10
Score 1.59 1.22 1.52 1.59 3.11 1.59 1.52 1.22 1.98 4.72 1.52 1.52 1.22 1.30 2.52 0 0
of the IDF for treatment (1.52) times the TF (1.0). The word hypertension occurs in documents 1, 2, 3, 4, 5, and 8. In documents 1, 3, and 4, it occurs twice, raising the TF to 1.30, thus contributing a score of 1.59 (1.30 times the IDF of 1.22) for each document, while in the remaining documents it occurs once, just contributing the score of 1.22. The word drug occurs in documents 5, 6, and 7, contributing weight as listed in Table 6.6. The scores for each document are summed in Table 6.7. Document 5 ends up ranked the highest, with a score of 4.72. This example shows the effectiveness and limitations of this approach. Document 8, dealing with glaucoma, is clearly unlikely to be relevant. Yet it ranks high owing to the presence of the words treatment and hypertension. Conversely, documents 6 and 7 are penalized because they lack the word hypertension, although they are clearly about the target topic, they use terms like antihypertensive therapy. This problem could be rectified by the use of subject headings, and in fact most systems already containing databases that include subject headings (e.g., MEDLINE) utilize them as if they were part of the document text.=
Chapter 7 Evaluation
A recurring theme throughout the book thus far is that the information retrieval (IR) world has changed substantially since the first edition, particularly with the advent of the World Wide Web. For this reason, one might have expected a substantial increase in the amount and quality of evaluation research. However, this is not the case. While a modest number of new evaluation studies have appeared since 1996, the growth of evaluation research has not paralleled the explosion of new content and systems. This may well be due to the overall success and ubiquity of IR systems in the Web era; that is, such systems are so ingrained in the lives of users that few believe that the need to evaluate them still exists. This is of course not the case, for much can be learned from looking at how these resources are used and from identifying areas calling for improvement. Therefore, one challenge for this chapter is to integrate results from the preWeb era into the modern picture, where content and systems are quite different. There is still value in looking at results from “older” systems and their users for several reasons. First, users of those systems had the same general goals and motivations as modern users. Furthermore, even if the early users did not have access to some of the recent innovations, they had to grapple with challenges such as controlled vocabulary terms versus text words, Boolean operators, and the desire to access full-text content from bibliographic databases. Another reason for looking at older studies is that they assessed in detail issues that are often not studied in today’s research. As in the first edition and a subsequent review article (Hersh and Hickam, 1998), this chapter is organized around six questions that someone advocating the use of IR systems might ask: 1. 2. 3. 4. 5.
Was the system used? For what was the sytem used? Were the users satisfied? How well did they use the system? What factors were associated with successful or unsuccessful use of the system? 6. Did the system have an impact? 219
220
II. State of the Art
7.1 Usage Frequency One of the best measurements of the value of IR systems is whether the applications are actually used by their intended audience. One can hypothesize about the pros and cons of different IR systems, but the discussion is moot if the system does not sustain the interest of users in the real world. This is certainly an important issue for developers of commercial systems, since unused systems are unlikely to last long in the marketplace. Clearly IR systems are used by many people. As noted in Chapter 1, nearly all physicians use the Internet, with those who have busier practices likely to use it more (Taylor and Leitman, 2001). In addition, a large majority of others who use the Internet have searched for personal health information (Taylor, 2002). Furthermore, the National Library of Medicine (NLM) reported that 244 million searches were done with its databases in the year 2000 (Anonymous, 2000g). This section focuses on usage by healthcare practitioners in clinical settings, since settings of that type have been assessed most prevalently. Of course, usage frequency alone has its limitations. For example, if only the quantity of searches performed is measured, one gains little insight into why the system was used, or how successfully. There are also some practical issues in measuring usage, especially when one is making comparisons between systems and/or settings. In the case of end-user searching, it is important to define the unit of usage. Is it a single search entered into the system, or is it the sum of a user’s efforts to find information on a single topic over one or multiple sessions? Furthermore, it may be difficult to demarcate topics, since the user’s interest is subject to change in the course of an interaction. There are two factors that classify usage studies: the setting (e.g., clinical locations, libraries, or other locations) and the type of tracking (e.g., direct monitoring of the system or user questionnaires). Because per-user statistics are typically not available from libraries or other locations, the focus here is on use of IR systems in clinical settings, tracked both by direct monitoring and user questionnaires. The latter type of measurement, based on personal recall, is potentially hampered by decreased reliability, especially when the users are being polled by the people by whose efforts the IR system was made available.
7.1.1 Directly Measured Usage There have been 10 evaluation studies that have measured usage in clinical settings by direct monitoring. Each of the settings was distinct from all the others, but in the aggregate, the following studies give a reasonable consistent picture of how often clinicians use IR systems when available in the clinical setting: 1. Horowitz et al. (1983) made MEDLINE available on terminals that were accessible to staff and students throughout the hospital and clinics at Beth Israel Hospital in Boston. 2. Collen and Flagle (1985) provided MEDIS, a system (no longer available) featuring access to MEDLINE references as well as the full text of several
7. Evaluation
221
journal titles and textbooks, at seven hospitals nationwide, with a mix of academic and community settings. 3. Simon (1988) made MEDLINE available to medical students in a Pediatrics clerkship. 4. Haynes et al. (1990) observed the use of MEDLINE in five sites at McMaster University Medical Centre: emergency room, intensive care unit, ambulatory clinic, and two inpatient hospital wards. 5. Tobia et al. (1990) made MEDLINE and the Computerized Clinical Information System (CCIS, Micromedex, Denver) available in Texas at a county center and at an academic medical center. 6. Abate et al. (1992) provided MEDLINE and other databases to three officebased practices, a clinical pharmacy group, and a university-based family medicine practice in West Virginia. Each site had one of the programs for 9.5 months, followed by a 2-month wash-out period, and then the other program available for another 9.5 months. 7. Osheroff and Bankowitz (1993) provided MEDLINE, medical textbook, drug references, and diagnostic expert systems to physicians at University of Pittsburgh. 8. Hersh and Hickam (1994) assessed a multiapplication workstation in the General Medicine Clinic at Oregon Health Sciences University, featuring access to MEDLINE, electronic textbooks, the Yearbook Series, and the decision support system Quick Medical Reference (QMR). 9. Pugh and Tan (1994) made CCIS available in two university-based emergency departments in British Columbia. 10. Royle et al. (1995) made MEDLINE available in a medical and hematology unit of McMaster University Medial Centre. Some of these studies also assessed user satisfaction and searching performance, the results of which are described in subsequent sections. Most of the studies were done well before the advent of today’s near-ubiquitous Internet. It is somewhat ironic that the Internet makes such usage studies more difficult to perform, now that in many cases users of databases are not required to log on, making usage increasingly anonymous. In fact, more recent studies assessing overall usage of systems in a German hospital (Haux, Grothe et al., 1996) and in a British health trust (Nankivell, Wallis et al., 2001) were unable to measure frequency of use by individual users. Also making modern-day studies more difficult is the presence of information on other electronic devices, such as handheld computers, and in other locations, such as the home. The results of the 10 studies are shown in Table 7.1. While the dissimilarities among them (i.e., different databases, clinical environments, specialties, and time period of observation) make direct comparison difficult, it can be seen that overall usage of systems is relatively small, especially in comparison to the quantification of information needs discussed in Chapter 2. In particular, these results show that IR systems are infrequently used relative to the known two-questions-per-three-patients information needs of clinicians (Gorman, 1995; Ely, Osheroff et al., 1999).
222
II. State of the Art
TABLE 7.1. Comparison of Studies of Measured Usage of IR Systems in Clinical Settings, Calculated from Data Presented in the Papers Referenced Users Study
Total
Horowitz et al. (1983) Collen and Flagle (1985) Medical students Nurses Residents Staff physicians Simon (1988) Haynes et al. (1990) Students Interns Residents Fellows Staff Tobia et al. (1990) Students Residents, fellows Pharmacists Clinicians Abate et al. (1992) Nurses Pharmacists Physicians Osheroff and Bankowitz (1993) Hersh and Hickam (1994) Interns Junior residents Senior residents Staff Pugh and Tan (1994) Royle et al. (1995)
3654 372
By categories
Usage per person-month
Length of observation in months
Total
36 2.7
0.3 6.7
58 35 152 127 104 158
7.4 5.0 8.2 6.0 2 8
1.7 2.2
30 22 45 14 47 506
3.2 3.0 2.6 1.5 0.9 5
1.7
129 167 115 95 38
0.4 1.7 3.7 0.8 19
1.6
11 12 15 8
0.8 0.2 3.8 0.5
31
10
12.5 1.3
8 9 8 6 13 29
Average
0.6 1.2 1.3 2.1 6.7 6
3.5 1.1
Another concern arising from these studies is the “novelty effect,” referring to the tendency of usage to fall off after the initial fanfare of a new technology has faded. Two of the studies actually measured usage over time and found this to be the case (Hersh and Hickam, 1994; Nankivell, Wallis et al., 2001). Indeed, one can plot the usage data in Table 7.1 versus the length of the study and discern a tendency for system usage to correlate inversely with length of observation (confirmed with a linear regression line, as shown in Figure 7.1). While this finding could be an artifact of the discordant study designs, it may indeed represent usage falling off when the excitement of a new technology fades. Some studies attempted to learn which resources clinicians use when more than one are available. Or, more generally, do clinicians prefer bibliographic or full-text resources? Four of the studies provided access to multiple online re-
7. Evaluation
223
sources. Tobia et al. found that the full-text CCIS product was used more often than MEDLINE in county and academic medical centers, whereas two other studies found a preference for bibliographic databases. The study by Abate et al. offered access to the full Bibliographic Retrieval Services (BRS) and Dialog systems, yet MEDLINE was used over 50% of the time, followed by the bibliographic database EMBASE, and the full-text journals on BRS. Likewise, Hersh and Hickam found that MEDLINE was used substantially more frequently (68.4%) than textbooks (19.2%), QMR (6.3%), or the Yearbook Series (2.0%). Even though they did not measure individual use, the more recent studies by Haux et al. and Nankivell et al. did assess differential usage of various resources. The former found that MEDLINE was used extensively in the German hospital (78.8%), with drug databases next (6.2%), and a variety of full-text resources extremely rare. Likewise, in the British health trust, MEDLINE was by far the most commonly used resource (60%), well above others such as Best Evidence, The Cochrane Database of Systematic Reviews, and Scientific American Medicine. Two studies of reported usage were excluded from Table 7.1. One measured the searching behavior of senior medical students who had campus-wide access to PaperChase but did not report the exact usage of all students (Pao, Grefsheim et al., 1993). This study did find that during their first 3 years of medical school, students averaged 1.36 searches per month. Usage was found to increase to 3.0 searches per month over a 5-year period, following a hands-on examination of searching ability early in the students’ fourth year. Another study reported usage of a system by clinicians but took place in a library (Simon, 1988). The authors
FIGURE 7.1. Plot of measured usage of IR system per user-month by length of time assessed. Linear regression line is added to show how usage varies with length of study.
224
II. State of the Art
made available a CD-ROM workstation with MEDLINE, PsycLIT, Micromedex, and other resources. They found that usage varied from 0.23 times per month for students to 0.14 for residents and 0.17 for faculty. MEDLINE accounted for over half of all usage, and the only full-text resource, Micromedex, was used 16% of the time. There are no studies of measured usage of health and biomedical IR systems outside the clinical setting. The closest to such studies comes from analysis of Web searching logs. Spink et al. (2002) have analyzed large numbers of queries posed to the Excite search engine. As these queries solely consist of what the user entered, the information needs behind them are unknown. Furthermore, nothing is known about the individuals who posed them, such as who they are, where they reside, or how many queries they posed. Their analysis found that in 1997, the proportion of “health or sciences” queries was 9.5% of all queries, while by 2001 that proportion dropped to 7.5%. Of course, since the number of queries per user was not known, this analysis could only determine the relative and not absolute numbers of health-related queries. Thus it cannot be determined whether health-related usage of the Web truly changed during that time.
7.1.2 Reported Usage Other studies have measured usage with questionnaires asking for recall of frequency of use. One of the most comprehensive studies was performed by the NLM in 1987 (Wallingford, Humphreys et al., 1990). A questionnaire sent to over 4000 MEDLINE account holders likely to be individual users had a response rate of over 70%. Over two-thirds of respondents were physicians, while the majority of the rest were research scientists. The respondents, who were evenly divided between academic and clinical settings, reported on average of 4.3 searches per month. Another NLM analysis of database users was carried out a decade later (Wood, Wallingford et al., 1997). This study identified specific user groups and asked subjects how frequently they used NLM databases. The mean usage for the various groups was as follows: • Healthcare providers, 1–3 times/month • Librarian/information services professionals, 10 times/month • Scientists, 1–3 times/month • Educators, 1–3 times/month • Patients/healthcare consumers, 1 time/month • Students, 1–3 times/month • Legal, 1–3 times/month • Media, 1–3 times/month In a similar analysis, the British Medical Association (BMA) Library made MEDLINE available on CD-ROM via dial-up to BMA members (Rowlands, Forrester et al., 1996). A survey of 61 users found that 44% were using the database weekly or more frequently and 34% were accessing it at least monthly. Upon surveying a random sample of physicians from all specialties and levels of training at New York Hospital–Cornell University Medical Center, Poisson (1986) found that 8% were already using online searching, with 63% interested in learning how to search and 29% not interested. She subsequently offered
7. Evaluation
225
training at one of the affiliated centers to the institution. Half the medical staff attended at least one training session. All who attended were subsequently surveyed to determine usage, with a majority (53.3%) reporting no searching done. The remainder reported that they searched less than once in 6 months (20%), once in 3 or 6 months (13.3%), or more than once in 3 months (13.3%). These results are consistent with the studies of measured use described earlier. In some studies, reported usage rates were higher than rates for directly measured use in other studies. Ludwig et al. (1988) made the MEDIS system available at Loyola Medical Center in Illinois in a few library and clinical sites across the center. Four months after the introduction of the system, stated frequency of use for all users, which included those from the medical, nursing, and dental fields ranging from the level of student to faculty, was fairly evenly distributed across four categories: 1 to 5 times per week, once a week, once a month, and no longer using. Markert (1989) made MEDIS available to students and faculty at 6 hospital sites and via modem from home at Wright State University. The median monthly usage was reported to be 10 or more times for medical students and full-time faculty, 7 to 9 times for residents, and 4 to 6 times for volunteer faculty. These results differ from those of Poisson and studies of measured usage above, already cited, perhaps as a result of overreporting of use or a population slanted toward researchers who spend more time in the library. Mullaly-Quijas et al. (1994) carried out a series of focus groups not only to determine how often online databases were used but what they were used for and what barriers existed to their use. Several groups of healthcare professionals, including physicians, nurses, pharmacists, dentists, and allied health personnel, were assessed in Nebraska, Missouri, and Utah. Reported usage was highly variable, most commonly in the range of monthly, but pharmacists were more frequent users. Familiarity with the various databases also varied widely.
7.2 Types of Usage In addition to knowing the frequency of system usage, it is also valuable to know what types of information need users address. Many of the studies described in Section 7.1 also investigated this issue. Since information resources, users, and settings were heterogeneous, direct comparison is difficult. But as with the studies cited, taken as a whole they give a relatively consistent picture of the kinds of use IR systems see in the healthcare setting. Six of the studies in Section 7.1 assessed the subject matter of the questions asked (Haynes, McKibbon et al., 1990; Abate, Shumway et al., 1992; Osheroff and Bankowitz, 1993; Hersh and Hickam, 1994; Royle, Blythe et al., 1995; Nankivell, Wallis et al., 2001). All but Royle et al. focused on the uses by physicians (with Abate, Shumway et al. also looking at use by pharmacists). In general, questions of therapy were found to be most frequent, followed by overview (or review) and diagnosis. The results of Hersh and Hickam are shown in Figure 7.2. Royle et al. found that the most frequently searched topic by nurses in their medical and hematology unit was leukemia (30%), followed by other diseases (26%), medications (19%), and psychosocial concerns (10%).
226
II. State of the Art Therapeutics
26%
Review
23%
Prognosis 2% Etiology 2% Complications Other/Unknown
5% Diagnasis
6%
19%
Author 6% Mechanisms 11%
FIGURE 7.2. Types of clinical question posed to an IR system in an ambulatory care setting. (From Hersh and Hickam, 1994. Copyright by and reprinted with permission of the Medical Library Association.)
Seven studies addressed the reason for using the system (Collen and Flagle, 1985; Markert, Parisi et al., 1989; Haynes, McKibbon et al., 1990; Silver and Dennis, 1990; Wallingford, Humphreys et al., 1990; Hsu, 1991; Haux, Grothe et al., 1996). The studies all varied in their categorizations of usage types, but in general, reasons for use revolved around patient care, personal education, and research. In general, searchers in clinical settings were more likely to search on patient care problems, while those in academic settings were more likely to search for research purposes. Furthermore, while all clinical groups were likely to search for patient care reasons, full-time faculty were more likely to search for teaching, research, or scholarly writing.
7.3 User Satisfaction Another method of evaluating the impact of an IR system is to measure user satisfaction. Of course, researchers are never certain that a user’s satisfaction with a system is associated with successful use of it. This may be especially problematic when systems are made available with great fanfare, without change, and in academic settings where peer pressure might motivate their use. Nonetheless, for computer applications in general, Nielsen and Levy (1994) performed a metaanalysis of studies in the human–computer interface literature that showed a general correlation between user satisfaction and successful use of systems. At least 15 studies, most already described in Section 7.1, have included some measurement of user satisfaction (Horowitz, Jackson et al., 1983; Collen and Flagle, 1985; Slingluff, Lev et al., 1985; Ludwig, Mixter et al., 1988; Haynes, McKibbon et al., 1990; Wallingford, Humphreys et al., 1990; Hsu, 1991; Abate, Shumway et al., 1992; Osheroff and Bankowitz, 1993; Hersh and Hickam, 1994; Pugh and Tan, 1994; Royle, Blythe et al., 1995; Haux, Grothe et al., 1996; Rowlands, Forrester et al., 1996; Nankivell, Wallis et al., 2001). Although diverse satisfactionrelated questions were asked in these studies, it was found in general that 50 to 90% of users were satisfied with the system provided them. When users were not satisfied
7. Evaluation
227
with systems, the general reasons were the time required to use them, concerns over the completeness of information, and difficulties in navigating the software. As noted earlier, one issue that often plagues new technology is the novelty effect. In addition to effect seen in Figure 7.1, the discontinuation of usage over time has been noted elsewhere. For example, Marshall (1990) followed up a group of “early adopters” of Grateful Med, revisiting them after 3 years. She found that one-third had given up searching, with the most frequent reasons cited for stopping listed as follows: • • • •
System too difficult to use (24.1%) Poor or inappropriate content (20.7%) System too slow (17.2%) Too busy to search (13.8%)
Another piece of evidence giving insight into the value of online searching was the observation by Haynes et al. (1991) that searching fell off by two-thirds when user fees were added. While few users pay for access to IR systems with modern systems, monetary value is certainly one criterion that could be used to assess the value of an IR system, and users’ unwillingness to pay for access may be an indication of dissatisfaction. More recently, McGowan and Richwine (2000) assessed the satisfaction of rural physicians given access to a number of bibliographic and full-text resources. About 71% of users cited inability to have ready access to the information and 78% stated inadequate time as impediments to using the service. About twothirds reported that they had a computer and the basic literacy to use it, although 61% never or seldom searched MEDLINE. The focus groups of Mullaly-Quijas et al. (1994) asked healthcare professionals about the barriers to using online databases. Most reported lack of time, lack of knowledge about computers and/or using the databases, and inaccessibility of computers. Many users were positive about the value of end-user searching but also reported that it would be desirable to have help from expert searchers available. The disparity between enthusiasm for the availability of clinical information resources and their subsequent infrequent use has been investigated. In a multistate telemedicine project, many resources from the medical library at the University of Washington were made available to rural clinicians (Rauch, Holt et al., 1994). When subsequent analysis showed that actual usage of resources was minimal (Masuda, Tarczy-Hornoch et al., 2001), users were surveyed to determine why enthusiasm for the resources had not translated into more frequent use. It was found that users valued paper-based resources and human-to-human communications more highly than electronic resources. In short, they had little incentive to use the resources and as a result did not do so often.
7.4 Searching Quality While usage frequency and user satisfaction are important components in any system evaluation, it is also important to understand how effectively users search with IR systems. As was discussed in Chapter 3, the most commonly
228
II. State of the Art
used measures used to assess the effectiveness of searchers and databases have been the relevance-based measures of recall and precision. Despite some controversy about the value of these measures in capturing the quality of the interaction with the IR system, considerable knowledge about IR systems has been gained by their use, although newer approaches to evaluation have provided additional perspective. This section of the chapter is divided into two parts: evaluations that focus on performance of the user and those that focus on performance of the system. As with many classifications in this book, the line between the two is occasionally fuzzy. Within each category, the studies are divided into those that focus predominantly on bibliographic databases, full-text databases, and Web resources. While the discussion focuses on health-related studies, a few important nonhealth evaluation studies are described as well.
7.4.1 System-Oriented Performance Evaluations System-oriented studies are those that focus on some aspect of the system. That is, even though the searches may have been originated by real users, the research question was oriented toward some aspect of system performance. Many systemoriented retrieval studies were undertaken in the late 1950s and 1960s, but two stand out as setting the historical groundwork for such evaluations. The first of these was actually a series of experiments, commonly called the Cranfield studies, conducted by Cleverdon and associates at the Cranfield College of Aeronautics in England (Cleverdon and Keen, 1966). While these studies have been criticized for some of the methods and assumptions used (Swanson, 1977), they provided a focus for retrieval performance research and the limitations of such studies. The second study, performed by Lancaster (1968), was the first IR evaluation to provide insight into the success and failure of IR systems. Commissioned by the NLM, this study assessed MEDLINE as it was available at the time: with searches composed by librarians on forms that were mailed to the NLM, which ran the actual searches and returned the results by mail. Preceding the advent of interactive searching, this access to MEDLINE was markedly different from what is available today. Lancaster’s study assessed a total of 299 searches from 21 different academic, research, and commercial sites in 1966–1967. In his protocol, users first completed a searching form and mailed it to the NLM, where librarians undertook a second, manual search on the topic, using Index Medicus. The results of both searches were combined and returned to the searcher by mail. The user then judged the combined set of results for relevance. The results showed that the recall and precision for the computer-based searches were 57.7 and 54.4% respectively, indicating that the manual searches identified many articles that the computer searches did not (and vice versa). For “value” articles (i.e., only those denoted “highly relevant”), recall and precision for the computer-based searches were 65.2 and 25.7%. Lancaster also performed a failure analysis, which is described shortly.
7. Evaluation
229
7.4.1.1 Bibliographic System Performance System-oriented evaluations of bibliographic databases have focused on issues such as comparability across different databases, comparability of different approaches within the same database, and optimal strategies for finding articles of different types. Several studies have assessed how well different databases cover specific topics. McCain et al. (1987) compared different databases for accessing the same topic in medical behavioral sciences. Of the five databases studied, three were bibliographic (MEDLINE, Excerpta Medica, and Psycinfo) and two provided topic citations (SCIsearch and Social SCIsearch). For each of the three bibliographic databases, researchers performed “descriptor” search, using controlled vocabulary terms and features, and a “natural language” search, using text words in the reference, and the results were pooled. The citation databases were subjected to a “natural language” search on the topic title and a “key citation” search, using a recent pertinent reference, and these results also were pooled. Ten researchers each proposed a topic in this subject area, and rated the retrieved references for relevance. The searches were performed by librarians. For each question and database, recall and precision were calculated, along with novelty, which is the proportion of relevant references not previously known to the user. The results (Table 7.2) lacked statistical significance but indicated a trend for best recall for topics in MEDLINE, though at a price of diminished precision. The citation index for social sciences (SSCI) outperformed the biomedical citation index (SCI) in all categories for this particular domain. Another investigation compared three databases (CINAHL, MEDLINE, and EMBASE) for researching three nursing research questions (Burnham and Shearer, 1993). The study found CINAHL to have the highest recall for two topics (nursing care of Crohn’s Disease and computers and privacy) and MEDLINE for the other (substance abuse in pregnancy). CINAHL and MEDLINE tended to be complementary. Although this study was small, its results emphasize the important point that searching success if often related to the choice of database.
TABLE 7.2. Comparative Study of Five Databases for Topics in Medical Behavioral Sciences Results (%) Database
Recall
Precision
Novelty
37 29 28 18
57 61 70 50
70 71 68 61
31
70
56
MEDLINE EMBASE PsycINFO Science Citation Index Social Science Citation Index Source: McCain, White et al. (1987).
230
II. State of the Art TABLE 7.3. Recall in Searching One to Five Databases for Topics Related to Occupational and Environmental Toxicology Number of databases
Range of recall (%)
1 2 3 4 5
15–59 52–84 74–93 84–99 100
Source: Gehanno, Paris et al. (1998).
A more recent study compared five databases (MEDLINE, BIOSIS, EMBASE, NIOSH-TIC, and TOXLINE) for two topics related to occupational and environmental toxicology (Gehanno, Paris et al., 1998). The analysis measured recall in a single database followed by combinations of two, three, four, and five databases. Table 7.3 lists the range of recall when searching is done in single and multiple databases. It can be seen that recall improves as the number of databases combined increases. Other research has focused on comparing different systems accessing the same database. Haynes et al. (1985) undertook a study comparing the performance and time required of 14 different access routes to MEDLINE available in 1986 for six clinical topics. They found that most systems yielded the same quantity of articles both directly and generally relevant, though there were substantial differences in cost, online time required, and ease of use. One outlier in terms of relevance was PaperChase, which was the least difficult to learn but retrieved significantly fewer relevant articles, with longer online time and higher cost. This study was redone nearly a decade letter, and again, substantial differences were found between systems, although this time PaperChase performed best for clinicians (Haynes, Walker et al., 1994). The mean number of relevant (1.1 to 8.4 per search) and nonrelevant (4.9 to 64.9 per search) citations retrieved varied widely, even though essentially the same search was being run in each system. Some researchers have looked at the wide variety of free MEDLINE services available on the Web. The results of Jacobs et al. (1998), focusing more on the failures of searching, are presented in Section 7.5.2. Klemencic and Todorovski (1999) compared the features of several free Web MEDLINE systems and evaluated them by measuring the search output with two queries. Many of the free MEDLINE systems were found to be limited in the features they provided, such as the fields that could be searched, the use of advanced MeSH features (e.g., explosions and subheadings), and the date span of the database. Variation in the latter made comparison of search outputs essentially impossible. Some studies in which the same database was searched have looked at specific features and their impact on retrieval performance. One focus has been the value of MeSH terms in searching MEDLINE. Hersh et al. (1994) compared MEDLINE retrieval using indexing of text (i.e., title and abstract) words only,
7. Evaluation
231
with the indexing of MeSH terms added to text words. Using natural language queries, precision at fixed levels of recall in a MEDLINE test collection was found to improve by about 10% with the MeSH terms also indexed. Srinivasan (1996a) has obtained comparable results. Another study compared the use of MeSH terms in two different commercial products, Dialog and Ovid (Hallett, 1998). Dialog was found to retrieve more references with the same query because of its “double posting,” whereby MeSH terms are searched on as whole phrases as well as the individual words within them. This study and one already cited (Haynes, McKibbon et al., 1985) show that different results can be obtained for the use of the same search terms against the same database. A related question that has been investigated is the effect of using abbreviations in searches. Federiuk (1999) assessed 20 common acronyms. She found that the Ovid system was able to map them all to appropriate MeSH terms. Searches were then entered for the MeSH term, the full text of the acronym as text words, and the abbreviation as a text word. The MeSH searches retrieved the highest number of documents in 18 of the 20 searches. However, the textword searches retrieved a substantial number of unique documents not retrieved by the MeSH terms. Since, however, the mere appearance of a term in a document does not guarantee relevance, further analysis is required to determine the importance of the documents not retrieved with the MeSH terms. A third line of system-oriented research with bibliographic databases has focused on the ability to retrieve articles of a specific type. This approach to searching is usually done in the context of evidence-based medicine (EBM), where the searcher is seeking to identify the “best evidence” for a given question. As noted in Chapter 2, what constitutes the best evidence varies according to the question being asked. In the case of articles about interventions, the most common type of question asked of retrieval systems (see Section 7.2), the best evidence comes from a randomized controlled trial (RCT). These studies are best accessed by the use of the term Randomized Controlled Trial in the Publication Type field. As noted in Chapter 5, this element was added in 1991. Other studies prior to 1991 that are likely to be RCTs have been tagged with the Publication Type Controlled Clinical Trial. Chapter 6 reported the development of best strategies for retrieving studies of high methodological quality in the areas of therapy, diagnosis, etiology, and prognosis, which have been incorporated into the Clinical Queries feature of PubMed (Haynes, Wilczynski et al., 1994). Additional research has focused on devising optimal strategies for retrieving systematic reviews using a frequency analysis of text words, MESH headings, and publication types (Boynton, Glanville et al., 1998). The ability to identify RCTs is particularly important in the construction of systematic reviews that include meta-analyses of studies. As noted in Table 6.4, the use of the RCT publication type is not perfect. A number of studies have assessed the ability to find RCTs in numerous topic areas. Dickersin et al. (1994), in a systematic review on studies of this question, found that no combinations of search strategies could retrieve all the RCTs identified by exhaustive hand searching. Studies also showed that precision fell drastically as levels of recall increased. More re-
232
II. State of the Art
cent studies in the areas of emergency medicine (Langham, Thompson et al., 1999), depression (Watson and Richardson, 1999), rheumatological disorders (Suarez-Almazor, Belseck et al., 2000), medical imaging (Berry, Kelly et al., 2000), and clinical nutrition (Avenell, Handoll et al., 2001) have shown that this problem persists. Other research has assessed identification of methodologically sound studies from databases without the benefit of MEDLINE indexing. Johnson et al. (E. Johnson, Sievert et al., 1995) found that while the MEDLINE publication types were valuable in identifying such studies, search strategies could be devised that enhanced their recall by using the full text of the document or its cited reference fields. Hersh and Price (1998), wishing to identify strategies to retrieve RCTs from unindexed abstracts of conference proceedings, compared a variety of different strategies and found that near-complete recall could be obtained at a price of very poor precision (10–15%). Both this study and a follow-up analysis of the analysis of Haynes et al. (Wilczynski, Walker et al., 1995) found that studies that were not RCTs were sometimes indexed as such and that some that were RCTs were not assigned the appropriate publication type. The latter study also found that studies of other types had similar problems (e.g., studies of diagnostic tests not meeting criteria for being methodologically sound having the word sensitivity or specificity in their title or abstract). 7.4.1.2 Full-Text System Performance The earliest comprehensive study of full-text databases was performed by Tenopir (1985), who assessed full-text searching of the Harvard Business Review on BRS. The searches consisted of 40 queries presented to two business school libraries. Tenopir formulated each search and searched on four different levels of full text: 1. 2. 3. 4.
Full text of the documents Abstract words only Controlled vocabulary terms (the documents also had human indexing) A union of title, abstract, and controlled terms
Relevance of the retrieved documents was judged by three experts from the business school. The results (see Table 7.4) showed that recall was much higher for fulltext searching but at a cost of markedly diminished precision. Searching less than the full text yielded better precision but less recall. Among the non-full-text types of searching, controlled vocabulary terms performed somewhat better than use of abstract words, but a combination of these, along with title words, achieved better recall without sacrificing precision. These results demonstrate that indexing more text of a document increases both quality and noise words. One of the purported benefits of the lexical–statistical systems in the next chapter is that weighting methods and relevance ranking help the user to sort out the increased quantity of documents retrieved via full-text searching. Tenopir’s results also demonstrate that the use of abstract words and controlled indexing terms can be complementary. Another well-known study of full-text retrieval was carried out by Blair and Maron (1985). These investigators used the IBM STAIRS system, a full-text,
7. Evaluation
233
TABLE 7.4. Results of Full-Text Searching in the Harvard Business Review, a Legal Document Database, and Full-Text Medical Databases Results (%) Database and condition Harvard Business Review Full text Abstract only Controlled terms Union of all Legal document database All articles Vital and satisfactory articles Vital articles only Medical databases MEDLINE Indexing terms Text words MEDIS CCML
Recall
Precision
Ref. Tenopir (1985)
73.9 19.3 28.0 44.9
18.0 35.6 34.0 37.0
20.0 25.3 48.2
79.0 56.6 18.3
Blain and Maron (1985)
McKinin, Sievert et al. (1991) 42 41 78 76
55 62 37 37
word-based, Boolean system, evaluated a legal document database of 40,000 documents. Fifty-one searches were posed by two attorneys and carried out by paralegal assistants. Searching was repeated until a satisfactory document collection was obtained for a query. After this, additional searching was done by logical (changing ANDs to ORs) and semantic (adding synonyms) expansion. The attorneys who originated the searches made relevance judgments using a four-point scale: vital, satisfactory, marginally relevant, or not relevant. The results (see Table 7.4) showed that recall was low, far below the 75% level that the attorneys felt was required for optimal searching results. Full-text searching has also been assessed in the medical domain. McKinin et al. compared searching in two full-text medical databases and MEDLINE (McKinin, Sievert et al., 1991). They took 89 search requests from a university medical library and performed each one on all three systems. Only documents present in all three were used for recall and precision calculations. The articles were judged for relevance by the original requester on a four-point scale: relevant, probably relevant, probably not relevant, not relevant. Their results paralleled those obtained by Tenopir (see Table 7.4), with full-text searching by wordbased Boolean methods leading to higher recall at the expense of lower precision in comparison to abstract (i.e., MEDLINE) searching. Gallagher et al. (1990) compared searching in a full-text database and in MEDLINE. Unlike the study of McKinin et al., no effort was made to limit searching to a common document set. Searches on 25 topics showed that MEDLINE searching retrieved nearly twice the number of total citations (15.6 per search for MEDLINE vs 8.5 for full text) and relevant citations (7.9 per search for MEDLINE vs 3.7 for full text). However, the full-text database was found to be current, and it answered some questions MEDLINE could not. Some ma-
234
II. State of the Art
terial in the full-text database, such as tables, charts, and bibliographies, was highly useful to searchers. 7.4.1.3 Web Searching System Performance The number of studies assessing performance of Web searching systems is surprisingly small. Many of them have focused on the clinical quality of information retrieved rather than measures of retrieval performance. Those that have looked at performance have tended to focus on the quantity of pages retrieved rather than their relevance. The most comprehensive general (i.e., nonmedical) analysis of Web sites focuses on the number of Web pages returned for a group of single-word searches (Search Engine Showdown, www.searchengineshowdown.com). A medical search engine “road test” also took this approach to comparing different clinically oriented Web catalogs (Anonymous, 1997b). Some studies have assessed the capability of Web resources. Hersh et al. (1998) had a medical librarian enter 50 queries previously known to have answers in the MEDLINE database (Gorman, Ash et al., 1994). The queries were entered into the metasearch engine Metacrawler, with the goal of finding pages oriented toward the healthcare professional (as opposed to the consumer). The results showed that only 26 (52%) of the queries had one or more applicable pages. The proportion of pages having content directly addressing the clinical question (precision) was only 10.7%. Another study of the ability of Web resources to answer clinical questions looked at how many questions could be answered by specific sites, be they general search engines or medically specific catalogs (Graber, Bergus et al., 1999). Ten questions were posed to nearly 20 sites. One site was able to answer six questions (MDConsult, www.mdconsult.com), while three were able to answer five (HotBot, www.hotbot.com; Excite, www.excite.com; HardinMD, www.arcade.uiowa.edu/hardin/md). Most of the medicine-specific Web catalogs performed poorly. One problem with studies of Web resources is the rapid change of the Web itself. Nonetheless, these studies do highlight the challenge of finding clinical information on the Web, especially for healthcare professionals. While the Web crawler search engines contain a great deal of nonprofessional and/or lowquality information, the Web catalogs have a hard time keeping track of all the potentially good sites. Furthermore, a great deal of valuable information is hidden in the “invisible Web,” virtually inaccessible to the novice user who does not know where to look or lacks a subscription.
7.4.2 User-Oriented Performance Evaluations Studies assessing the ability of users with IR systems have looked at a variety of measures to define performance; user’s self-reports of success were described earlier as user satisfaction studies (Section 7.3). A great many have focused on the retrieval of relevant documents, usually as measured by recall and precision,
7. Evaluation
235
although these measures have been criticized for being less pertinent to user success (see Chapter 3). Other studies have attempted to measure how well users are able to complete a prescribed task, such as answering a clinical questions. In summarizing the first two decades of research on user-oriented evaluation, Fenichel (1980) noted several consistent findings across studies: 1. There was a correlation between search success, measured in terms of recall, and “search effort,” which included number of commands used and time taken. 2. There was considerable variation across users, even with the same system and database. Even experienced users made mistakes that affected searching performance. 3. New users could learn to perform good searches after minimal training. 4. The major problems were related more to the search strategy than to the mechanics of using the system. Users made few errors related to use of the command language. 7.4.2.1 Assessing Users by Search Critique One method used to evaluate user searching is the critique of search statements by some expert standard, such as the judgment of an experienced librarian. A limitation to this approach is that the actual output of the search is not judged, and it is assumed that the expert searcher’s strategy will lead to the best results. As will be seen in the next section, searching strategies vary from one expert to the next (McKibbon, Haynes et al., 1990), and the strategies employing the most advanced techniques do not necessarily lead to the best results (Hersh and Hickam, 1994). It should also be noted that this sort of assessment is very close conceptually to the failure analyses reported in Section 7.5. The reason for including studies in this section is their focus on evaluating all searches done in an analysis, not just unsuccessful searches. Several studies using this approach were undertaken in a variety of health science university libraries, starting in the 1980s. J. Nelson (1992) analyzed transactions logs of 643 searches at the University of Southern California Norris Medical Library. While she found that a large majority of searches (84%) were successful, there was little use of MEDLINE features. For example, only 20% of searches used MeSH, while less than 10% used subheadings or explosions. Nelson also noted that despite the user preference for text-word searching, less than 10% employed more advanced aspects of such searching, such as truncation, the Boolean OR, or proximity operators. Most of these studies looked at the ability of medical students to learn effective searching strategies. Bradigan and Mularski (1989) gave a written test of searching concepts before and after MEDLINE training to second-year medical students. Topics on the test included expression of key concepts and application of Boolean operators. Test scores increased from 41% to 94% correct after training. Shelstad and Clevenger (1994) assessed the ability of third-year medical students to search MEDLINE for two surgical problems. The students had received MEDLINE training in the first year of their studies. Only half of the students made
236
II. State of the Art
appropriate use of MeSH terms with the Grateful Med software. Fewer than 10% made use of explosions or subheadings. All these features were determined to be necessary to carry out optimal retrieval on the topics which the students had been asked to search. As a result, virtually all students retrieved too few or too many references compared with the gold standard search by a librarian. Wildemuth and Moore (1995) also looked at the abilities of third-year medical students. Librarians evaluated students’ selection of search terms, use of Boolean operators, narrowing or broadening of the search, and use of system syntax. Ratings for each were, on average, at the median point (2.8 to 3.2) of a fivepoint scale. Students, on the other hand, highly rates that they found what they were looking for and that the search was an efficient use of their time. Rosenberg et al. (Rosenberg, Deeks et al., 1998) performed an RCT to assess the value of training in searching MEDLINE and retrieval of evidence for firstyear medical students. An 18-item scale was developed to score use of various search features. Items included use of database fields, synonyms, MeSH-related features (e.g., subheadings, explosions), Boolean operators, and limiters. Two questions were given for searching. Those who had training nearly doubled the number of features used, and this led to an increased amount of evidence retrieved for one (but not the other) question. Another group whose searching performance has been assessed is clinical pharmacists. Wanke and Hewison (1988) compared drug information searches of MEDLINE by pharmacists with those of medical librarians. Requests for searches over a 3-month period at Oregon Health Sciences University were given to both pharmacists and librarians. The search result printouts were then given to 2 judges (other drug information pharmacists), who designated which would be more useful in providing citations to answer the question. With 48 searches judged twice, 34 of the pharmacists’ searches were deemed better, versus 28 for the librarians and 34 ties (i.e., searches were comparable). 7.4.2.2 Assessing Users with Relevance-Based Measures One of the original studies measuring searching performance in clinical settings was performed by Haynes et al. (1990). This study also compared the capabilities of librarian and clinician searchers. In this study, 78 searches were randomly chosen for replication by both a clinician experienced in searching and a medical librarian. During this study, each original (“novice”) user had been required to enter a brief statement of information need before entering the search program. This statement was given to the experienced clinician and librarian for searching on MEDLINE. All the retrievals for each search were given to a subject domain expert, blinded with respect to which searcher retrieved which reference. Recall and precision were calculated for each query and averaged. The results (Table 7.5) showed that the experienced clinicians and librarians achieved comparable recall, although the librarians had statistically significantly better precision. The novice clinician searchers had lower recall and precision than either of the other groups. This study also assessed user satisfaction of the novice searchers, who despite their recall and precision results said that they were satisfied with their search out-
7. Evaluation
237
TABLE 7.5. Comparison of Grateful Med users. (Haynes, R., McKibbon, K., et al., (1990). Online access to MEDLINE in clinical settings. Annals of Internal Medicine, 112: 78–84. Reprinted with permission.) Results (%) Users Novice clinicians Experienced clinicians Medical librarians
Recall
Precision
27 48 49
38 49 58
Source: Haynes, McKibbon et al. (1990).
comes. The investigators did not assess whether the novices obtained enough relevant articles to answer their questions, or whether they would have found additional value with the ones that were missed. A follow-up study yielded some additional insights about the searchers (McKibbon, Haynes et al., 1990), which were described in the last chapter. As was noted, different searchers tended to use different strategies on a given topic. The different approaches replicated a finding known from other searching studies in the past, namely, the lack of overlap across searchers of overall retrieved citations as well as relevant ones. Figure 7.3 shows overlap diagrams, pointing out that the majority of both retrieved documents and relevant documents were retrieved by one searcher only. Thus, even though the novice searchers had lower recall, they did obtain a great many relevant citations not retrieved by the two expert searchers. Furthermore, fewer than 4% of all the relevant citations were retrieved by all three searchers. Despite the widely divergent search strategies and retrieval sets, overall recall and precision were quite similar among the three classes of users.
FIGURE 7.3. Overlap of relevant articles retrieved by three MEDLINE searchers. (Haynes, R., McKibbon, K., et al. (1990). Online access to MEDLINE in clinical settings. Annals of Internal Medicine, 112: 78–84. Reprinted with permission.)
238
II. State of the Art
A later study by the same group assessed different methods of training novice searchers to make them as effective as experts (Haynes, Johnston et al., 1992). It consisted of a randomized trial comparing the basic two-hour training session with the training session plus the addition of a clinical preceptor experienced in searching. There was no difference in searching ability between the two groups, as measured by average number of relevant references retrieved, but both groups improved their performance to the level of experienced searchers by their fourth online search. Another large-scale attempt to assess recall and precision in clinician searchers was carried out by Hersh and Hickam (1994). These authors attempted not only to assess the capability of expert versus novice searchers but also provided the latter with access to MEDLINE via Knowledge Finder (KF). This lexical– statistical system, which represents one of the first commercial implementations of that approach, utilizes non-Boolean “natural language” searches, with relevance ranking of the output. The output sets are usually much larger than those obtained with Boolean systems, since KF sets its default maximum output at 100 references. Hersh and Hickam also compared the performance of the experienced searchers using the full MEDLINE feature set and just text words from the title, abstract, and MeSH heading fields. As with the studies of Haynes et al., statements of information need were collected online and given for replication to experienced searchers, who were able to use either the NLM’s command-line-based ELHILL system or Grateful Med. Most opted for the former. Logs of all searching interactions were also kept. The KF system used in this study was a CD-ROM version containing MEDLINE references from 270 core primary care journals covering a period of 5 years. As with Haynes et al., relevance was assessed by clinicians blinded to the searcher. One problem with the results of this study (and in fact any study comparing Boolean and lexical–statistical searching) was the large retrieval set obtained by using KF. While advocates of this approach argue that a large output of relevance-ranked documents allows the searcher to choose their own recall and precision (i.e., there are usually more relevant documents near the top of the list, so the further one looks down the retrieval list, the more likely it is that recall will increase and precision will decrease), direct comparison of recall and precision with sets generated from Boolean retrieval is difficult. As seen in Table 7.6, the clinicians, who were novices, were able to retrieve spectacularly higher recall than any of the expert searchers, although they paid a price in precision (and most likely would not look at all 100 references on the retrieval list anyway). To give a comparison of the novice searchers with retrieval at a level more comparable to that of the experienced searchers, a second set of recall and precision values was calculated with KF’s default retrieval lowered to 15, the average size of Boolean retrieval sets. The levels of recall and precision were still comparable among all groups of expert searchers, with no statistically significant differences. Thus, the approach used by KF clearly showed the potential to be of value to searchers, certainly novices.
7. Evaluation
239
TABLE 7.6. Comparison of Knowledge Finder and ELHILL Users Results definitely relevant only (%) Group
Results definitely/ possibly relevant (%)
Retrieved
Recall
Precision
Recall
Precision
88.8
68.2
14.7
72.5
30.8
14.6
31.2
24.8
25.5
43.8
18.0
37.1
36.1
30.8
59.4
17.0
31.5
31.9
27.0
50.3
10.9
26.6
34.9
19.8
55.2
14.8
30.6
31.4
24.1
48.4
Novice physicians, using KF Novice physicians, KF top 15 Librarians, full MEDLINE Librarians, text words only Experienced physicians, full MEDLINE Experienced physicians, text words only
Source: Hersh and Hickam (1994).
Overlap among retrieval of relevant articles was also assessed, with results similar to those of Haynes et al. As shown in Table 7.7, over half of all relevant references were retrieved by only one of the five searchers, while another quarter were retrieved by two searchers. Well under 10% of relevant references were retrieved by four or five searchers. This study also compared the searching performance of experienced clinician and librarian searchers. It showed that the difference between both these groups and inexperienced clinician searchers was small and not statistically significant. Related to this finding, there appeared to be no benefit associated with the use of advanced MEDLINE searching features, since both experienced clinicians and librarians achieved comparable recall and precision using text-word searching only. In fact, experienced physicians showed a trend toward better recall when they used text words. There was a statistically significant difference for librarians using MEDLINE features over clinicians using MEDLINE features, indicating that these features are of most benefit to librarians. One problem with these studies was the unrealistic situation in which the li-
TABLE 7.7. Overlap Among Five Searchers Number of searchers 1 2 3 4 5 Source: Hersh and Hickam (1994).
Relevant references retrieved 957 474 190 99 42
(53.2%) (26.4%) (10.6%) (5.5%) (2.3%)
240
II. State of the Art
brarian searcher was assessed. As most librarians will note, their work involves more than just the search itself. An equally important aspect is the interview with the user, during which the information needs are explicitly gleaned. Indeed, the study by Saracevic and Kantor (1988b), discussed shortly, notes that performing this interview or having access to it doubles the intermediary searcher’s recall. However, most of these studies (and their searches) take place in clinical settings, where detailed interviews by librarians are impractical. Thus it is valid to compare the end user and the librarian in these settings, if only to use the latter as a point of reference for searching quality. A number of other studies have focused on recall and precision obtained by clinicians using IR systems. Because many of them focused on research systems and/or used simulated queries not employing real users, their results are presented in subsequent chapters. But those employing operational systems and real users are noted here. Probably the very first study to assess recall and precision by clinical end users was a very small study, with five users searching on the same topic (Poisson, 1986). Recall and precision varied widely, ranging, respectively, between 12 and 80% and 58 and 100%. Another study focused on medical students and their likelihood of obtaining relevant articles based on their searching experience (Pao, Grefsheim et al., 1994). Students who searched more frequently were more likely to retrieve relevant articles. Hersh and Hickam (1995) evaluated medical students searching an online version of Scientific American Medicine with a Boolean or natural language interface (as well as an experimental system whose results are discussed in Chapter 9). Twenty-one students searched on 10 queries, which were randomly allocated for each from the 106 queries of the same researchers’ study described earlier (Hersh and Hickam, 1994). The users obtained a slightly higher recall (75.3 vs 70.6%, not statistically significant) and slightly lower precision (14 vs 18.9%, not statistically significant) for the natural language interface. This study also analyzed the relationship between the number of relevant documents and recall, finding that a larger number of relevant documents led, on the average, to users obtaining a lower level of recall. Paris and Tibbo (1998) also compared Boolean and natural language searching with medical documents. Searches were conducted against a database of 1239 MEDLINE documents indexed by the term Cystic Fibrosis. The original queries were formulated by scientists and physicians knowledgeable in the field. From earlier research, the queries had been reformulated in an “optimal” Boolean format to yield the best retrieval. To create natural language queries, the Boolean operators and field limitations were removed. For analysis, the queries were divided into two types: those that used field limiters in the original Boolean queries and those that did not. The results were comparable for both sets. Recall was essentially 100% for queries of both types, most likely because the small document set afforded a broad query statement. Precision was higher for the Boolean searching (40–46% vs 37–39%) while E was lower (0.44–0.49 vs 0.54–0.55). Finally, an RCT compared the searching ability of obstetrics and gynecology residents given either a didactic tutorial, a hands-on tutorial, or no training at all
7. Evaluation
241
(Erickson and Warner, 1998). All groups achieved the same recall and precision, which averaged about 22 and 45%, respectively. 7.4.2.3 Assessing Users with Task-Oriented Measures As mentioned at the end of Chapter 3, a number of investigators have looked for alternatives to relevance-based measures for measuring the quality of IR system performance. One approach has been to give users tasks, such as answering a question. Egan et al. (1989) piloted this approach with a statistics textbook, finding significant performance differences with changes in the user interface. Mynatt et al. (1992) used a similar approach to assess the ability of college students to find answers to questions in an online encyclopedia. Hersh and colleagues have carried out a number of studies assessing the ability of IR systems to help students and clinicians answer clinical questions. The rationale for these studies is that the usual goal of using an IR system is to find an answer to a question. While the user must obviously find relevant documents to answer that question, the quantity of such documents is less important than whether the question is successfully answered. In fact, recall and precision can be placed among the many factors that may be associated with ability to complete the task successfully. The first study by this group using the task-oriented approach compared Boolean versus natural language searching in the textbook Scientific American Medicine (Hersh, Elliot et al., 1994). Thirteen medical students were asked to answer 10 short-answer questions and rate their confidence in their answers. The students were then randomized to one or the other interface and asked to search on the five questions for which they had rated confidence the lowest. The study showed that both groups had low correct rates before searching (average 1.7 correct out of 10) but were mostly able to answer the questions with searching (average 4.0 out of 5). There was no difference in ability to answer questions with one interface or the other. Most answers were found on the first search to the textbook. For the questions that were incorrectly answered, the document with the correct answer was actually retrieved by the user two-thirds of the time and viewed more than half the time. Another study compared Boolean and natural language searching of MEDLINE with two commercial products, CD Plus (now Ovid) and KF (Hersh, Pentecost et al., 1996). These systems represented the ends of the spectrum in terms of using Boolean searching on human-indexed thesaurus terms (CDP) versus natural language searching on words in the title, abstract, and indexing terms (KF). Sixteen medical students were recruited and randomized to one of the two systems and given three yes/no clinical questions to answer. The students were able to use each system successfully, answering 37.5% correct before searching and 85.4% correct after searching. There were no significant differences between the systems in time taken, relevant articles retrieved, or user satisfaction. This study demonstrated that both types of system can be used equally well with minimal training.
242
II. State of the Art
Further research by this group expanded the analysis to include nurse practitioner (NP) students and a myriad of other possible factors that could influence searching. Most of the results from these studies are presented in Section 7.5. However, the searching success rates are worth noting here. Each of these studies used only one IR system, the Ovid system used to search MEDLINE but with links to about 80 full-text journals. The first study focused solely on NP students and used multiple-choice questions from the Medical Knowledge Self-Assessment Program (MKSAP, American College of Physicians, Philadelphia) (Rose, 1998). Each of the 24 subjects answered three out of the eight questions used. Before searching, 25 of the 72 questions (34.7%) were answered correctly and after searching the total of correct responses increased to 42 out of 72 (58.3%). The second study assessed both medical and NP students, with 29 subjects answering three questions each out of a group of 30 (Hersh, Crabtree et al., 2000). The questions, which were phrased in a short-answer format, were obtained from three sources: MKSAP, the Cochrane Database of Systematic Reviews, and a set expressed in actual clinical practice (Gorman and Helfand, 1995). The main success-related results showed that medical students scored higher before and after searching but that both groups improved their scores by the same amount. In the final study, probably the largest study of medical searchers to date, 66 medical and NP students searched five questions each (Hersh, Crabtree et al., 2002). This study used a multiple-choice format for answering questions that also included a judgment about the evidence for the answer. Subjects were asked to choose from one of three answers: • Yes, with adequate evidence • Insufficient evidence to answer question • No, with adequate evidence Both groups achieved a presearching correctness on questions about equal to chance (32.3% for medical students and 31.7% for NP students). However, medical students improved their correctness with searching (to 51.6%), whereas NP students hardly did at all (to 34.7%). Table 7.8 shows that NP students changed with searching from incorrect to correct answers as often as they did from correct to incorrect. These results were further assessed to determine whether NP students had trouble answering questions or judging evidence (unpublished data). To assess this, a two-by-two contingency table was constructed that compared designating the evidence correctly (i.e., selecting yes or no when the answer was yes or no and selecting indeterminate when the answer was indeterminate) and incorrectly (i.e., selecting yes or no when the answer was indeterminate and selecting indeterminate when the answer was yes or no). As seen in Table 7.9, NP students had a higher rate of incorrectly judging the evidence. Thus, since medical and NP students had virtually identical rates of judging the evidence correct when answering the question incorrectly, the major difference with respect to questions answered incorrectly between the groups was incorrect judgment of evidence. Another group to assess ability to use an IR system successfully has been
7. Evaluation
243
TABLE 7.8. Cross-Tabulation of Number and Percentage of Answers Correct vs Incorrect Before and After Searching for All (A), Medical (M), and NP (N) Students Post search Presearch Incorrect A M N Correct A M N
Incorrect
Correct
133 (41.0%) 81 (36.3%) 52 (51.5%)
87 (26.9%) 70 (31.4%) 17 (16.8%)
41 (12.7%) 27 (12.1%) 14 (13.9%)
63 (19.4%) 45 (20.2%) 18 (17.8%)
Percentages represent percent correct within each group of students. Source: Hersh, Crabtree et al. (2002).
Wildemuth, Friedman, and associates. These researchers used a questionanswering approach to assess INQUIRER, a system containing factual databases on several biomedical topics in a medical school curriculum (Wildemuth, deBliek et al., 1995). The study yielded the following findings: • Personal knowledge scores (ability to answer questions before searching) were low. • Database-assisted scores (ability to answer questions with searching) were substantially higher. • There was no relationship between personal knowledge scores and databaseassisted scores; that is, a searcher’s prior knowledge was not associated with better ability to search. • Database-assisted scores persisted at a higher level long after the course was over and personal knowledge scores had returned to lower levels. A further analysis of the final point (in the bacteriology domain) showed that personal knowledge scores were low before the bacteriology course (12%), rose right TABLE 7.9. Cross-Tabularion of Number and Percentage of Evidence Judgments Correct vs Incorrect Before and After Searching for All (A), Medical (M), and NP (N) Students Answer Evidence Incorrect A M N Correct A M N
Incorrect
Correct
138 (42.6%) 84 (37.7%) 54 (53.5%)
0 (0%) 0 (0%) 0 (0%)
36 (11.1%) 24 (10.8%) 12 (11.9%)
150 (46.3%) 115 (51.6%) 35 (34.7%)
Percentages represent percent correct within each group of students.
244
II. State of the Art
after the course (48%), and decreased 6 months later (25%). However, the databaseassisted scores rose linearly over the three assessments from 44% to 57% (deBliek, Friedman et al., 1994). Thus information relevant to a problem can be retrieved from a database long after the ability to directly recall it from memory has faded. A subsequent study from this research showed that when INQUIRER was incorporated into a clinical simulation, there was a trend toward better performance when the system was used and a statistically significant benefit when the record containing the correct answer was displayed (Abraham, Friedman et al., 1999). Another study compared a Boolean and a hypertext browsing interface (Friedman, Wildemuth et al., 1996). While both led to improvements in database-assisted scores, the hypertext interface showed a small but statistically significant benefit. The study by Osheroff and Bankowitz (1993) described earlier also looked at the ability to answer clinical questions, although this investigation considered actual questions generated by real users. Of the 50 questions from eight users, 40% were deemed to be fully answered, 32% partially answered, and 28% not answered at all by the six information resources provided. More than half of the unanswered questions could be answered elsewhere by searchers using other information sources. A variant of the task-oriented approach has also been applied to health care consumers searching the Web (Eysenbach and Kohler, 2002). This study of 17 users given health questions to search found an answer was successfully obtained in 4 to 5 minutes. Although participants were aware of quality criteria for health-related Web sites (e.g., source, professional design, scientific language), they did not appear to apply these criteria when actually visiting sites.
7.5 Factors Associated with Success or Failure Although determining how well users can perform with IR systems is important, additional analysis focusing on why they succeed or fail is important, not only in figuring out how to best deploy such systems but also for the sake of determining how to improve them. This section focuses on two related groups of analyses. In the first group are studies attempting to determine the factors associated with successful use of systems, while the second group consists of analyses of why users fail to obtain optimal results.
7.5.1 Predictors of Success One of the earliest and most comprehensive analyses of predictors of success was from outside the healthcare domain, but its results set the stage for further work. Saracevic et al. (Saracevic and Kantor, 1988a,b; Saracevic, Kantor et al., 1988) recruited 40 information seekers, each of whom submitted a question, underwent a taped interview with a reference librarian to describe his or her problem and intended use of the information, and evaluated the retrieved items for relevance as well as the search in general. Each question was searched by nine intermediaries. Up to 150 retrieved items were returned to the users, who rated them as relevant, partially relevant, or not relevant to their information need.
7. Evaluation
245
All results were framed in the context of the odds that retrieved items would be judged relevant by the users. Some of the factors that led to statistically significantly higher odds of documents being judged relevant were as follows: 1. A well-defined problem posed by a user who was very certain that the answer would be found 2. Searches limited to answers in English and requiring less time to complete 3. Questions that were initially not clear or specific but were complex and had many presupposed concepts 4. Searchers who scored well on tests of word association (Remote Associated Test) and preferred to learn in abstract as opposed to concrete terms 5. Searches with increased evaluation of intermediate results, without too many terms or taking excessive time 6. Documents retrieved by multiple searches or searchers 7. Answers that had high utility for the user, as measured by benefits in time, money, or problem resolution One finding from this study cast a warning over studies using expert replication of end-user searches based on indirect information need statements by users, two of which were previously discussed. The investigators found that a tape of the interview between user and librarian enhanced recall, but recall was diminished by a written statement of the original question. This finding has implications for studies that compare clinician searchers with librarians based on the latter’s replication of searches using statements of information need, indicating that librarians may be at a disadvantage when they cannot interview the user. Of course, as was noted before, such interviews are likely to be impractical in busy clinical settings, especially for routine information needs. Another finding of interest in this study was a low overlap in search terms used (27%) and items retrieved (17%) for a given question, a finding similar to that of McKibbon et al. for healthcare searchers (McKibbon, Haynes et al., 1990). However, Saracevic and Kantor did determine that the more searchers a document was retrieved by, the more likely that document was to be judged relevant. This has been verified in healthcare searchers by a further analysis of the Hersh and Hickam MEDLINE study (unpublished data). Also assessing the factors leading to searching success was the study of Wildemuth et al. (1995) described earlier, which evaluated medical student performance in searching factual databases in bacteriology, toxicology, and pharmacology. The authors found that relevance-based measures (recall and precision), term overlap (students selecting terms overlapping with those known to lead to retrieval of records containing the answer), and efficiency (as measured by time) had a positive correlation with successful answering of questions, while personal domain knowledge (as measured by a test in which the system was not used) did not. The positive correlation of search success with relevance and the lack of correlation with personal domain knowledge was also found by Saracevic and Kantor. Another group assessed in the literature has been clinical pharmacists. As noted earlier, Abate et al. (1992) found that clinical pharmacists were much more likely
246
II. State of the Art
TABLE 7.10. Model of Factors Associated with Successful Question-Answering by Means of MEDLINE Searching and Evidence Judgment Variable Basic ID Question Demographics School Age Sex MedSpec YrsNurse NPSpec Computer experience CompHrs ProdSW OwnPC Modem Internet Computer attitudes PracHard EnjComp Searching experience LitSrch WebSrch WebMed TrainEBM HrdMsh UsedMsh HrdSH UsedSH HrdExp UsedExp HrdPT UsedPT Cognitive traits VZ2 RL1 V4 Search mechanics Time Stacks Sets Retrieved Viewed FTViewed User satisfaction Quis Relevance Helpful Justified Precision Recall
Definition User ID Question number School student enrolled (medical or NP) (Years) (Male, female) Medical specialty will be entering (from list, M students only) Years worked as a nurse (yars, NP students only) Nurse practitioner specialty (from list, NP only) Computer usage per week (hours) Use productivity software once a week (yes, no) Own a personal computer (yes, no) Personal computer has a modem (yes, no) Personal computer connects to Internet (yes, no) Practice easier or harder with computers (easier, harder) Enjoy using computers (yes, no) Literature searches per month (number) Web searches per month (number) Web searches for medical information per month (number) Ever had instruction in EBM (yes, no) Ever heard of MeSH terms (yes, no) Ever used MeSH terms (yes, no) Ever heard of subheadings (yes, no) Ever used subheadings (yes, no) Ever heard of explosions (yes, no) Ever used explosions (yes, no) Ever heard of publication types (yes, no) Ever used publication types (yes, no) Spatial reasoning test (score) Logical reasoning test (score) Vocabulary test (score) Time to complete question (minutes) Whether searcher went to stacks (true, false) Sets in MEDLINE search (number) Number of articles retrieved by user in terminal set(s) Total MEDLINE references viewed (number) Full-text document viewed (number) QUIS average for this searcher (number) Citations helpful answer (number) Citations justifying answer (number) User’s precision for retrieval of definitely or possibly relevant articles User’s recall for retrieval of definitely or possibly relevant articles
7. Evaluation
247
TABLE 7.10. (Continued) Variable Answers Order Answer Type PreAns PreCorr PreCert PostAns PostCorr PostCert Preferred
Definition Order question done by this search (2–6; first search ignored because all did same one) Answer to question (yes, no, or insufficient evidence to determine) EBM type (therapy, diagnosis, harm, prognosis) Answer before searching (yes, no, or insufficient evidence to determine) Answer correct before search (true, false) Certainty of answer before search (1 high, 5 low) Answer after searching (yes, no, or insufficient evidence to determine) Answer correct after search (true, false) Certainty of answer after search (1 high, 5 low) Who user would seek for answer (from list)
Source: Hersh, Crabtree et al. (2002).
to use an IR system than clinicians. The investigators also compared success rates for finding information. They found that less frequent searchers (those—mostly physicians—who did six or fewer searches in the 19-month observation period) had no success in finding information 44% of the time compared with 31% of the time for more frequent searchers. In their study of nurses’ searching, Royle et al. (1995) asked subjects whether their searches were successful. While 83% of searches were completed to the point of “answering the question,” only 42% were deemed “successful.” Factors correlating with success included taking more time but rated as worth the time, accessing a bibliographic (as opposed to full-text) database, being done for educational (as opposed to patient care) purposes, and being done on disease-related or psychosocial topics. The most comprehensive analysis of factors relating to searching success has been carried out by Hersh and colleagues. As noted in Section 7.4.2.3, three studies have investigated the factors associated with successful searching by medical and NP students (Rose, 1998; Hersh, Crabtree et al., 2000; Hersh, Crabtree et al., 2002). The remainder of this section focuses on the most recent of these studies because it was the largest and most comprehensively analyzed. The first task in looking for factors associated with successful searching is to develop a model of the various factors. Section 3.2.2 presented a general model of factors influencing successful searching. Actual operationalization of such a model requires modification of the model to the specific aspects of the study. For example, there needs to be an ultimate measure of success in using the IR system: namely, one or more dependent (outcome) variables. Likewise, there will be specific independent (predictor) variables specific to the study, such as the demographic characteristics of the users, the type of retrieval system, and the instruments used to assess various user attributes. Table 7.10 presents the model of factors for the study described in the third study by Hersh and colleagues (Hersh, Crabtree et al., 2002), which was refined from their initial two studies (Rose, Crabtree et al., 1998; Hersh, Crabtree et al.,
248
II. State of the Art
2000). In that model, the dependent variable was successful answering of the clinical question. The independent variables were grouped in several different categories. The first consisted of identifiers for the user and question searched, as well as the order of the question. The next category covered demographic variables, some of which were generic (e.g., age and sex), while others were specific to the study population (e.g., school enrolled and years worked as a nurse). There were also categories for computer experience, computer attitudes, and searching experience. The searching experience factors included not only general amounts of literature and Web searching but also specific knowledge of and experience with advanced features of MEDLINE. The model also included assessment of cognitive factors, since these have been shown to be associated with searching performance not only in the studies of Saracevic et al. cited earlier but in other work as well. Three factors were included in this model because they have been found to be associated with successful use of computer systems in general or retrieval systems specifically. • Spatial visualization: The ability to visualize spatial relationships among objects has been associated with retrieval system performance by nurses (Staggers and Mills, 1994), ability to locate text in a general retrieval system (Gomez, Egan et al., 1986), and ability to use a direct manipulation (3D) retrieval system user interface (Swan and Allan, 1998). • Logical reasoning: The ability to reason from premise to conclusion has been shown to improve selectivity in assessing relevant and nonrelevant citations in a retrieval system (Allen, 1992). • Verbal reasoning: The ability to understand vocabulary has been shown to be associated with the use of a larger number of search expressions and highfrequency search terms in a retrieval systems (Allen, 1992). The cognitive traits were assessed by validated instruments from the Educational Testing Service (ETS) Kit of Cognitive Factors (Ekstrom, French et al., 1976) (ETS mnemonic in parentheses): • Paper folding test to assess spatial visualization (VZ-2) • Nonsense syllogisms test to assess logical reasoning (RL-1) • Advanced vocabulary test I to assess verbal reasoning (V-4) Other important categories in the model included intermediate search results (i.e., results of searching performance that ultimately influence the user’s ability to successfully answer the question). One of these is search mechanics, such as the time taken, the number of Boolean sets used in the searching process, the number of articles retrieved in the “terminal” set, and the number of MEDLINE references and full-text articles viewed by the user. Another intermediate category is user satisfaction. In this study, user satisfaction was measured with the Questionnaire for User Interface Satisfaction (QUIS) 5.0 instrument, which measures user satisfaction with a computer system, providing a score from zero (poor) to nine (excellent) on a variety of user preferences (Chin, Diehl et al.,
7. Evaluation
249
1988). The overall user satisfaction is determined by averaging the scores of all the preferences. The next group of factors addressed the relevance of the retrieval set. These measures, including recall and precision, were considered to be intermediate outcome measures relative to the ultimate outcome measure of successfully answering the question. This is in distinction to the many retrieval studies that assess recall and precision as the final outcome measures. The final category of factors contains variables associated with the answer, such as the answer itself, the EBM type, whether the user gave the correct answer before or after searching, and the user’s certainty of the answer. With this model in place, the investigators proceeded to collect data for all the variables, ultimately aiming to determine which were associated with successful searching. The questions used for searching were taken from sources as described in Section 7.4.2.3. Subjects for the experiment came from a convenience sample of senior medical students from Oregon Health & Science University (OHSU) and NP students from OHSU and Washington State University, Vancouver. The general experimental protocol was to participate in three sessions: a group session for the administration of questionnaires and orientation to MEDLINE, the techniques of EBM, and the experiment, followed by two hands-on sessions in which the subjects would do the actual searching, read the articles, and answer the questions. At the group session, subjects were first administered a questionnaire on the personal characteristics and experience with computers and searching factors from Table 7.10. Next they were tested for the cognitive attributes measured by the ETS instruments. The session also included a brief orientation to the searching task of the experiment as well as a 30-minutes hands-on training session covering basic MEDLINE and EBM principles. The following searching features were chosen for coverage: MeSH headings, text words, explosions, combinations, limits, and scope notes. The overview of EBM described the basic notions of framing the appropriate question, determining which evidence would be most appropriate for a given question, and identifying the best searching strategies for finding such evidence. The hands-on sessions took place 2 to 4 weeks later in the OHSU library. All searching was done by means of the Web-based version of the Ovid system, accessing MEDLINE and a collection of 85 full-text journals. The logging facility of Ovid enabled all search statements to be recorded, as well as the number of citations presented to and viewed by the user in each set. In the two hands-on sessions, subjects searched six questions. For the first question of the first session, all users searched the same “practice” question, which was not graded. The remaining five questions (the last two from the first session and all three from the second session) were selected at random from the pool of 20 questions. Question selection was without replacement; that is, the same pool of questions was used for four consecutive searchers. Before searching, subjects were asked to record a presearch answer and a rating of certainty on a one (most) to five (least) certain scale for the questions on
250
II. State of the Art
which they would search. They were then instructed to perform their searching in MEDLINE and to obtain any articles they wished to read, either in the library stacks or in the full-text collection available online. They were asked to record on paper their postsearch answer, their certainty of the correctness of the answer (on a scale of 1 to 5), which article(s) justified their answer, and any article that was looked at in the stacks or in full-text on the screen. Upon completion of the searching, they were administered the QUIS instrument. Overall time taken was measured for each question. The search logs were processed to count the number of search cycles (each consisting of the entry of a search term or Boolean combination of sets) and the number of full MEDLINE references viewed on the screen. After all the hands-on searching sessions had been completed, the actual answers to the questions were determined by the research team. Generalized estimating equations (GEEs) were used in the statistical analysis because since each searcher had answered five different questions, individual questions were not independent (Hu, Goldberg et al., 1998). The recall–precision analysis was carried out by means of approaches most commonly used in the IR literature, including the use of domain experts to judge relevance, pooling of documents within a single query to mask the number of searchers who retrieved it, and assessment of interrater reliability. To calculate recall and precision from the myriad of Boolean sets generated by searching in each question, the researchers identified “end queries” of the search process, which were defined as the terminal point of a search strategy. This was the point at which the subject stopped refining (creating subsets of) a search and began using new search terms or new combinations of search terms. All relevance judgments were done by means of the MEDLINE record distributed in an electronic file, though judges were encouraged to seek the full text of the article in the library if necessary. Sixty-six searchers, 45 medical students and 21 NP students, performed five searches each. There were 324 searches analyzed. Several differences between medical and NP students were seen (Table 7.11). Use of computers and use of productivity software were higher for NP students, but searching experience was higher for medical students. Medical students also had higher self-rating of knowledge and experience with advanced MEDLINE features. The NP students tended to be older and all were female (whereas only half the medical students were female). Medical students also had higher scores on the three cognitive tests. In searching, medical students tended to have higher numbers of sets but lower numbers of references viewed. They also had a higher level of satisfaction with the IR system, as measured by QUIS. As noted in Section 7.4.2.3, the rates of correctness before searching for medical and NP students were virtually identical (32.3 vs 31.7%). But following searching, medical students had a higher rate of correctness than NP students (51.6 vs 34.7%). The goal of the statistical analysis was to determine the factors associated with successful searching, as defined by the outcome variable of correct answer after searching (PostCorr). After screening individual variables for their p values, using analysis of variance (ANOVA) techniques for continuous variables and chi-square tests for categorical variables, the researchers built
7. Evaluation
251
TABLE 7.11. Values of All Searching-Related Factors for All Searches, Stratified by Student Type Student type Variable n School (M) Age (years) Sex (M) CompHrs (hours) ProdSW (% true) OwnPC (% true) Modem (% true) Internet (% true) PracHard (% true) EnjComp (% true) LitSrch (monthly) WebSrch (monthly) WebMed (monthly) TrainEBM (% true) HrdMsh (% true) UsedMsh (% true) HrdSH (% true) UsedSH (% true) HrdExp (% true) UsedExp (% true) HrdPT (% true) UsedPT (% true) VZ2 (score) RL1 (score) V4 (score) Time (minutes) Stacks (Used) Sets (number) Retrieved (articles) Viewed (articles) FTViewed (articles) QUIS (score) Helpful (articles) Justified (articles) Precision (calculated) Recall (calculated) Order (2–6) PreCorr (% true) PreCert (1–5) PostCorr (% true) PostCert (1–5)
All
Medical
NP
66 68% 34 35% 8.4 79 92 86 73 68 88 5.7 9.0 2.5 57 91 73 87 60 79 49 56 22 12.7 12.6 22.6 32 28% 19.1 26 8.6 0.94 6.6 2.3 1.6 29% 18% 4.0 32 3.2 46 2.0
45 100% 31 51% 7.4 72 91 82 67 72 87 7.1 11.0 2.4 53 91 75 85 60 83 60 69 29 14.2 14.2 23.1 30 32% 21.6 23 7.1 0.87 6.8 2.3 1.7 30% 18% 4.0 32 3.2 51 2.0
21 0% 41 0% 10.7 95 95 95 85 60 90 2.6 4.5 2.7 67 92 67 92 58 72 25 28 5.0 9.2 9.0 21.6 39 20% 13.7 32 12.0 1.1 6.3 2.3 1.4 26% 20% 4.0 31 3.2 35 2.1
Source: Hersh, Crabtree et al. (2002).
a GEE model (Table 7.12). The final model showed that PreCorr, VZ2, Used2, and Type were significant. For the variable Type (EBM question type), questions of prognosis had the highest likelihood of being answered correctly, followed by questions of therapy, diagnosis, and harm. The analysis also found
252
II. State of the Art
TABLE 7.12. Values of Searching-Related Factors Stratified by Correctness of Answer, Along with Screening p Value for Statistical Analysis Variable
Incorrect
Correct
Screening p value
174 62% 34 29% 8.0 79 93 87 73 68 87 4.9 7.7 2.4 59 90 70 85 56 75 42 49 16 12.0 11.9 22.2 33 24 21 27 9.1 0.9 6.6 2.4 1.6 28% 18% 4.0 24 3.1 2.0
150 77% 33 42% 8.8 79 91 85 72 69 88 6.7 10.5 2.7 56 93 77 89 64 84 57 65 28 13.4 13.4 23.1 32 33 17 25 8.0 1.0 6.7 2.2 1.6 29% 18% 4.0 42 3.3 2.0
N/A .01 .55 .03 .48 .95 .62 .74 .84 .92 .88 .16 .05 .46 .69 .10 .14 .18 .16 .02 .01 .02 .02 .00 .10 .21 .41 .10 .29 .74 .14 .52 .78 .23 .57 .99 .61 .95 .00 .16 .66
n School (M) Age (years) Sex (M) CompHrs (hours) ProdSW (% true) OwnPC (% true) Modem (% true) Internet (% true) PracHard (% true) EnjComp (% true) LitSrch (monthly) WebSrch (monthly) WebMed (monthly) TrainEBM (% true) HrdMsh (% true) UsedMsh (% true) HrdSH (% true) UsedSH (% true) HrdExp (% true) UsedExp (% true) HrdPT (% true) UsedPT (% true) VZ2 (score) RL1 (score) V4 (score) Time (minutes) Stacks (Used) Sets (number) Retrieved (articles) Viewed (articles) FTViewed (articles) QUIS (score) Helpful (articles) Justified (articles) Precision (calculated) Recall (calculated) Order (2–6) PreCorr (% true) PreCert (1–5) PostCert (1–5) Source: Hersh, Crabtree et al. (2002).
that the VZ2 and School variables demonstrated multicollinearity; that is, they were very highly correlated, and once one was in the model, the other did not provide any additional statistical significance. The variable VZ2 was included in the final model because it led to a higher overall p value for the model than
7. Evaluation
253
School. Next, a similar analysis was done to find the best model using the 220 searches when the subject did not have the right answer before the MEDLINE search. The final best model was very similar to the model for all questions, with PreCorr obviously excluded. Again, VZ2 and School demonstrated high multicollinearity. In the recall–precision analysis, there were three relevance judgments made on question/document pairs on the scale of not relevant, possibly relevant, and definitely relevant. The reliability of the relevance judges, measured by Cronbach’s , was 0.81. Final document relevance was assigned according to the following rules: (1) if all judges agreed, the document was assigned that rating, (2) if two judges greed, the document was assigned that rating, and (3) if all three judges disagreed, the document was assigned the possibly relevant rating. The relevance judgments were then used to calculate recall and precision for each user/question pair. As seen in Table 7.11, there was virtually no difference in recall and precision between medical or NP students. Likewise, Table 7.12 shows no difference in recall and precision between questions that were answered correctly and incorrectly. A number of conclusions were drawn from this study. First, users spent an average of more than 30 minutes conducting literature searches and were successful at correctly answering questions less than half the time. Whereas medical students were able to use the IR system to improve question answering, NP students were led astray by the system as often as they were helped by it. The study also found that experience in searching MEDLINE and spatial visualization ability were associated with success in answering questions. A final finding was that the often-studied measures of recall and precision were virtually identical between medical and NP students and had no association with correct answering of questions. These results showed some similarities and some differences with respect to the prior studies. The original study, focused on NP students only, showed that attitudes toward computers were the strongest factor association with searching success (Rose, Crabtree et al., 1998). None of the demographic, experience, or cognitive variables were associated with success. The second study found that the most predictive factor of successful question answering was student type (medical vs NP); spatial visualization showed a trend toward predicting successful answering, but the result was short of statistical significance (Hersh, Crabtree et al., 2000). The second study differed from the final one in showing that both medical and NP students had their question-answering ability improved by the IR system. As with the final study, literature searching experience was associated with answering questions correctly. Factors that did not predict success in the second study included age, gender, general computer experience, attitudes toward computers, other cognitive factors (logical reasoning, verbal reasoning, and association fluency), Myer–Briggs personality type, and user satisfaction with the IR system. In summary, these studies demonstrated that students in clinical training were at best moderately successful at answering clinical questions correctly with the
254
II. State of the Art
assistance of searching the literature. Possible reasons for the limited success of question answering include everything from inadequate training to an inappropriate database i.e., a large bibliographic database instead of more concise, synthesized references) to problems with the retrieval system to difficulties in judging evidence.
7.5.2 Analysis of Failure The attempt to determine why users do not obtain optimal results with IR systems is called failure analysis. A number of such analyses have been carried out over the years. In his original MEDLINE study, Lancaster (1968) performed a detailed failure analysis, which he divided into recall (failure to retrieve a relevant article) and precision (retrieval of nonrelevant article) failures. For both types of failure, Lancaster cataloged problems related to indexing (i.e., problems with the indexing language or assignment of its terms) and retrieval (i.e., problems with search strategy). The particular problems, along with their frequencies, are shown in Table 7.13. More recent failure analyses have focused on end-user searching. Kirby and Miller (1986) assessed end-user searching on BRS Colleague (now defunct) at the Medical College of Pennsylvania. Library users who did their own searches were offered a free search on the same topic by an intermediary. Users deemed 39% of the searches “successful” and 61% “incomplete.” There were no differences between the two categories of searches in terms of time spent or system features used. The successful searches were ones likely to succeed with a simple search statement of two to three concepts. Incomplete searches were mostly due to problems of “search strategy,” such as failure to use MeSH terms or specify alternative approaches to formulating the question. Kirby and Miller and their colleagues also evaluated end-user searching with Compact Cambridge MEDLINE (now defunct) (N. Miller, Kirby et al., 1988). Search statements were analyzed for identification of errors and “missed opporTABLE 7.13. Recall and Precision Failures in MEDLINE Recall failures Indexing language—lack of appropriate terms (10.2%) Indexing—indexing not sufficiently exhaustive (20.3%), indexer omitted important concept (9.8%), indexing insufficiently specific (5.8%) Retrieval—searcher did not cover all reasonable approaches to searching (21.5%), search too exhaustive (8.4%), search too specific (2.5%), selective printout (1.6%) Precision failures Indexing language—lack of appropriate specific terms (17.6%), false coordinations (11.3%), incorrect term relationships (6.8%) Indexing—too exhaustive (11.5%) Retrieval—search not specific (15.2%), search not exhaustive (11.7%), inappropriate terms or combinations (4.3%) Inadequate user–system interaction (15.3%) Source: Lancaster (1968). (Courtesy of the National Library of Medicine.)
7. Evaluation
255
tunities,” where better strategies could have resulted in more documents retrieved. Searching errors included the following: 1. 2. 3. 4. 5. 6.
Designating a term as a MeSH term when it was not Entering subject terms in the author or unique identifier fields Misspelling a word Using a stop word or truncation symbol in a phrase search Inappropriate back-referencing of an earlier set Entering an author name in incorrect form
Missed opportunities included the following: 1. Nonuse of truncation 2. Failure to use appropriate MeSH heading 3. Incomplete specification of all fields (e.g., searching for a text word in the title or abstract field only) The analysis of 500 search statements entered by an unknown number of users found that 45% retrieved no postings. About 37% of all search statements had one or more errors, while 77% had missed opportunities. The most common errors were incorrect entry of MeSH headings (21% of all searches), subject searching in the unique identifier field (9%), and misspellings (6%). These results were similar to those obtained by Sewell and Teitelbaum (1986), who assessed use of the NLM command-line interface by pathologists and pharmacists at the University of Maryland Health Sciences Library. They found that most users employed search terms combined with the AND operator. While text words were generally used correctly, MeSH terms were used incorrectly 23% of the time. The most frequent reasons for missed opportunities were failure to use explosions and subheadings appropriately.
TABLE 7.14. “Missed Opportunities” in 58 MEDLINE Searches by Medical Students Missed opportunity Should have used MeSH term Made an illogical Boolean combination Should have used subheading Should have used a different proximity operator Should have exploded MeSH term Should have limited term to major descriptor Should have added synonyms with OR Should have used term truncation Should have limited term to a specific field Should have used broader term Should have used narrower term Should not have used MeSH term (none available) Should have used full database Should have limited search Other Source: Wildemuth and Moore (1995).
Frequency 90 34 31 26 24 14 12 11 10 7 6 3 2 2 2
256
II. State of the Art
Wildemuth and Moore (1995) assessed their medical student searches of MEDLINE, also using BRS Colleague, for missed opportunities. Table 7.14 shows the most common of these. By a large margin, the move that would have improved the search most often was use of a MeSH term, followed by better use of Boolean operators and addition of subheadings. One interesting subsequent finding came from a linear regression analysis that attempted to assess the association between ratings of search effectiveness and search behavior, including number of search statements, number of terms used, number of citations displayed, frequency of various search moves, prior searching experience, and science versus nonscience undergraduate major. None of these factors was found to be a strong predictor of searching success on the questions in this study. A number of other failure analyses focused on the NLM’s Grateful Med, which was designed for end users. A large study at the NLM focused on searches retrieving no articles (“no postings”) (Kingsland, Harbourt et al., 1993). This was found to occur with 37% of Grateful Med searches performed in April 1987 and 27% of searches from September 1992. The 1987 searches were analyzed in more detail, with the finding that 51% of searches used excessive ANDs, in that no documents contained the intersection of all search terms ANDed together by the searcher. Other reasons for empty sets include inappropriate entering of author names (15%), term misspellings (13%), punctuation or truncation errors (11%), and failed title searches (6%). The investigators did not assess how many “no postings” were due to an absence of material on the topic. Other errors made included the following: 1. Inappropriate use of specialty headings (e.g., using the term Pediatrics, which is intended to represent the medical specialty, to search for children’s diseases) 2. Incorrect use of subheadings (e.g., using Management instead of Therapy when searching for articles about treatment of a disease) 3. Not using related terms, either in the form of text words (e.g., adding a term like cerebr: or encephal: to the MeSH heading Brain) or MeSH crossreferences (e.g., adding terms like Bites and Stings or Dust to Allergens) Walker et al. (1991) assessed 172 “unproductive” Grateful Med searches at McMaster University in 1987–1988, dividing problems into the categories of search formulation (48%), the Grateful Med software itself (41%), and system failure (11%). While half the search formulation problems were due to an absence of material on the topic, the most common errors were found to be use of low postings terms, use of general terms instead of subheadings, and excessive use of AND. Problems specific to Grateful Med included inappropriate use of the title line (i.e., unwittingly typing a term on the title line, thus limiting retrieval to all articles with that term in the title) and the software’s automatic combining of words of the subject line(s) with OR, so that the phrase inflammatory bowel disease was searched as inflammatory OR bowel OR disease. Mitchell et al. (1992) assessed searcher failures of Grateful Med by medical students in biochemistry and pathology courses. An analysis of searches with no post-
7. Evaluation
257
ings showed that the most common error was failure to use MeSH terms that could have resulted in retrieval of relevant articles. The most common reasons for excessive postings were searching on only one concept and the OR of words on the subject line described in the preceding paragraph. A more recent study comparing MEDLINE systems on the Web demonstrates that many of the problems and challenges for both expert and novice searchers alike are still present. Jacobs et al. (1998) compared a number of Web-based interfaces to MEDLINE, giving criteria for evaluating them and carrying out a small descriptive study. In fact, they found that many of the features designed to make searching simple for users (often implemented by default and unable to be turned off) led to inappropriate results. Examples of problems included the following. 1. Default searching of all fields, leading to searches like head CT retrieving articles about the head by an author in the state of Connecticut (abbreviated CT) 2. Arbitrary cutoff of references displayed when using partial-match searching (e.g., at 200 or 1000 regardless of total number retrieved) 3. Elimination of the word A as a stopword in Vitamin A 4. Little help in finding MeSH terms 5. Inability to use subheadings Not all failure analyses have looked at bibliographic databases. In their study of full-text retrieval performance described earlier, McKinin et al. (1991) also assessed the reasons for full-text retrieval failures. About two-thirds of the problems were due to search strategy, in that the concepts from the search were not explicitly present in the document or an excessively restrictive search operator was used. The remaining third were due to natural language problems, such as use in the documents of word variants, more general terms, synonyms, or acronyms. In their assessment of clinicians attempting to use a computer workstation to answer clinical questions, Osheroff and Bankowitz (1993) analyzed the problems users had in dealing with questions they were unable to answer. The most common problem expressed (57% of the time) was poor interaction with the database, either because it was incomplete or because the user entered a poor search. Other problems included noncurrent database (14%), navigational difficulties with the software (15%), and no new information obtained (7%). Even users who obtained partial or full answers to their questions had occasional concerns that their answer was incomplete (63 and 10%, respectively) or complained that the workstation was difficult to navigate (32 and 10%, respectively).
7.6 Assessment of Impact It was noted in Chapter 3 that the true measure of an IR system’s success should be how much impact it has on the searcher’s information problem, be it assisting with a diagnosis or choosing the best therapy. As this chapter indicates, there
258
II. State of the Art
have been far more studies of the details of the user–system interaction than of how well that interaction assists in solving a problem, making a correct decision, and so on. This state of affairs is understandable, given that studies of impact are not only costly and time-consuming but also are potentially contaminated by confounding variables unrelated to the system. Many variables play a role in the outcome of a medical diagnosis and intervention, and even if the use of IR systems is controlled (i.e., physicians are randomized to one system or another), there may be other differences in patients and/or healthcare providers that explain differences in outcome independent of IR system use. The main approach to assessing impact has been the use of questionnaires asking providers questions such as whether the system led to a change in a decision, action, or outcome. The limitations of this approach, of course, are related to selective recall of those who reply to such surveys and/or potential differences among those who do and do not reply. Three studies have involved administering questionnaires to clinician users of hospital library services. King (1987) chose random samples of physicians, nurses (registered nurses and NPs only), and other healthcare professionals from eight Chicago-area hospitals to query them on the value of library services in their hospital. The sample sizes were chosen based on the relative numbers of each provider (i.e., 49% physicians, 40% nurses, and 11% other providers). Although the survey response rate was low (57%), it was found that while physicians used the library more often than nurses or other providers, all groups reported that information obtained was of clinical value, led to better-informed decisions, and contributed to higher quality care more than 90% of the time. Nearly three-quarters of each type of provider reported that the information would definitely or probably persuade the person to handle the case that prompted the library visit in a manner different from that initially contemplated. Marshall (1992) performed a similar study in 1992, assessing the impact of the hospital library on physician decision making in the Rochester, New York area. Physicians were recruited to use the library in their hospital and to complete a questionnaire describing its impact. Although Marshall’s response rate of 51% was low, like King’s, those who did respond indicated a generally positive role for the library. More than 80% indicated that they had handled some aspect of a case differently, most frequently in choice of tests, choice of medications, and advice given to the patient from the approach they had considered before consulting the library. Among the aspects of patient care the library information allowed the physicians to avoid were additional tests and procedures, surgery, hospital admission, and even patient mortality. Mathis et al. (1994) administered a similar survey to library patrons in Michigan and likewise found that 85% of searches were valuable to patient care, with 56% changing the way the case was handled. Among the frequent benefits were change in advice given to patients, avoidance of unnecessary tests and procedures, and modification of drug prescriptions. Other studies have also attempted to assess whether use of libraries or IR systems led to changes in patient care decisions. Veenstra (1992), for example, found
7. Evaluation
259
that a clinical medical librarian added to teaching services at Hartford Hospital was able to find information that affected patient care 40 to 59% of the time. In their study of Grateful Med introduced in clinical settings, Haynes et al. (Haynes, McKibbon et al., 1990) found that 47% of system uses led to finding information which changed the course of patient care. One of the most comprehensive assessments of the impact of MEDLINE was commissioned by the NLM, using the “critical incident technique,” in which users were prompted to recall a recent search that was effective or not (Lindberg, Siegel et al., 1993). The analysis of this survey focused on the 86% of searches deemed effective by a sample of 552 end-user physicians, scientists, and others. The most common impact of the information obtained was to develop an appropriate treatment plan (45%), followed by recognizing or diagnosing a medical problem or condition (22%), implementing a treatment plan (14%), and maintaining an effective patient–physician relationship (10%). As noted earlier, the problem with survey data is that such information depends on the memory of the surveyees and also, owing to incomplete response rates, may not represent a snapshot of the entire population. For this reason, Klein et al. (1994) attempted to determine whether MEDLINE searching had an impact on economic indicators, in this case hospital charges and length of stay (LOS). The investigators used a case-control approach for 192 hospital admissions where MEDLINE searching was known to have been done for patient care reasons. When matched for diagnosis-related group (DRG) and LOS, those that had “early” literature searches (done during the first three-quarters of the standard LOS) had statistically significant lower costs than those done “late.” Of course this study had many potential confounding variables, which do not detract from the comprehensive statistical analysis as much as they highlight the difficulty in doing such studies. For example, the reason for searching was unknown, and the later searches may have represented a patient becoming more ill later in the course of the hospitalization. Furthermore, as the authors themselves noted, there may have been other characteristics about the patients or their providers that were not known. A final problem with this study was the use of a case-control design when an RCT would have been more effective for assessing the benefits of the intervention. Nonetheless, more studies that attempt to answer questions such as the ones assessed in this study are needed to determine the real value of IR systems in healthcare settings. Research in other types of general-information intervention has similarly shown the effects to be modest at best. In an often-cited systematic review, Davis et al. (1995) found that traditional passive continuing medical education was ineffective in changing physician behavior. Likewise, Shiffman et al. (1999) found the benefits of using clinical practice guidelines to be variable (i.e., showing benefit in some interventions but not in others). Another systematic review, focusing on computer interventions in primary care, found the most effective ones to be targeted interventions for common conditions, such as preventive care measures (Mitchell and Sullivan, 2001). In keeping with the philosophy of EBM, could IR systems be evaluated by the
260
II. State of the Art
appropriate means to assess interventions (i.e., the RCT)? One major impediment to an RCT is that usage of IR systems is heterogeneous. That is, they are used for a variety of information needs. Furthermore, their impact on a given patient’s care is usually only indirect or at best one of numerous variables that affect patient outcomes. Other studies in medical informatics have demonstrated that welldesigned studies have been used successfully to assess the application of information technology in the healthcare setting for many years (e.g., Tierney, Miller et al., 1993; Bates, Leape et al., 1998; Evans, Pestotnik et al., 1998). But these studies have focused on specific clinical interventions guided by explicit rules in decision support systems. One possibility is to use “surrogate” outcomes, which assess not whether patients actually improve but, rather, whether physicians perform appropriate actions, such as ordering of tests or treatments. This approach has been used to assess the benefit of electronic medical records as documented by a systematic review (Jerant and Hill, 2000). Of course, it has been noted that surrogate outcomes do not always predict actual outcomes, so such findings must be used cautiously (D’Agostino, 2000).
7.7 What Has Been Learned About IR Systems? Although the beginning of this chapter lamented that the amount of evaluation research has not kept pace with the growth of IR systems and their use, the sum of research does give many insights into how systems are used, how often they are successful, and where they can be improved. The chapter ends with a review of the research findings in the context of the six questions around which the other sections were organized. While it is clear that IR systems will be used in clinical settings, their impact is modest, and they are used to meet only a small fraction of clinicians’ information needs. This does not mean that the systems are not valuable when they are used, but it does challenge the proponents of computer usage in medicine to implement systems that have more clinically pertinent information and are easier to use. Another consistent finding is that in most settings, bibliographic databases are still used more frequently than full-text resources, but this may change as more textbooks, journals, practice guidelines, and so on become accessible online. Studies on the type of usage are not surprising. Clinicians tend to pose patientoriented questions to IR systems, while those in academic settings tend to ask more research-related questions. Clinicians are most likely to search on questions related to therapy followed by overviews of diseases. User satisfaction with systems tends to be high, although there are some concerns that usage drops over time and/or when it becomes inconvenient. System-oriented studies of searching quality have shown that databases vary in coverage by topic. They also show that searching the same database with a different system gives divergent results, an outcome perhaps exacerbated by the new features modern systems have added to make searching simpler for end users.
7. Evaluation
261
In addition, achieving maximum recall, as needed for identifying RCTs for a meta-analysis, continues to be very difficult. It is likely that full-text searching leads to better recall, but at a significant price of lower precision. The capabilities of Web search engines are still largely unknown, but users must grapple with large amounts of information of varying quality, aimed at a diversity of audiences. User-oriented studies have shown that searchers are generally able to learn to search, but they make significant numbers of errors and miss many opportunities. Studies of recall and precision show that most searches do not come anywhere close to retrieving all the relevant articles on a given topic. Of course, most searchers do not need to obtain all the relevant articles to answer a clinical question (unless they are doing a systematic review). These studies also find considerable lack of overlap in the relevant articles retrieved when different users search on the same topic. This is important, especially since the quality of the evidence in studies can vary widely, as described in Chapter 2. These studies also show that the type of indexing or retrieval interface may not have a large impact on user performance. Task-oriented studies show that users improve their ability to answer clinical questions with IR systems, but performance is far from perfect. However quality of searching by users is assessed, systems take a long time to use. Large bibliographic databases such as MEDLINE may be inappropriate for most questions generated in the clinical setting, and the move to “synthesized,” evidence-based resources may help in this regard. Searching ability is influenced by a variety of factors. Although further research is needed to make more definitive statements, the abilities of healthcare personnel vary, with one or more specific cognitive traits (e.g., spatial visualization) possibly explaining the difference. Factors that may not play a significant role at all in search success are recall and precision. Although a searcher obviously needs to retrieve some reasonable amount of relevant documents and not be inundated by nonrelevant ones, the small differences across IR systems and users may not be significant. It is also clear that searchers make frequent mistakes and miss opportunities that might have led to better results. Finally, although healthcare IR systems are widely distributed and commercially successful, their true impact on healthcare providers and patient care is unknown. Demonstrating their benefit in the complex healthcare environment is difficult at best, with RCTs showing benefits in patient outcomes unlikely to be performed. On the other hand, as noted in the keynote address at the 1991 Symposium on Computer Applications in Medical Care by David Eddy, no one has ever assessed the impact of elevators on patient care, though they are obviously important. Analogously, no one can deny that medical IR systems are important and valuable, so further research should focus on how they can be used most effectively by clinicians, patients, and researchers.
This page intentionally left blank
Part III Research Directions
This page intentionally left blank
Chapter 8 Lexical–Statistical Systems
In Chapter 7 it was seen that the ability of information retrieval (IR) systems to find relevant documents for the user is far from ideal. Even if one accepts the limitations of recall and precision as evaluation measures, it is clear that new approaches to indexing and retrieval are needed to better steer users to the documents they need. In this and most of the remaining chapters of the book, various research approaches to IR will be described. The theory and implementation are explained in detail, but an underlying perspective of practicality and evaluation of these systems is maintained. The focus of this chapter is lexical–statistical approaches to IR, sometimes called automated retrieval or partial-match retrieval. These approaches can be described as lexical because the unit of indexing tends to be the individual word in the document and statistical because they involve operations like weighting of terms and documents. A basic approach for this type of indexing and retrieval was introduced in Chapters 5 and 6, respectively, because some of these methods are now used in state-of-the-art systems. These methods have actually been used in research systems for nearly 40 years; this long lag time in acceptance is due partly to their being better suited for end-user searching, which has become prevalent only in the last decade. The first section of this chapter focuses on how lexical–statistical approaches are evaluated. This material is followed by a basic description of the techniques used. The next section covers specific applications of these approaches, some of which correspond to Text REtrieval Conference (TREC) tracks (e.g., question answering and Web searching). The final section focuses on interactive evaluation of some of the systems and approaches described earlier in the chapter, most emanating from the Interactive Track of TREC. Lexical–statistical systems offer many appealing features, especially to novice end users who are less skilled in the use of controlled vocabularies, Boolean operators, and other advanced features of traditional retrieval systems. Lexical– statistical systems do not, for example, require the user to learn a controlled vo265
266
III. Research Directions
cabulary, which may express terms in ways not commonly used by clinicians (Walker, McKibbon et al., 1991; Hersh, Hickam et al., 1994) or exhibit degrees of inconsistency in the assignment of indexing terms (Funk and Reid, 1983). These systems also do not require the use of Boolean operators, which have shown to be difficult for novices (Slingluff, Lev et al., 1985; Borgman, 1986). With some additional features, such as relevance feedback, which also requires little effort on the part of the user, these systems have the potential to be quite valuable in busy clinical settings, where rapid access to information is required. The lexical–statistical approach itself is based on three assumptions about the words in queries and documents. These assumptions may not represent absolute “truth,” but they have been shown to lead to retrieval system performance comparable to that of any other method. The first assumption is that the words in a document represent its content. While Chapter 9 presents numerous examples of problems with this assumption, there is certainly some correlation between the words in a document and its conceptual content. The second assumption is that domain experts (i.e., end-user searchers) tend to use language of the domain, which is present in documents written by other domain experts. For example, a clinician is likely to use words in a query that are used by the writer of a clinically oriented paper. The third assumption is that the most important words are likely to be present in the documents sought and absent elsewhere. This forms the basis for weighting and ranking algorithms that allow lexical–statistical systems to perform relevance ranking, where the documents found in the search are ranked by their similarity to the query, as opposed to the usual nearly arbitrary order seen in Boolean systems.
8.1 Evaluation of Lexical–Statistical Systems The efficacy of many of the system-oriented approaches described in this chapter has been assessed by a general approach to evaluation that is detailed in this section. Except for some recent studies with real users to be described at the end of the chapter, virtually all evaluation of lexical–statistical systems has been based on “batch-mode” studies using test collections, which were introduced in Chapter 3. These collections typically contain a set of documents, a set of queries, and a binary determination of which documents are relevant to which query. The usual mode of comparing system performance is to generate a recall–precision table of the type presented in Chapter 3, with the most commonly used metric of mean average precision (MAP), which averages precision after each relevant document is retrieved (and assigns zero to relevant documents not retrieved) (Voorhees and Harman, 2000). Table 8.1 lists the Web sites for some of the most commonly used test collections in experiments of these types. Some of the test collections have been created with queries captured in the process of real interaction with a system, but others have been built by experimenters for the purpose of doing batch-style evaluation. Likewise, while some
8. Lexical–Statistical Systems
267
TABLE 8.1. The Most Commonly Used IR Test Collections and Other Resources for System-Oriented Research Name and description OHSUMED: test collection of MEDLINE records Retuers-21758 collection: original text categorization test collection research Retuers: new test collection for text categorization research TREC: test collections for IR research
Web address ftp://medir.ohsu.edu/ohsumed www.research.att.com/lewis/retuers21578.html
about.reuters.com/researchandstandards/corpus trec.nist.gov/data.html
relevance judgments have been performed by domain experts, others have not. Nonetheless, these collections have achieved high usage in the research community, and evaluations of IR system performance are typically not considered to be meaningful without their use. However, there are a number of problems with batch-mode studies and the test collections on which they are based. 1. Lack of real users: Simulating the behavior of users with batch-style studies does not guarantee that real users will perform identically in front of a computer. 2. Lack of meaningful measures: Recall–precision tables do not capture how meaningful the information being retrieved is to the user. Furthermore, research reports often do not provide analyses of statistical significance. 3. Unrealistic databases: Until TREC, most test collections were very small, on the order of a few thousand documents. There were concerns not only that such databases might have properties different from those of the large databases used in commercial systems, but also that retrieval algorithms themselves might not be scalable to large databases. 4. Unrealistic queries: Most queries in test collections are short statements, which in isolation do not represent the original user’s (or anyone else’s) information need. Also, recall from Chapter 7 that Saracevic and Kantor (1988b) found a twofold difference in recall when an intermediary searcher had access to a taped transcript of the user’s interaction with a librarian, showing that different results can occur when a searcher has access to multiple statements of the same information need. 5. Unrealistic relevance judgments: As was seen in Chapter 3, topical relevance judgments can be inconsistent. The main experiments in TREC have addressed some of these shortcomings. For example, the document collections used of real-world size. Table 8.2 lists the databases used for most of the original TREC tasks and tracks. However, these databases are unrealistic in others ways. For example, they are collections that a real-world searcher is unlikely to choose for searching. They also are small relative to the much larger databases that are now routinely available.
268
III. Research Directions
TABLE 8.2. The Document Collections for Most TREC Tasks and Tracks Disk 1
2
3
4
5
Collection Wall Street Journal, 1987–1989 Associated Press newswire, 1989 Computer Selects articles, Ziff-David Federal Register, 1989 Abstracts of U.S. Department of Energy publications Wall Street Journal, 1990–1992 Associated Press newswire, 1988 Computer Selects articles, Ziff-Davis Federal Register, 1988 San Jose Mercury News, 1991 Associated Press newswire, 1990 Computer Selects articles, Ziff-Davis U.S. patents, 1993 Financial Times, 1991–194 Federal Register, 1994 Congressional Record, 1993 Foreign Broadcast Information Service Los Angeles Times
Size (megabytes)
Documents
Mean number of words
267 254 242 260 184
98,732 84,678 75,180 25,960 226,087
434.0 473.9 473.0 1315.9 120.4
242 237 175 209 287 237 345 243 564 395 235 470 475
74,250 79,919 56,920 19,860 90,257 78,321 161,021 6,711 210,158 55,630 27,922 130,471 131,896
508.4 468.7 451.9 1378.1 453.0 478.4 295.4 5391 412.7 644.7 1373.5 543.6 526.5
The TREC queries are also an improvement over past test collections, since most (depending on the year) have consisted of detailed information needs statements. Such queries can be unrealistic too, however, especially since most researchers enter them directly into systems, which no user would ever do in practice. The structure of TREC ad hoc queries has varied by years. TREC-1 and TREC-2 featured very detailed information needs statements, along with key concepts related to the subjects of the queries. In TREC-3, the concepts were eliminated and rest of the query was shortened. The TREC-4 queries were shorter still, consisting merely of a one-sentence information need statement. Queries in TREC-5 and TREC-6 were lengthened to include the information need statement and a slightly longer narrative providing more detail. Again, the problem with these queries has not been so much that they do not reflect real information needs but, rather, that researchers enter them directly, some with more than 150 words, into systems, which again, no real user would ever do. Despite these limitations, useful information about the performance of systems has been obtained in TREC, especially when comparisons are made across systems. TREC has also required the developers of lexical–statistical systems to address efficiency issues for large databases. Indeed, the TREC experiments have come to define evaluation of English-language experimental retrieval systems. As noted in Chapter 3, TREC has evolved over the years, moving away from the ad hoc and routing retrieval tasks to a group of diverse tracks. Nonetheless, many retrieval systems and algorithms are still evaluated on the basis of how well they perform the TREC ad hoc task.
8. Lexical–Statistical Systems
269
8.2 Basic Lexical–Statistical Approaches Nearly 40 years of research has led to the evolution of a group of well-defined approaches to basic lexical–statistical document retrieval. Many of these approaches originate with Salton or his graduate students. Despite the conference’s caveats, the TREC experiments have helped to pinpoint the approaches that show the most promise. In particular, the TREC experiments have identified three lexical–statistical techniques that have improved results of every system to which they have been added. The first, which provides a minor improvement, is automatic phrase construction. The second, which provides more substantial benefit, is passage retrieval, where documents are retrieved based on smaller passages rather than the entire document. The final, which also provides substantial benefit, is query expansion, which adds new terms to the query from high-ranking retrieved documents. However, the benefit of these techniques has been questioned by research about document retrieval on the Web and with interactive users.
8.2.1 The Vector–Space Model The basic term-weighting and partial-match approach to indexing approach described in Chapters 5 and 6 was simple and effective. However, it was also limited in that certain advanced features, such as combining terms into phrases or utilizing already retrieved relevant documents to find more, are difficult to conceptualize. For this reason, the vector–space model was developed by Salton (1983). It should be noted that the use of the word vector in the term “vector–space model” does not imply that one must have a detailed grasp of vector mathematics to understand its principles. In the vector–space model, documents are represented as N-dimensional vectors, where N is the number of indexing terms in the database. Each term is one dimension of the vector. Appendix 3 lists the document vectors (with zero-value dimensions omitted) for the sample documents from Appendix 1. Table 8.3 lists the vector for document 5 for the query, drug treatment of hypertension (the word of is a stop word and is eliminated). Vectors can be binary or weighted, with term weighting represented by the length of the vector in a given dimension. Queries are also represented as vectors, so that the retrieval process consists of measuring the similarity between a query vector and all the document vectors in the database. The simplest method of doing this is to take the dot or inner product of the query and each document vector. When used with term frequency (TF) and inverse document frequency (IDF) weighting, this reduces to the approach outlined in Section 6.2.2. Table 6.6 showed the basic TF*IDF scoring for the foregoing query. The vector–space model also addresses a limitation of TF*IDF weighting, which is the inability to account for the length of documents. Documents that are longer have a higher number of words, hence the TFs for those words are increased, and a document highly relevant for a given term that happens to be short
270
III. Research Directions
TABLE 8.3. Vectors for Document 5 of Appendix 1 and the query, drug treatment of hypertension Document 5 AGENT ANGIOTENSIN ARRAY AVAILABLE BETA BLOCKER BLOOD CALCIUM CATEGORY CHANNEL CONVERTING DIURETIC DRUG ELEVATED ENZYME HYPERTENSION INCLUDE INCREASING INHIBITOR MAJOR PHARMACOLOGIC PRESSURE TREATMENT
TF*IDF 1.7 2 2 2 1.4 1.82 1.15 1.7 2 1.7 2 1.7 1.98 1.52 2 1.22 1.7 2 2 1.7 1.7 1.22 1.52
Query
1
1
1
will not necessarily have that relevance reflected in its TF. The solution to this problem is to use the cosine measure, which divides the dot product by the length of the query and document vectors. This gives more value to shorter documents. The cosine measure between a document and query was given in equation (2) in Chapter 6. Another value of the cosine measure is that it gives a measure of similarity on a scale of zero (no terms in common between query and document) to one (all terms in common between query and document). The cosine of two identical vectors is one and of two orthogonal vectors is zero. (Cosines can also vary from zero to minus one, but this does not occur in document retrieval systems because negative weights are not typically used. Nothing precludes their use, however.) The benefit of cosine normalization is not consistent across different databases. In a mix of MEDLINE references, for example, some nonrelevant documents lack abstracts and are very “short,” leading to inappropriately high weights from cosine normalization (Hersh, Buckley et al., 1994). In the TREC database, however, cosine normalization has been beneficial (Buckley, Salton et al., 1994b).
8. Lexical–Statistical Systems
271
The cosine calculation between the vectors for the sample query and document 5 is: Cosine(doc5,query)
(1.98 1) (1.22 1) (1.52 1) . . (1.7 22 . 1.522) (12 12 12)
4.72 0.33 14.52
(1)
Table 8.4 shows the cosine similarity values between the query and all the documents in Appendix 1. It can be seen that document 1 now ranks higher than document 4, whereas they were previously tied. This is because document 1 is shorter and has gained additional score owing to normalization. Cosine values can also be calculated between two documents. In this case, the preceding formula becomes t
Cosine(doci,docj)
TERMik * TERMjk
k1
TERM * TERM t
t
ik
k1
(2)
jk
k1
where TERMik and TERMjk represent the weights of the kth term in the vector for document i and document j, respectively. The cosine between documents is useful, for example, in the clustering of documents based on their content. This approach may also assist in the automated creation of hypertext links, which will be discussed shortly (Agosti, Crestani et al., 1997). In addition, clustering can be
TABLE 8.4. Cosine Similarity Between the Documents of Appendix I and the Query, drug treatment of hypertension Document
Cosine similarity with query
1 2 3 4 5 6 7 8 9 10
1.59/14.97 0.11 1.22/15.24 0.08 3.11/10.58 0.29 1.59/15.49 0.10 4.72/14.52 0.33 1.52/15.31 0.10 1.52/18.58 0.08 2.52/11.79 0.21 0/9.87 0 0/13.17 0
272
III. Research Directions
useful as an efficiency measure, where a group of similar documents can be represented as a single “centroid” vector (Voorhees, 1986). Queries can then be done against this centroid vector as opposed to each individual document vector, quickly filtering away documents with little or no similarity to the query. Terms themselves can also be represented as vectors, with the dimensions representing documents in which they occur t
Cosine(termi,termj)
DOCik * DOCjk k1
(3)
DOC * DOC t
t
ik
k1
jk
k1
where DOCik and DOCjk represent the weights of the kth document in the vector for term i and term j, respectively. With this cosine formula, term vectors can then be compared. This approach is useful for measuring term associations when automatically constructing thesauri and generating phrases.
8.2.2 Variations in Term Weighting For term weighting, only simple binary and TF*IOF weighting have been discussed so far. While the TF*IOF is a good general weighting scheme, in some document collections other schemes are found to work better. Much IR research over the years has focused on finding new and better weighting schemes. Salton and Buckley (1988) performed the last “pre-TREC” analysis of term weighting, assessing 287 distinct permutations of various weighting schemes across six older, smaller test collections in a classic batch-mode study. The best document weighting measure was found to be the TF*IOF measures presented in Chapter 5. The TREC experiments have led to a number of measures that give better MAP with its test collections, and these are described in this section. A similar massive comparison of different weighting techniques similar to that of Salton and Buckley was performed by Zobel and Moffat (1998), verifying the value of the new TRECbased approaches. 8.2.2.1 Variants of TF In the early TREC collections, a variant of TF*IDF was found to work better that replaced TF with a “normalized” version (Buckley, Allan et al., 1993): TF(term,document) ln(frequency of term in document) 1
(4)
With the OHSUMED test collection under SMART, the best weighting method was found to include another variant of TF, the augmented normalized term frequency (Hersh, Buckley et al., 1994): TF(term,document) 0.5 0.5
Frequency of term in document Maximum frequency of term in document
(5)
8. Lexical–Statistical Systems
273
Analysis with this test collection also found that document normalization was detrimental, since most MEDLINE references are similar in length and because of the large number of very short references (those without abstracts) that were mostly nonrelevant. Another weighting scheme mostly of historical interest is the term discrimination value (Salton, 1983). This value is obtained by calculating the “average” similarity for all documents in a collection, which is the average cosine between a document vector and all the other document vectors in a collection. This gives the density of the document space, which of course would be a maximum (1.0) when all documents are the same. By measuring the average similarity with and without each term, one can calculate for each term a discrimination value that is proportional to how much that term increases the average similarity. Terms that occur frequently will lead to decreased average similarity when removed from the collection, hence will have a low discrimination value, whereas uncommon terms will lead to an increased average similarity when removed and thus will have high discrimination value. This approach is very computationally intensive and does not offer any better performance than simpler approaches. 8.2.2.2 Okapi Weighting The TREC experiments have led to the discovery of two other term-weighting approaches that have yielded even better results. The first of these was developed by Robertson (1994). Based on the statistical model known as Poisson distributions, it is commonly called Okapi weighting, since it was developed in Robertson’s Okapi system (though it has adapted by many other systems). In essence, the Okapi weighting scheme is an improved document normalization schema, yielding up to 50% improvement in MAP in various TREC collections (Robertson, Walker et al., 1994). One version of Okapi’s TF is: Okapi TF
(Frequency of term in document)(k1 1) Length of document k1(1 b) k1b Frequency of term in document Average document length
(6)
The variables k1 and b are parameters set to values based on characteristics of the collection. Typical values for k1 are between 1 and 2 and for b are between 0.6 and 0.75. A further simplification of this weighting often used is (Robertson, Walker et al., 1994): Okapi TF 0.5 1.5
(Frequency of term in document) Length of document Frequency of term in document Average document length
(7)
Okapi weighting has its theoretical foundations in probabilistic IR, to be described shortly. As such, its TF*IDF weighting uses a “probabilistic” variant of IDF, to be covered below as well:
274
III. Research Directions
Okapi IDF log
Total number of documents number of documents with term 0.5 Number of documents with term 0.5
(8) The probabilistic model has also led to the newest theoretical approach to term weighting, known as language modeling, which will also be described soon. 8.2.2.3 Pivoted Normalization The second term-weighting approach to come from TREC is pivoted normalization (PN), developed by Singhal et al. (1996). Its major effect also comes from improved document normalization. Empirical analysis of the TREC collections showed that cosine normalization tended to result in shorter documents being overretrieved and longer documents being underretrieved. In particular, the probability of a document being retrieved, P(retrieval), was higher for shorter documents and lower for longer documents than the probability of a document being relevant, P(relevant), as shown in Figure 8.1a. The pivoted normalization approach defined the “pivot” as the point where P(retrieval) P(relevant) and the slope as a factor to “correct” the areas where P(retrieval) P(relevant), as seen in Figure 8.1b. A mathematically simplified version of PN is (Singhal, 2001): PN(term,document)
TF(term,document) * IDF(term) Length of document (1 slope) slope Average document length
(9)
A commonly used value for the slope is 0.2. Experiments in TREC using this approach have found MAP improvements of up to 20% (Singhal, Buckley et al., 1996).
8.2.3 Automatic Thesaurus Generation As already noted, one of the major problems in IR systems is synonymy, the presence of different words that mean the same thing. This is a problem in medical terminology in general, since there are many words that have identical meanings, such as high and elevated and kidney and renal. While Roget’s Thesaurus and other tools catalog such synonymy for general English, few such tools exist for technical domains, such as health care. Thesauri in general enhance recall by allowing different ways of saying the same thing. For example, it can be seen in the sample document collection (Appendix 1) that some documents use the word therapy while others use the word treatment. If these words could be designated as synonyms, then a query using one would retrieve documents indexed on either. Most of these approaches can be considered to be linguistic approaches and are described in the next chapter. But those based on purely statistical approaches are covered in this chapter.
8. Lexical–Statistical Systems
275
(a)
(b) FIGURE 8.1. (a) Probability of document retrieval. (b) Pivoted normalization. (From Singhal, Buckley et al., 1996.)
For the purpose of grouping synonyms, thesauri are usually organized by classes of terms, where all the terms in a class that have the same meaning. Some words may have multiple meanings and may exist in more than one class. For example, high might represent elevation in one class while meaning a state of euphoria in another. Some lexical–statistical systems have utilized simple manual thesauri. For example, Bernstein and Williamson (1984) devised a list of 80 synonym pairs in the healthcare domain that physicians are likely to interchange when they utilize an IR system. Other approaches, to be de-
276
III. Research Directions
scribed in the next chapter, have attempted to identify synonyms from large amounts of Web text (Turney, 2001) or terms in the UMLS Metathesaurus (Cimino, 1998a). Building a thesaurus for words or concepts requires human knowledge. The use of traditional thesauri in IR systems, especially word-based ones, presents additional challenges. This is because thesauri are not necessarily organized at the single-word level. That is, they may be constructed around multiword concepts. Or there may be some words that have multiword concepts for synonyms (e.g., hypertension and high blood pressure). Recognizing multiword concepts free text, however, can be a difficult task even with advanced natural language processing tools, as will be seen in Chapter 9. To surmount these problems, another area pursued by Salton and others has been automated thesaurus construction (Salton, 1983; Srinivasan, 1992). The goal of this process is to utilize term–term associations in a document collection or subset. First, a term–term similarity matrix is built, with the similarity measurement usually the cosine of term vectors. Single-link methods connect terms in a class, where a certain similarity threshold for the term and another in the class is exceeded. Clique methods require all terms in a class to exceed a threshold. It has been noted experimentally that terms in common classes exhibit high similarities, and thus high-frequency terms should not be used (Srinivasan, 1992). As one might expect, the “synonyms” generated by these algorithms are not always synonymous in the linguistic sense, since they really represent word–word associations. In the retrieval situation, the automatically generated thesaurus can be used like any other, to expand queries by adding associated terms. Of course, adding automated thesaurus capability to a fixed database does not usually enhance performance by much, since the term associations have been derived from associated terms already in the database. In fact, other query expansion methods, to be described shortly, have shown more benefit. Srinivasan has explored a variant of this approach with the several MEDLINE test collections (Srinivasan, 1996a,b), defining measures of association between the words in the title and abstract of the MEDLINE record and its assigned Medical Subject Headings (MeSH) terms. The technique and results are described in Chapter 9.
8.2.4 Latent Semantic Indexing A more recent approach to capturing semantic equivalence, also vector based, is latent semantic indexing (LSI) (Deerwester, Dumais et al., 1990). This method uses a mathematically complex technique called singular-value decomposition (SVD), which nevertheless can be easily understood in a qualitative sense. In this approach, an initial two-dimensional matrix of terms and documents is created, with the terms in one dimension and the documents in the other. The SVD process creates three intermediate matrices, the two most important being the mapping of the terms into an intermediate value, which can be thought to represent an in-
8. Lexical–Statistical Systems
277
termediate measure of a term’s semantics, and the mapping of this semantic value into the document. The number of intermediate values can be kept small, which allows the mapping of a large number of terms into a modest number of semantic classes or dimensions (i.e., several hundred). The result is that terms with similar semantic distributions (i.e., distributions that co-occur in similar document contexts) are mapped into the same dimension. Thus, even if a term does not cooccur with another, if it occurs in similar types of documents it will be likely to have similar semantics. In practical usage, an IR database might contain 50,000 documents and 100,000 words. The initial matrix will thus have 50,000 100,000 5,000,000,000 elements. By designating the reduction of semantic classes to many fewer dimensions (e.g., 200), the resulting semantic class–document matrix is 200 50,000 10,000,000 elements. While the optimal number of dimensions is not known, it has been shown for several of the small standard test collections that a few hundred is sufficient (Deerwester, Dumais et al., 1990). An initial significant limitation of this approach was its high computational requirements, but recent enhancements have made the process significantly faster, hence feasible with large document databases. Of course, LSI is also more efficient at query time, since fewer “term”–document weight calculations are necessary. But the major problem with LSI is that words often have more than one meaning, and there are no mechanisms to handle the separate “semantic spaces” that may occur for polysemous words. A couple of evaluation studies have shown performance enhancements of a few percentage points for LSI with small document collections (Deerwester, Dumais et al., 1990; Hull, 1994), but these benefits have not been realized with larger collections such as TREC (Dumais, 1994). A better use for this technique may be with the automated discovery of synonymy, to be discussed in the next chapter (Landauer and Dumais, 1997).
8.2.5 Phrase Construction Automatic thesaurus generation and LSI offer moderate help for the problem of synonymy. The converse problem is that of polysemy, the case of the same word meaning different things. One way to determine the meaning of a word is to consider its context, both in phrases and in sentences and paragraphs. For example, when the words colon and cancer occur very close together, as opposed to scattered across a document, there is contextual meaning; that is, the document is likely to be discussing the entity of colon cancer. Combining words into meaningful phrases in generally enhances precision. This is especially true when broad, high-frequency terms are combined. For example, high and blood and pressure are relatively common words and are likely to appear in many types of medical documents. But when these terms occur adjacently, they take on a very distinct meaning. Recognizing simple common phrases, however, is difficult to do algorithmically, especially without a dictionary or other linguistic resources. Furthermore, many phrases can be expressed in a variety of forms. For example, a document on high blood pressure might read, When blood
278
III. Research Directions
pressure is found to be high. . . . In addition, a single-word synonym might be substituted, such as elevated for high. As noted earlier, recognizing multiword phrases in free text is difficult. Thus Salton and others have investigated approaches to identifying important phrases based on statistical co-occurrences of terms (Salton, 1983; Fagan, 1989). The goal is to find words that commonly occur in close proximity to each other. As with automated thesauri, where the “synonyms” generated are not true synonyms in the linguistic sense, the phrases generated are not always grammatically or semantically sound. Methods for identifying linguistic phrases are covered in Chapter 9. The first step in automated phrase construction in selecting a “window” or span within which words must lie to be eligible to become a phrase. On first thought, one might choose adjacency as a requirement. Obviously, however, the terms in an important phrase may be separated by intervening words. It has been determined experimentally that a better approach is to use a five to ten intervening words or a sentence (Fagan, 1989). Words in phrases should also be restricted to those that occur with moderate to high frequency. The next step is to define a measure of cohesion, which indicates how likely terms are to co-occur. This is done by dividing the frequency of terms occurring together within the window by their individual occurrence frequencies: Cohesion(termi,termj) Constant
Frequency of termi and termj occurring together
Freque ncy of termi * frequ ency o f termj
(10)
When the co-occurrence frequency is high enough to overcome the individual frequency product, then the cohesion is high enough to permit designation the pair of terms as a phrase. This process can be done for phrases longer than two words, but at considerable computational expense. As noted earlier, the phrases designated by this approach are not necessarily those that make sense linguistically. For example, consider a collection of documents that had multiple phrases like, The diagnosis was cancer of the stomach. If the word cancer occurred less frequently than diagnosis or stomach elsewhere, the phrase diagnosis stomach is likely to have high cohesion and will be nominated as a phrase. This seems linguistically absurd, but an approach using a natural language parser to generate phrases performed worse than this approach (which probably says more about the problems of natural language processing than the benefits of this approach) (Fagan, 1987). To implement automated phrase generation, the phrase extraction routine is run after the initial indexing. Sufficiently cohesive phrases are identified and become indexing terms. When users query the database, an attempt is made to map queries into phrases when appropriate, and these phrases are used to add weight to matching documents. In the TREC experiments, a simpler method of phrase construction has been used. Buckley et al. (1993, 1994) designated phrases as any adjacent nonstop words that occur in 25 or more documents in the training set. This approach con-
8. Lexical–Statistical Systems
279
ferred modest performance benefit, although it still performs better than most of the linguistic approaches discussed in the next chapter. Another approach to phrase generation has been undertaken by the INDEX system (Jones, Gassie et al., 1990), which locates repeating nonunique phrases in documents and uses them as indexing terms. Phrases are ranked based on a formula WFN 2, where W is the number of unique words in the phrase, F is the frequency of the phrase in the document, and N is the number of unique non–stop words in the phrase. As with the foregoing approaches, a certain number of meaningless phrases are produced. This problem was addressed with a subsequent system, INDEXD, which used a dictionary in an effort to avoid meaningless phrases.
8.2.6 Passage Retrieval Another approach to capturing the importance of term proximity introduced in the 1990s is passage retrieval, where documents are broken into smaller passages, which are used to weight the document for retrieval by the query (Salton and Buckley, 1991). The goal of this method is to find sections of documents that match the query highly under the presumption that these local concentrations of query terms indicate a high likelihood of document relevance. Salton and Buckley claim that this process reduces linguistic ambiguity, since the smaller passage is more likely to ensure that words occurring together in the query are also occurring in the same context in the document. To give a medical example, the words congestive, heart, and failure are more likely to represent the concept congestive heart failure if they all occur in the same passage rather than scattered in separate parts of the document. The main problem to this approach is identifying appropriate passages and avoiding having highly relevant areas of documents span across passages. Callan (1994) identified three types of passage in documents that could be used to subdivide documents based on content: • Discourse passages: based on the structure of documents, such as sections and paragraphs • Semantic passages: based on changing conceptual content of the text • Window passages: based on number of words Interest in passage retrieval has grown with the availability of full-text documents, which provide more text for identifying their topical content. Most implementations start with a global match between query and document in the usual manner. This is followed by matching of the query against smaller portions of the document, be they sections, semantic areas, or window contents. Different weighting schemes may be used for the various subdocuments; for example, cosine normalization is typically not helpful at the sentence level, since there is less variation in length.
280
III. Research Directions
Salton and Buckley used discourse passages in their original experiments, which were found to work well with the highly structured text of an encyclopedia (Salton and Buckley, 1991), but less ably with the TREC data (Buckley, Allan et al., 1993). Hearst and Plaunt utilized a vector-based approach to identifying semantic passages based on abrupt changes in document vectors between text sections, a technique that showed modest performance gains (Hearst and Plaunt, 1993). Two groups of TREC have found that overlapping text window passages of 200 words provide the best MAP performance gain of around 10 to 15% (Broglio, Callan et al., 1994; Buckley, Salton et al., 1994b). Passages start 100 words apart, and each overlaps the next to avoid the breaking up potentially relevant passages. Other groups using slightly different approaches have also shown benefit (Knaus, Mittendorf et al., 1994; Kwok, Grunfeld et al., 1994; Robertson, Walker et al., 1994).
8.2.7 Relevance Feedback and Query Expansion Section 6.2.2 introduced the notion of relevance feedback, which aims to discover more relevant documents in the retrieval process by adding terms from documents known to be relevant to a modified query that is rerun against the document collection. Also described was a related approach called query expansion, where the relevance feedback technique is used without relevance information. Instead, a certain number of top-ranking documents are assumed to be relevant and the relevance feedback approach is applied. Assessing the benefit of relevance feedback can be difficult. Most batch-type studies use the residual collection method, in which the documents used for relevance feedback are ignored in the analysis and a new recall–precision table is generated. For each query, an initial search is performed with some method, such as cosine-normalized TF*IDF. Some top-ranking relevant and nonrelevant documents, typically 15 or 30, are chosen for modification of query vector. At this point a new recall–precision table could be generated with the new query vector. If, however, the improvements merely reflect reordering of the documents with the existing relevant ones ranked higher, the identification of new relevant documents may not result. In the residual collection method, only the residual documents from those not used in relevance feedback are used in the new recall–precision table. Another approach to evaluation of relevance feedback techniques came from the early routing task in TREC (which has since been replaced by the Filtering Track). Since this task provided “training” data (i.e., documents known to be relevant to the queries), relevance feedback techniques could be applied to the queries run against the unknown “test” data to follow. The most effective approaches were found to use the following iterative technique (Buckley, Mitra et al., 2000): 1. Designation of a “query zone” of a certain number (in this case, 5000) of documents that are related (e.g., in the same domain) but not all relevant to query (Buckley, Singhal et al., 1996)
8. Lexical–Statistical Systems
281
2. Expansion of terms weighted by the Rocchio formula, adding terms and phrases that enhance performance but below the point at which they provide diminishing returns, estimated to be around 300 to 500 (Buckley, Singhal et al., 1994a) or 5 to 10% of terms (Buckley, Singhal et al., 1996) 3. Adding pairs of co-occurring terms based on a measure of the terms’ correlation with relevance of the documents times the product of their correlation in relevant documents minus their correlation in nonrelevant documents (Buckley, Singhal et al., 1996): Pair weight Correlation of pair to relevance (Correlation in relevant documents Correlation in relevant documents)
(11)
4. Dynamic feedback optimization, or the iterative setting of Rocchio parameters based on training data, with optimal parameters found to vary in different collections among 2, 16–64, and 4–8 (Buckley and Salton, 1995) A variety of query vector modifications have been assessed in relevance feedback. In the older, smaller collections, the most effective measure was found to one called Ide dec-hi (Salton and Buckley, 1990). Another method found to work effectively for relevance feedback in TREC has been LSI. The best technique thus far is to create a “centroid” vector of all relevant documents from the training data and run it against test data (Dumais, 1994). The optimal role for LSI may be in a feedback situation, where the latent semantics of terms is uncovered by relevance data from documents already seen. Query expansion techniques have also been shown to be effective in TREC. Indeed, they may be viewed as complementary to passage retrieval. While passage retrieval is a precision-enhancing technique that aims to give higher rank to documents in which the query terms are concentrated, presumably promoting their context, query expansion is a recall-enhancing process aiming to broaden the query to include additional terms in top-ranking documents. Based on the increased likelihood of top-ranking documents being relevant, terms present in these documents but not entered in the query should lead to the discovery of additional relevant documents. In TREC-3, Buckley et al. (1994b) used the Rocchio formula with parameters 8, 8, and 0 (which perform less reweighting for expansion terms than in the relevance feedback experiments cited earlier) along with the addition of the top 500 terms and 10 phrases to achieve a 20% performance gain. Others in TREC have also shown benefit with this approach (Evans and Lefferts, 1993; Broglio, Callan et al., 1994; Buckley, Salton et al., 1994b; Knaus, Mittendorf et al., 1994; Kwok, Grunfeld et al., 1994; Robertson, Walker et al., 1994). Additional work by Mitra et al. (1998) has shown that use of manually created Boolean queries, passage-based proximity constraints (i.e., Boolean constraints must occur within 50–100 words), and term co-occurrences (i.e., documents are given more weight when query terms co-occur) improves MAP performance further still. Linguistic approaches have also been used for query expansion and are described in the next chapter.
282
III. Research Directions
8.2.8 Probabilistic IR An alternative model for lexical–statistical systems is the probabilistic model. This approach is not necessarily at odds with the vector–space model, and in fact its weighting approaches can be incorporated into the vector–space model. However, its approach is unique enough to warrant separate discussion. Indeed, it has been argued that the probabilistic approach is more theoretically sound than the other more empirical approaches to retrieval (Cooper, 1994). The theory underlying probabilistic IR is a model to give more weight to terms likely to occur in relevant documents and unlikely to occur in nonrelevant documents. It is based on Bayes’ theorem, a common probability measure that indicates likelihood of an event based on a prior situation and new data. Bayes’ theorem has of course been used extensively in the area of medical decision making. For example, one can determine the likelihood of a disease given preexisting knowledge and results of a test. A patient may have a 20% chance of having a disease, based on knowledge from the initial physical examination. But some more invasive test, depending on its accuracy, might update the probabilities for and against disease to give much better estimates. If the test is positive, for example, the likelihood of the disease might increase to 80%, but if negative, it might fall to 10%. The same approach can be used in IR. After all, an IR system is in essence a diagnostic system that attempts to “diagnose” queries with relevant documents. As with many real-word applications of probability principles, applying probability theory to IR does entail some assumptions that might not actually hold in reality. For example, as with all Bayesian approaches, the variables are presumed to be independent of each other. Just as medical findings in a disease are not independent of each other, neither are the terms that occur in a document. Furthermore, the relationship between a term and a relevant document is not necessarily fixed. It was seen in Chapter 3 that relevance often depends upon the situation. Meadow has asserted that there is no fixed relationship between a document and a query, and likewise there is unlikely to be a fixed relationship between a term and a query in a relevant document (Meadow, 1985). 8.2.8.1 Basic Probabilistic IR A number of relevance feedback formulas have been derived based on varying assumptions about the distribution of terms in relevant documents and nonrelevant documents. Table 8.5 lists the parameters for probabilistic feedback. The formula determined to be the most theoretically sound in the initial development of the model, which also achieved the best results experimentally, was (Robertson and Sparck-Jones, 1976): New term weight log
r/(R r) (n r)/(N n R r)
(12)
Probabilistic IR is predominantly a relevance feedback technique, since some relevance information about the terms in documents is required. Croft and Harper
8. Lexical–Statistical Systems
283
TABLE 8.5. Parameters for Probabilistic Feedback: N number of documents, R number of relevant documents for query q, n number of documents with term t, and r number of relevant documents with term t Document relevant
Document nonrelevant
Total
r Rr R
nr NnRr NR
n Nn N
Document term present Document term absent Total
suggested that the IDF be used as the initial term weight (Croft and Harper, 1979). In fact, modifying equation (12) can alter it to default to the IDF in the absence of relevance information: New term weight log
(r 0.5)/(R r 0.5) (n r 0.5)/(N n R r 0.5)
(13)
With no relevance information available (r R 0), the formula becomes log [(N n)/n], which is essentially the IDF. Probabilistic methods are usually applied by reweighting each search term in the query with equation (13) and then repeating the search. With older, smaller test collections, this approach did not perform as well as vector–space techniques. In Salton and Buckley’s (1990) relevance feedback experiments with six test collections, an approach similar to that described here achieved the best performance for probabilistic methods but was inferior to all the vector modification techniques. The authors tentatively attributed the failure to the disregarding of useful information in constructing the feedback query, such as query and document term weighting. In addition, the relevant retrieved items were not used directly to modify the query vector, but rather were used indirectly to calculate the probabilistic term weight. In the TREC experiments, as noted earlier, variants on the probabilistic approach have been shown to perform better than vector–space relevance feedback with the addition of query expansion (Broglio, Callan et al., 1994; Cooper, Chen et al., 1994; Kwok, Grunfeld et al., 1994; Robertson, Walker et al., 1994; Walczuch, Fuhr et al., 1994). As noted earlier, the Okapi weighting measure derives from the probabilistic model (Robertson, Walker et al., 1994). A modification to probabilistic IR is the inference model of Turtle and Croft (1991), where documents are ranked based on how likely they are to infer belief they are relevant to the user’s query. This method is also not necessarily incompatible with the vector–space model and in some ways just provides a different perspective on the IR problem. One advantage the inference model has is the ability to combine many types of “evidence” that a document should be viewed by the user, such as queries with natural language and Boolean operators, as well as other attributes, such as citation of other documents. Combining some linguistic techniques, described in Chapter 9, with slight modifications to TF*IDF weighting, passage retrieval, and query expansion, this approach has performed consistently well in the TREC experiments (Broglio, Callan et al., 1994).
284
III. Research Directions
8.2.8.2 Language Modeling A more recent application of probabilistic IR has been the use of language modeling. This approach was originally used in other computer tasks, such as speech recognition and machine translation, where probabilistic principles are used to convert acoustic signals into words and words from one language to another, respectively. A key aspect of the language modeling approach is “smoothing” of the probabilities away from a purely deterministic approach of a term being present or absent in a document in a binary fashion. Language modeling was introduced to the IR community by Ponte and Croft (1998). The argued for a different view of IR, based on the notion that documents can be thought of as predictors of queries. This approach led to a variety of mathematical expressions that generated new approaches to term weighting and document retrieval and resulted in modest performance gains with TREC data. A more generalized view of language modeling was put forth by Miller et al. (D. Miller, Leek et al., 1999), based on the use of Bayes’ formula. These authors note that the probability of a document being relevant for a given query, P(DR Q), can be expressed by Bayes’ formula as follows: P(DR Q)
P(Q DR) * P(DR) P(Q)
(14)
Since P(Q) is the same for all documents, it can be omitted from the formula. The next step is calculate P(DR Q). This is done by two measures: P(q D), the probability of a query term occurring in the document, and P(q GE), the probability of a query term occurring in general English. For the document set k, they are calculated as follows: P(q Dk)
Number of times q appears in Dk Length of Dk
(15)
P(q GE)
k number of times q appears in Dk k length of Dk
(16)
Then P(DR Q) is calculated by: P(DR Q)
[a P(q GE) a P(q D )] 0
1
k
(17)
qQ
The parameters a0 and a1 are probabilities in the model estimated from training data. Additional factors (with corresponding an) can be added to P(DR Q) by applying other features of documents, such as term bi-grams (phrases), differential weighting in different parts of the document (e.g., title, abstract), and weights from query expansion. A variety of enhancements to the basic approach have been demonstrated. Berger and Lefferty (1999) added automatically generated word synonyms cal-
8. Lexical–Statistical Systems
285
culated by means of a mutual information statistic that improved MAP on TREC data. Further work has added complexity to the model, leading to improved retrieval performance gains (Lafferty and Zhai, 2001; Lavrenko and Croft, 2001).
8.2.9 Stemming As noted in Chapter 5, most implementations of automated indexing and retrieval use stemming, whereby plurals and common suffixes are removed from words, based on the presumption that the meaning of a word is contained in its stem. Another advantage to stemming is a practical one: the size of inverted files can be decreased, since fewer words have to be stored. This also leads to more efficient query processing. Stemming has disadvantages as well. First, it is purely algorithmic, whereas language can be idiosyncratic. Thus most implementations do not handle grammatical irregularities. Second, in some instances the information in the stem may in fact confer meaning. Some aggressive stemmers remove suffixes like -itis, which are not medically insignificant. Later in the next chapter, some linguistic alternatives to stemming are described. A variety of approaches to stemming have been advocated, but the most common approach in IR has been affix removal stemming, whereby algorithms specify removal of prefixes or (usually) suffixes. The two most commonly used affix removal algorithms are those by Lovins and Porter. The Lovins stemmer is an iterative longest match stemmer that removes the longest sequence of characters according to a set of rules (Lovins, 1968). The Porter algorithm, on the other hand, has a series of rules that are performed if various conditions of the word length and suffix are met (Porter, 1980). Another stemmer that has been used in some systems is the S stemmer, whose rules were presented in Figure 5.5. How well do stemming algorithms perform? The data are confounded by a variety of different experimental parameters, such as type of stemmer, test collection, and performance measure used, but it is clear that the benefit of stemming is modest at best and can sometimes be detrimental (Harman, 1991). Two groups have compared stemming with no stemming in TREC ad hoc tasks, finding minimal improvement in performance with its use (Buckley, Salton et al., 1992; Fuller, Kaszkiel et al., 1997). Another analysis has compared multiple types of stemmers, not only the Lovins, Porter, and S stemmers but also some linguistically based stemmers described in the next chapter, finding minor improvements of a few percentage point increase in MAP overall (Hull, 1996). Hull’s analysis did find, however, that language idiosyncrasies cause a small number of queries to be affected in a large way, sometimes beneficially but other times detrimentally.
8.2.10 Implementations of Lexical–Statistical Systems Basic TF*IDF weighting and partial-match retrieval are now part of most commercial search engine products. Although the exact details are proprietary, many Web search engines use various aspects of lexical–statistical approaches. In ad-
286
III. Research Directions
dition to commercial systems, a number of free research systems are available, many of which were listed in Table 1.3. One of the first and certainly among the most widely used research systems is SMART, the original test bed for lexical–statistical techniques developed by Salton in the 1960s. First implemented in FORTRAN on a mainframe computer, it has undergone several reimplementations on various platforms and is currently written in C for Unix-based workstations. A major limitation of SMART is that the system is designed more for batch-style IR experiments than for interactive retrieval. Thus, the basic software has only a command-line interface, although various groups have implemented other interfaces. The current implementation was designed in a very modular fashion, and thus not only can various existing features (e.g., weighting algorithms, stop lists, stemming, relevance feedback, etc.) be modified, but new ones can be added. Some of the other research systems in Table 1.3 provide access to features not in SMART. For example, MG, provides the array of weighting functions assessed by Zobel and Moffat (1998). The Lemur Toolkit provides basic indexing and retrieval functionality as well as that of the growing area of language modeling (Lafferty and Zhai, 2001).
8.3 Specific Lexical–Statistical Applications Now that the basic methods of lexical–statistical systems have been described, attention can be turned to specific applications using these systems. As noted already, many of these applications correspond to tracks that have evolved in TREC.
8.3.1 Routing and Filtering Routing of documents and the optimal methods for it were described in Section 8.2.7. Document filtering is essentially the same process, although the view of the output is different. In routing, the output is evaluated by using the standard ranked retrieval approach (i.e., MAP). However, for real users, this measure is too crude, since it may be desirable to take into account the value of a relevant document and the penalty of a retrieved one. Section 3.4 described the basic approach to the evaluation of filtering. Many approaches to document filtering have employed techniques of text categorization, where the goal is to classify documents into a standard set of categories (Lewis, 1995). Most approaches to text categorization employ techniques from machine learning, a branch of artificial intelligence that uses algorithms to “learn” the correct approach to solving various problems (Mitchell, 1997). Machine learning has been applied to many problems, from medical diagnosis (Weiss and Kulikowski, 1991) to game playing. It also is used in a variety of IR-related problems, including routing and filtering, text categorization, and natural language understanding (see Chapter 9 for the latter). Some of the algorithms employed by machine learning, which will be introduced when their use is described
8. Lexical–Statistical Systems
287
in a specific application, include linear and other forms of regression, neural networks, and induction of rules. Since the early test collections for routing and filtering came from Reuters news service data, the task motivating initial text categorization research was the classification of news stories by topic, people, places, and so on. An early widely used resource was the Reuters-21578 collection, which has been superseded by a new collection of more recent documents and categories (see Table 8.1). Other document collections have been used for text categorization research in TREC and other settings, including MEDLINE records from the OHSUMED test collection (Robertson and Hull, 2000). For the latter, three sets of categories were defined: 1. Queries: 63 (out of the original 106) queries were used that had two or more relevant documents in the training set (the one-fifth of the collection representing the first year of the 5-year set) 2. MeSH terms: 4093 medium-frequency MeSH headings with at least four documents in the training set and one in the final quarter of the test data 3. Sampling of MeSH terms: a randomly selected set of 500 MeSH terms from the 4093 of item 2 The TREC Filtering Track simulates two types of filtering, adaptive and batch (Robertson and Hull, 2000). In adaptive filtering, the documents are “released” to the system one at a time, and only documents chosen for retrieval can be used to modify the filtering query. In batch filtering, all documents and their relevance judgments can be used at once. Similar to other TREC tasks and tracks, participants in the filtering track have used a variety of methods, which yield a wide spectrum of performance (Robertson and Hull, 2000). In general, most approaches have aimed to optimize document weighting and then identify a threshold that maximizes inclusion of relevant document and discards nonrelevant ones. Some have used machine learning techniques, such as neural networks or logistic regression, although their results have not exceeded simpler term-weighting approaches. Y. Yang (1999) has done a large-scale comparison of different approaches to text categorization. A baseline method used a simple word-matching approach between category words and documents, with no learning. The most effective methods, each achieving comparable performance, were as follows: 1. k-Nearest Neighbor (kNN): the documents most similar (i.e., the nearest neighbors) to the training documents were ranked and the top k documents were used to rank the best-fitting categories 2. Linear-Least Squares Fit (LLSF): a multivariate regression model was used to map document vectors into categories 3. Neural Network: a neural network was used to map document words into categories Text categorization techniques have also been applied in other medical examples. Yang and Chute (Y. Yang and Chute, 1994) have found the LLSF method
288
III. Research Directions
to be far more effective than simple word matching for retrieval of MEDLINE documents and classification of surgical reports into 281 International Classification of Diseases-9-CM (ICD-9-CM) categories. Larkey and Croft (1996) performed similar experiments with a collection of discharge summaries classified into ICD-9-CM codes, finding that a combination of kNN, relevance feedback, and Bayesian classifiers worked more effectively in combination than any single one individually. Another approach has used the measure of term strength, defined as the probability of finding a term in a document that is closely related to, or relevant to, any document in which the term has already occurred (Wilbur and Yang, 1996). Used mainly as a relevance feedback device, this approach has been found to perform comparable to LLSF. Perhaps the most important aspect of this approach is its use in the “related articles” feature of PubMed (see Chapter 6).
8.3.2 Web Searching With the growth of the Web, it not surprising that some research has focused specifically on Web searching. One unfortunate consequence of the commercialization of Web searching has been the development of proprietary algorithms that cannot be replicated and dissected in evaluations by researchers. A great deal of Web searching research has therefore focused on how users interact with commercial systems, and much of this will be described in Chapter 10. This section focuses on the Web Track at TREC as well as other system-oriented Web research. 8.3.2.1 TREC Web Track The general ad hoc retrieval task for TREC ended after TREC-8. At the same time, the Web Track emerged with a 10 gigabyte, 1.69-million-document collection called WT10g for TREC-9, which in essence served as the ad hoc track (Hawking, 2000). Queries were derived from a log of searches posed to the Excite search engine (www.excite.com), with additional narrative added by TREC assessors to make the search topic more explicit. Some of the original queries had misspellings, which were kept in the data to simulate the misspellings made by real users. One general goal of the Web track has been to determine whether the link information in Web data can lead to more effective retrieval than page content alone. To facilitate the latter, a link connectivity matrix was calculated and distributed with the data. For all the participants in the TREC-9 Web Track, the benefits of link data were found to vary from slight improvements to outright detriment (Hawking, 2000). Performance was found to be resilient for queries with errors, with little effect on MAP. The TREC 2001 Web Track used the same WT10g collection with two types of query: the standard ad hoc retrieval task and a home-page-finding task (Hawking and Craswell, 2001). Queries of the first type were created much as the TREC ad hoc queries were generated, and the output was judged by MAP. Queries of
8. Lexical–Statistical Systems
289
the second type were comparable to a “known-item” search task, with 145 home pages defined by assessors from the National Institute of Standards and Technology (NIST) and the output judged by two measures. The first was the mean reciprocal rank, as defined for the Question-Answering Track and presented in Section 3.4. The second was the percentage of the home pages that were correctly identified by systems within their top-10 ranking. Several of the home-pagefinding queries had more than one correct answer, since a number of Web sites have more than one URL pointing to it. The top-ranking ad hoc runs made use mainly of document content, while the best-performing home-page-finding runs made use of URL or anchor text. 8.3.2.2 Other Web Searching System Evaluation As noted in Chapter 7, most commercially oriented evaluations of Web searching engines (e.g., searchengineshowdown.com) measure “success” by sheer numbers of pages retrieved. There have been some attempts to use relevance-based measures to evaluate Web search engines. One limitation of these studies, of course, is the constantly changing nature of Web search engines. Nonetheless, these assessments do provide some insight into the capabilities of Web search engines. Recall from Chapter 7 that one assessment of a Web meta–search engine found for 50 queries derived from clinicians that the average precision was 10.7% (Hersh, Gorman et al., 1998), while another found that even the best clinical Web sites were able to answer only 50 to 60% of questions posed to them (Graber, Bergus et al., 1999). Several studies have attempted to compare commercial Web search engines. Since these studies focused on systems rather than users, their results are presented here. Again, a major limitation of these studies is the rapidly changing state of the Web. Indeed, most of the “early” (circa 1996–1999) did not include the Google search engine because it was in its infancy. A few early studies had a small number of queries or search engines and are not described in detail (Chu and Rosenthal, 1996; Ding and Marchionini, 1996; Tomaiuolo and Packer, 1996). Leighton and Srivastava (1999) compared five search engines accessed in 1997. Their queries consisted of 15 requests asked at the reference desk by undergraduates at a university library. The authors did the searching themselves, retrieving the top 20 pages from the 5 engines. They also performed their own relevance judgments, rating pages as being duplicate links, inactive links, or having a score from 0 (not relevant) to 3 (highly relevant). A formula was devised that gave the highest weight to relevance in the top 3 links (because they would appear on the user’s screen) followed by the fourth through tenth links (because they would appear on the first results page), and then the remaining 10 links (which would appear on the second results page). Five different scoring measures were calculated, based on different levels of relevance (scores between 1 and 3) and on whether duplicates were included. In all five experiments, the top 3 search engines—AltaVista (www.altavista.com), Excite, and InfoSeek (www.infoseek.com)—consistently outperformed the remaining 2, HotBot
290
III. Research Directions
(www.hotbot.com) and Lycos (www.lycos.com). The ranking of the top 3 search engines relative to each other varied across the 5 experiments. Lycos was found to have the highest number of dead links, while HotBot was found to have the most duplicates. Gordon and Pathak (1999) compared 8 search engines accessed in 1998. Their experiment actually used mediated searching, with information specialists searching on 33 topics submitted by members of the faculty at the University of Michigan School of Business, who filled out a five-part form detailing their information requests. The searchers constructed queries as they saw fit with each of the eight search engines, aiming to create one that retrieved the best 200 results for each. For each engine, after their final search was done, the top 20 documents were printed. Duplicates were eliminated and the results were then presented in random order to the faculty member who had requested the search. He or she was asked to rank each on a four-point scale of relevance: highly relevant, somewhat relevant, somewhat irrelevant, or highly irrelevant. Pages that were highly or somewhat relevant were considered to be relevant for the purposes of calculating recall and precision. These measures were calculated at each rank. Recall and precision varied widely over the first few documents but then settled into a pattern of rating (and achieving statistical significance at some document cutoff levels via Tukey’s test): 1. Highest for AltaVista and OpenText (www.opentext.com): precision at 10 and 20 documents between 36 and 40%, recall at 20 documents between 13 and 14% 2. Medium for Excite, Lycos, Infoseek, and HotBot: precision at 10 and 20 documents between 25 and 34%, recall at 20 documents between 7 and 11% 3. Lowest for Yahoo (www.yahoo.com): precision at 10 and 20 documents between 15 and 18%, recall at 20 documents between 5 and 6% Another finding of this study was a very low overlap among all pages retrieved and relevant pages retrieved. No matter whether the top 20 or top 200 documents retrieved were being analyzed, the percentage of all documents and relevant documents retrieved by only one search engine was always greater than 90%. Hawking et al. have also assessed commercial search engines and attempted to compare their results with those obtained in the TREC Web Track. In their first study, they began by comparing the output of 20 search engines accessed in 1999 with 54 topics from the TREC-8 Web Track (Hawking, Craswell et al., 2001b). Relevance assessments were done by individuals with university degrees but no specific expertise in IR, using a binary (relevant/nonrelevant) scale. The top 20 non-dead-link pages were collected from each engine and assessed with a variety of precision-oriented measures: • MAP [called TREC-style average precision (TSAP) in the paper] • Mean reciprocal rank of the top relevant document • Precision at 1 to 5 documents retrieved The results of the commercial systems, which included metasearch engines (Metacrawler, www.metacrawler.com, and Inquirus, www.neci.nec.com/lawrence/in-
8. Lexical–Statistical Systems
291
quirus.html), directories (Yahoo and LookSmart Australia, www.looksmart.com.au), and all the major Web crawler search engines. Although the performance of different engines varied with different measures, rank orders tended to be similar; that is, those ranking high or low with any one measure tended to rank high or low with all the measures. Statistical analysis showed the top half of the systems have insignificant differences in performance, but Northern Light (www.northernlight.com), Inquirus, and Google (www.google.com) tended to rank the highest consistently. The authors next proceeded to compare the same measures with the results of systems that had participated in the TREC-8 Web Track. The document collection used was from a smaller crawl of the Web at a different point in time, 1997, which severely limited this study. The best TREC systems, however, were found to have statistically significant better results with all the foregoing measures. In a follow-up study, Hawking and colleagues (2001a) compared most of the same engines with the 54 topic relevance queries and 106 new queries of the type looking for an online service, such as ordering flowers or finding animal sounds. In these experiments, Google performed the best for both types of query, showing results statistically significantly better than all systems except Northern Light and FAST (www.alltheweb.com). Singhal and Kaszkiel (2001) took exception to the comparison of commercial search engines with TREC results by Hawking et al. Noting that the Web page collections in the two studies were vastly different, Singhal and Kaszkiel performed a new study of Web-style versus TREC-style comparisons. Also pointing out that Web searchers are often looking for specific pages, they chose to evaluate a page-finding task. For queries, they selected at random 100 pagefinding queries from a large log of Excite queries, defined as those with the name of an entity and the words homepage or website present. The latter words were stripped out and searching success was measured by identifying the home page of the entity. The document collection consisted of a 217-gigabyte, 18-million-page crawl done in October 2000. Pages listed in the output of the commercial search engine output were manually added to the collection. The output was scored based on the rank of page to be found. The “TREC algorithm” consisted of using the PN weighting scheme with a second pass over the documents incorporating query expansion (the top 10 documents assumed to be relevant). The specific algorithm was deemed to be representative of TREC because when it was run against the TREC-7 and TREC-8 ad hoc tasks, the results achieved were comparable to the best performing systems from those years. The results showed that two commercial search engines performed best, with the number of queries identifying the target page in 85 to 95% of the queries. Two other search engines found the target page about 70% of the time. The TREC algorithm found the target page only 50%, although a variant omitting the query expansion step improved its performance to 60%. The top two performing engines were found to be aided by RealNames (www.realnames.com), a commercial service that links queries of this type to home pages. Nonetheless, these com-
292
III. Research Directions
mercial systems clearly outperformed a TREC algorithm known to be successful in ad hoc retrieval tasks. The authors speculated that query expansion was detrimental to the effectiveness of the TREC algorithm because it tended to transform a highly specific query into a more general one. For example, a query seeking a specific health insurance company, Horizon Blue Cross Blue Shield (www.bcbsnj.com), was changed into a general query about health insurance companies. 8.3.2.3 Analysis of the Web Several lines of research have investigated different aspects of the Web, with most focusing on its link structure. One line has looked at the “bibliographic” aspects of the Web, attempting to determine what meaning can be extracted from linkages in much the same way that author citations and journal impact factors provide an insight into scientific productivity. Recall from Chapter 6 that the Google search engine uses links into a page as a means to rank output in its search engine. Ingwersen (1998) has proposed a Web impact factor (WIF), which is defined as the proportion of pages that link to a site from some defined area of Web, such as a site or a group sites devoted to a specific topic. It is defined mathematically as follows: WIF
Number of pages linking to a site Total number of pages in site
(18)
Two kinds of pages contribute to the denominator, external-link pages from outside the site and internal-link pages from within the site. Most analyses have used the “external WIF” to omit internal links within a Web site. As an example, consider a Web site of an academic medical informatics program. The external WIF would be calculated by all pages that point to that site (e.g., 10) divided by some universe of pages considered to be in the realm of medical informatics (e.g., 1,000). The external WIF would be 10/1000 0.01. The external WIF can be calculated from Web search engines (limited insofar as the engines do not completely cover the entire Web). For example, the AltaVista query +link: -url: will return all Web pages that point to a specific Web domain except those within the domain. The number of such pages divided by the number of pages in a site gives the external WIF. Almind and Ingwersen (1998) used the WIF to determine that Denmark was less visible on the Web than other Nordic countries. While Smith (1999) found no correlation between research output (measured in numbers of papers published) and the external WIF for Australasian universities, Thelwall (2000) did find a positive trend between the WIF and a measure of general rating for 25 British research universities. However, Thomas and Willett (2000) found such an association when they narrowed the view to British library and information science departments. Smith also considered whether the WIF could serve as a sort of journal impact factor for Web-based journals and found it to be unreliable.
8. Lexical–Statistical Systems
293
Other Web research has looked at its mathematical properties. Two researchers have noted that the distribution of in-links and out-links on the Web follows a power law. That is, the probability of a page having k links is P(k) k
(19)
where has been found experimentally to vary between 2 and 3 (Albert, Jeong et al., 1999; Broder, Kumar et al., 2000). Albert et al. also measured the “diameter” (average number of links from one randomly selected page to another) of the Notre Dame University Web site and found it to be 18.6. Broder et al. (Broder, Kumar et al., 2000) found that the larger Web was not completely interconnected and instead consisted of four roughly equal components described in Section 1.5.3: a strongly connected core (SCC), IN links to the SCC, OUT links from the SCC, and tendrils not connected to any of the first three (Figure 1.6a). Their analysis also showed that the links from the SCC to the OUT pages tended to be higher in quantity and more diverse than those coming to the SCC from the IN pages. They also found the directed diameters of the SCC to be 28 pages. Section 1.5.3 also introduced the notions of hubs and authorities on the Web. Kleinberg (1998) actually developed the mathematical means to define the latter based on linkages from the former. Although it uses different calculations, the Google search engine also can be viewed as determining authority based on the number of in-links of pages (Brin and Page, 1998). 8.3.2.4 Automated Generation of Hypertext Another interesting research problem relevant to hypertext and the Web is whether automated means can assist with the laborious process of manually designating hypertext linkages. Section 1.5.1 described several taxonomies of links. Allan (1997) notes that links can also be divided into those most appropriately assigned by manual and automated links. He states that the former can be idiosyncratic and are beyond the means of current text processing approaches, although a number of automated types of links can be generated, including the following: • Revision links: tracking versions of an object • Summary and expansion links: the former begins a summary on a topic and the latter expands to greater detail • Equivalence links: strongly related documents • Comparison and contrast links: identifying similarities and differences across texts • Tangent links: topics related in an unusual manner • Aggregate links: grouping together related documents The process of identifying such links begins with a similarity calculation across all documents and its parts (i.e., paragraphs, sentences, etc.) by means of a cosine similarity measure. Specific links can be generated (or suggested) based on properties of the connectedness of the documents. For example, tangential links can be identified by their disconnectedness with other documents that are other-
294
III. Research Directions
wise linked. Likewise, aggregate links can be constructed from nontangential links that are highly interconnected. Agosti et al. (Agosti, Melucci et al., 1995) use a three-layered approach to identify candidate documents for potential linkage. The base level is the document. The next level consists of the index terms that comprise the documents. The top level contains concepts whose linkages to the index term level is determined by automated classification techniques. Another significant challenge in automated hypertext construction lies in devising the means to evaluate its output (Melucci, 1999). One problem is that what constitutes a “good” hypertext linkage is not known, whereas there is at least general agreement on what comprises a relevant document. Manually generated hypertext linkages could be used, but there would need to be proof that they had been built in a consistent manner by different individuals. Another approach might be a “task-oriented” one along the lines of some of the studies presented in Chapter 7, but studies looking explicitly at automatically generated hypertexts have not been done, especially on the Web.
8.3.3 Question Answering Although the name of this field, like the subject of this book, is “information retrieval,” the reality is that most systems and approaches are concerned with retrieval of documents. Despite work on computer-generated answering to questions dating back decades [e.g., the LUNAR system to answer questions in a database about lunar rocks from space missions (Woods, 1970)], the IR community has only recently shown an interest in this area. Beginning with TREC8, a Question-Answering Track was developed that attempted to move systems from document retrieval toward information retrieval (Voorhees and Tice, 2000). The original TREC-8 Question-Answering Track consisted of all the document collections of TREC disks 4 and 5 (see Table 8.2) except the Congressional Record. The questions for answering came for a variety of sources, including track participants, NIST staff and assessors, and an Internet frequently asked questions (FAQ) log. Sample questions included Who was the first American in Space? and Where is the Taj Mahal? Track participants had to provide not only the document that answered the question, but also a span of text that contained the answer, with NIST assessors judging the submitted strings for correctness. The entire process had to be automatic; that is, users could not take part. In TREC-8 there were two subtasks: answers limited to 50-byte span and answers limited to 250-byte span. In general, results of the first year varied widely. The best results for both tasks showed an MRR of around two-thirds. While most groups doing both tasks scored higher on the 250-byte task (i.e., it was easier), the best overall MRR came from one group’s 50-byte task. When the answer was found, the document containing it tended to be ranked high. In general, traditional best-match document retrieval approaches worked reasonably well in the 250-byte task, but more linguistic processing was required for the 50-byte task. Examples of linguistic knowledge used for the latter included trying to identify
8. Lexical–Statistical Systems
295
people or organizations for “who” questions and time designations for “when” questions. (Linguistic approaches used in the Question-Answering Track and other areas of TREC are described in Chapter 9.) Some of the wrong answers to the sample questions above demonstrated the challenge of the task. In looking for answers to where the Taj Mahal was located, many of the newswire stories in the collection referred to the Taj Mahal Casino in Atlantic City, New Jersey, not the tomb in Agra, India. Likewise for other question, one document referred to former California Governor Jerry Brown as someone “who has been called the first American in space” and some searches returned that answer accordingly. The track continued in TREC-9 (Voorhees, 2000), which used a larger document collection (all newswire of all disks from the Associated Press, Wall Street Journal, San Jose Mercury News, Financial Times, Los Angeles Times, and Foreign Broadcast Information Service). In addition, the number of questions was increased to 693, and were taken from Internet searching logs. The base set of questions numbered 500, and 193 “syntactic variants” added to determine the ability to answer the same question phrases in different ways. Once again, there were subtasks with 50-byte and 250-byte limits. In the TREC-9 track, in addition, answers had to be supported by the document: that is, an answer had to be explicitly stated, with supporting evidence. The TREC-9 results resulted in overall lower MRR scores, likely indicating that the questions were more difficult. The range of the top 10 MRR scores for the 50byte task varied from 0.23 to 0.58, with 34 to 66% of answers not found. For the 250-byte task, the top 10 MRR scores varied from 0.37 to 0.76, with 14 to 51% of answers not found. The performance of one group far exceeded that of the rest of the participants (Pasca and Harabagiu, 2001). Their approach utilized a variety of steps, including use of an answer-type taxonomy to identify a named entity (such as person or quantity), passage retrieval to narrow the likely location of the answer in the document, and machine learning techniques to extract the answer from the location. Details are provided in Chapter 9. In TREC-2001, the document collection stayed the same but other modifications were made (Voorhees, 2001). The track was driven in part by 5-year plan for question answering under the auspices of several U.S. government research agencies (Burger, Cardie et al., 2000). In the TREC-2001 track, only the 50-byte limit was used. Furthermore, the track was divided into three tasks: 1. Main task: the same task as earlier tracks, modified by the possibility that no answer was present (i.e., the correct answer was “no answer”) 2. List task: the answer consisted of a specified number of instances of a particular kind of information (e.g., nine countries that import Cuban sugar) 3. Context task: a series of questions that require systems to track discourse (or context) of the questions, for example: a. How many species of spiders are there? b. How many are poisonous to humans? c. What percentage of spider bites in the United States are fatal?
296
III. Research Directions
TABLE 8.6. Graesser’s Taxonomy of Questions Question 1. Verification 2. Comparison 3. Disjunctive
4. Concept completion
5. Definition
6. Example
7. Interpretation
8. Feature specification
9. Quantification
10. Causal antecedent
11. Causal consequence
12. Goal orientation
13. Enablement
14. Instrumental/ Procedural
Abstract specification Is a fact true? Did an event occur? How is X similar to Y? How is X different from Y? Is X or Y the case? Is X, Y, or Z the case? Who? What? When? Where? What is the referent of a noun argument slot? What does X mean? What is the superordinate category and some properties of X? What is an example of X? What is a particular instance of the category? How is a particular event interpreted or summarized? What qualitative attributes does entity X have? What is the value of a qualitative variable? What is the value of a quantitative variable? How much? How many? What caused some event to occur? What state or event causally led to an event or state? What are the consequences of an event or state? What causally unfolds from an event or state? What are the motives behind an agent’s actions? What goes inspired an agent to perform an action? What object or resource enables an agent to perform an action? How does an agent accomplish a goal? What instrument or body part is used when an agent performs an action? What plan of action accomplishes an agent’s goal?
Examples Is an F-test a type of statistic? Did it rain yesterday? In what way is Florida similar to China? How is an F-test different from a t-test? Do the mountains increase or decrease the rain in Oregon? Did he order chicken, beef, lamb, or fish? Where are the large population densities in North America? Who wrote the song? What did the child steal? What is a factorial design? What does interaction mean?
What is an example of an ordinal scale? What experiment supports this claim? Does the graph show a main effect for “A”? What happened yesterday? What is George like? What color is the dog?
How many rooms are in the house? How much profit was made last year? How does warm air get to Ireland? Why is the kite going backward?
What happens to the warm winds when they reach the mountains? What are the consequences of double-digit inflation? Why did Roger move to Chicago? What was the purpose of the city’s cutting taxes? What device allows you to measure an earthquake? What do I need to bake this fish? How does a person perform long division? How do you move a mouse on a computer?
8. Lexical–Statistical Systems
297
TABLE 8.6. (Continued) Question 15. Expectational 16. Judgmental
17. Assertion
18. Request/ Directive
Abstract specification Why did some expected event not occur? The questioner wants the answer to judge an idea or to give advice on what to do. The speaker expresses that he or she is missing some information. The speaker directly requests that the listener supply some information.
Examples Why wasn’t there a war in Iraq? Why doesn’t this doll have a mouth? What do you think about the new taxes? What should I do to stop the fight? I don’t understand what this message on the computer means. I need to know how to get to the Newark airport. Please tell me how to get a printout of this file.
Source: Graesser, Byrne et al. (1992).
The results of the main task showed the MRR to be comparable to prior years, with the range of MRR in the top 10 performers varying from 0.35 to 0.68 and with 31 to 49% of answers not found (Voorhees, 2001). The list task run was scored by a measure of accuracy (i.e., the proportion of instances correct), with the range of the top ten varying from 15 to 76%. Only a small number of groups participated in the context task. Interestingly, incorrectness on early questions did not seem to affect performance on later ones, indicating that the initial question could identify a set of documents small enough to permit later questions to be answered correctly even if earlier ones had received wrong responses. The 5-year plan for question-answering research will likely ensure continued development of new experiments and systems (Burger, Cardie et al., 2000). The road map document notes that such systems must be timely and accurate with their answers, meet the needs of users, provide complete answers to questions, and deliver information relevant to the context of the user. A broad diversity of question types will be considered, such as those in the taxonomy put forth by Graesser (1992) and listed in Table 8.6.
8.3.4 Text Summarization Another important application of automated IR is text summarization, which is the process of distilling the most important information from documents to produce a shorter version for specific users and/or tasks (Mani and Maybury, 1999). There are a variety of commercial text summarization systems available, including the AutoSummarize facility in the Microsoft Word application (Microsoft Corp., www.microsoft.com). There are two types of output from text summarization systems: 1. Extract: summary consisting of material copied from document 2. Abstract: summary with some material not present in document, such as more general terms or paraphrasing of content
298
III. Research Directions
Mani and Maybury (1999) note three important characteristics of automatically generated summaries: 1. Reduction: the compression or condensation rate, measuring the length of the summary, usually in words, divided by the length of the original source 2. Informativeness: how relevant the information to the user 3. Well-formedness: how well extracts avoid gaps, ambiguous anaphors, and incoherent reductions in lists and tables and the readability of output produced from abstracts Of the two summarization tasks, extraction has proved easier than abstraction. It has also been easier to evaluate, since after a test collection of “correctly” extracted sentences has been built, variants of recall and precision can be used to measure whether correct sentences are extracted and noncorrect ones are not extracted, respectively. Some of the features used to determine which sentences to extract include the following: 1. Location: in document or specific part (e.g., section, title, introduction, or conclusion) 2. Thematic: presence of statistically important terms in sentence (e.g., those with high TF*IDF) 3. Fixed phrases: Those giving cues (e.g., “in summary”) or emphasis (e.g., “in particular”) to important sentences 4. Weight: added to text units containing terms in title, heading, opening paragraph, or user’s interests 5. Cohesion: text units connected based on repetition or presence of synonyms Most document summarizers (Kupiec, Pedersen et al., 1995; Mani and Bloedorn, 1998) use machine learning techniques to determine the optimal use of these features. Abstraction usually requires natural language processing techniques of the types described in Chapter 9. Documents are usually processed to slot important features into a template or to recognize key concepts. Evaluation usually requires some sort of human judge to assess the quality of the output.
8.4 User Evaluation of System-Oriented Approaches Throughout the chapter, concerns have been raised over the batch-oriented approach to evaluation of lexical–statistical IR systems and its shortcomings. The results of studies that have assessed and compared these systems in real-world settings indicate that these methods can be implemented and used effectively, especially by novice users. In this section, these studies and the lessons learned from them are described.
8.4.1 Early Studies One of the first studies to evaluate a lexical–statistical system with real users was performed with CIRT, a front end to an online library catalog at City University
8. Lexical–Statistical Systems
299
in London that featured Boolean and natural language word-based searching, with the latter using term weighting, relevance ranking, and probabilistic relevance feedback (Robertson and Thompson, 1990). In the evaluation study, end users were assisted in using the system by librarian intermediaries, randomized to either Boolean or weighted searching. Users were given offline prints and asked to provide relevance judgments for up to 50 documents. Both user and intermediary filled out questionnaires to document subjective aspects of the system. The results showed essential equivalence between the systems in terms of recall, precision, user effort, cost, and subjective user interactions. As noted in Chapter 7, Hersh and colleagues have compared Boolean and natural language searching with bibliographic and full-text databases using both relevance-based and task-oriented measures (Hersh, Elliot et al., 1994; Hersh and Hickam, 1994; Hersh and Hickam, 1995). These studies have shown minimal differences in searching performance between the two approaches. Turtle (1994) has achieved different results upon comparing Boolean and natural language searching in two legal databases. Forty-four natural language information need statements were given to expert searchers, who were asked to use the Westlaw Inference Network (WIN) system searching to create Boolean queries. They were allowed to iterate with the system to find an optimal strategy, performing an average of 6.7 searches against the system. Both the natural language statements and the Boolean queries were then run against the databases. Relevance judgments from earlier experiments were used to calculate recall and precision. In contrast to foregoing studies, Turtle’s results showed a marked benefit for natural language over Boolean searching, although no statistical analysis was performed. Nonetheless, recall and precision at 20 documents was about 24 to 35% higher for the two databases. A clear methodological limitation in this study was its direct comparison of the Boolean and natural language output, since WIN’s Boolean output was given in reverse chronological order instead of being ranked. It was unknown whether these results were due to the lack of ordering of the Boolean sets. But since the studies described earlier found no difference in the two types of searching in operational tasks, the ordering of Boolean sets may have had more impact on recall and precision results than the user’s ability to successfully interact with the system. Some investigators have attempted to assess which lexical–statistical system users will actually employ when given a choice. Dumais and Schmitt (1991) assessed an interactive LSI system that allowed two interactive search methods: a LookUp function that allowed the user to enter a new query and a LikeThese function that provided a new query based on LSI-based relevance feedback. Fiftyseven college students searched 10 questions each in a newspaper database. Students were more likely to use LikeThese searches, with or without LookUp searches. The LikeThese searches obtained a higher number of relevant documents in the top-10-ranked items. A look at usage of relevance feedback has also been performed in the interactive portion of the TREC experiments. Users of both the Okapi (Robertson, Walker et al., 1994) and INQUERY (Koenemann, Quatrain et al., 1994) systems were found to use relevance feedback about once
300
III. Research Directions
per search. Clearly, more studies need to be done in operational settings to determine the benefit and role of all the lexical–statistical techniques that have shown to be beneficial in non-real-world searching environments.
8.4.2 The TREC Interactive Track As noted in Section 8.1, the TREC experiments have for the most part used batchstyle searching evaluation with no human involvement in the query process. While some manual modifications of queries have been allowed in various TREC tracks, the emphasis has always been on building automated systems. Nonetheless, most groups in the ad hoc task that employed manual modification of queries found it to enhance results. When users modified INQUERY’s automated search statements (produced from processing of the query text), a 15.5% improvement in MAP was seen (Callan, Croft et al., 1995). Likewise, Cooper et al. (1994) had human intermediaries transform queries into word lists by searching on a nonTREC newspaper database, which resulted in a 10% MAP improvement in searches on the TREC database. 8.4.2.1 Methods There has always been a nidus of interest in interactive evaluation at TREC (Over, 2000). The Interactive Track was originally formed for TREC-3. In the first few years, the track used the queries and documents from the main TREC tasks. In TREC-3, several groups attempted to compare the manual construction of routing queries with the automated approaches. Unlike the gains seen from human modification of ad hoc queries, routing queries constructed by humans fare less well in terms of MAP. Koenemann et al. (1994), for example, taught librarians how to use INQUERY to develop routing queries based on the training data, which were then run against the test data. Their performance was much poorer than the automated runs with INQUERY (Broglio, Callan et al., 1994). Likewise, Beaulieu et al. (1996) found that for the Okapi system, automated runs performed better than manual ones. The human difficulties in these experiments were attributed to the difficulty users had in building training queries. The Interactive Track became more independent starting with TREC-6, where an instance recall task was employed (Hersh, 2001). Recognizing that the number of relevant documents retrieved was not a good indicator of a user’s performance with an IR system, the track moved toward a task-oriented approach. The approach chosen was to measure how well users could define instances (called “aspects” at TREC-6 but “instances” subsequently) of a topic. For example, How many stars have been discovered by the Hubble Telescope? or How many countries import Cuban sugar? The database chosen for searching was the Financial Times 1991–1994 database of the overall TREC collection (see Table 8.2), consisting of 210,158 documents (articles). In the searching experiments, users were asked to save the documents
8. Lexical–Statistical Systems
301
that contained instances and record the specific instances identified. As with the development of document relevance judgments, assessors at NIST devised a set of “correct” instances and the documents they occurred in based on the results submitted from participating sites. This allowed the primary dependent variable, instance recall, to be measured: Instance recall
Number of relevant instances in saved documents (20) Number of relevant instances for topic
One of the challenges for the Interactive Track has been to define the degree of standardization of experiments across sites. Certainly one of the values of TREC in general has been the standardization of tasks, queries, and test collections. Counter to this, however, are not only the varying research interests of different participating groups, but also the diversity of users, settings, and available systems. The design of the TREC-6 Interactive Track attempted to measure the effect of the system, minimizing differences between sites and searchers. Sitespecific effects were minimized by use of a common control system, the NIST zPRISE system (see Table 1.3), which was installed by every group. Searcherspecific effects were minimized by a Latin squares design for topics and the order in which they were searched by each subject. Unfortunately, the use of a common control system introduced problems of its own. Not only were most participating groups interested in comparing their own systems against controls of their own choosing, but a statistical analysis showed that the approach used in TREC-6 was incapable of filtering out site-specific interactions (Lagergren and Over, 1998). In the TREC-7 Interactive Track, the common control system was abandoned and each site was free to use its own control system. Beginning with TREC-6, a number of other standardized procedures were undertaken. In particular, common data were collected about each user and each of their searches. The data elements were collected from common questionnaires used by all sites or extracted from search logs (Table 8.7). Searches were also given an explicit amount of time for searching: 20 minutes in TREC-6 and TREC8 and 15 minutes in TREC-7. Sites were also asked to provide a narrative of one user’s search on one topic each year. The instance recall approach was used for three successive years of the Interactive Track. With TREC-9, a new task was adopted: question answering (Hersh and Over, 2000). While questions from the noninteractive TREC-8 QuestionAnswering Track were considered for use, preliminary assessment found them too easy for the interactive task. Instead, a number of candidate questions were developed by the participating research groups inspired more by the data at hand than by any systematic considerations. Four types were considered and tested to gauge their suitability: • Find any n X’s (e.g., Name three US Senators on committees regulating the nuclear industry.) • Comparison of two specific X’s (e.g., Do more people graduate with an MBA from Harvard Business School or MIT Sloan?)
302
III. Research Directions
• Find the largest/latest/ . . . n X’s (e.g., What is the largest expenditure on a defense item by South Korea?) • Find the first or last X, (e.g., Who was the last Republican to pull out of the nomination race to be the candidate of his/her party for US president in 1992?) In the end, eight questions were chosen, four of each of the first two types. Examples included Name three US national parks where one can find redwoods and Is Denmark larger or smaller in population than Norway? Questions of the last two types were deemed too difficult to find or create and, given their “superlative” nature, seemed less likely for users to be able to do successfully. The TREC Interactive Track continued many of the other standard approaches adopted for TREC-6 through TREC-8. One change was an expansion of the collection used to include more collections from the Associated Press, Wall Street Journal, San Jose Mercury News, Los Angeles Times, and the Foreign Broadcast Information Service. However, the same user questionnaires were employed and the searching logs were again processed to obtain most of the data listed in Table 8.7. The correct answers to the questions were determined by NIST assessors, with searcher results judged as either completely correct, partially correct, or incorrect. Most sites required the answer to be completely correct in their data analysis. Most participating research groups have assessed a small number of research questions in their experiments over the years in the Interactive Track. Hersh and colleagues, for example, compared Boolean and natural language searching interfaces in TREC-7 and compared weighted schemes shown to be more effective in batch studies with real users in TREC-8 and TREC-9. This group also attempted to determine the factors associated with successful searching, similar to the experiments they performed using MEDLINE described in Section 7.4.2.3 (Rose, Crabtree et al., 1998; Hersh, Crabtree et al., 2000; Hersh, Crabtree et al., 2002). 8.4.2.2 Comparing Boolean and Natural Language Searching The studies comparing Boolean and natural language also built on work initiated earlier in the medical domain and discussed in Chapter 7 (Hersh, Elliot et al., 1994; Hersh and Hickam, 1994; Hersh and Hickam, 1995). The TREC-7 study of Hersh et al. (Hersh, Turpin et al., 2001) compared these two approaches with the instance recall task. A group of highly experienced information professionals (mostly librarians) was recruited to take part. The Web-based interfaces used are shown in Figures 8.2 and 8.3. Both interfaces used the MG system (see Table 1.3) as the retrieval system. Similar to the recent MEDLINE studies (Hersh, Crabtree et al., 2000; Hersh, Crabtree et al., 2002), three additional cognitive tests (VZ-2, RL-1, and V-4) and QUIS were administered to users. As seen in Table 8.8, the users performed comparably on both systems, similar to most earlier studies comparing these two types of interface. However, there
8. Lexical–Statistical Systems
303
TABLE 8.7. Common Variables Collected by All Research Groups for the TREC Interactive Track Variable Demographic Gender Age Experience Years Point Catalogs CD-ROM Online WWW Frequency Enjoy Cognitive traits FA1 Satisfaction post topic Familiar EasyStart EasyUse Satisfied Confident TimeAdequate Satisfaction postsystem SysEasyLearn SysEasyUse SysUnderstand Satisfaction postexperiment Understand TaskSim TaskDiff Search mechanics Saved DocRec Time Terms Viewed Seen Cycles
Definition Male vs female In years Years experience of online searching (1, least; 5, most) Experience with point-and-click interface (1, least; 5, most) Experience using online library catalogs (1, least; 5, most) Experience using CD-ROMS (1, least; 5, most) Experience searching commerical online systems (1, least; 5, most) Experience searching Web (1, least; 5, most) How often searching done (1, least; 5, most) How enjoyable searching is (1, least; 5, most) Controlled associations test to assess associational fluency (TREC-7 only) User familiar with topic (1, least; 5, most) Search was easy to get started (1, least; 5, most) Search was easy to do (1, least; 5, most) User was satisfied with results (1, least; 5, most) User had confidence that all instances were identified (1, least; 5, most) Search time was adequate (1, least; 5, most) System was easy to learn to use (1, least; 5, most) System was easy to use (1, least; 5, most) User understand how to use system (1, least; 5, most) User understand nature of experimental task (1, least; 5, most) Task had similarity of other searching tasks (1, least; 5, most) Systems were different from each other (1, least; 5, most) Documents saved by user Document recall (relevance defined as having one or more instance) Time in seconds for search Number of unique terms used for topic Number of documents viewed for topic Number of documents seen for topic Number of search cycles for topic
were other differences between the systems. Topics using the natural language interface resulted in more documents being shown to the user (viewed). However, fewer documents were actually selected for viewing (seen) when that interface was used, probably because users had to spend more time scrolling through the document titles shown. This group of highly experienced searchers also clearly preferred the Boolean interface, no doubt in part owing to their familiarity with it.
304
III. Research Directions
FIGURE 8.2. The natural language interface for TREC Interactive Track experiment of Hersh et al.
8.4.2.3 Do Batch and Interactive Experiments Give the Same Results? In TREC-8 and TREC-9, Hersh and colleagues addressed the question of whether results from batch searching experiments are comparable to those obtained by real users. That is, are the results found when batch-style experiments and measures like MAP are used congruent with those obtained by real users with retrieval tasks using the same systems. To test this question, three-part experiments were performed. First, previous TREC Interactive Track data were used to determine the “best-performing” system under batch experiments. This was done by creating a test collection that designated documents with instances as relevant and running traditional batch experiments to generate MAP results. These experiments found that Okapi weighting (TREC-8) and Okapi plus PN weighting (TREC-9) performed much better than “baseline” TF*IOF weighting. Next, the interactive user experiments were carried out, with the best-performing system serving as the “experimental” intervention and the TF*IOF system as the “control.” In the final step, the batch experiments were repeated with the data generated from the new interactive experiments, to verify the results held with the topics employed in the user
8. Lexical–Statistical Systems
305
FIGURE 8.3. The Boolean interface for TREC Interactive Track experiments of Hersh et al.
TABLE 8.8. Outcomes of TREC Interactive Track Study Comparing Boolean and Natural Language Searching Variable
Boolean
Natural language
Instance recall Total search terms Documents viewed Documents seen Documents saved Postsystem easy to learn (1–5) Postsystem easy to use (1–5) Postsystem easy to understand (1–5) Postsystem QUIS average Postexperimental easier to learn (number of subjects) Postexperimental easier to use (number of subjects) Postexperimental liked better (number of subjects)
34.6% 6.59 141.1 14.68 5.32 3.45 2.95 3.42 4.61 17 19 19
34.2% 6.15 241.4 13.58 4.65 2.92 2.13 3.04 4.09 6 4 4
Source: Hersh et al. (1998).
306
III. Research Directions
experiments. (The experiments had to be done this way to accommodate the general approach of TREC in determining results from data generated in current experiments. Thus, the batch experiments calling for the data from the interactive experiments could not take place until those experiments had been performed.) In the TREC-8 instance recall task, Hersh et al. (1999) found in the first batch experiment that Okapi weighting performed best, achieving an 81% better MAP than TF*IOF weighting. So for the user experiments that followed, Okapi weighting served as the experimental system and TF*IOF weighting as the control. All subjects used the natural language interface (see Figure 8.2) to the MG system. Highly experienced searchers were again employed. The results showed a better than average instance recall for Okapi over TF*IOF weighting, 29 vs 33%. However, as seen in Figure 8.4, all of the benefit was seen in one query, and the results were not statistically significant. In the final batch experiments, the Okapi weighting achieved a 17.6% benefit with the data that had actually been used by the interactive searchers. These experiments also found a positive and statistically significant linear relationship for instance recall with the following variables: • The number of documents saved as having instances by users • Document recall (defining a relevant document has one having one instance or more) • The number of documents selected for viewing by the user (seen)
FIGURE 8.4. Instance recall (mean and standard deviation) achieved by systems employing Okapi and TF*IDF weighting for six interactive queries (Hersh, W., Turpin, A., et al. (2001). Challenging conventional assumptions of automated information retrieval with real users: Boolean searching and batch retrieval evaluations. Information Processing and Management, 37: 383–402, Elsevier Science.). The benefit for the Okapi weighting in batch experiments is listed as a percentage value above the interactive mean and standard deviation. (Courtesy of Elsevier.)
8. Lexical–Statistical Systems
307
The same experiment was repeated in TREC-9 by Hersh et al. (2000), using the question-answering task. In the initial batch experiments here, however, a combination of Okapi plus PN weighting achieved the highest MAP over the TF*IOF baseline, 58%. The user experiments thus employed Okapi plus PN weighting as the experimental system. The same user types as well as the same natural language interface (Figure 8.2) to the MG system were used. In the user experiments, the proportion of questions answered correctly was virtually identical between systems, 43.0% for TF*IOF weighting and 41.0% for Okapi plus PN weighting. In both the TREC-8 and TREC-9 experiments, use of the same user interface precluded measuring user satisfaction differences between systems. The results of the final batch experiments were comparable to those of the TREC-8 experiment: the Okapi plus PN weighting achieved a 31.5% MAP improvement over the TF*IOF weighting. A follow-up experiment attempted to determine why the “improved” (Okapi weighting in TREC-8 and Okapi plus PN weighting in TREC-9) systems failed to lead to better performance over the TF*IOF system in the hands of real users (Turpin and Hersh, 2001). Analysis of the search output found that the users retrieved more relevant documents as well as more instances in the top 10 documents with the improved systems. However, the overall number of relevant documents retrieved was comparable for both systems, although the improved system retrieved fewer nonrelevant documents. Further analysis showed that the average length of documents retrieved by the improved systems was over twice as long. This was to be expected, given that the document length normalization in Okapi and PN weighting give additional weight to longer documents. The most likely explanation from these experiments was that the TF*IOF system provided users access to shorter documents, albeit fewer relevant ones, enabling users to achieve comparable performance with two outcomes measures: instance recall and correctness of answering questions. These results have significant meaning for the myriad of batch experiments performed in TREC and other IR evaluations. Namely, the MAP of systems without human users in the loop is a very limited and possibly misleading measure of performance. Of course the TREC Interactive Track experiments are limited to certain tasks, topics, users, and databases. However, further research must determine how well the results of batch experiments correlate with the performance of real users.
8.4.2.4 Relevance Feedback Other groups have used the TREC Interactive Track to assess different research questions. Belkin et al. (2000) have attempted to determine the utility to users of relevance feedback. Again taking the perspective that something shown to have value in batch experiments will not necessarily benefit users, this group has assessed a variety of approaches to relevance feedback. Their initial experiments (TREC-5) looked at the value of basic relevance feedback, that is, the ability to designate documents as relevant and to add some of their terms to the query to generate a new list of document output. Their observational study found that
308
III. Research Directions
users preferred interactive to automatic relevance feedback; that is, they wanted to use relevance feedback as a device to add new terms to their queries rather than have it blindly suggest more documents. The next experiments (TREC-6) added features to the system, including the ability to specify the use of documents for feedback in a positive or negative way. In addition, the user was given control over terms added to the query; that is, relevance feedback was used as a term suggestion device. Users were randomized to have access to only the positive feedback functions or to both the positive and negative feedback functions. There was a trend toward improved instance recall with the latter system as well as the number of documents saved, but the results were not statistically significant, possibly owing to the small sample size. A follow-up experiment with similar interfaces (TREC-7) showed similar results. Further experiments (TREC-8) compared the use of positive relevance feedback in a term-suggestion manner with a technique called local context analysis (LCA) (Xu and Croft, 1996), which derives a set of terms from the top-ranking documents to suggest a different group of terms for use in the feedback process. Because the latter does not require users to make relevance judgments about documents, there is less work for users. Although the instance recall results were virtually identical for the two systems, users selected more suggested terms in the LCA system for use in their queries, perhaps because they spent less time reviewing and designating relevant documents in the regular relevance feedback approach. Other researchers have investigated relevance feedback as well. Robertson et al. (1998) found that it slightly enhanced instance recall in the Okapi system, though a simple TF*IDF approach outperformed both. Yang et al. (2000) assessed different approaches to relevance feedback. The main focus of their experiments was to compare the use of the whole document for relevance feedback versus selection by the user of specific passages in documents. They hypothesized that the user would do better with the latter, more focused approach. Their results showed no significant difference, but a definite trend in favor of the wholedocument approach. 8.4.2.5 Presentation of Search Results Other research in the TREC Interactive Track has looked at the presentation of search results. Wu et al. (2000) used the instance recall tasks to assess the value of users’ clustering of documents (TREC-7) or classifications of terms (TREC-8). The value of clustering was assessed by comparing a system that grouped the document list by clusters with one that just provided a straight ranked list. The list approach yielded better instance recall than the cluster approach, but the difference was not statistically significant. Feedback showed that users liked the cluster approach but had difficulty understanding exactly how it worked. To analyze the value of term classification, Wu et al. chose an approach that
8. Lexical–Statistical Systems
309
prompted the user to categorize the output of the original natural language query by one of the search terms. For the selected term, a set of related terms was shown from WordNet, a lexical resource described in the next chapter (Fellbaum, 1998). In essence, WordNet contains synsets that represent all the polysems for a given word. After the user chose a category, the documents retrieved in the original search that matched each polysem in the set were shown under the term. Again, the results showed no differences between the two approaches. In the TREC-9 question-answering interactive task, Wu et al. (2001) identified a user interface difference that affected results. They compared two systems that differed in how they displayed their document output (or “surrogates”). One system displayed “answer-indicative sentences” that showed three sentences containing the highest number of words from the query. The other displayed the title and first 20 words of the document. Users had a higher rate of answering questions correctly with the former system (65% vs 47%). A follow-up experiment with a different locally developed test collection showed comparable results. Other researchers have investigated presentation of search results as well. In the TREC-7 instance recall task, Allan et al. (1997) found that separate “aspect windows” allowing a user to maintain different queries and result sets did not improve performance over a baseline TF*IOF system, but a 3D interface highlighting documents belonging to the different instances produced an improvement that was significant. Belkin et al. (2000) compared displays that showed single documents versus 6 documents tiled at a time, with no differences in the results.
8.5 Summary This chapter has summarized the major research carried out under the guise of “automated information retrieval.” The common thread has been a focus on algorithmic approaches to improving indexing and retrieval, with a particular focus on techniques that utilize words in documents and statistical measures applied to them. The evaluative results, mostly coming from the TREC initiative, show consistent benefit for a variety of approaches, such as certain types of term weighting, passage retrieval, and query expansion. However, the sections describing the results of Web and interactive searching results should temper those conclusions, since it is not clear that the approaches studies provide much benefit for real users. Continued user-oriented evaluation must take place to define the optimal uses of these techniques.
Chapter 9 Linguistic Systems
Chapter 8 covered a number of powerful methods for indexing and retrieval based largely on word stems in text. In this chapter, attention is turned to another major area of information retrieval (IR) research, which is the application of linguistic methods. These methods are based on techniques called natural language processing (NLP), which derived from the field of computational linguistics. The chapter begins with an overview of language and computational linguistics. The focus then turns to NLP methods used in IR. As in Chapter 8, a perspective of scalability and evaluation is maintained.
9.1 Rationale for Linguistic Systems in IR Although considerable success in indexing and retrieval can be obtained with the use of matching word stems in queries and documents, individual words do not contain all the information encoded in language. One cannot, for example, arbitrarily change the order of words in a sentence and fully understand the original meaning of that sentence (i.e., He has high blood pressure has a clear meaning, whereas Blood has pressure he high does not). The problem of single words begins with words themselves. Many words have one or more synonyms, which are different words representing the same thing. Some common examples in health care include the synonyms high and elevated as well as cancer and carcinoma. Another frequent type of synonym, especially prevalent in health care, is the acronym, such as AIDS. Sometimes acronyms are embedded in multiword terms (AIDS-related complex) or other acronyms (ARC, which stands for AIDS-related complex). Conversely, many words also exhibit polysemy; which describes a word that has more than one meaning. Consider the word lead, which can represent a chemical, a component of an electrocardiogram, or a verb indicating movement. In discussing polysemy, words are noted to have different senses or meanings. Common words often have many senses. In the Brown Corpus, the 20 most commonly used nouns in English have an average of 7.3 senses, while the 20 most common verbs have 12.4 senses (Kucera and Francis, 1967). 310
9. Linguistic Systems
311
There are also problems beyond the synonymy and polysemy of single words. Words combine together to form phrases, which take on meaning beyond the sum of individual words themselves. For example, the words high, blood, and pressure combine in a phrase to take on a highly specific meaning. Furthermore, phrases exhibit synonymy and polysemy as well. For example, another way of describing the disease high blood pressure is hypertension. But the phrase high blood pressure also exhibits polysemy, inasmuch as it can indicate the disease (which is diagnosed by three consecutive elevated blood pressure readings) or a single measurement of elevated blood pressure. These problems continue up to the most complex levels of language. Thus, certain large phrases have identical words with completely different meaning, such as expert systems used to improve medical diagnosis and medical diagnosis used to improve expert systems, as well as those that have the same meaning but share no common words, such as post-prandial abdominal discomfort and epigastric pain after eating. In sum, these problems highlight the biggest obstacle to computer-based understanding of text, which is the ambiguity of human language. All human language is inherently ambiguous, and the major challenge of computational linguistics is to devise algorithms that disambiguate language as well as possible to allow useful computer applications. Of course, even if one could unambiguously understand a passage of text, there is also the problem of representing its meaning. The artificial intelligence community has long struggled with means of representing knowledge, which has hindered the development of large-scale knowledge-based systems, especially in complex domains like health care (D. Evans, Cimino et al., 1994). A related problem lies in deciding how to represent the information in language. This is usually done by a formal representation schema, such as a semantic network (Sowa, 1991) or conceptual graph (Sowa, 1984). Such formalisms have achieved widespread use in the artificial intelligence community, yet even they have degrees of ambiguity.
9.2 Overview of Linguistics The field concerned with the use and representation of language is linguistics. The subfield concerned with computer programs to understand and generate natural language is computational linguistics. It is a practically oriented field, aiming to develop applications in areas such as speech understanding, questionanswering systems, database querying, and, or course, IR systems. As will be seen, the goal of complete and unambiguous understanding of language has proved quite difficult to attain, and the success of linguistic methods in IR has been modest. There are a number of excellent references in linguistics and its computational aspects (Gazdar and Mellish, 1989; Allen, 1995). This large field cannot be covered in a single chapter of an IR textbook, so this chapter will highlight the gen-
312
III. Research Directions
eral aspects of computational linguistics along with their specific uses and limitations in IR. This section introduces the fundamental definitions and concepts in linguistics to allow understanding of their use in IR.
9.2.1 Branches of Linguistics It has been noted that linguistics is the study of language and how it is used to communicate in the broadest sense. There are three recognized branches of linguistics, which share very little in common except for an interest in the use and understanding of language: 1. Theoretical linguistics deals with producing structural descriptions of language. It attempts to characterize organizing principles that underlie all human languages. 2. Psycholinguistics deals with how people produce and comprehend natural language. It attempts to characterize language in the ways that it explains human behavior. 3. Computational linguistics looks at utilizing language to build intelligent computer systems, such as the types of application listed in Table 9.1. Each branch of linguistics deals with language in a different way, but all recognize the different levels of language, starting with the sounds humans make to the complex meaning that is conveyed, as listed in Table 9.2. Most use of linguistic methods in IR focuses on the middle levels of morphology, syntax, and semantics. Phonology is of concern mostly in speech recognition systems, while problems of pragmatics and world knowledge lack solutions that would allow their use in IR.
9.2.2 Overview of English The next step is to give an overview of the components of English that are addressed in IR systems, beginning with the most basic units and building upward. The lexical–statistical approach to IR considers words or word stems to be the most basic units of written language, but of course they are not. A number of words are composed of more basic units, which are called morphemes. Many words are composed of roots and affixes, which can be prefixes and suffixes,
TABLE 9.1. Applications of Computational Linguistics Abstracting: automated summarization of texts Data extraction: codifying the information in texts Information retrieval: retrieval of texts Machine translation: conversion of texts in one language to another language Question answering: answering factual queries User interface: performing computer tasks based on natural language input Writing assistance: analysis of texts for spelling, grammar, and style
9. Linguistic Systems
313
TABLE 9.2. Levels of Language 1. Phonology: analysis of sound units that make up words, most useful in speech understanding systems 2. Morphology: analysis of parts of words, useful in verb tenses, noun singular/plural. Also helpful in breaking down complex words (i.e., appendic-itis) and equating noun and verb forms (i.e., verb to treat vs noun treatment) 3. Syntax: analysis of relationship of words in a sentence to each other. How words are grouped into phrases, what words modify each other. 4. Semantics: meaning of words, phrases, and sentences. What real-world objects each represent. 5. Pragmatics: how context affects interpretation of sentences. 6. World knowledge: general knowledge of world that must be present to understand discourse.
and roots. An example of a word with a root, a prefix, and a suffix is pretesting, with the root test-, the prefix pre-, and suffix -ing. Some words are composed of bound morphemes, which are units that cannot occur alone. For example, in the word arthroscopy, which refers to the medical procedure of viewing the inside of a joint with a fiber-optic scope, both the prefix arth- and the suffix -oscopy must be attached to another morpheme. Many bound morphemes have many roots that they can attach to, such as -itis, which can combine with virtually any body part to indicate inflammation of that part. The morphemes of a word come together to form lexemes, which are the basic word units. Words of different types vary in their value as indexing terms in IR systems. Particles, prepositions, and determiners are less valuable than nouns, verbs, and adjectives, which is why many words from the former group are likely to be on stop word lists. Words come together to form phrases. As will be seen, phrases reduce the ambiguity of documents and lead to better retrieval in some instances. The phrase that is generally most useful in document retrieval is the noun phrase (NP). NPs vary in complexity from simple one-word nouns to those containing several nouns and adjectives. NPs can be pronouns as well as names or proper nouns (e.g., Bill Hersh, Oregon Health & Science University). In a multiword NP, the main noun is called the head, and the other words are called modifiers. The head is always a noun, but the other words can be either specifiers or qualifiers. Specifiers indicate how many objects there are or how they relate to the talker/writer or listener/reader. They can be quantifiers (any, all), demonstratives (how object is related to speaker, such as this or that, and whether object is identifiable from the situation such as a and the), possessives (my, your), ordinals (first), and cardinals (one). Qualifiers occur after any specifiers but before the NP head. They consist of adjectives, adjectified verbs, or other nouns. Adjectified verbs are verb participles that act as adjectives (the bruised leg). Likewise, nouns can modify other nouns (the heart valve). The two other important types of phrase are the verb phrase (VP) and prepositional phrase (PP). VPs contain a head verb and optional auxiliary verbs. Head and auxiliary verbs combine in different ways to form tenses (simple past, past
314
III. Research Directions
perfect). Some verbs take additional words called particles that modify the verb. These overlap with prepositions, but they must immediately follow the verb or object NP. PPs consist of a preposition and NP. They qualify others parts of sentences, and can be attached to a verb (I gave the stethoscope to Dr. Jones) or noun (I gave the stethoscope from Dr. Jones). Phrases combine to form sentences, which ask, state, or describe things about the world. There are four basic moods of sentences: 1. Declarative: I am in good health. 2. Yes/no question: Am I in good health? 3. Wh-question: Who is in good health?, Why am I in good health? 4. Imperative: Be in good health! The simplest sentences consist of a single NP (the subject) and a verb phrase. The next simplest sentences contain NP-VP-NP, or subject, verb, and object. But sentences can also be very complex, such as a sentence embedded within another (The patient who was just diagnosed with heart disease felt very depressed.).
9.2.3 Phases of NLP As just defined, NLP consists of the computer programs that process language. While NLP techniques make use of many levels of linguistics, often in concert with one another, there are three distinct phases. Parsing is the processing of sentences into syntactic categories with the aid of morphological knowledge. Semantic interpretation is the attachment of meaning to syntactic interpretation of sentences. Contextual interpretation is the understanding of sentences in context. 9.2.3.1 Syntax and Parsing In parsing, a sentence is analyzed to determine its syntactic structure. This structure is specified by a grammar, a set of allowable rules that define how the parts can come together to form larger structures. Each of the words must contain a syntactic category or part of speech (POS) that defines the structures in which it can participate. The parsing process in this section is demonstrated by using the sentence, High blood pressure causes heart disease. TABLE 9.3. A Sample Lexicon Word high blood pressure cause heart disease
Syntactic category ADJ N N V N N
Semantic type Relative measure Body substance Measurement Movement verb Body part Pathologic process
9. Linguistic Systems
315
TABLE 9.4. A Sample Grammar S NP NP NP NP
NP VP NOUN NP ART NP ADJ NP NOUN
PP VP VP VP VP
PREP NP VERB VERB NP VERB NP PP VERB PP
Before parsing can begin, a lexicon of words (and, if desired, bound morphemes) is required. A lexicon should also contain the words’ allowed syntactic categories (e.g., lead is a noun and verb), any idiosyncratic features (e.g., the past participle of the verb to lead is led), and, for reasons to become apparent when discussing semantics, semantic type information. Table 9.3 contains a small lexicon for words in the sample sentence. A grammar consists of a set of rewrite rules stating that a symbol can be replaced by a sequence of other symbols. Symbols that cannot be further decomposed are called terminal symbols, while those that can be further composed are nonterminal symbols. A context-free grammar (CFG) has only a “single nonterminal symbol on the left side” rule. CFGs are important because they can represent a good deal (but not all) of natural language and can be parsed efficiently. Most programming languages have CFGs. A small grammar to parse the sample sentence is shown in Table 9.4. Noam Chomsky defines four types of grammars, often called the Chomsky hierarchy, which are based on their “generative capacity” or ability to handle grammatically complex language (Chomsky, 1965). These are listed in Table 9.5. Chomsky contends that only type 1 and 0 grammars can describe a natural language like English. Unfortunately, from a computer processing point of view, these grammars are the hardest to parse, since they are ambiguous; that is, more than one different correct combination is possible. There are a number of approaches to parsing, but they all fall into two general categories, top down and bottom up. In the former, the rewriting begins with the highest level structure, the sentence (S), and moves down to smaller structures. In the latter, the process begins by rewriting individual words and building up to larger structures, all the way to S. A top-down parse of the sample sentence is shown in Table 9.6. TABLE 9.5. The Chomsky Hierarchy Type 3, regular grammars: can process only regular expressions, which can have at most one nonterminal symbol on the left side of the rewrite rule. Can be represented by a discrete finite automaton. Type 2, context-free grammars: left side of rewrite rule can contain only single nonterminal symbol. A typical context-free grammar is a programming language. Type 1, context-sensitive grammars: more than one symbol can appear on left side of rewrite rules, but there must be more symbols on right than left. Type 0: No restrictions on rules. Source: Chomsky (1965).
316
III. Research Directions
TABLE 9.6. Top-down Parsing of the Sentence, High Blood Pressure Causes Heart Disease S NP VP NP ADJ NP ADJ high NP NOUN NP
NOUN blood NP NOUN NOUN pressure
VP VERB NP VERB causes NP NOUN NP
NOUN heart NP NOUN NOUN disease
The sample lexicon and grammar in Tables 9.4 and 9.5 are obviously quite limited. The lexicon is very small and does not include all the possible parts of speech for the words it contains. For example, cause can be a noun and pressure can be a verb. Likewise, the grammar does not handle tense information of verbs or plurality of nouns. Thus as indicated by this small example, lexicons and grammars for general English tend to be very large. And since language itself is a dynamic entity, building a complete lexicon or grammar is essentially impossible. Furthermore, in the case of grammars, rewrite rules alone are insufficient to represent natural language. Much more complex rules are required to properly parse natural language. The structure most commonly used in an augmented transition network (ATN), introduced by Woods (1970). A variety of computationally efficient approaches have been developed for ATNs, typified by Earley’s algorithm (Earley, 1970). In recent years, there has been an effort to standardize the labeling of parts of speech. Table 9.7 shows one such effort, which comes from the Penn Treebank (www.cis.upenn.edu/treebank/). This project has also generated a large collection of POS-tagged data, with over 3 million words of text from a variety of sources (M. Marcus, Santorini et al., 1994). The largest single collection in the Penn Treebank is the Wall Street Journal, with over 40,000 sentences. A retagged version of the Brown Corpus is also available in this collection. These resources provide not only the materials for a comprehensive lexicon, but also statistics about the English language that aid some of the newer corpus-based approaches to parsing (see Section 9.2.4). An important large lexicon for health care is the SPECIALIST lexicon, developed as part of the Unified Medical Language System (UMLS) (McCray, Srinivasan et al., 1994). Entries in the lexicon can be single words or multiword terms, with the latter consisting of terms that are “atomic” (i.e., have a single meaning). Words or terms that have spelling variants are grouped into single lexical records. Depending on the POS, the base form is either the singular form for nouns, the infinitive form for verbs, or the positive form for adjectives and adverbs. Each lexical entry includes the syntactic category, inflectional variation (e.g., singular and plural for nouns, the conjugations of verbs, the positive, comparative, and superlative for adjectives and adverbs), and allowable complementation patterns (i.e., the objects and other arguments that verbs, nouns, and adjectives can take). The lexicon denotes 11 syntactic categories: verbs, nouns, adjectives, adverbs, auxiliaries, modals, pronouns, prepositions, conjunctions, complementizers, and determiners.
9. Linguistic Systems
317
TABLE 9.7. The Syntactic Categories From the Penn Treebank Category CC CD DT EX FW IN JJ JJR JJS LS MD NN NNS NNP NNPS PDT POS PRP PP RB RBR RBS RP SYM TO
Definition
Category
Definition
Coordinating conjunction Cardinal number Determiner Existential there Foreign word Preposition/subordinate conjunction Adjective Adjective, comparative Adjective, superlative List item marker Modal Noun, singular or mass Noun, plural Proper noun, singular Proper noun, plural Predeterminer Possessive ending Personal pronoun Possessive pronoun Adverb Adverb, comparative Adverb, superlative Particle Symbol (mathematical or scientific) To
UH VB VBD VBG VBN VBP
Interjection Verb, base form Verb, past tense Verb, gerund/present participate Verb, past participle Verb, non–third person singular present Verb, third person singular present Wh-determiner Wh-pronoun Possessive wh-pronoun Wh-adverb Pound sign Dollar sign Sentence-final punctuation Comma Colon, semicolon Left parenthesis character Right parenthesis character Straight double quote Left open single quote Left open double quote Right close single quote Right close double quote
VBZ WDT WP WP WRB # $ . , : ( ) " ‘ “ ’ ”
Source: M. Marcus, Santorini et al. (1994).
The SPECIALIST lexicon contains over 163,000 entries with over 268,000 strings. Its focus is biomedical, containing not only technical words from the domain but also the general English words that are also used in biomedical texts (McCray, 1998). Technical words come mainly from the UMLS Metathesaurus. For general English, the lexicon contains the 10,000 most frequent words from the American Heritage Word-Frequency Book (Carroll, Davies et al., 1971) and the 2000 base words from the Longman Dictionary of Contemporary English (Anonymous, 2001j). (The latter is an important resource in many NLP applications because it defines all English words from those 2000 base words.) Each lexical record in the SPECIALIST lexicon has the following fields (called slots in the documentation): • • • •
base—indicates the base form spelling_variants—to indicate variations in spelling entry—for the unique identifier (EUI) of the record cat—indicating POS
Table 9.8 shows the unit lexical records for anesthetic. The base form anesthetic and its spelling variant anaesthetic appear in two different
318
III. Research Directions TABLE 9.8. SPECIALIST Lexicon Unit Lexical Records for anesthetic {base=anesthetic spelling_variant=anaesthetic entry=E0330018 cat=noun variants=reg variants=uncount } {base=anesthetic spelling_variant=anaesthetic entry=E0330019 cat=adj variants=inv position=attrib(3) position=pred } stative
lexical records, one an adjective and the other a noun. The details of the other field are provided in the lexicon documentation. 9.2.3.2 Semantics and Case Frames Parsing, of course, only identifies the syntactic categories of words in a sentence (i.e., nouns, verbs, noun phrases, etc.). It does not tell what these components of the sentence actually mean. To achieve understanding of language, methods to uncover its semantics are needed. This task, which is considerably more difficult than parsing, has proven impossible for unrestricted natural language. Not only do individual words have different meanings or senses, which is problematic enough with IR systems, but bigger phrases introduce even more complexity. This section describes two approaches used in traditional NLP to handling semantics: case frames and semantic grammars. Before semantic analysis can begin, the lexicon must have another data element, the word’s semantic type. This is a category that indicates the general meaning of the word. With a semantic type category, restrictions can be placed on the type of phrases allowed to form. For example, measurements typically require a substance (blood) and a measured attribute (high). Thus, for example, one can say high blood pressure, but not high intelligence pressure, which is grammatically correct but semantically meaningless. The semantic typing of words in phrases permits specification of allowable templates. In the sample sentence, the template of a pathological process can be recognized, with a measurement (pressure), a substance (blood), and an attribute (high). Another template in that sentence is a disease process (disease) and a location (heart). Semantic typing can also be useful in making the parsing process more efficient, in that various parse trees can be ignored if it can determined that a phrase is grammatically correct but semantically meaningless.
9. Linguistic Systems
319
Now all that is needed to tie the meaning together is a way of representing that one of the NPs (high blood pressure) causes another (heart disease). The most common approach is to use a case frame, which is a template that represents the relationship(s) between concepts. In this case, the verb causes (or an equivalent, such as leads to) indicates a type of relationship between a pathological process and disease. Figure 9.1 shows the case frame. It should be noted that the verb does not necessarily need to be causes. The case frame may also accept leads to or is caused by. Of course, languages contain far more complex phrases and relationships, and the case frame approach, like every semantic approach, has been difficult to scale up to large subject domains. Many conceptual relationships in these domains are far more complex than the simple example used here. Furthermore, domain knowledge often depends on general understanding of the world. For example, while blood pressure is usually measured by a sphygmomanometer (the cuff that goes around the arm), it can be measured in many other ways, such as by the insertion of a cannula into a blood vessel. Case frames have certainly not achieved success in the heterogeneous world of document databases. Not only do documents contain quite variable language (which is often a goal of writers), but different parts of a document contain information at different levels of importance. For example, an article on treatment of congestive heart failure might digress into aspects of diagnosis or etiology that might not be central to the focus of the article. Term frequencies from lexical–statistical methods give an approximation of what is important by measuring how often words occur. Interpreting the semantics of such an article is far more difficult. Despite the inability to handle all the semantics in documents, some partial semantic methods have found modest success in IR. Some systems described later in this chapter use semantic information in attempting to disambiguate document and/or query words. Others exploit word and/or concept synonyms to broaden the chance of successful retrieval. An alternative approach to handling semantics is to use a semantic grammar, where the semantic is encoded in rewrite rules. Table 9.9 shows a simple semantic grammar for handling a query about the blood levels of various drugs. The rules specify semantic instead of syntactic categories. This grammar could handle queries such as, What is the blood level of theophylline? causes causes
pathologic-state disease measurement location pressure
substance blood
heart
process disease
attribute high FIGURE 9.1. A sample case frame for the sentence, High blood pressure causes heart disease.
320
III. Research Directions TABLE 9.9. A Simple Semantic Grammar S what is the BLOOD-LEVEL of DRUG BLOOD-LEVEL blood levellevelvalue DRUG theophyllinedigoxin . . .
Semantic grammars are most useful in highly restricted domains where there is minimal variation in language. The original approach was implemented in a natural language front end for a database management system (DBMS) (Hendrix, Sacerdoti et al., 1978), and the example in Table 9.9 implies use in a DBMS. Friedman has also utilized this approach in attempting to extract data from clinical narratives (Friedman, Alderson et al., 1994), as described in more detail in Chapter 11. 9.2.3.3 Context Clearly, even the understanding of semantics is not enough for complete understanding of language. Oftentimes the meaning of language, especially that in documents, is based on relationships between sentences. For this, one must move to the level of pragmatics, which is concerned with how sentences are used in different contexts, and real-world knowledge, so that one can infer facts that may not be obvious from an understanding of a specific domain alone. One aspect of context is discourse analysis, which looks at the relationships between concepts across sentences. An extensively investigated area is that of reference, which is concerned with how phrases in sentences connect to realworld objects. Anaphoric references are NPs that refer to objects mentioned earlier in a sentence or in an earlier sentence, such as: High blood pressure causes heart disease. It also increases the risk of stroke. Nonanaphoric references are NPs that were not mentioned earlier but are introduced for the first time, such as the first sample sentence above. Beyond pragmatics, one must also grapple with real-world knowledge, which provides the basis for understanding language, especially in technical domains. There is unfortunately no comprehensive store of human knowledge, and in fact it may be impossible to build one with the techniques that have been devised so far. The biggest effort in this area so far is the Cyc Project, which had as its original goal to encode all human knowledge, including common sense, but now has recast its aim to represent all the knowledge in a large encyclopedia (Lenat and Guha, 1990).
9.2.4 New Approaches to NLP In the 1990s, the focus of NLP research began to evolve as it was realized that the earlier methods did not lead to general language understanding nor even scale up to allow real-world solutions to language-related computational problems. Recognizing that it was difficult to build and maintain lexicons, grammars, and other linguistic knowledge resources, researchers shifted their focus from “de-
9. Linguistic Systems
321
terministic” NLP to that which was “empirical” or “corpus based.” In the latter approach, which achieved better results, algorithms were developed to extract linguistic knowledge (e.g., POS tagging, grammars, etc.) from natural language corpora rather than requiring system developers to manually encode it. Most approaches have used machine learning techniques that add probability information to syntactic and semantic processes, aiming to generate the parse most likely to be correct rather than trying to deterministically identify it (Brill and Mooney, 1997). The approaches were found to be successful in computer-based speech recognition in the 1980s (Bahl, Jelineck et al., 1983) and have diffused to other applications of NLP. The probabilistic approach has been highly successful when applied to syntactic parsing. Charniak (1997) notes that a mathematical construct, the Hidden Markov Model (HMM), is particularly effective. The HMM approach follows the optimal probabilistic path through a sentence, aiming to optimize the quantity
arg max t1,n
i p(titi1)p(witi)
(1)
where t1,n is the optimal set of tags across the sentence and ti is the ith tag of the word wi. At each step through the parse, two probabilities are calculated: p(titi1), the probability of a correct tag for the current word given the correct tag for the prior word, and p(witi), the probability of the word given that tag. Mathematically, this approach has similarity to the language modeling approach to IR discussed in Chapter 8. In fact, a similar smoothing process is used in this type of parsing to overcome zero probabilities when no corpus data are available to provide a probability in a given situation. Some of the techniques include estimating probabilities based on clues from word endings (e.g., those ending in -ing are likely to be progressive verbs). The HMM approach generates probabilistic context-free grammars (PCFGs), which are similar to CFGs but have probabilities associated with each rule. Additional success has been achieved with the use of lexicalized parsing, in which additional statistical information is used relating the constituents of phrases to the heads of the phrases. A related approach that has demonstrated high accuracy is transformational tagging, in which the derivation of rules from corpus statistics is used in place of probabilities. Brill (1992) uses an approach for POS tagging that derives rules (e.g., change the tag from A to B if the tag of the preceding word is C) that minimize the number of errors based on the corpus statistics. Although there are many possible rules that could be derived from this approach, the approach can be controlled to keep the number small (i.e., 200 or 300). The Brill tagger achieves comparable accuracy to and runs much faster than probabilistic taggers. Both the approaches to parsing just described yield accuracy comparable to that of older deterministic approaches and do not require the maintenance of large grammars to address every possible variation to handle general English grammar. This approach has also found success in semantic interpretation (Ng and Zelle, 1997) and has begun to show promise in information extraction (Cardie, 1997), which is covered in Chapter 11. One area of semantic interpretation highly rel-
322
III. Research Directions
evant to IR is word-sense disambiguation (WSD), that is, determining the sense to which a polysem belongs. A health example is the word cold, which can refer to a temperature or an illness. There are a number of resources that identify the different senses of words, including dictionaries as well as the lexical database WordNet (Fellbaum, 1998). The UMLS Metathesaurus also designated sense for different medical concepts. The same HMM approach described for parsing can be used for WSD. It has also had success in limited domains of case-framestyle mapping of natural language queries of databases.
9.3 Feasibility of NLP in Information Retrieval A number of researchers have studied documents, queries, and other text to assess the feasibility of linguistic methods. Such studies and analyses are valuable because they not only determine which methods are likely to work but can also quantitatively determine which are likely to yield gains in performance.
9.3.1 Lexical Knowledge McCray (1998) has reviewed lexical knowledge and its impact on biomedical information processing. Her analysis began with defining a lexical item: whereas a word is the smallest freestanding unit of meaning in speech, basic units of meaning can vary from morphemes within a unit to multiword terms. Words of two particular types in biomedical text were singled out as noteworthy: neoclassical compounds, which are single words consisting of several bound morphemes combining forms derived from Greek or Latin (e.g., esophagogastroduodenoscopy) and acronyms, which are short forms used for convenience (e.g., EGD, which stands for esophagogastroduodenoscopy). McCray also noted the problem of multiword terms that act as an atomic unit and have a specific meaning. For example, blood bank (which can also be written as bloodbank or even bloodbank) has a specific lexical meaning. At this point, the distinction of lexical items from phrases begins to blur, since many phrases take on such atomic meaning. McCray noted that terms represent the important meanings in a technical field and that much of the terminology resides in nouns and NPs. She also observed that acronyms are formed from common NPs and may be used more commonly than their fully expanded forms (e.g., AIDS). McCray further identified some of the challenges to language processing by noting spelling conventions, also called orthography. She noted the synonymy/polysemy problems of the same word having different spellings (e.g., the American and British variants dyspnea and dyspnoea) and different words having the same spelling (e.g., the temperature cold vs the disease cold). She also elucidated the importance of punctuation, noting that it both separates items from each other and serves specific grammatical functions. Punctuation also causes problems in language processing—for example, when it is used inconsistently, particularly with hyphens, as in the blood-bank example.
9. Linguistic Systems
323
In attempting to lay the foundation for how lexical knowledge can be used to improve language processing computationally, McCray noted that knowledge of pluralization rules, and knowing where idiosyncracies exist, could help uncover meaning. Likewise, the derivational process, whereby new words are formed from existing words, can give clues to meaning. For example, in nominalization, nouns are derived from verbs, usually by adding the suffixes such as -ment or -ion. An example is the noun treatment, derived from the verb to treat. The SPECIALIST lexicon contains a variety of auxiliary programs that process words to their root or normalized form, similar to but more sophisticated than the stemming routines described in Chapter 5. McCray’s paper described experiments using Dorland’s Illustrated Medical Dictionary (Dorland, 2000) in which the lexical processing tools increased the ability to match Metathesaurus terms to terms in the dictionary. She closed by noting that controlled vocabularies are important for sharing and processing of biomedical data and must not only reflect important terminology in a field but also allow for natural variation in language use. One form of lexical knowledge extraction that has generated interest is the detection and disambiguation of acronyms. While lists of acronyms from many domains are plentiful, (e.g., in the UMLS Metathesaurus), identifying and matching them to the proper term are more challenging. Larkey et al. (2000) developed a series of algorithms to identify acronyms on Web pages. They developed a complex set of rules, but essentially the algorithms were of two broad types. The canonical algorithm found acronym/expansion pairs in forms such as “expansion(ACRONYM)” or “expansion or ACRONYM,” whereas the contextual algorithm found acronyms occurring in the vicinity of the expanded form but not in any canonical pattern. The algorithms had a trade-off between recall and precision but seemed to work best for acronyms longer than two letters and when additional Web pages were used for processing. Pustejovsky et al. (2001) have developed a system called ACROMED, which detects acronyms, and a companion system POLYFIND, which attempts to disambiguate those that are ambiguous. ACROMED identifies the location of acronyms by means of a pattern-matching parser similar to the canonical algorithm of Larkey et al. It was assessed by using a standard collection of 155 acronyms derived from 86 MEDLINE abstracts and judged by a biology expert. The baseline algorithm achieved a recall of 63% and precision of 90%. With the additional use of syntactic information requiring the nearby phrase to be an NP, performance improved to a recall of 72% and precision of 99%. Pustejovsky and colleagues compared their system to that of Larkey et al. and found the general-purpose system of the latter did not perform as well as ACROMED, which had been optimized for the biomedical domain. POLYFIND works by applying a variant of TF*IDF weighting for the query of the expanded form to the abstract, with those ranking highest likely to contain the acronym in its appropriate sense. Another potential form of lexical knowledge that might improve IR is spelling correction. McCray (1998) noted that spelling errors arise for a number of reasons and have been difficult to correct on an automated basis. Computer algo-
324
III. Research Directions
rithms for detecting and correcting spelling errors have been around for quite some time (Peterson, 1980), and such functionality is a common feature of word processing applications. Some IR systems (e.g., the Google search engine, www.google.com) detect potential spelling errors and highlight them to the user. Of course, spelling errors are plentiful in documents to be retrieved as well, and determining how to handle them is more challenging.
9.3.2 Term Dependency Studies A number of studies in the mid-1980s assessed the effect of dependencies among terms in phrases to enhance retrieval performance. Croft (1986), for example, noted that if users manually designated phrases in natural language queries from a small test collection, and those phrases were then used along with words as indexing terms, a 5 to 10% improvement in performance was obtained. Smeaton (1986) derived dependencies from documents utilizing syntactic information to derive NPs and the NP portions of PPs. Using Croft’s queries, he demonstrated slightly better performance. Sparck Jones (1999) notes that much early work was based on small test collections in which the short (e.g., title and abstract) documents made it unlikely that there would be sufficient linguistic constructs (e.g., NPs) to provide matching against comparable constructs in queries to improve retrieval performance. She conjectures that the new, longer full-text documents in the TREC collections might provide a richer substrate for linguistic-based indexing, although she also notes that the queries in TREC, at least in most years, are also longer, which provides word-based systems with more data to match on as well.
9.3.3 Effect of Ambiguity Krovetz and Croft (1992) ran a set of experiments that yielded data on the effect of ambiguity on IR system performance. One experiment assessed the effect of word-sense mismatch between queries and documents in two small test collections. Examination of the top 10 ranked documents from 45 queries from each collection revealed that about 10% of the word matches were sense mismatches. Most frequently reasons for sense mismatches were due to stemming (such as arm and army) and general words being used in a technical sense (such as address and window in the computer domain). About half the mismatches had all words in the document sense mismatched, while the other half had both a correct and an incorrect sense present. Sense mismatches were much more likely to occur in nonrelevant documents, indicating that lexical ambiguity is a significant problem in IR and that efforts to control it may result in better performance. Another experiment with these collections assessed the different number of senses present in document words. About 40% of the words were used in more than one sense in the collection. The authors felt that the words that would be most worth disambiguating would be those with a uniform distribution of senses across the two or more senses present, with the sense distribution skewed but the
9. Linguistic Systems
325
query sense falling into one of the minority senses. Their analysis showed that about 16% of words fell into the former category while 8% fell into the latter. Upon removing all documents with sense mismatches of this type, the researchers found only small improvements, since just the top 10 documents were considered, and the mismatches due to stemming were not corrected. Sanderson (1994) investigated the role of WSD in retrieval performance by artificially introducing ambiguous terms (based on concatenating two to five document or query words together) into a test collection. He found that performance dropped only minimally with the addition of these terms. In further experiments, he found that greater than 90% accuracy was required before disambiguation began to have a beneficial effect. While these experiments were somewhat artificial, they indicate that WSD has little effect in IR systems unless it is highly accurate. Indeed, as will be seen shortly, attempts at disambiguation have yet to demonstrate performance improvement. Although there have not been explicit experiments assessing the impact of ambiguity on medical IR systems, a number of analyses have addressed their presence. Weeber et al. (2001) recently analyzed the 34.5 million phrases detected in the 409,337 citations added to MEDLINE in 1998. They found that over 4 million (11.7%) mapped ambiguously into more than one UMLS Metathesaurus concept. In an analysis of a concept-mapping program, Nadkarni et al. (2001) found that over 200 Metathesaurus concepts themselves were redundant (i.e., really represented the same concept). They also noted that over 5% of phrases detected in a corpus of medical record text mapped into more than one concept (e.g., cervical).
9.3.4 Ability to Recognize Nominal Compounds Despite the inconclusiveness of the foregoing data, there is a suggestion that being able to disambiguate senses should lead to better performance. Such disambiguation, however, requires knowledge about words and the concepts in which they occur. Gay and Croft (1990) assessed the ability to recognize the meaning of compound nominals (multiword NPs) in documents. Using a small test collection, they were able to devise frames to capture complex nominal categories, based on properties of the head noun. For example, a transitive event category allowed the head to contain a word used to describe transitive events, such as analysis (e.g., data analysis), management (e.g., database management), and construction (e.g., dictionary construction). Building the templates for structures of these types was time-consuming, and the benefit to the retrieval process was unclear. Therefore, the evaluation experiments focused on how well the templates could recognize the target compounds in their correct meaning. They found that of the compounds used to encode the knowledge base itself, only 82% could be correctly interpreted. When they tried to interpret the same compounds from new text, only 65% were understood correctly. They also attempted to look at how well just word proximity was able to recognize the correct concept. For 17 different two-word phrases, they found that
326
III. Research Directions
requiring adjacency led to a 94% rate of the concept being correct when it was found, but at a price of it not being found 28% of the time that the concept was expressed in the sentence. When the adjacency requirement was relaxed to allow words in the concept to be up to three words apart, the rate of the concept being correct fell to 84%, although the rate of it being missed fell to 11%. The conclusion of this work was that building frames to recognize the meaning of compound nominal phrases is a time-consuming process of dubious value to retrieval effectiveness, at least compared with just looking for the presence of terms within a window of proximity. These results highlight the problems of applying linguistic techniques that work in constrained domains to more heterogeneous information. Warner and Wenzel (A. Warner and Wenzel, 1991) attempted to determine the syntactic and semantic reasons that lead to difficulty in recognizing NPs in documents. The long-term goal of this work was to determine which linguistic features were most important to account for in applying NLP techniques to bibliographic retrieval. Their experiment began with identifying 18 NPs for searching in the ERIC educational database. Broad searches were done to retrieve all documents that expressed the conceptual meaning of these phrases. The next step was to analyze the retrievals to create a classification scheme for identifying the factors leading to linguistic variation in the concepts (Table 9.10). Finally, all instances of the concepts were coded according to the classification. The most common general category for linguistic variation was the structural/lexical category. The most frequent problems within this category were as follows: NP spread across a PP, one or more adjacent nouns inserted into an NP, and coordination of the concept across conjunctions. Frequent problems in the morphology category were both inflectional and derivational (see Section 9.4.1). Other work has attempted to assess the accuracy of NPs output by various parsers. Tolle and Chen (2000) compared four different NP parsers for their ability to identify relevant NPs in a collection of CANCERLIT documents. The highest recall for relevant NPs was obtained by a commercial parser called NPTool (Lingsoft, www.lingsoft.fi) at 71.7%. Precision for this tool was slightly lower TABLE 9.10. Classification of Reasons for Linguistic Variations in Noun Phrases 1. Morphology: words next to each other a. Same order 1. Inflectional—plurals 2. Derivational—other suffixes b. Different order 1. Same endings 2. Different endings 2. Structural/lexical a. Lexical clue b. Syntactic constructions 1. Coordination 2. Prepositional phrase Source: A. Warner and Wenzel (1991).
3. One or more adjectives or nouns inserted 4. Relative clause 5. Copula construction 6. Semantically related words 3. Punctuation a. Semicolon b. Period c. Colon d. Comma e. Other punctuation 4. Other
9. Linguistic Systems
327
than some of the other tools, assessed at 34.3%. The performance of the parser developed by these researchers, the Arizona Noun Phraser, was found to be significantly enhanced by use of the SPECIALIST lexicon.
9.4 Linguistic Techniques in Ad Hoc Retrieval It has been suggested that selective use of NLP in IR may be feasible. For example, recognition of nouns and how they come together in noun phrases may give a better clue to the underlying concepts discussed in a document. Likewise, having some semantic information may allow recognition of synonym equivalence and disambiguation of polysemous terms. This section describes a number of different approaches that have been undertaken to exploit linguistic properties of IR databases.
9.4.1 Morphological Analysis The goal of morphological analysis is to reduce words to their root form. The simplest form of morphological analysis is stemming, that is, the application of a set of rules to words to remove plurals and common suffixes. This approach, however, does not utilize any linguistic information about words. It is also unable to recognize the meaning of words or to handle exceptions when a suffix should not be removed. Stemmers make two types of error, those of commission and those of omission (Krovetz, 1993). In the former, the stemmer reduces words of different meaning to the same root form. The Porter stemmer, for example, reduces hepatitis (infection or inflammation of the liver) and hepatic (an adjective to describe something about the liver, such as hepatic injury) to the same root form hepat, an obvious error. Errors of omission occur when the stemmer fails to reduce words of the same meaning to a common form. The Porter stemmer does not recognize the equivalence of thorax (the chest cage) and thoracic (an adjective to describe something in the chest cage). Thus the phrases thoracic injury and injury of the thorax are not recognized as equivalent. In a linguistic approach to morphological analysis, the syntactic category of the word is identified, so that grammatically appropriate reduction of the word to root form can be done, or not done, depending on the circumstances. This requires a procedure to determine the syntactic category of the word as well as a dictionary to guide the appropriate semantic reduction of the word. There are two types of morphology: inflectional and derivational. Inflectional morphology refers to syntactic changes in a word, such as verb tense and pluralization, with no change in syntactic category. Derivational morphology refers to other suffixes of words, which may change not only syntactic category but meaning as well. In traditional NLP, inflectional morphology is usually done during the parsing process. Plural nouns are converted to singular form, while different verb tenses are identified with the word itself are converted to the infini-
328
III. Research Directions
tive form. Derivational morphology may be done during parsing or semantic analysis. Recognition of adjectified verbs (e.g., the lacerated scalp) is likely to be done during parsing, while semantic processes are required to recognize the equivalence of the phrases thoracic injury and injury of the thorax. One of the most comprehensive approaches to morphology in IR has been carried out by Krovetz (1993). His initial approach used the Longman Dictionary of Contemporary English to modify the Porter stemmer. This dictionary contains all the allowable suffixes for a word. The algorithm checked before firing a rule whether the word was already in the dictionary as a root. If it was, the stemming process would end. If the word was not a root, it would be necessary to carry out the rule and check again before the next rule. This approach was still problematic, owing to the nonlinguistic nature of the Porter rules (i.e., there were still errors of commission and omission). Krovetz next analyzed the morphological properties of several IR test collections and implemented approaches to stemming based on analysis of the most frequent morphological problems. He developed a stemmer that addressed both inflectional and derivational morphology. The inflectional portion of the stemmer converted plural nouns to singular form and past participles to present tense, and also removed the suffix -ing. The dictionary ensured that idiosyncratic forms (e.g., calorie and its plural calories, irregular verbs, etc.) were converted to the appropriate stem. The derivational portion of the stemmer was built based on an analysis of common suffixes noted in four IR test collections. The 10 most common suffixes were -er, -or, -ion, -ly, -ity, -al, -ive, -ize, -ment, and -ble. Five other suffixes were added: -ism, -ic, -ness, -ncy, and -nce. Since the Longman Dictionary lists forms of different meaning, a suffix was not removed if the form occurred in the dictionary. Unfortunately for the new stemming approach, the gains in recall–precision were only modest, similar to most other stemming experiments such as those described in Chapter 8 (Harman, 1991; Hull and Greffenstette, 1996). However, whereas nonlinguistic stemming just left root forms of documents, which were difficult to use further, Krovetz’s algorithm allowed some recognition of meaning which could be used to disambiguate word senses, as will be described below.
9.4.2 Parsing and Syntactic Analysis Of all the NLP techniques, probably the best understood method is parsing. It is therefore not surprising that a number of researchers have attempted to enhance IR systems with parsing and other types of syntactic analysis. The motivation for this approach is based on the assumption that by understanding the components of a sentence (i.e., the phrases), one can better understand the underlying conceptual content and presumably better match it to query statements. Early approaches to parsing suffered from two major deficits. First, parsing was viewed as a deterministic process yielding a “correct” parse. There was no attempt to handle the ambiguity arising from multiple or incomplete parses. The second problem was related to computer hardware: until the 1990s, most machines did
9. Linguistic Systems
329
not have the power to parse databases of the size typically found in IR. Part of this hardware limitation was confounded by the first problem, the deterministic approach to parsing, which in general was computationally intense. 9.4.2.1 Use of a Standard Parser for Indexing Terms Fagan (1987) pursued a line of research that compared the generation of phrases from word frequencies (statistical phrases, described in Chapter 8) and from parsing (syntactic phrases, with a focus on NPs). He modified the SMART system by adding a parser that derived NPs from the text, which were used along with words to index documents. The same procedure was used for queries. This approach was shown to improve slightly on single-word indexing alone (1.2–8.7%), but performed less well than statistically generated phrases (2.2%–22.7%). Further analysis provided additional insights: • Better results were obtained when parser output was limited to two-word phrases consisting of the NP head and one modifier. This was most likely because the terse queries were unlikely to have complex NPs. • The use of term-frequency (TF) weighting improved performance for phrases of both types. • Both phrases were more effective when used in addition to single-word indexing rather than as a substitute. Salton et al. (1990) subsequently investigated the parser used in Fagan’s experiments to determine its effectiveness. They first analyzed its ability to parse a book chapter on text compression. They noted that about one-third of sentences were parsed perfectly, while another third were parsed with only minor problems that would not make a difference in IR settings (e.g., a noun was tagged as an adjective or vice versa). However, the final third had major problems that likely would affect retrieval (e.g., the proper meaning of the phrase was not captured). Salton et al. did find that applying some normalization rules, such as keeping all phrases with capitalized or italicized components or those with a minimum frequency, improved the quality of indexing phrases. Their final experiment compared statistical and syntactic phrase generation by measures of phrase precision (proportion of acceptable statistical/syntactic phrases in a document) and phrase recall (proportion of statistical/syntactic phrases identified in a manually defined set of phrases for a document). They found in general a slight increase in each measure for syntactic over statistical methods (no statistical analysis was performed) but concluded that the benefit for syntactic over statistical phrases was negligible. Since statistical methods were far more efficient with resources, both in terms of computer algorithm complexity and the human effort required to build parsers and lexicons, Salton et al. deemed them preferable to syntactic methods. 9.4.2.2 Parsing Plus Rules for Syntactic Disambiguation Fagan’s and Salton’s experiments showed that parsing alone was unlikely to confer significant benefit to IR systems. Others, however, have developed new ap-
330
III. Research Directions
proaches to parsing that sacrifice the goal of unambiguous parsing for speed and other benefits. The simplest approach has been to use a POS tagger to assign syntactic categories that allow use of other techniques for identifying phrases. One such example was the FASIT system, which attached syntactic categories to words and utilized rules applied to the tagging to derive NPs (Dillon and Gray, 1983; Burgin and Dillon, 1992). The goal of the FASIT parser was to identify NPs that provided more context than single words. The overall indexing process consisted of concept selection and concept grouping. FASIT selected concepts by a three-step method: 1. Syntactic tagging involved assignment of individual words and punctuation to about 100 possible syntactic categories. 2. An exception dictionary prevented stemming of certain words; the remaining words were stemmed, with the syntactic category determined by the suffix removed. For example, removing -ed designated a word as a past-tense or pastparticiple verb, while removing -s designated the word as a third-personsingular verb or plural noun. 3. Many words were left with multiple tags, so syntactic disambiguation was provided by about 270 rules. One rule was that a preposition is never followed by a past-tense verb. The rules disambiguated some but not all of the multiply tagged words. Concept selection occurred by matching from 100 concept forms or templates in a dictionary. For example, a verb participle and noun formed a concept, as in diseased heart. FASIT grouped concepts by transforming them into canonical form and merging the forms into concept classes, which could be used to represent synonyms. Canonical forms were determined by statistical operations. First, all words were stemmed. Next, prepositions, conjunctions, and general nouns (common words such as people, activities, papers) were removed and the remaining words were alphabetized. Concepts were then grouped by a measure of association with all other concepts: sij measure of association(concepti,conceptj) si sj sij
(2)
where sij is sum of weights in concepts i and j. The weight of individual words in a concept was cfk N wki log tfk nk
(3)
where wki is weight of word k in concept i, cfk is frequency of word k in all concepts, tfk is frequency of term k in text, N is total number of concepts, and nk is number of concepts with word k. This approach gave the highest weight to terms that group well, which were likely to be synonyms. For retrieval, FASIT applied the same procedure to queries and used a vector– space model to determine similarity between queries and documents. FASIT was initially evaluated with collection of 250 library science master’s papers and 22
9. Linguistic Systems
331
natural language queries (Dillon and Gray, 1983). It performed slightly better than a SMART-like system that featured a 68-word stop list and stemming of plurals, although no statistical analysis was done. A major problem in FASIT was disambiguation. If ambiguous words were eliminated, concepts were missed, while if they were kept erroneous phrases were sometimes selected. Techniques were developed to improve disambiguation, but they did not improve retrieval performance (Burgin and Dillon, 1992). 9.4.2.3 Partial Parsing Other IR systems have been based on parsers that do not aim for complete parsing but instead focus on the NP, the area in which the content is likely to represent that present in queries. As such, these systems have focused on the recognition of NPs and their use in indexing and retrieval. The CLARIT system, for example, is designed to recognize NPs, identifying their boundaries rather than completely parsing the entire sentence (D. Evans, Lefferts et al., 1992). The CLARIT parser applies lexical tagging and an inflectional morphology analyzer to reduce terms to root form. The grammar then identifies phrases, with particular emphasis on identifying NPs. The parser can identify just simplex NPs, which consist of the NP head and its modifiers, or both simplex and complex NPs, the latter of which contain posthead PPs, relative clauses, and VPs. An example of CLARIT parsing for an AIDSLINE abstract is shown in Figure 9.2. CLARIT has several additional features to enhance retrieval after parsing has been completed. Most simply, it can combine the phrases plus individual words into a vector–space approach, matching them against phrases and words in the user’s query for retrieval. Another feature is thesaurus discovery, which can be based on the top-ranking documents (query expansion) or ones denoted by the user to be relevant (relevance feedback). CLARIT also has a comprehensive user interface that allows the user to generate thesaurus discovery terms, add or delete them from the query, and vary their weighting in the query. The thesaurus discovery process can be applied to a small number of relevant documents or an entire collection. Its goal is to derive a first-order thesaurus (a list of terms) that contains the NPs most representative of the documents. The resulting term list can be used not only to augment queries in the IR setting, but also to derive terminology from a domain, a process of value in other settings, such as the building of clinical vocabularies to be discussed in Chapter 11. Thesaurus discovery operates by giving the NPs in a collection of documents a score between zero and one based on their frequencies of the whole phrases and the component words in the collection, along with the rarity of the term in general English (based on the Brown Corpus). The scoring process enables the size and specificity of the thesaurus to be controlled. The participation of CLARIT in the TREC experiments has shown that this type of approach can scale up to very large document collections (D. Evans and Lefferts, 1993). In terms of performance, however, its approach has not exceeded that of the most advanced lexical–statistical systems in TREC. Its true perform-
332
III. Research Directions
TITLE: Some economic implications of introducing control measures to reduce the spread of HIV infection in sub-Saharan Africa.1 OBJECTIVE: This paper looks at the demographic and economic implications of introducing measures to control the spread of HIV infection in a country in sub-Saharan Africa. METHOD: The consequences of introducing control measures at different times is explored using a simple mathematical model of the transmission dynamics of HIV infection that incorporates both epidemiological and demographic processes. The controls considered include the screening of blood for transfusions and measures to reduce the probability of the sexual transmission of HIV either by a reduction in sexual partner change rates or by the adoption of “safer sexual practices”. RESULTS and CONCLUSION: The costs of these measures and the corresponding number of cases of HIV infection and AIDS prevented are used to show the costs and benefits of prevention. Results show the importance of the early introduction of control measures; any delay substantially increases the severity and potential costs of the future epidemic. CLARIT indexing terms: hiv transfusion hiv infection sub saharan africa transmission dynamic sexual transmission demographic process introduce measure economic implication introduce control measure future epidemic
sexual partner change rate prevention probability safe sex practice simple mathematical model early introduction control measure potential cost control blood
FIGURE 9.2. An AIDSLINE abstract with CLARIT indexing. (Courtesy of Clarvoyance, Pittsburgh.)
ance, however, may be compromised by the batch nature of the TREC experiments, since CLARIT is designed to be a very interactive system and may perform better when interacting with a user. TREC-5 included an NLP Track that was designed to assess the efficacy of systems using linguistic approaches. The test collection consisted of a subset of the full TREC collection, along with usual TREC-style queries. The CLARIT group attempted to determine whether their approaches provided improved retrieval performance over simple word-based ones (Zhai, Tong et al., 1996). Two
9. Linguistic Systems
333
particular techniques were studied. The first was the use of lexical atoms, which were common two-word phrases known to represent single concepts (e.g., hot dog). About 13,000 of these were generated in an automated process from a collection of documents from the Wall Street Journal (D. Evans and Zhai, 1996). The second technique was syntactic parsing, with several approaches studied, including (e.g., heavy industry construction group): • Head-modifier pairs: that is, NP heads paired with other modifiers in the NP (construction industry, heavy construction, industry group) • Adjacent subphrases: that is, longer spans of the NP short of the full NP (heavy construction industry) • Full NPs (heavy industry construction group) The input to the system consisted of TREC queries subjected to the same techniques. The output was ranked using a TF*IDF weighting on the words, lexical atoms, and phrases. One experiment found that the addition of lexical atoms improved mean average precision (MAP) by about 3% for batch-processed queries and by about 7% for the same queries that had been manually modified. When the same experiments were performed on TREC-4 data, the results were less beneficial, with improvements of around 1 and 2%, respectively, for experiments of the same type. The experiments on syntactic phrases compared a baseline of just words with 1. Words and head-modifier pairs 2. Words, head-modifier pairs, and adjacent subphrases 3. Words, head-modifier pairs, adjacent subphrases, and full NPs The first approach gave the best performance improvement over the word-only baseline, increasing MAP by 13%. Each of the latter two approaches improved performance by about 9%. Another approach to partial parsing has been used by Strzalkowski et al. (1999). Similar to CLARIT, their system aims to recognize simple and complex NPs in concert with other IR techniques, such as term weighting. The architecture of their system separates each technique into “streams” so that each can be relatively weighted to optimize system performance and isolated for analysis in experimentation. The most complex of these streams is the headmodifier pair stream, which aims to extract NP heads and pair them with other components of the NP. This process is based on the Tagged Text Parser (TTP), which functions like an ordinary grammar-based parser but has a built-in timer that limits the time spent on parsing any one sentence (Strzalkowski, 1992). If a complete parse cannot be obtained, a “skip-and-fit” strategy is pursued, with an attempt to identify NPs and relative clauses. Prior to analysis by the TTP, document sentences undergo POS tagging and linguistic stemming to reduce inflected words and convert nominalized verb forms to their root forms. The final step in this stream is the pairing of various modifiers with the heads of the NPs.
334
III. Research Directions
The system of Strzalkowski et al. has a variety of other streams, which include the following: • Simple NP stream: unnormalized sequences of adjectives, participles, and other nouns leading up to an NP head • Name stream: recognition of names based on the identification of words as proper nouns in the lexicon • Stems stream: single words from the text, filtered for stop words and with simple stemming applied (higher weight is given to words that have only one sense in WordNet under the assumption they are unambiguous and therefore reliable discriminators) • Unstemmed words stream: unmodified single words • Fragments stream: 1024-word passages in a document Each stream contributes a component to the overall retrieval score for the document, with its differential contribution weighted by the equation
Document score A(i) score(i,d) [0.9 nstreams(d)]
(4)
i
where A(i) is the coefficient for stream i, score(i,d) is the normalized score for the document within the stream i, and nstreams(d) is the number of streams in which the document has been retrieved (above a certain rank, such as the top 1000 documents). Another version replaces the final element of equation (4) with a measure of precision if such information is available (e.g., from training data or some other estimate). The coefficients A(i) are estimated or derived from training data for the specific retrieval system and test collection. The system of Strzalkowski et al. also used query expansion by manual or automated techniques. The manual approaches provided a user interface to enter portions of text from words to multiple paragraphs from relevant documents. The automated techniques identified key concepts from the query (especially when long TREC-style queries are available) and added paragraphs containing those concepts into expanded query. The results of their ad hoc runs using 50 TREC queries and the Wall Street Journal document subset are summarized in Table 9.11. As can be seen, better results were obtained with long and manually expanded queries regardless of whether NLP was used. The NLP system gave a slight improvement in performance over SMART. Experiments on the routine task gave similar minor improvements over the SMART baseline.
9.4.3 Word-Sense Disambiguation Sections 9.4.3 and 9.4.4 move from a concern with the application of selective syntactic approaches in IR to a focus on the use of selected semantic approaches. One of the simplest of these is WSD. As noted so far, many words, especially commonly used ones, have multiple senses (e.g., AIDS and hearing aids, EKG leads and lead poisoning). Inappropriate retrieval may occur when
9. Linguistic Systems
335
TABLE 9.11 TREC-5 NLP Track Results of Strzalkowski et al.a Method SMART PN phrases with short queries SMART PN QE phrases with short queries NLP with full queries NLP with automatically expanded queries SMART PN QE phrases with manually expanded queries SMART PN QE phrases with NLP manually expanded queries SMART PN QE phrases with full queries NLP with manually expanded queries
Mean average precision 0.15 0.21 0.21 0.22 0.24 0.27 0.30 0.32
aThe
systems compared were SMART with pivoted normalization (PN) weighting, query expansion (QE), statistical phrases, and the NLP system described Section 9.4.2.3 with manually or automatically expanded queries. Source: Strzalkowski, T., Lin, F., et al. (1999) Evaluating Natural Language Processing Techniques in Information Retrieval, 113–146, in Strzalkowski, T., ed. Natural Language Information Retrieval. Dordrecht, The Netherlands. Kluwer. Reprinted with kind permission of Kluwer Academic Publishers.
a query uses one sense and a document uses another. This problem is frequent (Krovetz and Croft, 1992), and efforts to eradicate it may not be successful (Sanderson, 1994). One approach to WSD has been to use resources that categorize words into their senses. One such resource is WordNet, a semantic network type of system containing English words, their synonyms, and the senses in which they occur (Fellbaum, 1998). Each set of synonyms in WordNet is called a synset. Words can belong to more than one synset; for example, the word cold can be a disease, a temperature, and an emotional state. Synsets are categorized into a semantic hierarchy, with the top-level terms called unique beginners (Fellbaum, 1998). Voorhees (1993) analyzed WordNet from the standpoint of improving IR system performance. She noted that while the average number of senses per word is 1.26, some words, which include some used in language with high frequency, have a large number of senses, as many as 27. Likewise, while the average synset contains 1.74 words, the largest one contains 38 words. Two researchers have used WordNet in attempts to develop algorithms that disambiguate the sense of query words to enhance the retrieval documents where words in the query are used in the same sense as they are in the documents. Voorhees (1993) developed an approach that attempted to identify the hood of a synset, which was essentially the largest span of the noun hierarchy and its narrower terms that contained a single synset. The synset identifier became a component of query and document vectors in SMART, with cosine matching providing a ranked retrieval output. Unfortunately, Voorhees found that this approach degraded retrieval performance overall, although selected queries demonstrated benefit. Further analysis indicated that most queries in these older, smaller test collections were too short to allow disambiguation of their words. Additional experimentation with the longer queries in the TREC collection found that this automated approach did not confer benefit either, although some of the lessdeveloped queries were able to obtain benefit by manual expansion with synsets
336
III. Research Directions
selected by the user for inclusion (Voorhees, 1994). Richardson and Smeaton (1995) used a somewhat similar approach to WSD with WordNet and obtained comparable results. While their work demonstrated that they could perform WSD reasonably well, this advantage did not translate into improved retrieval performance. Another approach to WSD does not use any lexical resources (Schutze and Pedersen, 1995). Instead, a thesaurus is created via automatic means using singular-value decomposition (i.e., latent semantic indexing) to determine “synonym” terms. Based on the observation that words occurring close to other words help define its sense, a context vector is created based on clustering techniques that allow words to be disambiguated. Experimental results on a Wall Street Journal subset of TREC data found the sense-based approach to improve MAP by 7.4% alone over the word-based baseline and by 14.4% when used in conjunction with regular word-based vectors.
9.4.4 Semantic Approaches While the word sense disambiguation methods were designed to handle polysemous words, other approaches have been implemented to handle the problems of synonymy, which is problematic in all domains and is certainly a detriment to IR systems in health care. Table 5.6 showed a sample of synonymous terms from the MeSH and SNOMED vocabularies used in medicine. In this section we describe three different approaches that have attempted to map terms into controlled vocabularies. 9.4.4.1 Lexical Mapping of Text into Controlled Vocabularies The domain specificity of synonyms is sometimes helped by the presence of manually constructed thesauri. Medicine, in particular, has a number of thesauri that contain important terms and their synonyms. Their limitation, however, is that they are often constructed for purposes other than document retrieval. For example, the ICD-9 classification is used mainly for coding medical diagnoses for billing purposes. Many of its terms are not those that physicians are likely to use in documents or queries. Even the MeSH vocabulary, which is used for manual indexing and retrieval of medical literature, has a number of expressions of terms that would not normally be used by clinicians. Aronson (1996) has noted that text strings can map into controlled vocabulary terms in a variety of ways: • Simple match: text string matches vocabulary term exactly • Complex match: string maps into more than term (e.g., intensive care medicine mapping into Intensive Care and Medicine • Partial match: text string maps into part of a term, which can occur in several ways: 1. Normal: string maps into part of phrase (e.g., liquid crystal thermography mapping into Thermography)
9. Linguistic Systems
337
2. Gapped: string maps with gap either in string or term (e.g., ambulatory monitoring mapping into Ambulatory Cardiac Monitoring) 3. Overmatch: beginning or end of term includes additional words not in string (e.g., application mapping into Heat/Cold Application or Medical Informatics Application) • No match: no part of text maps into any term A number of researchers have develop approaches for mapping concepts from free text into controlled vocabularies for a variety of applications, including IR. One of the earliest efforts to do this in IR was the SAPHIRE system (Hersh, 1991), which utilized the UMLS Metathesaurus. The Metathesaurus provided synonym linkages across medical vocabularies (e.g., the MeSH term Hypertension was linked to the ICD-9 term Elevated Blood Pressure). Since many terms in these vocabularies contained synonyms, the grand sum of all synonymous terms was large, which enable SAPHIRE to recognize a wide variety of string forms that mapped into concepts. The original version of SAPHIRE employed a concept-matching algorithm in both indexing and retrieval (Hersh, 1991). The algorithm took as its input any string of text, such as a document sentence or a user query, and returned a list of all concepts found, mapped to their canonical or preferred form. This was done by detecting the presence of word-level synonyms between words in concepts (e.g., high and elevated) as well as concept-level synonyms between concepts (e.g., hypertension and high blood pressure). The concept-matching process was purely semantic, with no syntactic methods (e.g., parsing) used. Individual words in the string were stemmed and converted to a canonical representation. Starting with the first word, the algorithm then attempted to find the longest from the vocabulary that matched all the succeeding words. When a term was found, the process repeated from the next word in the string after that term, continuing until the end of the string. The concept-matching algorithm required a vocabulary of concepts and their synonym forms. The concepts for SAPHIRE’s vocabulary originated from the Metathesaurus. Although some reorganization of the vocabulary was required for use by SAPHIRE, no alteration of the content was necessary. This vocabulary allowed SAPHIRE to recognize about 130,000 medical concepts and an equal number of synonyms for those concepts. In SAPHIRE’s indexing process, the text to be indexed for each document was passed to the concept-matching algorithm. The indexing terms for each document were the concepts matched, which were weighted with the inverse document frequency (IDF) and term frequency (TF) redefined for concepts. For retrieval, the user entered a natural language query, and the text was passed to the concept-matching algorithm. Each document with concepts in the list then received a score based on the sum of the weights of terms common to the query and document, similar to the TF*IDF approach described in Chapter 6, but using concepts instead of words. The resulting list of matching documents was then sorted based on the score.
338
III. Research Directions
FIGURE 9.3. SAPHIRE user interface.
The original SAPHIRE user interface is shown in Figure 9.3. Query text was entered in the uppermost text box. A wild-card character could be used to have words completed for the user when, for example, he or she was unclear on the exact spelling. The concept-matching algorithm extracted all concepts from the query statement and returned them in a list, which was displayed in the middle of the search window and included the number of documents in which the concept occurred. All words that did not map into concepts were discarded. Matching documents were shown at the bottom of the window, with the weights normalized such that the highest ranking document was given a score of 100. Documents could be viewed by double-clicking on their titles. Initially the top 10 documents were displayed, with the user able to add more in increments of 10. If the users desired to modify the search, additional terms could be added by entering more text, or existing terms could be deleted by double-clicking on a term and verifying its deletion in a dialog box. SAPHIRE was comprehensively evaluated with databases of several different types and in both batch and interactive modes. The first study compared batch queries entered into SAPHIRE with interactive Boolean searching by librarians and physicians (Hersh and Hickam, 1992). SAPHIRE achieved better recall and precision than the physicians did, though the system’s performance was not as good as that of the librarians. The next study compared SAPHIRE with conventional MEDLINE searching of MEDLINE documents (Hersh, Hickam et al., 1994). It used previously searched topics and judgments of relevance from a clinical evaluation of Grateful Med at McMaster University (Haynes, McKibbon et
9. Linguistic Systems
339
al., 1990). In this study, recall and precision for SAPHIRE fell between expert and novice clinician searchers, although none of the differences among the groups was statistically significant. A failure analysis identified some recurring patterns for false negative and false positive retrievals. The most common causes for failure to retrieve relevant documents were the presence of synonyms not recognized by SAPHIRE (e.g., the form in the document was not in the Metathesaurus) or terms that were present but at a different level of granularity (e.g., a query contained the term antibiotic but the document used the actual drug name). The most common reason for retrieval of nonrelevant documents was the presence of most or all query terms in the document, but with a different focus or relationship between the terms. Another study compared SAPHIRE with word-based Boolean searching of Yearbook Series extended abstracts (Hersh and Hickam, 1993). It was the first to implement interactive searching by real users, a group of 16 senior medical students who searched on 10 questions generated on medical rounds at the University of Pittsburgh. The database used was six volumes from the Yearbook Series, each document consisting of the original title and abstract for an article, along with expert commentary. Each student searched half the questions with SAPHIRE and the other half with a Boolean system. There was no statistically significant difference in recall or precision between the two systems. The three studies just described generated three test collections, which were used for the next study to compare SAPHIRE and some derivations of lexical– statistical approaches for searching AIDSLINE, MEDLINE, and Yearbook Series documents (Hersh, Hickam et al., 1992). This study compared SAPHIRE with a lexical–statistical system alone, the lexical–statistical system and SAPHIRE combined, and a version of SAPHIRE with a different concept-matching algorithm that eliminated the exact word order requirement of SAPHIRE’s original concept-matching algorithm and instead required only that words in a concept be adjacent. This algorithm also allowed partial matching as long as over half of the words were present, aiming to overcome the problem of synonyms not matching the exact form in the Metathesaurus. The lexical–statistical system showed the best overall performance. The performance of the combination of the lexical– statistical system and SAPHIRE was intermediate between the two programs alone, while the version of SAPHIRE with the new concept-matching algorithm performed worse, owing to excess inappropriate concept matching. A later study (Hersh and Hickam, 1995) compared SAPHIRE with both wordbased Boolean searching and lexical–statistical searching by means of a different type of database, the internal medicine textbook, Scientific American Medicine (SAM). The textbook was subdivided into 6623 “documents.” In the study, 21 senior medical students were recruited to search on 18 queries each, half with one of the three systems and half with another. These queries were generated by internal medicine faculty and housestaff in the OHSU General Medicine Clinic. Each query was searched using each of the three systems. As with the Yearbook Series study, there was no difference in recall or precision among the three systems (see Table 9.12).
340
III. Research Directions
TABLE 9.12 Comparison of SAPHIRE with Lexical–Statistical and Boolean Systems for Searching the Textbook Scientific American Medicine Definitely relevant documents System Boolean Word based SAPHIRE
Definitely and possibly relevant documents
Recall (%)
Precision (%)
Recall (%)
Precision (%)
70.6 75.3 66.9
18.9 14.8 16.1
64.2 66.9 61.6
28.7 25.6 26.2
Source: Hersh and Hickam (1995).
In all, SAPHIRE showed performance nearly comparable to, but not better than, word-based methods. While its use of synonyms was shown to be beneficial in some queries, the inability to map free text into vocabulary terms hindered it in others. The results of these experiments led to a major revision in SAPHIRE’s approach, replacing the exact-match requirement of the algorithm with one allowing partial matching and ranking of concepts (Hersh and Leone, 1995). The pseudocode for this algorithm is shown in Figure 9.4. The new algorithm begins by breaking the input string (which can be a sentence or phrase from a document or a user’s query) into individual words. Words are designated as common if they occur with a frequency above a specified cutoff in the Metathesaurus. The purpose of designating words as common is to reduce the computational overload for words that are occasionally important in some terms but occur frequently in others, such as the word A in Vitamin A or acute in acute abdomen. Since the words A and acute occur commonly in many other terms, calculating weights for these additional terms added a large and unnecessary computational burden. For each word in the input string, a list of Metathesaurus terms in which the word occurrs is constructed. The Metathesaurus term lists for common words contains only terms that also occur in one or more of the noncommon words in the input string. Thus if the string is acute abdomen, the common word acute would contain only the term acute abdomen and not the term acute leukemia. Once the term lists for each word are created, a master term list is created, containing any term that occurs in one or more individual word lists. Terms in which less than half the words occur in the input string are discarded. (Thus, a partial match has to have half or more of the words from the term in the input string.) The terms are then weighted based on formulas that gives weight to terms that are longest, have the highest proportion of words from the term in the string, and have the words of the term occurring in close proximity to each other. Terms that match all the words in the input string exactly are given additional weight. The partial matching approach was, of course, less amenable to the completely automated indexing that was the goal of the original version of SAPHIRE. As such, the system was put to use as the front end to find terms for a Web cat-
9. Linguistic Systems
341
Input: string of words common word cut-off (default = 270) CUI (default = false) size cut-off (default = 40) weight cut-off (default = 1.0) Output: list of Metathesaurus terms with weights for each word in string word_frequency = number of Metathesaurus concepts that word occurs in if word_frequency is greater than the common word cut-off then word is common else create a list of all Meta terms that contain this word for each word in string that is common create a list of all Meta terms that contain this word alone or that contain this word and at least one word that is not common create a master list of all Meta terms that occur in any of the words if all terms occur in a single concept then return concept with weight = 11 else for all Meta terms in the master list twis = number of words from this term in string wit = number of words in term if(wit + 1) div 2 < twis then discard this term for all Meta terms in master list term_words_present = twis/wit log_term_length = log(wit) + 1 intervening_words = number of words between first and last words of term in string log_intervening_words = log(intervening_words + 1) + 1 weight = term_words_present*log_term_length/log_ intervening_words sort Meta terms in master list if CUI is true then eliminate all LUIs which have a common CUI with a higher-ranked LUI eliminate all terms below size cut-off eliminate all terms below weight cut-off FIGURE 9.4. Pseudocode for revised SAPHIRE concept-matching algorithm.
342
III. Research Directions
alog (Hersh, Brown et al., 1996), for matching of terms from other languages (Hersh and Donohoe, 1998), and so on. More detail on the latter are provided shortly. A number of other researchers have developed systems that map free text into controlled vocabulary terms for document retrieval. R. Miller et al. (1992) developed CHARTLINE, later called PostDoc, a system that uses a similar sliding window approach to match words in text to UMLS Metathesaurus terms. The system was initially used to generate MeSH terms for literature searches based on the concepts in patient narratives. Another well-known approach to mapping text into Metathesaurus terms is the MetaMap system (Aronson, 1996, 2001). Metamap’s general approach is to process input text to generate a wide variety of possible word variants, which are then used to generate candidate strings for potential matching. It begins by using a POS tagger to assign syntactic tags from the SPECIALIST lexicon to words, which are then grouped into NPs. Next, all possible variants of words within the phrases are generated based on knowledge resources associated with the SPECIALIST lexicon, including lists of acronyms, abbreviation, and synonyms as well as rules for derivational morphology. Table 9.13 lists the types of variant generated. Each variant receives a distance score, which indicates the extent of variation from the original word. These scores are used to evaluate candidate terms that are selected for matching. The evaluation scoring process is based on four factors: 1. Centrality: the word that is the NP head gets a score of 1; all other words get a score of 0. 2. Variation: how much the variants in the Metathesaurus string differ from the original text string words, based on the formula 4 Variation Distance 4
(5)
3. Coverage: the amount of words that the Metathesaurus string and the text string have in common, consisting of two components: a. Meta span, the number of words in text string present in Metathesaurus term b. Phrase span, the number of words in Metathesaurus term present in text string
TABLE 9.13 Variant Generation in MetaMap Variant Spelling variant Inflection Acronym/abbreviation Expansion of acronym/abbreviation Synonym Derivational variant Source: Aronson (2001).
History code
Distance score
p i a e s d
0 1 2 2 2 3
9. Linguistic Systems
343
The coverage value is calculated by giving the meta span twice as much as weight as the phrase span. 4. Cohesiveness: based on the largest number of connected words in the Metathesaurus and text strings, consisting of two components a. Meta cohesiveness, the sum of squares of the connected components divided by the square of the length of the string b. Phrase cohesiveness, the sum of squares of the connected components divided by the square of the length of the phrase The cohesiveness value is calculated by giving the meta span twice as much as weight as the phrase span. Each Metathesaurus term’s score is normalized to between 0 and 1000, and the Meta candidates are mapped to the disjoint parts of the NP. The program has a variety of parameters, such as how extensively the variant process will be performed, which can be set to tune the algorithm for appropriateness to the function to which it is being applied. An example of MetaMap with the input text ocular complications is shown in Aronson’s report (Aronson, 1996). The variants generated for the word ocular are shown in Table 9.14. As can be seen, variants come from synonyms (eye), inflectional variation (eyes), and derivational variation (oculus), with each given an appropriate distance score. Table 9.15 shows the candidate Metathesaurus terms generated from the variants. The evaluation scoring for the two terms Complications and Eye is demonstrated in the following steps 1. Centrality: a score of 1 is given to the term Complications and 0 to the term Eye. 2. Variation: the term complications has no variant, so is given a distance variant score of zero, while the term Eye has a distance value of 2 and is given a distance variant score of 4/(2 4) 2/3. 3. Coverage: both Complications and Eye have a meta span and phrase span of 1, so their coverage value is (2/3) (1/1) (1/3) (1/2) 5/6. 4. Cohesiveness: both complications and Eye have a meta cohesiveness and phrase cohesiveness of 1, so their cohesiveness value is (2/3) (1/1) (1/3) (1/4) 3/4.
TABLE 9.14. Variants of Word ocular in MetaMap Processing ocular {[adj], 0 “ ”} eye {[noun],2 “s”} eyes {[noun], 3 “si”} optic {[adj], 4 “ss”} ophthalmic {[adj], 4 “ss”} ophthalmia {[nou], 7 “ssd”} oculus {[noun], 3 “d”} oculi {[noun], 4 “di”} Source: Aronson (2001).
344
III. Research Directions
TABLE 9.15. Candidate Terms and Their Weighting from Input Ocular Complications to MetaMap Metathesaurus term Complications (Complication) Complications (Complications Specific to Antepartum or Postpartum) Complicated Ocular Eye Eye NEC Ophthalmic Optic (Optics) Ophthalmia (Endophthalmitis)
Weight 861 861 777 694 638 638 611 611 588
Source: Aronson (2001).
The final evaluation weight for Complications is [0 2/3 2 (5/6) 2 (3/4)]/6, which normalizes to 861, and Eye is [1 1 2 (5/6) 2 (3/4)]/6, which normalizes to 638. Since the terms are disjoint with the respect to the original text string, the evaluation is applied to the combined candidates with the final weighting of (1 2/3 2*1 2*1)/6, which also normalizes to 861. These terms are returned as the best match for the input string. MetaMap has been applied to many tasks. It was assessed to augment ad hoc retrieval by replacing phrases in queries and documents of a MEDLINE test collection and found to result in a 4% improvement in average precision. It was also used to assist in query expansion, as described in the next section, and better performance gains were achieved. MetaMap has also been used in a variety of information extraction tasks, as described in Chapter 11, which also discusses other concept-mapping algorithms that have been employed for information extraction. 9.4.4.2 Statistical Mapping of Text into Controlled Vocabularies Another use of controlled vocabularies in linguistic systems has been to associated words in the text with controlled vocabulary terms that have been assigned to documents with those words. This approach was originally used with a large physics database that had manually assigned index terms (Furh and Knorz, 1984). A number of healthcare applications have used this approach. The MeSH term selection function of the Ovid (www.ovid.com) retrieval system maps the user’s input to the MeSH terms that occur most frequently when those words are used in the title and abstract of the MEDLINE record. Another approach in the healthcare domain has been implemented by Yang and Chute, who used a technique of linear-least-squares fit to measure the association between words that occur in MEDLINE abstracts and the assigned MeSH terms (Y. Yang and Chute, 1994). This technique required a training set of documents that were used to derive the associations. Using a partitioned MEDLINE test collection, Yang and Chute found enhancement over baseline SMART
9. Linguistic Systems
345
performance, but only comparable performance to SMART using relevance feedback. Likewise, the PINDEX project used probabilistic techniques were used to derive the likelihood that a MeSH term was assigned to MEDLINE records based on the presence of words in the abstract (G. Cooper and Miller, 1998). This system worked by deriving all one-word, two-word, three-word, and four-word sequences in the title and abstract of a MEDLINE record. Since these records also had assigned MeSH terms, a probability could be derived that a MeSH term was associated with a phrase. This was determined by the conditional probability of the phrase X being present (occurring with frequency m) when the MeSH term Y is present (occurring with frequency k): k P(YX) m
(6)
Indexing terms were selected when k/m exceeded 0.1. An average of 5 MeSH terms were selected per MEDLINE record using this approach. Both PINDEX and PostDoc have been assessed for their ability to assign MeSH terms to clinical narratives. The details of these evaluations are presented in Chapter 11. However, the PINDEX/PostDoc approach could also be applied to IR applications. 9.4.4.3 Semantically Assisted Query Expansion Another possible use of semantic information is to expand queries with semantically-appropriate terms, such as synonyms and other related terms. D. Evans and Lefferts (1993) introduced expansion via automatically constructed thesauri in TREC, showing modest gains over the TF*IDF baseline but not exceeding other nonlinguistic approaches, such as improved term weighting or query expansion. Voorhees (1994) has attempted to use WordNet along its hierarchical and synonym links to expand queries, but as with her WSD experiments, no benefit was seen. Another approach to finding phrases is to replace parsing with algorithms for identifying word–phrase associations. The INQUERY system has a component called PhraseFinder, which generates a list of associations across a document collection (Jing and Croft, 1994). As with many of the parser-based approaches already described, all words are tagged with one of seven possible parts of speech: noun (N), adjective (J), adverb (R), verb (V), verb-ing (G), verb-ed (D), and cardinal number (I). PhraseFinder designates phrases, which are defined as any collection of words in a sentence meeting certain rules, such as {NN, JN, N}, which identifies a phrase as a double noun, an adjective–noun pair, or a single noun. It then measures the association frequency between individual words and all phrases inside a window of a defined number of sentences. PhraseFinder is used in the retrieval setting to expand queries. After the user enters a query, PhraseFinder provides a ranked list of associated phrases for the database being searched. The set of phrases can contain phrases that subsume the query words (duplicates), contain words not in the original query (nondupli-
346
III. Research Directions
cates), or both. Experimental results with TREC data have shown the use of both duplicates and nonduplicates, along with a phrase rule of only NPs {NNN, NN, N}, to achieve the best performance. It has also been found with TREC data that the optimal window for association is 10 sentences and that a sample of data to build the association list works as effectively as the entire collection. Human interaction with this portion of the INQUERY system has led to substantial performance gain, partly because spurious PhraseFinder phrases were deleted (Broglio, Callan et al., 1994). This approach has also achieved modest success in the healthcare domain. Srinivasan (1996a,b) used MEDLINE test collections to define measures of association between the words in the title and abstract of the MEDLINE record and its assigned MeSH terms. In this approach, each MEDLINE record is represented by two vectors, one for words and one for MeSH terms. Initial word-based queries against MEDLINE are expanded to include MeSH terms, which are then used for retrieval by means of the MeSH vectors. Experimental results showed modest performance gains (e.g., 8–9% in average precision), but even better results were obtained by query expansion using the MeSH terms from top-ranking relevant documents. Researchers at NLM have also explored semantically assisted query expansion, processing queries by using MetaMap to identify Metathesaurus terms, which are in turn used to expand queries by means of the INQUERY system (Aronson and Rindflesch, 1997). The NLM workers achieved results comparable to those of Srinivasan, as seen in Table 9.16. Hersh et al. (Hersh, Price et al., 2000) have also attempted to expand queries with the UMLS Metathesaurus using the larger OHSUMED test collection. Metathesaurus concepts were manually selected for the collection’s 106 queries. Queries were first expanded by adding all words occurring in synonym terms for the concept. They were then expanded further by adding all words from all terms in concepts related to the query concepts, including parent terms, children terms, and related terms. Queries were also expanded by using the words in the text of the concept definition. While the expansion by words from synonym terms degraded performance only slightly, attempts at expansion incorporating other concepts (i.e., parents, children, and related), resulting in significantly worse results. TABLE 9.16. Results of Average Precision Using MeSH Terms and Thesaurus-Based Query Expansion in MEDLINE Documents by Srinivasan and Aronson and Rindflesch Method Text-only, no MeSH terms MeSH terms MeSH terms with thesaurusbased query expansion Adding document feedback to thesaurus-based query expansion
Srinivasan
Aronson and Rindflesch
0.52 0.56 0.57
0.52 0.57 0.60
0.60
(not done)
Source: Srinivasan (1996a,b); Aronson and Rindflesch (1997).
9. Linguistic Systems
347
Using the words of the definition also degraded performance. One implication of these results which has not been explicitly tested, is that the automatic explosion of MeSH terms, done in many MEDLINE systems and at least one Web search engine (Suarez, Hao et al., 1997), could be detrimental. French et al. (2001) have also assessed approaches to MeSH-based query expansion using OHSUMED. They used a probabilistic formula to associate words and MeSH terms from documents retrieved by a query. They first tried to assess whether the augmented queries could improve collection selection when the OHSUMED collection was artificially subdivided into subcollections. The purpose of these experiments was to investigate the important task of choosing among different collections when many are available for searching. Although the researchers’ queries improved collection selection over the original queries, this did not lead to overall improved document retrieval. French et al. did find that expanding the queries with one to three MeSH terms resulted in improved document retrieval.
9.5 Other Retrieval Applications of Linguistic Systems Section 9.4 focused on ad hoc retrieval. However, linguistic systems have been used for other IR applications. This section describes two such applications, crosslanguage retrieval, and question answering.
9.5.1 Cross-Language Retrieval English is the lingua franca of online information. Most international scientific conferences and publications use English as their required language. (Indeed, a German colleague has suggested that if English is not a scientist’s first language, it is always his or her second.) An analysis by the Web Characterization Project of the Online Computer Library Center (wcp.oclc.org) has found that 73% of all Web pages are in English. The next most common languages are German (7%), Japanese (5%), French (3%), and Spanish (3%). Cross-language information retrieval (CLIR, also called translingual retrieval) is done when query and documents are in different languages. Despite the preponderance of English in science and on the Web, CLIR capability is desirable in certain instances. Government or business analysts may, for example, desire intelligence from political or business documents in non-English-speaking countries. In healthcare, researchers performing meta-analyses may seek to retrieve randomized controlled trials written in languages other than English. (This task is made somewhat easier by the availability in most major bibliographic databases of translations into English of non-English titles and abstracts.) Crosslingual capabilities can also be helpful in an educational sense, enabling nonEnglish-speaking individuals to learn English medical terminology. Research in cross-language retrieval has been an interest at TREC, where the Cross-Language Track has assessed a variety of languages, including European
348
III. Research Directions
languages, Chinese, and Arabic. An independent TREC-like activity has emerged for CLIR in European languages, the Cross-Language Evaluation Forum (CLEF, www.clefcampaign.org), leaving non-European languages to TREC. Another forum focusing on Japanese-related CLIR is the National Information Infrastructure–National Center for Science Information Systems Test Collection for IR Systems (NTCIR, research.nii.ac.jp/ntcadm/index-en.html) Project. CLIR is typically assessed via test collections (Braschler, Harman et al., 2000). Experiments can be monolingual (i.e., queries are expressed in one language against a document collection in another) or multilingual (i.e., queries are expressed against documents in more than one language). In the case of multilingual collections, documents may be parallel (i.e., document-for-document translations exist across languages) or comparable (i.e., documents contain similar conceptual content but are not exact translations). In general, experiments in parallel corpora have found that performance, measured by MAP, is typically 50 to 75% of what would be obtained when queries and documents are in the same language (Voorhees and Harman, 2000). Oard (1997) has classified and reviewed the major approaches to CLIR. He notes that CLIR can be broadly classified into two types of retrieval, controlled vocabulary and free text. The former consists of documents that have been manually indexed into a controlled vocabulary, which itself has been translated into different languages. Although a number of commercial bibliographic databases have thesauri that have been translated into multiple languages, research into their use for CLIR has been minimal (Soergel, 1997). Of note, the MeSH vocabulary from health care has been translated into over 20 languages (S. Nelson, Schopen et al., 2000). Some of the terms from these MeSH translations (only from Roman alphabet or transliterated languages) have begun to appear as “synonyms” in the UMLS Metathesaurus. For example, the Spanish term corazon appears as a synonym for the English term heart. Free-text CLIR is further divided by Oard into corpus-based and knowledgebased techniques. Corpus-based techniques take advantage of parallel or comparable corpora, attempting to learn appropriate translations. One approach uses LSI to “shrink” the conceptual space across multilingual collections (Landauer and Littman, 1990). Other work has focused on techniques to align documents across languages at varying levels of similarity (Braschler and Schauble, 2000): • • • • •
Same story Related story Shared aspect Common terminology Unrelated
Once these alignments have been determined, algorithms can match queries to similar documents or portions of them automatically. Knowledge-based free-text CLIR techniques typically employ machinereadable resources, such as dictionaries. The earliest work with this approach came from Salton (1970), who used a word–word dictionary to translate words from one
9. Linguistic Systems
349
language to another. This approach was also used by Hull and Greffenstette (1996) for French queries of the (English) TREC collection. Another approach has been to use EuroWordNet, a version of WordNet extended to several European languages (Gilarranz, Gonzalo et al., 1997). One limitation of simple word-to-word mapping entails the polysemy of words, since word sense may differ across languages as well as within a language. Some approaches to overcome this problem have included the use of POS tagging and document alignment (Davis and Ogden, 1997) as well as query expansion (Ballesteros and Croft, 1997). The amount of CLIR-related work going on in the healthcare domain has been small, perhaps because of the predominance of English in scientific publications and meetings. Most work has focused on the development of multilingual resources to assist retrieval. Hersh and Donohoe (1998) found that when nonEnglish terms appeared as synonyms in the UMLS Metathesaurus, the SAPHIRE approach could be used to map words in other languages to Metathesaurus concepts, with two modifications. First, there had to be a means of handling nonEnglish characters. Most non-English European languages are diacritical characters, such as umlauts and accents, that are not part of the “7-bit” English ASCII code on which the Metathesaurus is based. Many of these characters are represented in the upper half of “8-bit” ASCII code, as designated by an international standard, the ISO Latin-1 Character Set (ISO 8879). So that non-English words could be used to retrieve words from the Metathesaurus, a translation table had to be added that converted 8-bit ASCII codes into their 7-bit transliterations. The second modification was made necessary in weighting the output, because some foreign languages, particularly German, combine multiple word phrases into single words. Such combinations cause the original algorithm to inappropriately downweight the terms, since part of the weighting algorithm is based on the proportion of words common to the input and matched term. Table 9.17 shows some example queries. TABLE 9.17 Cross-Language Translations Using SAPHIRE German to English: natriumbicarbonat für diabetische ketoazidose CUI C0011880 C0074722 C0022638
String Diabetic Ketoacidosis Sodium Bicarbonate Ketosis
Vocabularies MeSH MeSH Dorland’s, ICD-9, MeSH
Spanish to English: la náusea y el vómito CUI C0042963 C0027487
String Vomiting Nausea
Vocabularies ICD-9, MeSH, SNOMED Dorland’s, ICD-9, MeSH, SNOMED
English to French: surgery for appendicitis CUI C0038894 C0003615
String CHIRURGIE APPENDICITE
Source: Hersh and Donohoe (1998).
Vocabulary French MeSH French MeSH
350
III. Research Directions
One use of this approach was to provide a multilingual entry point to the CliniWeb Web catalog, allowing users to enter query terms in multiple languages to find clinical Web pages (Hersh, Brown et al., 1996). The main benefit was allowing users to retrieve English documents from the Web by entering terms in their native language. There was actually no reason why documents in other languages could not be retrieved, although there are no non-English documents in the CliniWeb database. Figure 9.5 shows a German query entered into CliniWeb with the resulting suggested terms. Although Hersh and Donohoe (1998) did not evaluate document retrieval, they did carry out a formative evaluation of SAPHIRE’s direct language translation capability. An American librarian fluent in German and a German medical documentation specialist were asked to type into the system a variety of German medical terms they encountered in their everyday work. Both entered about two dozen queries, observing a number of generalizations in the following respects. 1. Unlike their English counterparts in the Metathesaurus, many German terms did not have both plural and singular forms present. It was concluded that a stemming algorithm to handle common suffix variants was necessary. 2. Many German terms in the Metathesaurus were in Latin as opposed to Ger-
FIGURE 9.5. German query, natriumbicarbonate toazodose, to CliniWeb.
für
diabetische
ke-
9. Linguistic Systems
351
man form. For example, the latinate Myokardinfarkt was present as opposed to the German Herzinfarkt. Clearly an increased number of synonyms would be necessary as well. 3. German words that came together in phrases to form single words (e.g., Oesophagusvarizenblutung or esophageal bleeding) were generally not present in the Metathesaurus). This may represent a necessary enhancement for future versions of the Metathesaurus if more comprehensive foreign-language coverage is desired. Bodenreider and McCray (1998) performed a similar analysis of the French translation of MeSH that had been to the Metathesaurus. They found a number of limitations that would require enhancement before the Metathesaurus could be useful for French-language applications: 1. Very few French synonyms (only 5.5% of terms) were present. 2. Only MeSH was present, and terms in MeSH are known to be expressed to facilitate indexing and retrieval of documents, not necessarily in a form suitable for other applications. For example, articles and prepositions were often omitted in the French terms and nouns were not adjectified when used as adjectives. These characteristics would make use of NLP tools difficult with such terms. 3. Removal of diacriticals or ligatures (connected characters) as well as capitalization of all non-English terms to use a 7-bit character set led to ambiguous terms. An example of the former is the word côte (rib) and côté (side) being translated to cote (quotation). An example of the latter is POMPE potentially representing the word pump or the proper noun Pompe (as in Pompe’s Disease). Bodenreider and McCray concluded, like Hersh and Donohoe, that 8-bit representation as well as a richer source of terms and synonyms was required. Hersh and Zhang (1999) developed a Chinese-based interface to the UMLS Metathesaurus. Asian languages like Chinese differ greatly from Roman languages in that they contain many more than the basic 26 or so alphabet characters. In addition, they generally do not use spaces to indicate word boundaries. (In the case of Chinese, each character is essentially a whole word.) Although not in the medical domain, Kwok (1999) experimented with a number of different approaches to cross-language Chinese IR. His research demonstrated that the best performance increment was obtained without attempting to identify word boundaries at all, using instead simple bi-grams of words in the queries and documents. Some modest additional improvement was seen with the addition of simple rules to identify short words. Using a Chinese translation of MeSH, Hersh and Zhang (1999) implemented an interface that used a commercial software package for entering Chinese characters. Upon entering the characters, the user was presented with a list of all Chinese MeSH terms containing one of more the entered characters and their English translations. The user then selected an English term, which led to the page that showed the English term and its Chinese translation as well as (in English)
352
III. Research Directions
information from the Metathesaurus, including the MeSH tree address(es), synonyms from all vocabularies, parent term(s), child term(s), definition, CliniWeb links (if they existed for the term), and a PubMed MEDLINE search for review articles. Hersh and Zhang hypothesized that this system might be most useful in the teaching English medical terminology to non–English speakers, allowing translation of terms and subsequent browsing of information on the Web. Hiruki et al. (2001) adapted the basic Chinese system to Japanese, a language whose writing is even more complex than Chinese. They note that the Japanese writing system has four sets of characters: kanji (Chinese characters), the phonetic syllabaries hiragana and katakana, and Latin letters (romaji). Hiragana is used for Japanese words that cannot be expressed in kanji or have grammatical functions. The use of katakana is restricted to foreign names and words borrowed from other languages. Romaji characters are used for abbreviations and symbols, as in the scientific literature. Although any Japanese word can be written phonetically with one of the two syllabaries, all the character sets are routinely used in combination to avoid ambiguity from the language’s many words that are pronounced alike. As with Chinese, spaces are not used to separate words. Medicine in Japan uses a mixture of Chinese, Japanese, and Western (mostly German and English) terms (Izumi and Isozumi, 2001).
FIGURE 9.6. Japanese query for hypertension to the J-MeSH tool.
9. Linguistic Systems
353
The Japanese–English MeSH translator (J-MeSH tool) of Hiruki et al. uses a Web browser interface and enable users to accurately find equivalent English medical terms for Japanese medical terms. It accomplishes this by first mapping a Japanese term to candidate terms from a Japanese translation of MeSH, then retrieving the English equivalent of the desired candidate term by way of the Metathesaurus concept unique identifier (CUI). Figure 9.6 shows the terms retrieved from entry of the Japanese word for hypertension. When one of the hyperlinked English terms is selected, another page is shown with the Japanese search term, the English equivalent, the English definition, and parent and child terms in the MeSH hierarchy. This page also provides links that submit the term as a query to CliniWeb, PubMed, or Google. The user can also browse up and down the MeSH hierarchy by clicking on the hyperlinks of the parent and child terms, or return to the candidate terms table page to select a different candidate term. Figure 9.7 shows the page that appears if the user has selected the term Pulmonary Hypertension. In an approach to CLIR that made use of the Metathesaurus, Eichmann et al. (1998) attempted to determine how well Spanish and French queries could re-
FIGURE 9.7. Display of Japanese information for the term pulmonary hypertension in the J-MeSH tool.
354
III. Research Directions
trieve English documents. They used the OHSUMED test collection, with its queries translated manually into Spanish and French. They then set out to assess algorithms that used the Metathesaurus to automatically translate those queries back to English and retrieve MEDLINE references from the collection. Their basic approach was to use the SMART system to create a transfer dictionary. This was done by creating “documents” consisting of the Spanish or French word along with all CUIs in which it appeared. This allowed a query to “retrieve” English concepts, with the output ranked by a TF*IDF weighting scheme. Of course, they did not want to include all concepts retrieved, so they developed several strategies to refine the output: 1. Full matches: only terms that contained all words in the query were retained. 2. Partial matches: the shortest phrase matching the remaining output was retained. 3. Word-based translation: based on a dictionary of exact translations for words not fully matched. 4. Addition of remaining Spanish words. They also used a di-gram similarity measure to map words in the queries not present in the Metathesaurus to morphological variants that were present. The results of their experiments showed that the best approach for Spanish and French was to combine all the refinements just listed except the partial matches, with achievement of 75 and 64% of the English baseline average precision for Spanish and French, respectively. A failure analysis provided some further insight. First the researchers noted that there were more Spanish terms than French terms, which could potentially explain the better Spanish results. However, reducing the coverage of the Spanish terms to those also present in French did not change the average precision results. Like Bodenreider and McCray (1998), Eichmann et al. concluded that French coverage in the Metathesaurus was problematic. Another interesting finding was that more French queries performed better than Spanish queries over the English baseline, although many also performed worse.
9.5.2 Question Answering As noted in Chapter 8, the best approach in the TREC Question Answering Track made use of a number of linguistic approaches, so discussion of it was deferred to this chapter (Pasca and Harabagiu, 2001). It should be recalled that this is an open-ended question-answering task, so no answer is guaranteed to be present in the text. The system of Pasca and Harabagiu provides a variety of features that contribute to its success. The first step is to process the question to determine the semantic category of its expected answer type. One or more such categories are determined, which correspond to intermediate nodes in WordNet. The answer will then likely occur as one or more leaf nodes underneath the intermediate nodes. The next step is to use a retrieval process that identifies paragraphs within documents to find potential answers. This is followed by an approach that
9. Linguistic Systems
355
pinpoints the likely location of the answer, the paragraph, based on a training process that has used a machine learning (neural network) approach. Finally, a cache of earlier questions is maintained, along with an algorithm to detect reformulations of the same or very similar questions. This approach achieved a nearly doubled score (mean reciprocal rank 0.60) in the 50-byte task and onethird higher score (mean reciprocal rank 0.78) in the 250-byte task.
9.6 Summary This chapter has shown that while a great deal has been learned about language and the ability of computer systems to understand it, there are still significant barriers to using NLP to enhance IR systems. It is not clear that linguistic approaches will be able to outperform the state-of-the-art lexical–statistical approaches from Chapter 8 in ad hoc retrieval, although further research is still warranted. Promising results may occur in retrieval applications of other types, such as CLIR and question answering.
Chapter 10 Augmenting Systems for Users
Chapters 8 and 9 focused on two categories of methods to build information retrieval (IR) systems. This chapter covers a variety of other approaches, including those that aim to improve the user interface as well as those that attempt to organize beyond the single system in the case of digital libraries. As with rest of the book, much of the “research” that appeared in this chapter in the first edition has now migrated to the earlier state-of-the-art chapters in this edition.
10.1 Content One way to improve systems for users is to facilitate better access to content. Chapter 4 described a variety of approaches made possible by modern tools such as computers with graphical interfaces and the networks on which they run. A key attribute of innovation in content delivery has been the ability to integrate across resources, although barriers between proprietary collections of content are still an impediment. This section focuses on attempts to improve content delivery beyond what is available commercially, looking at methods innovated in the past but still not implemented in state-of-the-art systems or those still under research.
10.1.1 Early Innovations A number of researchers addressed the issue of organization of content early on, even when the technology did not facilitate it owing to the restrictions due to slow network connections, text-only screens, small storage capabilities, and so on. One of the early comprehensive attempts to manage the volume of text in a database was the Hepatitis Knowledge Base, initiated by Bernstein et al. (1980) at the National Library of Medicine (NLM). Focusing on viral hepatitis, these researchers created a hierarchical structure of concepts and explanatory text in the area. After this structure was developed, individual portions of 40 review articles and a textbook were selected and added to form a single body of information. The result was a hierarchical database of concepts at the highest levels, 356
10. Augmenting Systems for Users
357
summary-type text at intermediate levels, and detailed text at lower levels. A consensus panel of 10 hepatologists oversaw the selection of concepts and insertion of text. An update process was also implemented to ensure that the database would stay current. After the database was developed, a retrieval system was implemented to allow rapid access (Bernstein and Williamson, 1984). At the core of the system was a lexical–statistical engine that used a stop word list and a stemming algorithm. A manually constructed 80-term thesaurus was used to convert common synonyms to a normalized form. Finally, a weighting scheme for ranking was developed that used a process analogous to TF*IDF but added extra weight to the intermediate-level paragraphs from terms at higher levels to enhance the score of this middle-level information considered to be most important. An evaluation of the system with over 60 queries showed that 85 to 95% of the text materials needed for the answer were within the top few retrieved paragraphs. Another attempt to make the primary literature more accessible focused on representing the information more formally. This seemed to have particular promise for articles that described therapeutic interventions, where the disease and the intervention to treat it could be modeled explicitly. In the realm of cancer, for example, diagnoses are usually based on clinical stages of the disease, which not only indicate prognosis but also lead to different treatment decisions. Thus localized breast cancer is known to respond best to the combination of a surgical procedure called lumpectomy and radiation therapy, whereas more advanced disease requires more aggressive surgery and chemotherapy. For the physician confronted with choosing the optimal therapy for a patient with breast cancer, as with many clinical situations, the task can be daunting. Rennels et al. (1987) implemented the Roundsman system to address this problem. Its goal was to take a description of a patient as well as a treatment proposal and provide a critique based on the literature. The system provided a narrative based on the clinical studies that are appropriate to the patient’s presentation. Roundsman’s abilities stemmed from a database containing a library of clinical studies that have formal representations of their clinical attributes. Table 10.1 shows what Rennels called the population description for breast cancer. These were the attributes of the disease that patients present with and that TABLE 10.1. Population Descriptions in Roundsman POPULATION-DESCRIPTION with clinical-stage-set (I II) t-set (T0 T1a T1b T2a T2b) n-set (N0 N1a) path-n-set (UNKNOWN) m-set (M0) menopausal-status-set (PRE POST) age-lower-bound 20 age-upper-bound 80 Source: Rennels, G., Shortliffe, E., et al. A computational model of reasoning from the clinical literature. Computer Methods and Programs in Biomedicine, 24:139–149, Elsevier Science.
358
III. Research Directions
populations from interventive studies have. The system attempted to match the patient to the appropriate studies, so that only studies that used similar patients were considered. On the input side, the system was given a description of the patient and the treatment proposal. Since Roundsman was a critiquing system, it had to determine not only whether the studies in the library matched the particular patient, but also whether they contained the therapy to be critiqued. A distance metric determined whether the study was close enough to be retrieved. If the studies could be matched to the patient and the proposed therapy, a narrative was produced dynamically, tailored to patient and therapy. The THOMAS project of Lehmann and Shortliffe, while not implementing the patient–study matching aspect of Roundsman, took the study result analysis a step further by assisting the physician in constructing a decision–analytic model that used a Bayesian approach to combining the physician-user’s prior probabilities along with statistical data from the appropriate studies (Lehmann and Shortliffe, 1991). A major advantage of THOMAS was that it allowed a physicianuser who understood the principles but not necessarily the computational details of Bayesian statistics to use that approach to obtain expert assistance. Another early innovative system was Explorer-1, which implemented a hypertext version of The Brigham and Women’s Hospital Handbook of Diagnostic Imaging (Little Brown, Boston), a handbook designed to assist clinicians in the selection of radiological tests (Greenes, Tarabar et al., 1989). The text of the handbook was broken down into nodes. There were also additional nodes for clinical algorithms and what-if analyses to calculate disease probabilities. A simple word-based searching mechanism provided access to specific nodes, while an overview window could trace the user’s history through the nodes. Explorer-1 subsequently evolved into DeSyGNER (Decision Systems Group Network of Extensible Resources), a complete architecture for knowledge management, defined by Greenes and Deibel as the “authoring, management, selective retrieval and annotation of knowledge, for such purposes as education, problem solving, and decision making” (Greenes and Deibel, 1991). The DeSyGNER architecture was implemented to handle both adaptive and nonadaptive knowledge. Nonadaptive knowledge consists of predetermined content, such as text and images, while adaptive knowledge consists of procedures, such as data analysis and expert systems. The goal of DeSyGNER was to provide a “building block” approach to hypermedia resources, based on a core kernel that stored and retrieved different knowledge entities as well as facilities for sharing, updating, and versioning. The kernel allowed the construction of new entities that took advantage of the lowlevel kernel services. The knowledge entities of DeSyGNER consisted of the simple entities from Explorer-1, such as hypertext nodes and simple figures, but also more complex ones such as color images, animations, video sequences, other simulations, and adapters to external resources, such as the QMR decision support system (R. Miller, Masarie et al., 1986). The entities were structured into knowledge management applications by compositional tools, such as page layout, dynamic book layout, or other means for aggregating and/or relating enti-
10. Augmenting Systems for Users
359
ties. The kernel services handle data storage, data dependencies (i.e., linking, updating), and interaction with the user.
10.1.2 Linkage The early researchers whose work was described in Section 10.1.1 recognized the importance of linkage of information, even if the technology at the time did not permit optimal solutions. Work continues in the area of linking various aspects of content, some of which was described in state-of-the-art systems described earlier in the book. For example, the genomics databases described in Section 4.4.2 demonstrate how information from gene sequences (GenBank) to the scientific papers describing them (MEDLINE) to textbook descriptions of diseases they occur in (OnLine Mendelian Inheritance in Man) can be integrated. A glimpse of the same approach for clinical information was hinted in some of the aggregated systems described in Section 4.5, but the barriers resulting from the proprietary status of the information have hindered integration of this kind of information. The remainder of this section discusses two research areas that could serve as foci for integration of clinical information: linkage of knowledge-based content to the electronic medical record (EMR) and linkage of users to human knowledge. 10.1.2.1 Linkage to the Electronic Medical Record Most IR applications are stand-alone applications; that is, the user explicitly launches an application or goes to a Web page. A number of researchers have hypothesized that the use of IR systems can be made more efficient in the clinical setting by embedding them the EMR. Not only would this allow their quicker launching (i.e., the user would not have to “switch” applications), but the context of the patient could provide a starting point for a query. Cimino (1996) reviewed the literature on this topic and noted that embedding had been a desirable feature since the advent on the EMR. More recently, however, the ability to link systems and their resources via the Internet, particularly using Web browsers, has made such applications easier to develop and disseminate. Cimino noted that the process of linking patient information systems to IR resources consisted of three steps: 1. Identifying the user’s question 2. Identifying the appropriate information resource 3. Composing the retrieval strategy Humphreys (2000) notes that newer technologies enhance the prospects for linking the EMR to knowledge-based information systems. In particular, the Internet and the Web reduce the complexity of integrating disparate legacy systems, provide standards that facilitate development of applications, and allow users of all types to access resources from a variety of locations from the home to the clinical setting. She notes that three levels of integration are required to achieve this vision:
360
III. Research Directions
1. Technical connections: the gamut of pure technology-related issues that allow integration, such as hardware, software, telecommunications, access integration, and so on 2. Organizational connections: the means by which organizations license clinical applications and the information they access 3. Conceptual connections: the standardization of the structure of the information and the terminology to describe it Most IR systems provide a simple mechanism for identifying the user’s question: they provide a query box to enter it. Since, however, the EMR contains context about the patient, such as diagnoses, treatments, test results, and demographic data, it is conceivable this information could be leveraged to create a contextspecific query. Some early approaches looked at extracting information from dictated reports (Powsner and Miller, 1989; R. Miller, Gieszczykiewicz et al., 1992) but were limited by the nonspecificity of much of the data in those reports. Cimino et al. (1993) developed generic queries that were based on analyses of real queries posed to medical librarians. They subsequently developed infobuttons that allowed the user to retrieve specific information. For example, an infobutton next to an ICD-9 code translated the code into a MeSH term using the Unified Medical Language System (UMLS) Metathesaurus and sent a query to MEDLINE (Cimino, Johnson et al., 1992). Likewise, an infobutton next to a laboratory result generated a MEDLINE search with the appropriate term based on whether the result was abnormal or not (Cimino, Socratorus et al., 1995). The approach has also been extended to radiology results and full-text resources (Zeng and Cimino, 1997). It has even been adapted to allow patients to view some of their own results in a prototype patient-based record system (Baorto and Cimino, 2000). One challenge for linking clinical information to knowledge-based resources is determining what information to offer the user. This is especially so for clinical narratives which contain a wealth of words and concepts. Mendonca et al. (2001b) assessed narrative reports to determine how to promote the most important concepts on which a user is likely to search. They found that TF*IDF weighting of concepts extracted by a natural language processing system was effective in promoting those of most interest to real human users. These authors have also developed the means to formulate data extracted from other parts of the clinical record into the types of well-formed questions required in evidencebased medicine (Mendonca, Cimino et al., 2001a). Selecting the appropriate information resource and devising a query to it are also challenging. One system that attempted to provide access to a variety of information resources was SmartQuery (Selsky, Eisenberg et al., 2001). This application provided an interface on top of a commercial EMR product NetAccess, which was a Web-based front end to the Lifetime Clinical Record (LCR), a mainframe-based reporting system from Siemens Medical Systems (SMS, www. smed.com). SmartQuery provided context-sensitive links from patient-specific data viewable in NetAccess to online medical knowledge resources that were either freely available or licensed to the local institution. The overall goal of Smart-
10. Augmenting Systems for Users
361
Query was to enable clinicians to use information sources of different types, such as electronic textbooks, research articles, and clinical guidelines, to find answers to clinical questions; information overload was limited by allowing each user to tailor his or her queries and specify which resources should be accessed. The system was configured to access five information resources: • MEDLINE: via PubMed • Best Evidence (American College of Physicians, www.acponline.org): a collection of extended, structured abstracts of important journal literature • Harrison’s Online (McGraw-Hill, www.mcgraw-hill.com): an electronic version of the well-known internal medicine textbook • National Guidelines Clearinghouse (Agency for Healthcare Research & Quality, AHRQ, www.guideline.gov): a bibliographic database of clinical practice guidelines • CliniWeb: a collection of human-reviewed and MeSH-indexed Web pages (Hersh, Brown et al., 1996) Queries appropriate to each of the resources were fashioned from the terms selected by the user. Each required a different query formulation process based on the interface it provided. For example, MeSH terms were specified as such to PubMed and CliniWeb but just passed as text strings to Harrison’s On-Line, which provided only a text-word query mechanism. SmartQuery added buttons and checkboxes to the patient data displays in NetAccess. Checkboxes were shown next to each lab test identifier, and above displays of dictated reports. To use SmartQuery, the user checked the boxes next to items relevant to his or her question and then clicked an Add button that caused a MeSH term corresponding to the data underlying the displayed information to be added to the list of MeSH terms created from the ICD-9 codes. The user was also able to enter additional terms via a text box. The top of the page contained a series of checkboxes for the different information sources available. As shown in Figure 10.1, the user sent a query by checking the terms of interest and the information sources he or she wanted. Data specific to the patient were collected from a variety of sources within that patient’s record, including the problem list, laboratory test results, and dictated reports. Each source provided specific challenges. Although the NetAccess system did not contain an electronic problem list, ICD-9 codes were used as a proxy for a real patient problem list. The UMLS Metathesaurus was used to translate the ICD-9 codes. When no exact translations were found in a given Metathesaurus term, an attempt to find a broader term in the hierarchy was made. Translating data from the format of laboratory data to query terms posed two challenges. First, an abnormal value demonstrated not only that the test had been done but that it conveyed specific information about the patient (e.g., the serum calcium level was low). Furthermore, some names of tests did not readily form usable queries, such as Meas ICA, Wh B for a calcium test. The researchers manually developed a table for common lab tests that mapped laboratory test identifiers to useful query terms, including terms to be used when a flag
362
III. Research Directions
FIGURE 10.1. Page to select terms and resources from SmartQuery.
indicated abnormally high or low values. For example, if the test Meas ICA, Wh B was selected, SmartQuery detected whether the most recent result was normal, high, or low and then offered the term Calcium, Hypercalcemia, or Hypocalcemia. SmartQuery also provided processing of dictated reports, since the list of ICD9 codes from earlier encounters would not reflect problems new to the current episode of care. As a result, the most recent history and physical (H&P) dictation was used as a source of MeSH terms. Since extracting terms from the entire H&P would have yielded many nonuseful terms, processing was limited to selected sections labeled chief complaint, reason for admission, assessment, impression, plan, or assessment and plan, since these were likely to contain information about significant diagnostic and therapeutic issues relevant to the care of the patient. The text in each section was preprocessed to remove section headings, expand or remove many common abbreviations (e.g., UTI was expanded to urinary tract infection, whereas bid was removed). A barrier method approach was used to send each phrase in the resulting text to SAPHIRE to get a list of associated MeSH terms. The list was further filtered by removing terms from a list of common but nonuseful terms (such as Normal or Increase) and those belonging to certain UMLS semantic types (such as Biomedical Occupation or Discipline). An example of the output of SmartQuery is shown in Figure 10.2. Because some resources, especially MEDLINE and Harrison’s Online, could return an
10. Augmenting Systems for Users
363
overwhelming number of search results, only the top five results from these resources were displayed on the results Web page. The results were displayed with hyperlinks so that the user could link directly to each. A growing number of commercial EMR vendors have begun introducing linkages to contextual knowledge-based information within their products. A major research question remaining to be addressed is how valuable this approach is to the busy clinician, in particular whether the process helps or hinders the busy clinician. One formative evaluation demonstrated that using the infobutton approach to link radiology reports to knowledge resources resulted in a 10-fold reduction in keystrokes (Zeng and Cimino, 1997). Ultimately, however, evaluation must demonstrate that useful information can be provided in real-world settings superior to a stand-alone retrieval application. 10.1.2.2 Linkage to Human Knowledge Recall from Chapter 2 that clinicians have frequent information needs, on the order of 2 questions for every 3 patients seen, yet they pursue answers to only one-third of them (Gorman, 1995). This research also showed that the most common source clinicians turned to for answers was another human, most often a colleague or consultant in their referral chain. It was shown in Chapter 7 that relative to overall information needs, computer-based knowledge resources have been used modestly (i.e., the average user seeks answers to clinical questions with online resources only
FIGURE 10.2. Search results in SmartQuery from query and resources selected in Figure 10.1
364
III. Research Directions
a few times per month). One likely reason for this is the time it takes to obtain an answer: upward of 30 minutes when one is using MEDLINE and journal literature (Hersh, Crabtree et al., 2000). It is possible that the move toward synoptic information resources, particularly those that adhere to principles of evidence-based medicine, may increase the usage of online knowledge resources (Haynes, 2001). Another approach to providing knowledge-based information to clinicians might involve the development of technologies that recognize the value of person-to-person consultation and facilitate it. This approach is much less developed than the myriad of online information resources, especially when used in the clinician-to-clinician mode. There are a great many online patient-to-clinician consultation services. Probably the largest of these is NetWellness, which has over 17,000 answered questions in its database (Guard, Fredericka et al., 2000). A query of “online medical consultations” to the Google search engine (last performed February 1, 2002) yielded dozens of such services, with over half offered commercially. Some early clinician-to-clinician consultation services were developed in Iowa (Bergus, Snift et al., 1998) and The Netherlands (Moorman, Branger et al., 2001). The former used e-mail for physician communications while the latter used an option within the EMR. A different approach has been taken by other who offer online consultations for a fee [e.g., Partners Medical System (econsults.partners.org) and The Cleveland Clinic (www.eclevelandclinic.org)]. A telemedicine project in Australia is attempting to provide online consultation for clinicians in underdeveloped countries (eSMART, www.coh.uq.edu.au/coh/ projects/telemedicine/esmart.html). Another project to facilitate online consultation has been the Professional’s Information Link (PiL) (Hersh, Miller et al., 2002). PiL was motivated by a combination of the unmet needs of a group of clinicians in rural Oregon and their desire for answers to their questions by specialists. These clinicians already had access to a telephone-based consultation service but found it insufficient. The central feature of PiL has been to provide answers to questions that arise during care, especially for topics on which local expertise is not readily available. Questions are forwarded to an academic medical center, where the aim is to provide an answer within two business days. The “asynchronous” (i.e., non-real-time) nature of PiL makes it more convenient both for busy rural clinicians and academic specialists. In addition, communications between both groups are facilitated by a medical librarian who forwards incoming questions to appropriate specialists, assists them in finding additional resources to augment answers, keeps them cognizant of the desired turnaround time, and returns them all to the individual who asked the question. Figure 10.3 depicts the flow of information through the system.
10.2 Indexing Another focus of research to augment IR systems has been on the indexing process. While Salton (1983) argued that attempts at human indexing were hopelessly flawed owing to the inconsistency of human indexers, others have noted that hu-
10. Augmenting Systems for Users
365
FIGURE 10.3. Information flow in the Professional’s Information Link (PiL).
man indexers do add value by providing a focus on the most important conceptual aspects in documents (Swanson, 1988a). Inasmuch as many individuals adhere to the latter view, a great deal of research has focused on tools to aid in indexing. While Chapter 9 described some systems that used approaches to recognize concepts as a step in the retrieval process [e.g., SAPHIRE (Hersh, 1991) and MetaMap (Aronson, 2001)], the systems described in this section were designed to enhance the NLM’s indexing process. This section also describes the Information Sources Map (ISM), a now-defunct component of the UMLS project that attempted to index whole databases, as well as the Semantic Web project.
10.2.1 Early Approaches Initial efforts to augment the indexing process focused on the content of documents as well as the databases in which they resided. As noted in Chapter 5, despite the detailed NLM protocols for human indexing of MEDLINE, indexers have substantial inconsistency (Funk and Reid, 1983). Likewise, indexers do not follow the protocols reliably (Crain, 1987). Humphrey (1992) asserted that human indexers had trouble going from text phrases to identifiable indexing terms and, once terms were found, coordinating them into a group of meaningful descriptors. To address these problems and to assist the human indexer, she initiated the Medindex system, a knowledge-based indexing assistant designed to facilitate correct and consistent indexing for Medical Subject Headings (MeSH) terms at the NLM (Humphrey, 1988). The system used a frame-based approach, with slots to hold related information and procedures that were invoked when the frame or one of its slots was instantiated. The selection of indexing terms (via frames) invoked prompts for additional information, aiming to suggest a coherent set of indexing terms and appropriate subheading assignment.
366
III. Research Directions
Humphrey provided an example with the term Estrogen Replacement Therapy. In Medindex, the system would suggest terms for the associated problem, in this case Osteoporosis, Postmenopausal, and the substance Estrogens. The latter action would invoke another procedure, which prompted for purpose in relation to Estrogens (for) Osteoporosis, Postmenopausal. In this situation, the choices would be subheadings attached to MeSH terms, such as Osteoporosis, Postmenopausal/Prevention and Control. Medindex also contained tools to manage the complex knowledge base that was required for its operation. The system grew to cover one-quarter of MeSH, about 4000 terms (Humphrey, 1992). Explorations were also undertaken to adapt Medindex to the retrieval environment, so that the complex relationships stored in the indexing process, which were not represented in the MeSH indexing output, could be made available during searching (Humphrey, 1988). MedIndex has now given way to the NLM Indexing Initiative described in Section 10.2.2. Roberts and Souter (2000) have also explored means to automated controlled vocabulary indexing in medical journal articles. They began with an analysis of some general observations about the indexing process: • The whole article is not read, so indexers must focus quickly on the crucial parts, which usually include the title, introduction, and results. • Indexing, however, is based on the entire article, not just the title and abstract • Indexing is not a word-mapping process; rather, it is based on the significant concepts in the article and their interrelationships. • Concepts are indexed only if the article contains significant new information about them, not just speculation. • Indexing is specific, so that the most specific term is always used. This led them to devise a process to experiment with automated indexing by using a subset of a database on complementary medicine (Roberts, 1995). Since they had only electronic text of the title and abstract, their analysis was limited to these portions of the articles. They conjectured that titles usually contained nouns and thus utilized them minus stop words. Abstracts were noted to contain important material not in the title, such as additional detail about the subject of the study, but also other content not relevant for indexing, such as statistical data. They adopted the approach of requiring words suitable for indexing in the abstract to occur three or more times, except for check tags, which were allowed to occur as infrequently as once. Their process of identifying controlled vocabulary terms was driven by rules applied to the words resulting from the foregoing process applied to titles and abstracts. Single- and multiword phrases were compared against controlled vocabulary terms with several minor variations: as they were, with “s” added to the end, with the last letter removed, and with the words disease and disorder interchanged at the end. If these rules did not result in an exact match, the best partial match was obtained, based on the number of words common to the phrase and vocabulary term. The evaluations were somewhat marred by the arbitrary se-
10. Augmenting Systems for Users
367
lection of records “appropriate” for a given process. However, only 57% of the suggested terms were also selected by human indexers, although the analysis also identified instances of incorrect human indexing (and the automated system found some of those appropriately).
10.2.2 NLM Indexing Initiative In recent years, the NLM has undertaken to explore whether fully or partially automated approaches can be effective for indexing its databases, by means of a new approach, the Indexing Initiative (IND). The motivation for IND comes from the realization that not only is human indexing expensive, but individuals trained to do it well are increasingly difficult to find and maintain. The entire effort is an experimentally driven effort to determine the best approach to indexing, especially as tools from the UMLS project and other efforts mature and increasing amounts of full text are available (Aronson, Bodenreider et al., 2000). A project Web site provides details on all the approaches that have been assessed, including those that have been discarded (ii.nlm.nih.gov). The schema of the approaches currently being employed in IND is shown in Figure 10.4. The input to the process consists, for now, of titles and abstracts,
Title + Abstract Phrases Noun Phrases
Trigram Phrase Matching
PubMed Related Citations
MetaMap UMLS concepts
Rel. Citations
Restrict to MeSH
Extract MeSH descr.
MeSH Main Headings
Clustering
Ordered list of MeSH Main Headings FIGURE 10.4. Overview of components of NLM Indexing Initiative. (Courtesy of National Library of Medicine.)
368
III. Research Directions
though full text will be considered when it is more consistently available. The output is an ordered list of MeSH main headings. The components of the process are isolated and parameterized so that the relative behavior of each can be analyzed. There are three main pathways for generating MeSH terms, which utilize MetaMap, trigram phrase matching, and PubMed-related citations, respectively. MetaMap has been described extensively (Aronson, 1996, 2001). Section 9.4.4.1. In IND, MetaMap is used to generate a set of ranked Metathesaurus terms based on two factors: frequency and relevance. The frequency factor f is measured as follows: number of times concept occurs f 13
(1)
The relevance factor is determined by four components: the MeSH tree depth m, the word length w, the character count c, and the MetaMap score mm, which are measured as follows: number of periods in MeSH tree address m 9
(2)
number of words in term w 26
(3)
number of characters in term c 102
(4)
raw MetaMap score mm 1000
(5)
Each of these above factors is normalized by applying a sigmoidal function that results in its value varying from 0 to 1, then the factor is relatively weighted based on experimentally derived parameters. The trigram phrase-matching algorithm breaks the text up into all possible phrases consisting of one to six contiguous words, omitting punctuation. The resulting strings have their individual words broken up into trigrams of three characters each, with extra weight given to certain derivations (demonstrated by the example DNA sequence selectivity): • The first trigram of the word (e.g., dna, seq, and sel) • The first character of the word (e.g., d##, s##) • The first letters of adjacent words (e.g., d s, s s) These trigrams are then measured against trigrams produced by the same process applied to strings in the Metathesaurus, using a cosine vector similarity, with all terms achieving a score above 0.7 included. The two processes just described have their Metathesaurus output restricted to MeSH in the following manner: • If the Metathesaurus concept has a direct mapping to a MeSH term, use that term.
10. Augmenting Systems for Users
369
• If the Metathesaurus concept has an associated expression (ATX), which consists of one or more MeSH terms that express a non-MeSH Metathesaurus concept, then the appropriate MeSH terms from the ATX are added. • If the Metathesaurus concept can be mapped to one or more broader MeSH terms, use those terms. • Otherwise attempt to match the Metathesaurus concept to nonhierarchically related terms. Depending on the step at which process is halted when unsuccessful, this algorithm can result in high precision (stopping early) or high recall (stopping late). Experimentally, an intermediate approach has been found to work best. The third approach in IND is to process the output from the PubMed Related Citations function, which yields other MEDLINE records that are similar. The top-ranking 20 citations for the given record are used, with terms exceeding a specified frequency included. The output of the three processes provide a list of ranked MeSH terms, which undergo a clustering process that reranks the terms based on a variety of factors. Although a formal evaluation has not yet been carried out, initial results find that the human-selected terms are generally identified by the process, although many other terms are suggested as well. Further studies will aim to identify how to best apply the approaches in the operational indexing process.
10.2.3 Information Sources Map Another issue addressed by NLM researchers was the ability to index entire databases. A problem always faced by users of IR systems is that a well-phrased search may be executed against the wrong database. Since most other medical databases are not as well known as MEDLINE, searchers may try to use MEDLINE for a topic that is best searched in a different database. The ISM was designed to address this problem. The two major goals of the ISM were to describe electronically available information sources in machine-readable format, so that computer programs can determine which sources are appropriate to a given query, and to provide information on how to connect to those sources automatically. The ISM was essentially a database about databases, indexed by terms in the Metathesaurus. Some of the data fields from the ISM are listed in Table 10.2. There were two basic types of data elements. The first type contained free text, such as a narrative describing the contents and a sample record. The second type contained controlled vocabulary terms, semantic relationships, and intended audience. The latter had the potential to be used by computer programs to automatically steer users to appropriate databases. Masys (1992) assessed the ability of the ISM data to steer users to the appropriate database. He had an expert librarian generate MeSH terminology to 50 clinical queries. Additional UMLS data were used by identifying the UMLS semantic type (from the Metathesaurus) and semantic relationship (from the Se-
370
III. Research Directions
TABLE 10.2. Some Data Elements from the UMLS Information Sources Map NA_ PO_ ADD CIT DEF MN_
Name Provider Address of provider City of provider Short narrative description of database Relevant MeSH headings
SRL STY TYP UF_ USE SAM
Semantic type relationships Semantic types Intended audience Update frequency Probability of use by discipline Sample output
mantic Network) when two types were present and linked by that relationship. A panel of two physicians and a librarian designated the relevant database for each search, with recall and precision calculated based on the “retrieval” of a database from matching the query to the ISM. The MeSH term and semantic types alone had high recall (86 and 91%, respectively), though the former had much higher precision (81%) than the latter (51%). Combining the MeSH term with the semantic relationship significantly lowered recall (32%) but greatly enhanced precision (91%). Combining all three with AND resulted in even lower recall (31%) but the highest precision (96%), while using OR resulted in the highest recall (99%) but low precision (47%). The most developed interface to the ISM was the NetMenu system from Yale University (Clyman, Powsner et al., 1993). After the user entered a query, the system would suggest databases most likely to have relevant material. This was done by matching the Metathesaurus term, semantic type, and/or type of information source from the user’s query with those specified in the ISM. P. Miller et al. (1995) summarized the experiences and lessons learned with this system. They found that seamless access to disparate IR systems across a network was difficult. There was a conflict between attempting to achieve consistency by restricting specific systems to a lowest common denominator functionally and the desire to exploit specific and sometimes advanced features of these systems. For example, most databases do not have MeSH subheadings, yet MEDLINE searching can be enhanced by their use. Miller et al. also found difficulties in indexing whole information resources, especially large ones covering many topics, such as MEDLINE or a medical textbook. The NLM also developed a system around the ISM called Sourcerer in the early evolution of the Web (Rodgers, 1995). Similar to NetMenu, a user’s query was first processed against the ISM to determine appropriate resources and then searching was allowed within those resources to retrieve specific documents. Follow-up research at NLM compared the indexing coverage of the ISM with natural language processing over full-text documents (Wright, Nardini et al., 1999). It was found not only that the latter provided a richer diversity of concepts from the index process, but also that the concepts in the ISM were too general to aid in selection of sources. Mendonca and Cimino (1999) also evaluated the ISM, assessing its indexing and how well it facilitated connection to its included databases. They found that while the resource for the most part supported the purpose of the ISM, it required further attention to granularity, completeness, and accuracy of indexing. The ISM component of the UMLS was ultimately abandoned. One of the reasons for this was the sheer increase in volume of information sources. The ISM
10. Augmenting Systems for Users
371
was undertaken in the pre-Web era, when the number of databases one could access was modest. Keeping track of the additional databases and Web sites that proliferated with the growth of the Web, however, was difficult if not impossible. Furthermore, the ability of search mechanisms to link to or index multiple sites made the need for an ISM-like system less compelling.
10.2.4 Semantic Web Throughout this book, “indexing” has by and large referred to attributes that describe what content is about (e.g., its subject material or format type) or how it can be accessed (e.g., where it resides or how much it costs). Some researchers have developed a much broader view of what the objects residing on the Web can do. They note that today’s content on the Web is mostly designed for reading. But the content could be expanded to include meaning and, as a result, to give computational tools the ability to understand that meaning and act upon it. This notion has been termed the Semantic Web by Tim Berners-Lee, the founder of the original Web (Berners-Lee, Hendler et al., 2001). In their article about it, Berners-Lee et al. provide an example that includes a medical situation. The phone rings, automatically reducing the volume in his entertainment system, and his sister reports that their mother is being referred to a medical specialist. At the doctor’s office, the mother’s Semantic Web had already taken the diagnosis, downloaded education-appropriate reading material about the treatment, and made an appointment with an appropriate specialist who is accepted by the woman’s insurance plan, resides within 20 miles, and is rated high on a trusted rating service. The brother will need to bring his mother to the specialist, so he activates his Semantic Web agent to interact with his mother’s and the specialist’s to find the appropriate time. The key notion in these interactions is that the Semantic Web has knowledge about the meaning of the system’s needs, activities, and interactions with others. An even simpler and more pure IR Semantic Web action might be a system that selects retrieval of material that is appropriate not only subjectwise, but also at the proper level (e.g., an evidence-based synopsis for a clinician who happens to be in the middle of a busy clinic day) and from an information provider with which the user has a valid account. In their original description of the Semantic Web, Berners-Lee et al. (2001) note that a limitation of the current Web is that, as with computers in general, there is no way to process the meaning or semantics of the underlying data. The Semantic Web requires knowledge representation to define objects (e.g., content, activities, etc.) and rules for their interactions. Knowledge representation has been studied by researchers in the field of artificial intelligence (AI) for years (Musen, 1992), but just as the original Web compromised with the more rigid early hypertext systems, the Semantic Web compromises with the detailed representations and their underlying rules and schemas developed by the AI community. Since the original description of the Semantic Web, a great deal of work has developed it further, as cataloged on the project’s Web site (www.semanticweb.org).
372
III. Research Directions
Two key technologies for the Semantic Web were introduced in Chapter 5: eXtensible Mark-Up Language (XML) and the Resource Description Framework (RDF). In essence, XML is the syntax of the Semantic Web and RDF is the semantics. XML provides the format to express the meaning of data in a manner that is consistent and can be processed by a computer program (Anonymous, 2001v). RDF provides the framework to represent that meaning (E. Miller, 1998). It was seen in Chapter 5 that RDF could express content representation schemes, such as Dublin Core Metadata (DCM) (Weibel and Koch, 2000). However, RDF can also express more complex forms of meaning, such as the location of a specialist and his or her office hours. These objects and relationships are usually described as an ontology. In their overview of ontologies, Gruninger and Lee (2002) note that ontologies are used in computer science for three purposes: 1. Communication: between humans, systems, or humans and systems 2. Computational inference: representing and manipulating data, plans, and other structures 3. Reuse and organization of knowledge: structuring data and plans for use and reuse A key challenge for Semantic Web ontologies, as with any other distributed computing effort, is determining how standards that allow sharing and collaborative applications can be developed (Kim, 2002). The Semantic Web site contains tools for building and maintaining components of the Semantic Web, such as editors to develop ontologies and metadata and databases to store and retrieve them. Semantic Web activity in the healthcare domain is in its infancy. A tool developed for designing and implementing ontologies in health care, Protégé, has been used in a variety of domains, including some outside health care (Noy, Sintek et al., 2001). The HealthCyberMap project has attempted to meld data from a geographic information system (GIS) with ICD-9 diagnosis codes to information specific to diseases and regions (KamelBoulos, 2001). Metadata is composed based on DCM but extended with additional fields, including the location of the publisher and the rated quality of the information resource.
10.3 Retrieval Although a large number of research approaches to improving retrieval have been devised, very few of them have been implemented in large-scale systems and, as a result, most systems are accessed via the state-of-the-art means described in Chapter 6. Most IR systems provide little assistance to the user beyond simple online help. Some research systems have added assistance aiming to help the user choose search terms better or use Boolean operators more effectively. A number of systems, some in the medical domain and others not, have been designed to serve as expert assistants. These systems typically have rules that recognize the factors that cause poor searches and suggest alternatives to the user.
10. Augmenting Systems for Users
373
This section describes a number of innovative approaches to enhancing retrieval, some of which are likely to find their way into large-scale systems in the future. Hearst (1999) groups user interface enhancements into four categories, which are adopted in this section: 1. 2. 3. 4.
Starting the search process Query specification Viewing results in context Aids for relevance feedback
A great deal of user evaluation comes from the field of human–computer interaction (HCI). B. Shneiderman, the author of a leading textbook in the HCI field (Shneiderman, 1998), notes that all computer usability (not just IR systems) must take human diversity into account, from physical abilities and physical workplaces to cognitive and perceptual abilities. He has formulated a set of rules that should govern the design of human–computer interfaces: 1. Strive for consistency in terminology, prompts, menus, help screens, color, layout, capitalization, and fonts. 2. Enable frequent users to use shortcuts, such as abbreviations, special keys, hidden commands, and macro facilities. 3. Offer informative feedback. 4. Design dialogs to yield closure (e.g., by organizing sequences of actions into groups). 5. Offer error prevention and simple error handling. 6. Permit easy reversal of actions. 7. Support internal locus of control. 8. Reduce short-term memory load. Although a variety of techniques have been used to assess human–computer interaction, one particular approach commonly used is usability testing (Rubin, 1994).
10.3.1 Starting Points Hearst (1999) notes that traditionally, a starting point has been provided by listing available resources. However, this approach is often inadequate, especially for users who are unaware of the structure of the resource(s) or the topics they cover. An alternative approach is to provide an overview of the contents. A variety of approaches have been devised, and the simplest one—the ability to browse through a classification hierarchy—is used quite commonly on the Web (e.g., by Yahoo!, www.yahoo.com). some systems have provided the ability to browse not only hierarchically, but also by traversing other thesaurus links (Korn and Shneiderman, 1996). Another approach has been to cluster documents topically. One of the best-known approaches of this technique was the Scatter/Gather (Cutting, Pedersen et al., 1992), which grouped documents into topically oriented clusters (based on word similarities). The user was able to focus on specific clusters to
374
III. Research Directions
further narrow them down, aiming to create an ideal document set. While the original Scatter/Gather approach was text based, a variety of graphical displays have been derived to demonstrate the clustering of topics. One interface employed in the healthcare domain used Kohonen maps, which cluster the concepts in a two-dimensional view (Chen, Houston et al., 1998). Another approach to assisting with starting points is to help the user select the appropriate source when a retrieval system searches over multiple databases. This is known to be a challenging research problem (Bartell, Cottrell et al., 1994), but a number of medical meta–search engines have been developed. One example was seen in the ISM portion of the UMLS project described earlier. Other approaches have attempted to help the user focus on certain resources, such as those with a high likelihood of being evidence based. The TRIP database (www. tripdatabase.com) provides word searching over 55 resources in its database, including many of high-profile full-text journals, along with other prominent evidence-based resources, such as the Cochrane Database of Systematic Reviews (www.cochrane.org) and the National Guidelines Clearinghouse (www. guideline.gov). SUMSearch (sumsearch.uthscsa.edu) provides similar coverage of
FIGURE 10.5. SUMSearch interface. (Courtesy of R. Badgett.)
10. Augmenting Systems for Users
375
evidence-based resources but provides more detailed searching options. Its interface allows focusing of results to question type (e.g., treatment), resource type, and other limits. Its searching algorithm attempts to adjust for too many or too few results retrieved (see Figure 10.5). For excessive results, searching in large databases (e.g., PubMed) is restricted, while for inadequate results, searching in other databases is added. An evaluation of SUMSearch showed that it did not increase frequency or satisfaction of searching among medical students (Badgett, Paukert et al., 2001).
10.3.2 Query Formulation A number of early systems attempted to assist the user with constructing queries. CONIT was a system that aimed to act as an intelligent intermediary for novice searchers (R. Marcus, 1983). It performed such tasks as assisting with the syntax of various search systems and mapping from a user’s natural language question text to Boolean statements of controlled vocabulary terms or text words. Mapping was done by stemming each natural language term in the query and taking the OR of each term that a stem mapped into, followed by an AND of all the different terms in the query. CONIT-assisted searches were found to achieve performance comparable to that of searches assisted by human intermediaries. CANSEARCH was designed to assist novice physician searchers in retrieving documents related to cancer therapy (Pollitt, 1987). The user did no typing and used only a touch-screen to navigate menus related to cancer sites and therapies. (Recall that MEDLINE has a particularly obtuse method of representing cancers, making the newer system all the more valuable.) Once the proper menu items were chosen, a MEDLINE search statement was formed based on rules in the program. The CANSEARCH main menu had the user select the specific cancer, while submenus allowed the designation of more detail, such as the treatment. For instance, if the user chose the site as breast cancer and the therapy as cisplatinum, then the resulting search statement passed to MEDLINE would be Breast Neoplasms/Drug Therapy AND Cisplatinum/Therapeutic Use AND Human. An additional method to assist the user has been reformulation of the query. Gauch and Smith (1991) developed an expert system that broadened the query until a certain number of documents had been retrieved. This is done by a number of means, such as the following: • Adding terms sharing the same stem (e.g., if the user entered cough, adding terms such as coughs and coughing) • Adding synonym terms (e.g., adding cancer when carcinoma was used as a search term) • Broadening proximity operators (e.g., changing a within-sentence operator to within-paragraph or whole-document searching) • Adding broader terms from a thesaurus (e.g., adding antibiotics when penicillin was used as a search term)
376
III. Research Directions
The most comprehensive effort to provide expert search assistance in the medical domain came from the COACH project at the NLM (Kingsland, Harbourt et al., 1993). COACH was added as an expert assistant to the now-defunct Internet Grateful Med. The rules used by COACH were based on an analysis of failed searches done by end users on the NLM system, as described in Chapter 7. Recall that the biggest problem found was searches with null retrieval (no documents returned). The most common reason for null retrieval was excessive ANDs, such that no documents had all the terms intersected together. Other mistakes commonly made included inappropriate use of specialty headings, improper use of subheadings, and failure to use related terms. COACH was activated from within Internet Grateful Med after a poor search was obtained. It offered two main modes of operation: assisted increase or assisted focus. The former was invoked when the search yielded no or only a few references. In that instance, COACH may have recommended reducing the number of search terms (or at least those connected by AND), using proper specialty headings where appropriate, or adding related terms or synonyms. The assisted focus mode was called on when an excessive number of references were retrieved. It may have recommended adding a subheading to or designating as a central concept one or more of the search terms. Another system of historical interest that aimed to implement hypermedia capabilities with the clinician in mind was Frisse’s Dynamic Medical Handbook project (Frisse, 1988). This system transformed a well-known reference, the Washington University Manual of Medical Therapeutics (Little Brown, Boston), used widely to assist in therapeutic decisions in internal medicine, into a dynamic resource. This reference had a rigid hierarchical structure that lent itself well to a hypermedia design. A combination of lexical–statistical IR and hypertext-based methods led the user to an appropriate starting point for browsing, at which point he or she explored linked nodes to find information. Frisse also noted several user interface features that were necessary to address the ways in which medical handbooks were typically used. These included the following. 1. 2. 3. 4.
Highlighting: to emphasize important concepts and passages Annotating: to add explanatory information Page turners: to rapidly move back and forth between sections Path tracers: to mark the path that led to a section, to preserve content discovered along the way 5. Bookmarks: to allow the user to return later to a specific place 6. Clipboard: to keep information “photocopy,” with the source and context specified, to allow the user to return to it 7. Agenda keeper: to keep a list of future readings and tasks In the development of the Dynamic Medical Handbook, Frisse noted that even for a hypertext of modest size, such as the Washington Manual of Medical Therapeutics, browsing alone was inadequate for retrieving information. He viewed the query as a means to find an optimal starting point for browsing. Frisse also
10. Augmenting Systems for Users
377
postulated that the conventional isolated-document view of IR was inadequate for a richly linked hypertext. Thus he developed an approach to retrieval that began with a conventional lexical–statistical approach but modified document weighting to account for terms in linked nodes (Frisse, 1988). Typical of a word–statistical IR system, the indexing units were single words and the documents retrieved by a query were those that had one or more words from the query. The uniqueness of Frisse’s approach was the use of two weighting components to rank nodes. The intrinsic component of the weight consisted of the usual TF*IDF weighting for the words common to each node and the query. The extrinsic component, however, was the weights of all immediately linked nodes divided by the number of such nodes. Thus the formula for the weight of a node was:
1 WEIGHTi WEIGHTij y j
d WEIGHTd
(6)
where WEIGHTi was the total weight of node i, j was the number of search terms, WEIGHTij was the weight of all the search terms j in node i, y was the number of immediately linked nodes, d was the index number of immediately linked nodes, and WEIGHTd was the weight of each linked node. Since the Dynamic Medical Handbook contained only hierarchical links, this weighting approach had the effect of giving more weight to documents “higher” in the hierarchy. The rationale for this was that these documents were more general and thus tended to serve as good starting points for browsing. Frisse and Cousins (1989) subsequently implemented a relevance feedback mechanism based on belief networks that allowed the retrieval of more nodes based on designation of relevance for visited nodes.
10.3.3 Providing Context A number of approaches attempt to provide the user with some sort of context about the document. Hearst (1999) notes that the simplest approach has been through the use of document surrogates that show brief portions of the text. Many Web search engines, for example, show representative sentences from the pages that contain the query terms and words surrounding them. Another state-of-theart approach has been to highlight query terms in the document. Hearst (1996) has developed the TileBars representation for providing context: the user enters each query topic on a single line and the retrieval interface shows a tile bar next to each document, with the intensity of each tile representing the presence of the topic with each passage of the document. The presence of all topics within a passage indicates a portion of the document that likely will be important to view because all the terms occur in it. Another approach to showing context has been by the organization of search results via some classification, such as a table of contents or term hierarchy. The SuperBook hypertext statistical textbook organized search results via its table of contents, which would provide users a context for the portion of the book retrieved (Egan, Remde et al., 1989).
378
III. Research Directions
Alternatively, in the healthcare domain, the DynaCat system has used UMLS knowledge and MeSH terms to organize search results (Pratt, Hearst et al., 1999). The goal of DynaCat is to present search results with documents clustered into topical groups, such as the treatments for a disease or the tests used to diagnose it. The first step in DynaCat’s process is to map the user’s queries into one of nine query types. For example, queries of the type treatment-problems might come from search statements that ask for the complications or adverse effects of a treatment. Other query types include problempreventive actions, problem-risk-factors, problem-tests, problem-symptoms, symptoms-diagnosis, problem-treatments, problem-prognostic-indicators, and problem-prognoses. Each query type organizes search results around MeSH terms in the MEDLINE record for the retrieved documents that contain one of a small number of UMLS semantic types. For example, the treatment-problems query type organizes results around MeSH terms that are of the semantic type Disease or Syndrome. If the retrieved MEDLINE records have MeSH terms of this semantic type (e.g., Lymphedema), then the search output will be categorized around those terms. Other terms of different semantic types (e.g., Mastectomy) will not be used in categorization. The search results are presented to the user grouped under the terms selected for categorization. For example, all the articles indexed with the MeSH term Lymphedema are listed under the term in a category, as shown in Figure 10.6. Likewise, articles classified under other terms are displayed as part of the categories for those terms. An evaluation with breast cancer patients and their family members showed that DynaCat allowed them to find more answers to questions in a fixed amount of time and resulted in higher user satisfaction than two other systems, one providing clustering by a table of contents view and the other showing a best-match ranking of the query (Pratt and Fagan, 2000). Further work has attempted to generalize the approach beyond the breast cancer domain (Pratt and Wasserman, 2001). Another approach to providing context to the user has been the Cat-a-Cone system (Hearst and Karadi, 1997). This system provides a means to explore term hierarchies by using cone trees, which rotate the primary term of interest to the center of the screen and show conelike expansion of other hierarchically related terms nearby. Figure 10.7 shows an example using the MeSH hierarchy. The user can either view the list of all documents indexed with the primary term or search within those documents by entering additional search terms. The basis of Cat-aCone derives from several of the information visualization tools developed by the Xerox PARC User Interface Research Group (Anonymous, 2002b). Similar to Cat-a-Cone is the TopicMap system, available from HighWire Press (www.highwire.org), which provides a hierarchy (different from MeSH but incorporating many of its terms) that allows browsing of topics to choose a particular one for searching. The TopicMap interface, shown in Figure 10.8, displays nearby topics with the one of focus at the center of the display. When other
10. Augmenting Systems for Users
379
FIGURE 10.6. Output from DynaCat search on risk factors for breast cancer. (Courtesy of W. Pratt.)
topics are clicked, they move to the center of the display. When the center topic is clicked, a page opens that displays articles indexed on that topic. The search can be limited to a core set of journals, all HighWire journals, or all HighWire journals plus MEDLINE. Another means of providing context is to demonstrate the relationship of documents to other documents. One approach that has been shown to be effective is not graphical, and in fact relies on reference tracing, in which additional documents are retrieved for the user based on bibliographic citations in ones already obtained and found to be relevant. Chapter 2 demonstrated how citations in papers form networks, showing progression in an area of science. Since authors cite papers relevant to their work, it may be that these cited papers are relevant to someone’s searching. Citation retrieval can be thought of as a form of relevance feedback, since it requires at least one relevant paper for the process. Citation retrieval can be backward (i.e., papers cited by the relevant one are added to the retrieval list) or forward (i.e., papers that cite the relevant one are added to the list). Figure 10.9 depicts this process graphically. Reference tracing is not a new idea. It was recognized in the 1960s that the
380 FIGURE 10.7. Cat-a-Cone interface. (Courtesy of M. Hearst.)
10. Augmenting Systems for Users
FIGURE 10.8. TopicMap view for accessing medical journals. (Courtesy of Highwire Press.) © 2002 Inxight Software, Inc.
FIGURE 10.9. Graphical depiction of reference tracing.
381
382
III. Research Directions
bibliographic references in scientific papers could be useful indicators of the significance (Westbrook, 1960) and content (Kessler, 1963) of those papers. In fact, networks of documents and citations were advocated as having many uses in characterizing scientific communications and progress (Price, 1965). The most practical searching tool to arise out of this early work was the Science Citation Index (SCI), which was described in Chapter 4 (Garfield, 1964, 1979b). The use of citations to enhance IR systems has also been advocated since the 1960s. Kessler (1963, 1965) found that papers related by bibliographic citations were also related by subject matter. Salton (1963) noted that this type of correlation was significantly greater than what would be expected for random document sets. These observations were verified by Trivison (1987), who found that 77% of references cited from a collection of information science literature contained related terms that could be used for retrieval. If citations provided an alternate means of retrieval, how well would they perform in actual systems? This was first evaluated by Salton (1971), who noted with a small test collection that reference tracing could enhance retrieval performance. Pao and Worthen (1989) evaluated MeSH term searching and citation searching for a cystic fibrosis database. They noted that although MeSH term searching produced a higher number of relevant references per search, there were about 14% of relevant references that could be retrieved only by citation tracing. These results confirmed earlier work by Griffith et al. (1986) demonstrating that reference tracing could complement retrieval by indexing terms, increasing recall by up to 33%. Pao (1993) performed the most comprehensive study of the value of citation searching in health care (Pao, 1993). She looked at 85 searches and expanded them by the SCI, finding that citation links added at least one relevant items to 85% of the searches. Overall, the citations linked to an average of 24% more relevant materials. Over half the searches had only a handful of additional citations (10), and of these almost half were relevant. Pao also noted that citation linking can be effective in adding extra relevant citations without excessive loss of precision.
10.3.4 Assisting Relevance Feedback and Query Expansion Operational systems in the healthcare domain employing relevance feedback or query expansion have been few. Probably the most prominent approach has been the Related Articles feature of PubMed described in Chapter 6. Another approach has been to use thesaurus relationships to automatically expand query terms. While some research has shown modest benefits with this approach (Srinivasan, 1996c; Aronson and Rindflesch, 1997), other investigations have not (Hersh, Price et al., 2000). One system that employs these approaches is Medical World Search, initially a research system (Suarez, Hao et al., 1997) and now available commercially (www.mwsearch.com). Medical World Search performs searching over a subset of Web sites determined to be of high quality by an editorial process. It matches user query terms to UMLS Metathesaurus concepts and then allows
10. Augmenting Systems for Users
383
synonym terms and/or more specific terms (hierarchically narrower terms) to be added to the query.
10.4 Devices This book has largely ignored the types of device on which IR systems operate. Almost implicit from this omission is that IR will take place on a standard desktop or laptop computer. However, there is growing interest in accessing knowledge-based information from devices of different kinds, in particular more portable ones, such as handheld devices. Although these devices are widely used in various aspects of healthcare, and some would argue that this alone demonstrates their value, very little research has assessed their effectiveness. This section covers not only handheld devices but also intermediate-sized devices, such as tablets. The major concern with portable devices has been screen size. A variety of studies have demonstrated that users read 15 to 25% more slowly from screens that are one-third to two-thirds the size of regular screens (Duchnicky and Kolers, 1983; Resiel and Shneiderman, 1987), although others have shown no differences (Shneiderman, 1987; Dillon, Richardson et al., 1990). One of the nodifference studies found that users had to navigate more and were more likely to want a different screen size (Dillon, Richardson et al., 1990). A more recent study compared a Web browser on a larger (1024 768 pixels, approximately 30 lines of content) and a smaller (640 480 pixels, approximately 15 lines) display (Jones, Marsden et al., 1999). Users were given four tasks to complete, with two requiring focused searches and two needing more general searches. As shown in Table 10.3, searches were much more successful with the larger screen. These users were also twice as likely to indicate that the smaller screen impeded task performance. Further analysis showed that users of the smaller screen required many more navigational moves within pages than users of the larger screen. Users of the larger screen, however, were more likely to follow links across pages, whereas users of the smaller screen more often returned to the search interface to query the database again. The authors concluded that
TABLE 10.3. User Success With Larger and Smaller Screen Sizes Percentage correct General task Focused search General search
Specific task
With small screen
With large screen
Find share price for given company Evaluate performance of company over time Find continent with the most public holidays in May 1998 Select any intriguing public holiday
40 30
10 0
50
20
80
70
Source: Adapted from Jones, Marsden et al. (1999).
384
III. Research Directions
Web pages likely to be viewed on smaller devices should place navigational features and key information near the top of the page and reduce the amount of information on pages.
10.4.1 Handheld Devices Handheld computers are generally considered to be those that fit in the palm of a hand or in a shirt pocket. They were popularized by the commercial success of the Palm Pilot (Palm Computing, www.palm.com), but those running the Windows CE operating system (Microsoft, www.microsoft.com) have been gaining in usage. Recent surveys show widespread usage by clinicians. One survey of internal medicine physicians showed that 47% were then using them and another 20% would begin using them in the next year (Anonymous, 2002a). These physicians stated that the medical applications they use with their handhelds included drug information (80%), references for normal lab values (32%), medical textbooks (21%), and billing or coding (21%). A survey of family medicine residency programs found that (67% of residency programs used handheld devices, with the applications used comparable to those cited by the internists (Criswell and Parchman, 2002). A growing number of projects have been employing these devices to enter and/or look up patient clinical data (e.g., Poon, Fagan et al., 1996; Duncan and Shabot, 2000; Chen and Cimino, 2001). While the widespread adoption of handheld devices would imply that they are used successfully, no studies have assessed how well users search knowledge-based resources with them. Perhaps more importantly, no research has attempted to elucidate which types of search and information resources in health care are best suited for this platform and which are ill suited. Given that the most popular applications appear to be drug references, disease outlines, and other list-type applications, it may be that these simpler types of application are best suited for these devices. Figure 10.10 shows the search interface for Epocrates (www.epocrates.com), one of the most popular drug references available on the Palm Pilot, which essentially consists of an alphabetical list of drugs from which to choose. Other applications that consist of more complex information and/or require more complex searching interfaces may be difficult on the small screen of a handheld. Handhelds also have limitations in terms of memory capacity and network connectivity that restrict the types of information for which they are best suited (though the latter may change as wireless networks increase in bandwidth). Some research outside healthcare has looked at means of using handheld devices to facilitate searching. One approach has been to assist the user in developing a small collection of highly relevant information that resides on the handheld and is accessed in “disconnected” mode (Aridor, Carmel et al., 2002). This content consists of the 100 or so most authoritative pages on a topic along with the titles and URLs of other pages that are available when the device is “reconnected” via either synchronization or connection to a network. This approach is probably most effective for a user who works in a relatively narrow domain as opposed, for example, to a primary care clinician. Another approach used vary-
10. Augmenting Systems for Users
385
FIGURE 10.10. Epocrates search interface. (Courtesy of OCLC.)
ing degrees of text summarization to reduce the amount of text displayed (Buyukkokten, Kaljuvee et al., 2002). Methods used included the following: • Incremental display of sentences (e.g., one, a few, or many) • Keywords, displayed individually or with other words for context • Summaries, with application of text summarization techniques A user evaluation showed that although different techniques worked better for different information tasks, all reductions in text size decreased the workload for the user. Other approaches to summarizing Web pages have been developed in the Digestor (Bickmore and Schilit, 1997) and OCELOT (Berger and Mittal, 2000) projects.
10.4.2 Tablet Devices If handheld devices are too small, then perhaps a compromise might be devices that are somewhat larger but still smaller than conventional laptop or desktop computers. Microsoft has adapted the operating system used on handheld devices, Windows CE, to tablet devices (Spring, 2001). These devices, also known as reading appliances, are aimed at providing functionality that allows users to interact with electronic documents the way they interact with papers ones (Schilit, Price et al., 1999). This includes having the functionality to search, annotate, and organize such documents. Golovchinsky et al. (1999) have assessed the IR capabilities of such devices and their documents, finding that when users could se-
386
III. Research Directions
lect text passages, performance exceeded that obtained with standard relevance feedback approaches. Further work with these devices has led to emphasis on electronic books or eBooks, with the development of standards and specifications (www.openebook.org). Further work remains to determine the value of these devices in healthcare settings.
10.5 Digital Libraries Discussion of IR “systems” thus far has focused on the provision of retrieval mechanisms to access online content. Even with the expansive coverage of some IR systems, such as Web search engines, they are often part of a larger collection of services or activities. An alternative perspective, especially when communities and/or proprietary collections are involved, is the digital library (DL). DLs share many characteristics with “brick and mortar” libraries, but also take on some additional challenges. Borgman (1999) notes that libraries of both types elicit different definitions of what they actually are, with researchers tending to view libraries as content collected for specific communities and practitioners alternatively viewing them as institutions or services. As evidence that DLs are a topic of important concern, it should be noted that the U.S. President’s Information Technology Committee published three reports in 2001, one of which covered the topics of DLs (Anonymous, 2001e). The other two focused on the use of information technology to enhance health care (Anonymous, 2001p) and education (Anonymous, 2001s). The DL report stated that the full potential of DLs has not been realized, noting that the underlying technologies (particularly the Internet) were developed with federal leadership, that archives face both technical and operational challenges, and that the issue of intellectual property cannot be ignored. The report recommended that support for research be expanded, large-scale testbeds be developed, all federal material be put online, and the government lead efforts to develop digital rights policies.
10.5.1 Overview of Libraries Libraries have traditionally performed a variety of functions, including the following: • Acquisition and maintenance of collections • Cataloging and classification of items in collections • Serving as a place where individuals can go to seek information with assistance, including information on computers • Providing work or studying space (particularly in universities) DLs provide some of these same services, but their focus tends to be on the digital aspects of their content. There are a number of organizations concerned with libraries. The American Library Association (ALA, www.ala.org) focuses on libraries in general, while
10. Augmenting Systems for Users
387
the Special Library Association (SLA, www.sla.org) focuses on scientific and technical libraries. An organization that has emerged to address the scientific aspects of DLs is the Joint Conference on Digital Libraries (JCDL, www.jcdl.org), sponsored by the Association for Computing Machinery (ACM, www.acm.org) and the Institute of Electrical and Electronics Engineers (IEEE, www.ieee.org). As noted in Chapter 1, the Medical Library Association (MLA, www.mlanet.org) addresses the issue of medical libraries. Resources for DLs include books by Lesk (1997) and Arms (2000) and Digital Library Magazine (www.dlib.org). Much research in the area has been funded by the National Science Foundation (NSF) Digital Libraries Initiative (dli2.nsf.gov), which has participation by several other federal agencies, including the NLM. Probably the main function of libraries is to maintain collections of published literature. They may also store nonpublished literature, such as letters, notes, and other documents, in archives. But the general focus on published literature has implications. One of these is that, for the most part, quality control can be taken for granted. At least until the recent past, most published literature came from commercial publishers and specialty societies that had processes such as peer review which, although imperfect, allowed the library to devote minimal resources to assessing their quality. While libraries can still cede the judgment of quality to these information providers in the Internet era, they cannot ignore the myriad of information published only on the Internet, for which the quality cannot be presumed. The historical nature of paper-based traditional libraries also carries other implications. For example, items are produced in multiple copies. This frees the individual library from excessive worry that an item cannot be replaced. In addition, items are fairly static, simplifying their cataloging. With DLs, these implications are challenged. As noted in Chapter 2 and as described further shortly, there is a great deal of concern about archiving of content and managing its change when fewer “copies” of it exist on the file servers of publishers and other organizations. A related problem for DLs is that they do not own the “artifact” of the paper journal, book, or other item. This is exacerbated by the fact that when a subscription to an electronic journal is terminated, access to the entire journal is lost; that is, the subscriber does not retain accumulated back issues, as is taken for granted with paper journals. Another major function of libraries has been to provide access to their collections. The traditional means for accessing collections was the card catalog: users picked a subject or author name and flipped through cards representing items in the library. Card catalogs have largely been replaced by online public access catalogs (OPACs), which are similar to but have differences from document retrieval systems (Hildreth, 1989). The main difference emanates from the general differences between books and “documents.” That books tend to be larger and to cover more subject material is exemplified by envisioning the difference between a medical journal article on a specific disease and its treatment and a book on internal medicine. Thus, catalogers of books tend to use much broader indexing terms. In many ways, their challenges are similar to those faced by the NLM in attempts to build the UMLS ISM.
388
III. Research Directions
A major challenge for libraries is managing the physical size of their collections. Lesk notes that many aspects of libraries mimic the exponential growth curve documented in the growth of scientific literature by Price (1963), such as the number of volumes in libraries. Related to this challenge is the fact that libraries tend to be public or (even in private organizations) quasi-public entities that do not generate large revenue streams. Cummings et al. (1992) have documented that the costs of maintaining collections have exceeded the resources that libraries generally have available, resulting in a reduction of their purchasing power. Despite the potentially lower costs of production of electronic publication, the benefit of such publication has not yet been realized (see Section 2.6). Because libraries are traditionally resource poor, another major activity they undertake is sharing of collections. Few libraries can maintain complete collections, so most participate collaboratively with other libraries to attaining materials via interlibrary loan (ILL). This of course potentially sets up conflict, as the needs of publishers to maintain revenues to continue being in business must be weighed against the desire of libraries to reduce costs by sharing. The right to protection of intellectual property is enshrined in the U.S. Constitution (Article I, Section 8, Clause 8), which states that Congress has the power to “promote the progress of science and useful arts, by securing for limited times to authors and inventors the exclusive right to their respective writings and discoveries.” Copyright law for the United States is detailed in Title 17 of the United States Code (uscode.house.gov/title_17.htm). Within this law, however, is a provision for fair use (Section 107), which allows copyright work to be reproduced for purposes of “criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research.” In particular, there are four factors to be considered whether an activity constitutes fair use: 1. 2. 3. 4.
Purpose of use: educational vs commercial Nature of work: photos and music more protected than text Amount of copying: should be for individuals Effect of copying: out-of-print materials easier to justify than in-print
Fair use guidelines vary among libraries. A Web site that provides pointers to a wide range of material on the topic has been developed by Stanford University Libraries (fairuse.stanford.edu). The actual interpretation of fair use varies from library to library. Lesk (1997) lists some typical guidelines. Copying guidelines might limit users to 1000 words or 10% of book, a single journal article, or one illustration or image per book or journal. For libraries requesting materials from other libraries, the guidelines might limit copying to five articles from the most recent five years of a journal. Publishers have responded to fair use guidelines by creating the Copyright Clearance Center (CCC), which attempts to standardize the process of royalty payments for use of journal articles that are reproduced. Royalty payments are usually listed at the bottom of the first page of journal articles. They apply to individual use and not reproduction in other published works. The CCC acts as a clearinghouse, with libraries and copy shops forwarding royalties to CCC, which distributes them to publishers.
10. Augmenting Systems for Users
389
Another important goal of libraries is the preservation of library collections. In traditional libraries, the aim is for survival of the physical object. There area number of impediments to preservation: • Loss, theft, and general deterioration from use • “Perfect binding,” or the use of glue in binding instead of sewing, which reduces the longevity of books • Acid paper, that is, materials printed on paper produced by means of an acid process, mainly between 1850 and 1950, which has led to an international resolution eschewing further use of such processes (Anonymous, 1989) DLs have their own set of preservation issues, which are described in Section 10.5.5.
10.5.2 Definitions and Functions of DLs A number of authors have attempted to define exactly what is meant by the term “digital library.” Borgman (1999) reviewed all the definitions put forth by others and concluded that there were two competing views: a research-oriented view, with DLs defined to represent specific collections of content and/or technologies, and a service-oriented view, with DLs thought to represent a set of services provided to a specific community, such as a university or a company. Humphreys (2000) recently examined DLs from the standpoint of being able to provide information in the context of the EMR, as described in Section 10.1.2.1. Zhao and Resh (2001) note that the Internet transforms publishing, hence DLs. Among its effects are the following: • More efficient access to knowledge: for example, via searching and access to full-text content • New knowledge representations: access to hypertext, multimedia, and ultimately the Semantic Web • Interdisciplinary integration: linkages across journals and other resources • Transformation of production processes: streamlining of the peer review and editing processes • Transformation of the consumption process: digital libraries bypassing conventional libraries and authors bypassing publishers These authors surveyed a number of librarians and found great enthusiasm for Internet publishing. One concern about digital libraries is access to the professionals who have always aided users of physical libraries. For example, reference librarians are still key to assisting researchers, especially when exhaustive searching is required, such as in systematic reviews. One challenge to the role of library professionals is an economic one: as the amount of online content increases in availability to users, and it is used directly with increasing frequency, the amount of time that can be devoted to any user and/or resource shrinks. However, the value of professional assistance to users cannot be denied. One proposal that caused a stir when it was promoted in an Annals of Internal Medicine editorial is the notion of a new information professional for the clinical setting, the informationist
390
III. Research Directions
(Davidoff and Florance, 2000). The medical library community responded to this call by touting the virtue of clinical librarianship and affirming the value of library science training (Schacher, 2001). This term was actually introduced by Garfield (1979a) in the context of having information professionals who understood the laws of information science, and advocated in the early 1990s by Quint (1992). A recent special issue of Journal of the Medical Library Association responded to how librarians might take up the challenge in this new era (Shearer, Seymour et al., 2001), with additional elucidation on how education in library and information science (Detlefsen, 2002) and medical informatics (Hersh, 2002) might be structured. As noted, the major goals of libraries are provision of access to collections, management of intellectual property issues, and preservation of materials. Libraries that are predominantly digital have unique challenges for all three of these.
10.5.3 Access to Content Earlier chapters have focused on the importance of metadata in providing access to online content. From the DL (and commercial publishing) perspective, the view of metadata is broader than what has been focused upon so far, which is the access to content based on resource type or conceptual content. The larger view incorporates the notion that content must be made available not only reliably, but also in a manner that allows use of intellectual property to be appropriately tracked and expensed. The latter is addressed in Section 10.5.4. This section focuses on access to individual items, collections of items, and the metadata that describes them. 10.5.3.1 Access to Individual Items Probably every Web user is familiar with clicking on a Web link and receiving the error message: HTTP 404 - File not found. DLs and commercial publishing ventures need mechanisms to ensure that documents have persistent identifiers so that when the document itself physically moves, it is still obtainable. The original architecture for the Web envisioned by the Internet Engineering Task Force was to have every uniform resource locator (URL), the address entered into a Web browser or used in a Web hyperlink, linked to a uniform resource name (URN) that would be persistent (Sollins and Masinter, 1994). The combination of a URN and URL, a uniform resource identifier (URI), would provide persistent access to digital objects. The resource for resolving URNs and URLs was never implemented on a large scale. Various approaches for permanence of Web objects identifiers have been developed. The Persistent URLs (PURL) system uses the redirect feature of the Hypertext Transfer Protocol (HTTP) (Weibel, Jul et al., 2002). A PURL is actually a URL that uses a resolver address in the domain portion of the URL and a name after the domain portion. The resolver address is a PURL server (written in source code freely distributed by the Online Computer Library Center, OCLC, www.oclc.org) that processes the name and returns a URL that points to the current location of the object. For example, the PURL http://
10. Augmenting Systems for Users
391
purl.oclc.org/OCLC/PURL/FAQ is processed by the PURL server to return a URL that redirects the Web browser to the actual location of the object, as depicted in Figure 10.11. Another approach to persistent identifiers is the Handle System (Sun and Lannom, 2002). This system not only allows persistent naming but also tracks multiple exact copies in different locations of a digital object as well as different versions (e.g., a scientific paper, a preliminary draft of it, and a version with supplemental data). A limitation of this system is the requirement of a special browser plug-in. One approach that has begun to see widespread adoption by publishers, especially scientific journal publishers, is the Digital Object Identifier (DOI) (Paskin, 1999). A key aspect of the DOI initiative is the International DOI Foundation (IDF, www.doi.org), a membership organization that assigns a portion of the DOI to make it unique. The DOI system consists of four components: 1. 2. 3. 4.
Enumeration: the location of the identifier, the DOI Description: metadata of the entity association with the DOI Resolution: the means to resolve the identifier to actually located the object Policies: rules that govern the operation of the system
The DOI has recently been given the status of a standard by the National Information Standards Organization (NISO) with the designation Z39.84. The DOI itself is relatively simple, consisting of a prefix that is assigned by the IDF to the publishing entity and a suffix that is assigned and maintained by the entity. For example, the DOI for articles from the Journal of the American Medical Informatics Association have the prefix 10.1197 and the suffix jamia.M####, where #### is a number assigned by the journal editors. Likewise, all publications in the Digital Library of the Association for Computing Machinery (www.acm.org/dl) have the prefix 10.1145 and a unique identifier for the suffix (e.g., 345508.345539) for the paper (Hersh, Turpin et al., 2000a). The DOI system is designed to work within the Handle System as its adoption spreads. The portion of the prefix to the left of the period (10) indicates to the Handle System that the identifier is a DOI. Until the Handle System is adopted more widely, publishers are encouraged to facilitate resolution by encoding the DOI into their URLs in a standard way. For example, the paper referred to in the preceding paragraph (Paskin, 1999) is located by the URL http://doi.acm. org/10.1145/345508.345539.
FIGURE 10.11. Persistent URL (PURL) architecture. (Courtesy of OCLC.)
392
III. Research Directions
10.5.3.2 Access to Collections With myriad resources online, there has always been a desire to provide seamless access. In the pre-Web era, a standard was developed to provide a standard means for IR clients and servers to interact with each other. Called Z39.50, it aimed to enable any server to allow searching on its collection (with appropriate restriction based on access rights) and at the same time to allow any client (with the proper access rights) to search on any server (P. Miller, 1999). This separation of the user interface from the back-end IR system allowed users on different platforms and with different clients to access the same collections. This approach, however, limited the amount of retrieval capabilities, since each of the disparate components, client and server, needed to understand the functionality of the other. Early versions of the protocol, for example, did not provide for natural language searching capabilities. The momentum for Z39.50 was also hampered by the early search engines of the Web, which developed their own mechanisms for sending queries from Web page forms to IR systems. Another challenge for providing access to collections is the ability to link across disparate documents. In some fields, such as computer science, large amounts of scientific papers have been placed online by their authors and others. (This has generally not been the case for health care, where copyright restrictions have adhered to with greater rigor.) Lawrence et al. (1999) have exploited the large amount of online content to build ResearchIndex (formerly called CiteSeer), which uses a technique called autonomous citation indexing (ACI) to link the citations as well as the postings of the documents themselves to provide a comprehensive DL of scientific work in various areas of science. 10.5.3.3 Access to Metadata As noted throughout this book, metadata is a key component for accessing content in IR systems. It takes on additional value in the DL, where there is desire to allow access to diverse but not necessarily exhaustive resources. One key concern of DLs is interoperability (Besser, 2002). That is, how can resources with heterogeneous metadata be accessed? Arms et al. (2002) note that three levels of agreement must be achieved: 1. Technical agreements over formats, protocols, and security procedures 2. Content agreement over the data and the semantic interpretation of its metadata 3. Organizational agreements over ground rules for access, preservation, payment, authentication, and so forth One approach to interoperability gaining favor is the Open Archives Initiative (OAI, www.openarchive.org) (Lagoze and Van de Sompel, 2001). This project had its origins in the E-Prints initiative, which aimed to provide persistent access to electronic archives of scientific publications (Van de Sompel and Lagoze, 1999). While the OAI effort is rooted in access to scholarly communications, its methods are applicable to a much broader range of content. Its fundamental activity is to promote the “exposure” of archives’ metadata such that DL systems
10. Augmenting Systems for Users
393
can learn what content is available and how it can be obtained. The OAI recognizes two classes of participants: 1. Data providers that expose metadata about their content 2. Service providers that harvest metadata to provide value-added services These participants agree to adhere to the OAI protocol that provides a low-cost, low-barrier approach to interoperability. Each record in the OAI system has an XML-encoded record. Each of these records contains three parts: 1. Header: a unique identifier and a datestamp indicating creation or latest modification 2. Metadata: a set of unqualified DCM tags describing the resource 3. About: an optional container for information about the metadata, such as its schema The DCM tags are explicitly unqualified because the focus of the metadata is on discovery (i.e., providing systems with a description of what is there) as opposed to description (i.e., providing a mechanism for more detailed searching). The “about” container can be used to describe more about the metadata or, as suggested by Lagoze and Van de Sompel (2001), provide information about rights for access or terms and condition for usage. With this framework, the OAI protocol then allows selective harvesting of the metadata by systems. Such harvesting can be date based, such as items added or changes after a certain date, or set based, such as those belonging to a certain topic, journal, or institution. The harvesting protocol allows six different activities, as listed in Table 10.4. A number of other protocols address interoperability. The Simple Digital Library Interoperability Protocol (SDLIP) focuses on integrating the offerings of different DLs (Paepcke, Brandriff et al., 2000). One innovation of this protocol is the maintenance of metadata elements for searching paradigms of different types (e.g., Boolean or natural language). The protocol can then take a search statement and format it for a given search engine. Another protocol that has been implemented in the Digital Object Architecture of the Corporation for National Research Initiatives (CNRI, www.cnri.reston.va.us), which focuses on normalizing heterogeneous metadata to make it amenable to management by a DL (Blanchi
TABLE 10.4. Allowable Actions in Open Archives Initiative Harvesting Protocol Action GetRecord Identify
ListIdentifier ListMetadataFormats ListRecords ListSets
Description Retrieve a single record Retrieve information about a repository including, at a minimum, the name, base URL, version of OAI protocol, and e-mail address of the administrator of the resource Retrieve list of identifiers that can be harvested from a repository Retrieve metadata formats from a repository Harvests records from a repository, with optional argument for filtering based on attributes of records, either date based or set based Retrieve set structure in a repository
Source: Lagoze and Van de Sompel (2001).
394
III. Research Directions
and Petrone, 2001). Many of these projects have associated toolkits of source code to allow their use. One complete DL software system, implemented as opensource software, is Greenstone, developed by the New Zealand Digital Library Project (www.nzdl.org) (Witten, Bainbridge et al., 2001).
10.5.4 Intellectual Property As with other DL-related concerns, intellectual property issues have already been described at various places in this book. Intellectual property is difficult to protect in the digital environment because although the cost of production is not insubstantial, the cost of replication is near nothing. Furthermore, in circumstances such as academic publishing, the desire for protection is situational. For example, individual researchers may want the widest dissemination of their research papers, but each one may want to protect revenues realized from synthesis works or educational products that are developed. The global reach of the Internet has required intellectual property issues to be considered on a global scale. The World Intellectual Property Organization (WIPO, www.wipo.org) is an agency of the United Nations attempting to develop worldwide policies, although understandably, there is considerable diversity about what such policies should be. Probably the ideal method for intellectual property protection is to encode rules for it in the object’s metadata. The schemes described earlier in the book (e.g., DCM) do this at best in a rudimentary way. The DOI system attempts to encode metadata that includes the means by which objects are accessed and thus protected. The metadata schema for the DOI system is the Interoperability of Data in e-Commerce System (INDECS, www.indecs.org). This system defines a small kernel of metadata as well as additional metadata that can be used. The kernel includes the following attributes: • • • •
Identifier: the DOI Title: the name of the entity Type: the type of resource of the object Mode: the “sensory” mode by which the object is perceived (e.g., text, audio, video) • Primary agent: the creator of the object • Agent role: the role the primary agent played in creating the object The DOI Application Profile (DOI-AP) defines the scope of INDECS metadata for a given application. It may include information about events or situations. Events may be classified as creating or using, that is, entailed in developing the object or in defining the terms of its use, including price. Situations define the relationship between creating events and intellectual property policies. The INDECS approach is not the only one that has been devised to handle management of intellectual property issues. The Safe Dealing approach devised by researchers at IBM Corp. (www.ibm.com) defines an approach for “rolebased” access to resources based on certification of the user, the organization to which the user belongs, and the provider (Gladney and Cantu, 2001). A commercial initiative broader in scope than DLs but relevant to their business aspects is the Universal Description, Discovery and Integration (UDDI, www.uddi,org)
10. Augmenting Systems for Users
395
project, which aims to create standards for business transactions on the Web as part of the growing Web Services effort (Coyle, 2002).
10.5.5 Preservation There are a number of issues related to the preservation of DL materials. One concerns the size of such materials. Although hard disk space is, in modern times, considered “cheap,” the computer size of objects becomes important in determinations of how to store massive collections as well as transmit them across networks. Table 10.5 lists the relative sizes of various DL elements. Lesk (1997) has compared the longevity of digital materials. He has noted that the longevity for magnetic materials is the least, with the expected lifetime of magnetic tape being 5 to 10 years. Optical storage has somewhat better longevity, with an expected lifetime of 30 to 100 years depending on the specific type. Ironically, paper has a life expectancy well beyond all these digital media. Rothenberg (1999) has referred to the Rosetta stone, which provided help in interpreting ancient Egyptian hieroglyphics and has survived over 20 centuries. He goes on to reemphasize Lesk’s description of the reduced lifetime of digital media in comparison to traditional media and to note another problem familiar to most long-time users of computers, namely, data can become obsolete not only owing to the medium, but also as a result of its format. Both authors point out that storage devices as well as computer applications, such as word processors, have seen their formats change significantly over the last couple of decades. Indeed, it may be harder in the future to decipher a document stored in the WordStar word processing format than an ancient stone or paper document.
TABLE 10.5. Relative Sizes of Collections and Components Size Kilobytes (103 bytes)
Megabytes (106 bytes)
Gigabytes (109 bytes)
Terabytes (1012 bytes) Petabytes (1015 bytes)
Exabytes (1018 bytes) Source: Lesk (1997) and IDC Corp. (www.idc.com).
Example (size) Printed page (1) Scanned page (30) One second of speech (10) Small book (500) Medical X-ray (2) The Bible (5) Large medical textbook (10) Two-hour radio program (50) Oxford English Dictionary (500) Digital movie (10) MEDLINE database (40) Books on the floor of a library (100) World Wide Web (circa 1998) (2) Library of Congress (20) A scanned national library (1) World disk production (1995) (15) World tape production (1995) (200) World disk production (2001) (540) World disk production (2004) (3)
Chapter 11 Information Extraction
The general goal of information retrieval (IR) systems is the retrieval of documents from textual databases, which will then be read and applied to the task for which they were retrieved, such as a search for more information on a disease by a clinician or a patient or an attempt by a researcher to identify earlier studies. Sometimes, however, there is a desire to do more with textual data. In particular, at times when the aim is to actually extract facts or knowledge from the text. In the first edition of this book, the chapter corresponding to this one (Chap. 11, The Clinical Narrative) focused exclusively on processing the text of the clinical narrative (e.g., the patient history and physical report or the discharge summary). While there was interest at that time in extracting information from textual data of other types, the vast increase in the amount of online text available, especially on the Web, along with new approaches to process it, have led to increased interest in deriving information from other forms of text. The name usually given to the process of identifying facts and knowledge from large collections of text is information extraction (IE). In their overview article, Cowie and Lehnert (1996) note that IE can make use of IR techniques to narrow down the amount of information for the IE process, but the IE is fundamentally different in that it aims to provide the user with facts and knowledge rather than with documents. It should be noted that the Question-Answering Track (see Section 8.3.3) of the Text REtrieval Conference (TREC) does start to move IR systems and research toward IE. Another name that has been given to the IE process is text mining (Liddy, 2000). In this view, IE is one aspect of data mining, which is sometimes called knowledge discovery from databases (KDD). The genomics community has begun to call text mining of the published literature “mining the bibliome” to demonstrate its similarity to data mining of the genome (Alfred, 2001). Whatever one chooses to call it, IE draws on techniques from many areas. It makes use not only of IR techniques, but also of natural language processing (NLP) and other aspects of artificial intelligence. Cowie and Lehnert (1996) note that IE systems typically combine a number of components that operate at different levels of text processing, as listed in Table 11.1. This chapter focuses on IE in the realm of health and biomedicine. The applications are grouped on the basis of the classification of information from Chap397
398
III. Research Directions
TABLE 11.1. Components of Information Extraction with Techniques Used Level Text Word Noun phrase Sentence Intersentence Template
Description
Technique used
Select relevance of text or passages Tag parts of speech of words Recognize important concepts Map phrases into structure showing relationships among them Recognize and unify expressions Format output to predefined form
Information filtering Part-of-speech tagging Parsing Semantic tagging Discourse reference Output generation
Source: Adapted from Cowie and Lehnert (1996).
ter 1 into patient-specific and knowledge-based categories. IE from patientspecific information occurs mainly from the clinical narrative, which was the focus of this chapter in the first edition. IE from knowledge-based information has occurred in a number of areas, but the most prominent area of focus in recent years has been the literature of genomics and molecular biology.
11.1 Patient-Specific Information It was noted in Chapter 2 that IR focuses mostly on knowledge-based information. Patient-specific information is quite different from knowledge-based information, because it is generated and used for different purposes. Patient-specific information is produced as a result of an encounter between a patient and the healthcare system. Its main purpose is to document that encounter not only for clinical reasons, but for financial and legal reasons as well. With the increasing computerization of medical records, as well as the incentives to control costs and ensure quality, there is increasing desire to tap the information in the clinical record for other purposes, such as real-time decision support, retrospective outcomes research, and quality assurance. There is good reason to attempt to extract such information, for the coded information in charts, usually generated for billing purposes, does not capture the richness or complexities of the actual patient and the course of his or her disease (Jollis, Ancukiewicz et al., 1993). Research has also shown a discordance between the data in coded and free-text portions of the medical record (Stein, Nadkarni et al., 2000). If clinical narrative data are to be used in this fashion, however, they must be encoded to allow application in decision support rules and/or database analysis. Thus if the information in clinical narratives is going to be used for these purposes, IE techniques must be employed to extract such encoding. This section covers the processing of this narrative information.
11.1.1 Challenges in Processing the Clinical Narrative For general IR tasks, the goal of processing a text is to select descriptors that represent the subject matter. Whether the traditional approaches of human in-
11. Information Extraction
399
dexing and word indexing that were discussed in Chapter 5 or the newer innovations such as term weighting and linguistic techniques that were introduced in Chapters 8 and 9, respectively, the goal of these indexing processes is to identify the topical content of the document so that it will be retrievable by someone searching for documents on that topic. As noted, however, the goal of processing the clinical narrative is usually different. While document retrieval is occasionally the goal of searching patient reports, the goal is more likely to be retrieval of specific factual information, such as whether a patient had a particular symptom, physical finding, or test result. The requirement for accuracy is much higher, however, because this information is used in the care of individual patients (e.g., alerting the clinician to the presence of some potentially dangerous combination of attributes) or groups of patients (e.g., assessing the outcomes of a population treated with a drug having potentially serious side effects). While the consequences of an inappropriate indexing term in an IR system are modest (e.g., leading to a false hit in retrieval), the consequences of erroneous fact extraction from a clinical narrative can be an inappropriate recommendation in the care of a patient or an incorrect assessment of the efficacy of a treatment in a population. Another problem is that while documents in journals and textbooks are typically edited, spell-checked, and otherwise polished for easy reading, clinical narratives are usually written or dictated quickly in a telegraphic, elliptical style with misspellings and grammatical incompleteness. As a result of these problems, Hripscak et al. (1995) have noted that such information is usually “locked” in the clinical narrative. Problems in processing the clinical narrative occur in all three phases of NLP described in Chapter 9: parsing, semantics, and contextual interpretation. The major challenge for parsing the clinical narrative arises from the incomplete sentences that predominate in clinical texts. For example, Marsh and Sager (1982) assessed a set of hospital discharge summaries and found that about half the sentences were syntactically incomplete. Table 11.2 lists the major categories of incomplete sentence types they found, in decreasing order of frequency. Another problem related to parsing is that words may not be in the lexicon. There are a variety of reasons for this, from spelling errors to names of people, devices, or other entities to novel words that clinicians create when they generate narra-
TABLE 11.2. Main Categories of Syntactic Incompleteness in Medical Records Category Deleted verb and object (or subject and verb), leaving a noun phrase Deleted tense and verb be Deleted subject, tense, and verb be Deleted subject Source: Marsh and Sager (1982).
Example Stiff neck and fever Brain scan negative Positive for heart disease and diabetes Was seen by local doctor
400
III. Research Directions
TABLE 11.3. Amount and Proportion of Words in Categories Averages Category Initials and embedded metacharacters In one of six medical vocabularies In names list or Unix spell checker Algorithmically recognizable Recognizable in context Otherwise unrecognized Correctly spelled real word Probably correctly spelled Incorrectly spelled Garbage word Unknown Total
Amount 1,344 42,721 32,100 12,592 7,311 28,925 12,912 9,101 6,171 70 671 124,993
(1.1%) (34.2%) (25.7%) (10.1%) (5.8%) (23.1%) (10.3%) (7.3%) (4.9%) (0.1%) (0.5%) (100%)
Documents
Frequency
157.7 827.5 75.6 15.0 9.1
158.1 1658.5 140.6 18.2 12.2
23.7 5.8 2.2 1.4 1.6 311.6
28.1 6.6 2.4 1.4 1.7 613.9
Source: Hersh, Campbell et al. (1997).
tives. Hersh et al. (1997) analyzed all the words in a corpus of 560 megabytes of clinical narratives from a university teaching hospital. The 238,898 documents contained a total of 124,993 unique words, where classified into categories as shown in Table 11.3. Only about 60% of the words occurred in one of six major medical vocabularies including the Unified Medical Language System (UMLS) Metathesaurus, a large list of names, or the Unix spell checker utility. A small percentage of characters were artifacts of the system (initials and embedded metacharacters), but the remaining nearly 40% were content words that a lexicon based on the aforementioned medical words, general words, and names would not contain. An analysis of these words discovered that they could represent the following categories: • Algorithmically recognized by a grammar [e.g., 3yr (age) or 3x5 cm (dimension) • Recognized in context (e.g., Dr. William Hersh or Lloyd Center) • Medical words not in any vocabulary (e.g., dipsticked or righthandedness) • Incorrectly spelled or otherwise unknown Semantically, there are problems with words that are used differently in medical language and general English usage. Macleod et al. (1987) found that some words are used different in medical narratives, such as the word appreciated, which acts as a synonym for detected or identified (e.g., PMI not appreciated). In addition, they noted that other words were used idiosyncratically, such as eye drops (drops is a noun, not a verb) and mass felt at 3 o’clock (the mass is felt at the position of 3 o’clock, not the time). Some words were also difficult to semantically interpret owing to the syntactic incompleteness. For example, May halt penicillamine could be interpreted that to indicate that penicillamine may be discontinued or that it will be definitely discontinued in May.
11. Information Extraction
401
Medical narrative language is also full of synonymy, leading D. Evans to assert that the “surface strings” of clinical findings cannot capture the underlying structure (Evans, 1988). He points to the phrases epigastric pain after eating and postprandial stomach discomfort, which mean the same thing yet have no words in common. A related problem is that clinical narratives typically use many abbreviations, some of which can be ambiguous. For example, the abbreviation PCP can stand for the drug phencyclidine, the disease Pneumocystis carinii pneumonia, or an individual, the primary care physician. Contextually there are many problems as well. Medical charts are typically full of elliptical sentences (e.g., Complains of chest pain. Increasing frequency, especially with exertion. Usually associated with shortness of breath and nausea/vomiting). The example series of sentences represents a single clinical entity, which is chest pain due to angina pectoris, but the components of the finding are spread across three sentences. Another contextual problem, also of concern in IR, is that parts of the document may not be relevant for IE. For example, a section of narrative may be engaging in discussion or speculation. While some types of narrative (e.g., history and physical reports or discharge summaries) are well structured, others (e.g., progress notes) are not. The repercussions of extracting discussion or speculation as “facts” could be detrimental to systems that used the extracted data operationally. Fortunately there are some aspects of clinical narratives that do make processing easier. The first is that they follow a fairly regular grammar, which linguists called a subgrammar. Thus, even though the wording is cryptic and the sense of words ambiguous, there is some regularity to the use of language in clinical narratives. Sager asserts that virtually all clinical narrative statements can be represented by one of six information formats, a fact exploited heavily in the Linguistic String Project, described in Section 11.1.2.1 (Sager, Friedman et al., 1987). Another aspect of the clinical narrative that can help with processing is the predictable discourse, especially in portions like the physical examination, where most physicians follow a consistent pattern in presenting the findings. Archbold and Evans (1989), for example, discerned a number of regularities. They first determined that most physical findings could form a propositional template of up to seven components: 1. Major topic: main topic of section (e.g., neurologic) 2. Minor topic: subtopic of section (e.g., motor) 3. Method: procedure used to obtain finding (e.g., slit lamp examination) 4. Site: location of finding (e.g., right upper quadrant) 5. Attribute: property of site being discussed (e.g., rate for heart, size for liver)
402
III. Research Directions
TABLE 11.4. The Maximum Probability Path Through the Physical Examination 1. Start of physical exam general blood pressure, eye, oral, pharynx, skin, vital 2. General HEENT sign, skin, vital 3. HEENT neck ear, eye, fundus, hearing, nicking, oral, pharynx, pupil, sclera, tongue, visual field 4. Neck chest blood vessel, bruit, jugular venous distention, lung, mandible, trachea 5. Chest heart breast, breath, electrocardiogram, X-ray 6. Heart abdomen back, lift, point of maximal impulse, S3, S4, systolic ejection murmur, thrill 7. Abdomen rectum bowel, edge, fetus, leg, liver, mass, muscle, organomegaly, pelvis, periumbilical, pulse, quadrant, spleen 8. Rectum extremity back, genitalia, prostate, stool, urinary 9. Extremity Neurologic affect, edema, foot, hand, mentation, mood, periphery, plantar, range, motion, status, urinary 10. Neurologic end affect, cranium, mood, muscle, nerve, proprioception, psychologic, Romberg, strength, tendon 11. End Source: Archbold and Evans (1989).
6. Value: value at the site of the attribute (e.g., 10 cm for site liver and attribute size) 7. Qualifier: comment that modifies value (e.g., appears to be) Archbold and Evans also noted a maximum probability pathway through physical exam topics, such that the next general area of the exam could be predicting with regularity. They did find, however, that subtopics within a topic could be in any order. Table 11.4 lists the ordered topics and subtopics within them. A final finding was that a majority of attributes occurred only within certain topics, such as thyromegaly in Neck and gallop in Heart.
11.1.2 Approaches to Extraction from the Clinical Narrative The approaches undertaken to extract content from the clinical narrative have varied in scope and domain. Simple, domain-specific approaches focus on a specific area and can usually handle many of the idiosyncrasies of that area but are difficult to generalize to other domains. Comprehensive approaches, on the other hand, scale better but are much harder to build and maintain. This section covers a number of approaches that have been implemented and are described in the literature. After describing some early approaches, this section focuses on two NLP systems that have achieved operational use in real electronic medical record (EMR) systems. For each system, the major features are described, along with evaluation results. Since the evaluations have been small in size and have utilized different data sets, there is unfortunately no way to compare the different systems. There are actually a number of challenges to evaluation research for the clinical narrative (Friedman and Hripcsak, 1998). One challenge is determining metrics for such
11. Information Extraction
403
evaluation. A common approach is to adapt recall and precision, where recall becomes the proportion of findings correctly determined from all correct findings and precision becomes the proportion of findings correctly determined from all findings suggested by the system. Recall and precision can be calculated at the level of individual findings across all documents in the collection or at the level of individual documents. However, as with IR, these are limited measures in that they do not give the entire picture of how an IE system might perform for a realworld task. Another challenge to evaluation systems for processing the clinical narrative is that patient record text is much more difficult to share across research sites text owing concerns about privacy and confidentiality of the personally identifiable health information it contains. 11.1.2.1 Early Approaches The best-known early effort in clinical text processing was Sager’s Linguistic String Project (LSP) (Sager, Friedman et al., 1987). This effort was based on the notion of sublanguage analysis, in that technical documents in a single field (such as clinical medical narratives) were found to utilize only a subset of English grammar and vocabulary. If these sublanguages could be recognized and incorporated into algorithms, accurate extraction could occur without having to process general English. Sager et al. (1987) noted that most statements in the medical record could be reduced to six information formats: 1. 2. 3. 4. 5. 6.
General medical management Treatment other than medication Medication Test and result Patient state Patient behavior
These formats contained enough semantic information to allow the extracted information to be loaded into a relational database. Appropriate information formats could be qualified by various modifiers, such as time, levels of uncertainty, and severity. The key to each information format’s operation was the lexicon, which contained words with their English and sublanguage classifications. The classes not only represented syntactic information about the words but also placed semantic restrictions on them that enable the information formats to be interpreted semantically. The LSP lexicon contained 40 healthcare sublanguage classes and 14 English semantic subclasses. An example of the Medication information format is shown in Figure 11.1. The top-level tree contained slots for the classes INST (institution), PT (patient), MEDINFO (medical information), and VTR (treatment verb). Each slot allowed all terms from each of its classes. The VTR class, for example, allowed verbs about treatment (e.g., treated, injected). These verbs were distinct from the VMD class of medical management verbs (e.g., examined, admitted). The medical information slot was actually a subtree that
404
III. Research Directions
FIGURE 11.1. The Medication information format for the statement, Patient was treated by Ampicillin 500 mg tid orally. (Sager, Friedman et al., 1987. Courtesy of Addison-Wesley.)
had four slots for the classes H-RX (medication name), QN (dose), H-RXFREQ (medication frequency), and H-RXMANNER (route of administration). The narrative processing utilized four steps for each sentence (Friedman, 1987): parsing, sublanguage selection, regularization, and information formatting. In parsing, the words were reduced to their syntactic categories according to an English grammar modified for the syntax of medical documents (Sager, 1981). The output was a parse tree (or more than one parse tree if syntactic ambiguities could not be resolved) that contained all words in the sentence categorized syntactically. The parser could handle a variety of complex syntactic structures, such as conjunctions (e.g., swollen feet and ankles) and ambiguously placed modifiers (e.g., swelling in knees and swelling over last few days in knees). In the next step, sublanguage selection, words were slotted into their sublanguage categories to further clarify meaning. The first step in sublanguage selection was detection of semantic patterns based on syntactic categories (e.g., allowing the distinguishing of pain over several days vs pain over the upper abdomen). This was followed by word sense disambiguation based on allowable sublanguage sequences and similar proper placement of modifiers. After sublanguage selection was regularization, in which words were normalized into informationally equivalent forms. For example, pain in left leg, left leg pain, and painful sensation in left leg were all mapped into a common form. This operation also expanded conjunctions (e.g., knee is red and swollen to knee is red and knee is swollen) and gave connectives a uniform operator–argument structure. The final step was information formatting. The appropriate information format (from the foregoing list) was selected, and the regularized parse tree was put into that template. From this point, the information could be further processed or loaded into a database. For the latter, an entity–relationship data model was developed that mapped the findings and their attributes into one of four types of medical “facts”: clinical (e.g., history, exam, diagnosis), laboratory, treatment, and response (Sager, Lyman et al., 1994a). This model allowed relations between facts of different types, such as treatment–response and diagnosis–treatment. The LSP technique was ported to a number of different domains, such as
11. Information Extraction
405
asthma discharge summaries (Sager, Lyman et al., 1994a), radiology reports (Grishman and Hirschman, 1978), and lipid metabolism articles (Mattick, 1987). It was also translated to two different languages, French (Sager, Lyman et al., 1989) and German (Oliver, 1992). Work was also undertaken to map information formats to various coded terminologies, such as the Systematic Nomenclature of Medicine (SNOMED) vocabulary (Sager, Lyman et al., 1994b). One evaluation of the LSP system was performed for asthma discharge summaries (Sager, Lyman et al., 1994a). A list of 13 important details of asthma management was developed, with the measures of recall and precision adapted based on a gold standard of human review of the documents. The 59 discharge summaries were divided into a training set and a test set, with the former used to update the dictionary, modify the grammar, and develop the database queries. The recall for the testing set was 82.1%, while the precision was 82.5%. When minor errors (e.g., a misplaced word or part of a finding not retrieved) were eliminated, the recall and precision rose to 98.6 and 92.5%, respectively. In all, the LSP project was certainly the most comprehensive early attempt at medical narrative interpretation. Its main limitation was the lack of a knowledge base to normalize terms. Thus the system had no way to recognize two conceptually similar but differently worded text passages. Another problem was that, like all NLP systems, the complex dictionary, grammar, and other structures were difficult to maintain and required updating by language experts for new domains. Another early approach was Canonical Phrase Identification System (CAPIS) (Lin, Lenert et al., 1991). Instead of attempting to extract all findings in a clinical narrative, CAPIS just aimed to identify findings specified by the user. In particular, CAPIS worked best with more highly structured portions of the narrative, such as the physical examination. This part always has the patient as the subject and does not require interpretation of any temporal information. Extraction of findings began with a lexical analysis routine that broke out clauses between punctuation marks and function words. This was followed by a parsing process that used a finite-state machine. A useful feature of the parser was the ability to recognize negation information. When a negation word was found, all findings that followed were assumed to be negative until an exception word (e.g., but) or end of sentence. A thesaurus was used to make known synonym substitutions among words and phrases. The next step was the matching routine, which matched the phrases against the findings list, which contained canonical representations of the findings. In addition to exact matches, the system allowed best partial matches when an exact match could not be made, with matches to longer terms in the findings list given preference. The initial evaluation of CAPIS was performed in the domain of patients with gastrointestinal bleeding, showing a recall of 92% and precision of 96% for 156 findings in 20 reports. A failure analysis indicated both simple fixable errors, such as grammatical incorrectness in the documents, to complex errors resulting from the inability to process more complex findings spread across several clauses. CAPIS was extended to other domains and combined with other tools. In one implementation it was reconfigured to process the impression portion of chest
406
III. Research Directions
X-ray (CXR) reports to determine the presence of lesions (e.g., potential malignancies) that required follow-up (Zingmond and Lenert, 1993). In combination with a learning algorithm based on training data, the system was found to be as sensitive as radiologists in designating the need for follow-up. In another implementation, the system was revised to detect a set of findings in admission summaries that would determine the appropriateness of the recommendation for coronary artery bypass graft (CABG) surgery (Lenert and Tovar, 1993). Another early system of note was MedSORT-II, which used a more pure computational linguistic approach (D. Evans, 1988). MedSORT-II began with a basic lexicon of words, atomic phrases (e.g., Wilson’s Disease), and abbreviations (e.g., AIDS). The lexicon also designated bound morphemes, which were parts of words that must be bound to other morphemes to be a lexical unit, such as -oscopy. Each lexical item also had a semantic type. These basic lexical items formed more complex concepts based on rules of composition, which were statements in the form of generic declarative semantic structures that specified all and only the legal combinations of basic concepts. These more complex concepts ultimately formed a set of higher level concepts and their associated generic forms, containing representations making explicit all the assumptions and all the associated context of medical data. A clinical finding was defined as the class of observations that minimally had to contain a method, a locus, and a result, in particular so that the result derived from the application of the method to the locus, as seen in Table 11.5. These elements are tied together by a semantic network. The fundamental units (atoms) are lexemes. Each lexeme has, at a minimum, a syntactic category and semantic type. One or more lexemes can be combined to form simple concepts, each of which also has a semantic type. More complex concepts can be built from these simpler concepts, and these also have a semantic type. This architecture can be extended all the way up to the most complex clinical findings. 11.1.2.2 MedLEE Despite the LSP’s heavy reliance on syntactic methods, part of its ability to handle domain-specific language processing came from its knowledge about the semantics of terms in medical usage. It might have been possible to set aside the use of complex syntactic information and just focus on semantic inTABLE 11.5. A Finding from MedSORT-II no pin-prick sensation in calf ==> | | | [pin-prick] | | [calf] | | | [sensation] | [absent] Source: Evans, D. (1988). Pragmatically-structured, lexical semantic knowledge bases for unified medical language systems. Proceedings of the 12th Annual Symposium on Computer Applications in Medical Care, Washington, DC. IEEE. 169–173. © 1988 IEEE. Reprinted with permission.
11. Information Extraction
407
formation and relationships. This sort of approach was developed by computational linguists (Hendrix, Sacerdoti et al., 1978; DeJong, 1979; Lebowitz, 1983) and formed the basis of the Medical Language Extraction and Encoding (MedLEE) system to perform clinical finding extraction without extensive parsing (Friedman, Alderson et al., 1994). In MedLEE, the rewrite rules for the syntax are replaced with semantic categories. The first domain of use for MedLEE was radiology reports. A radiology finding consists of the central finding, a body location, and finding modifiers. After findings are identified in reports, they are stylistically regularized and mapped to controlled terminology using a thesaurus. MedLEE consists of four components that process text to obtain structured output. The first component is the preprocessor, which processes the input text to determine word and sentence boundaries, resolve abbreviations, and perform lexical lookup of syntactic and semantic categories. For example, if the word abdominal is encountered, it is determined to be a body location category and mapped to the target form abdomen. The semantic categories containing various findings, locations, and modifiers are listed in Table 11.6. In the next component in MedLEE’s processing, parsing, findings are recognized based on the semantic grammar. The semantic grammar can also recognize negation. A simplified version of the grammar is presented in Table 11.7. The terminal nodes represent literal strings or semantic categories. The following component is phrase regularization, in which noncontiguous phrases are converted to a regularized form. For example, adenopathy in the left hilum is mapped to a form that can equivalently map left hilar adenopathy. In the final component, encoding terms are mapped into some sort of controlled structure. This process is assisted by a knowledge base of synonyms for both single- and multiword terms. MedLEE has synonyms not only for clinical terms, but also for the modifier terms, aiming to regularize levels of certainly and degree as well. The final output of processing a finding is shown in Table 11.8. The first evaluation of MedLee assessed its ability to recognize findings from four diseases in 230 randomly selected CXR reports (Friedman, Alderson et al., 1994). Physicians who read the actual reports and denoted the findings provided the gold standard. Recall and precision were found to be 70 and 87%, respectively. When additional terminology was added to the queries to enhance their retrieval capability, recall increased to 85%, while precision remained unchanged. A further analysis of the data found that orienting the grammar to identifying the largest well-formed segment of the sentence led to better performance than attempting to process the entire sentence (Friedman, Hripcsak et al., 1998). Hripscak et al. (1995) assessed MedLEE based on its ability to automatically detect clinical decisions that might be used in decision support applications. Six radiologists and six internists were given radiology reports and asked to state whether six clinical conditions were present or absent. The level of disagreement between the radiologists, the internists, and the radiology text processor was assessed via a “distance” measure, defined as the number of conditions per report
408
III. Research Directions
TABLE 11.6. MedLEE’s Semantic Categories for Findings, Locations, and Modifiers Semantic category Bodyloc Certainty
Cfinding
Change
Connector Degree
Descriptor Device
Disease
Position Pfinding
Procedure Quantity Recommend Region Status
Technique
Description Terms denoting a well-defined area of the body or a body part. Terms affecting the certainty of a finding. This class modifies status and changes terms in addition to findings. Terms denoting a complete radiology finding because they implicitly or explicitly contain a finding and a body location. Terms denoting a change in findings where the change is an improvement or worsening of a finding but not the start or end. Terms that connect one finding to another Terms denoting the severity of a finding. These terms can also modify change certainty, and other degree words. Terms qualifying a property of a body location or finding. Terms denoting surgical devices that are evident on the radiology report. Terms denoting a disease. These terms are based on the disease axis in SNOMED 3. Terms denoting orientation. Terms denoting a partial finding. These terms must occur along with a body location to be a complete finding. Terms denoting a therapeutic or diagnostic procedure. Terms representing nonnumeric quantitative information. Terms denoting recommendations. Terms denoting relative locations within a body location. Terms denoting temporal information other than an improvement or worsening of a finding. Terms denoting information related to the manner in which the radiographic examination was obtained.
Source: Friedman, Alderson et al. (1994).
Examples Hilum, left lower lobe, carotid artery Possible, appears, no evidence of
Cardiomegaly, widening of the mediastinum, pleural effusion Worsening, improving, increase
May represent, indicative of, suggests Mild, severe, moderate
Linear, large, enlarged Sternotomy wire, SwanGanz catheter, surgical wires Asthma, cardiomyopathy, sickle-cell anemia Transverse, anteroposterior, lateral Opacity, lesion, markings
Bronchoscopy, mastectomy, radiation therapy Many, few, multiple Clinical correlation, follow up, repeat X-ray Upper, lower, mid Chronic, active, resolved
Expiratory film, poor inspiration
11. Information Extraction
409
TABLE 11.7. MedLEE’s Semantic Grammara
("." | ";") { | {} {} disease | cfinding | pfinding | descriptor [] [] [] [] | [negation] certainty | negation conjunction | [auxverb] [negation] certainty degree [negation] change {} bodyloc {} | conjunction in | on | at | along | near | under region {} conjunction
aTerms
in angle brackets represent nonterminal symbols. Terms in square brackets are optional. Terms in quotes are literals. Terms in curly braces are optional. Plain terms represent either terminal words or semantic categories. Source: Friedman, Alderson.
in which there was disagreement on the conditions. The average distance across all physicians was 0.24, and the confidence interval for this measurement contained MedLEE’s average distance from each physician, which was 0.26. Thus the system’s performance fell within the normal level of variation among physicians. This study also found that the sensitivity for detecting the six conditions was 81% for the physicians and 85% for MedLEE. The specificity was 98% for both. This study also looked at the distance of lay persons as well as a system using simple word matching (i.e., no NLP), finding much greater distance for each.
TABLE 11.8. Processing of the Sentence, Heart shows extensive enlargement by the Semantic Grammar of MedLEE: the Central Finding is Determined from the Mapping of heart and enlargement; the Degree Modifier Maps from extensive; the Certainty Modifier Maps from the Term show [Finding Str] (Central) [cardiomegaly] (Bodyloc Mod) [heart] (Degree Mod) [high degree] (Certainty Mod) [moderate certainty]. Source: Friedman, Alderson et al. (1994).
410
III. Research Directions
Since one concern about the performance of EMR and NLP systems is that they do not generalize to new institutions or locations, this study was repeated using different physicians and reports in a different city (Hripcsak, Kuperman et al., 1998). Somewhat similar to the original study, internists and radiologists judged the presence or absence of seven clinical conditions in 200 CXR reports. In this analysis, the unmodified version of MedLEE was found to perform outside the confidence interval for all physicians (0.34 vs 0.23–0.34). Modifying MedLEE’s lexicon and grammar with other CXR reports as training data did not decrease the distance, although training it on the physicians’ designation of the conditions did improve it to within the confidence interval (0.32). Another concern about NLP systems is their use in real-word tasks, as opposed to the artificial ones described so far. As such, MedLEE has been employed to provide decision support to clinicians in the identification of patients with tuberculosis (TB) who require follow-up treatment and respiratory isolation. Jain et al. (1996) analyzed reports of 171 culture-positive TB patients to identify seven clinical findings, the presence of any one of which indicated the diagnosis. Rules were set up in MedLEE to detect the presence or absence of these findings. MedLEE was found to agree with manual coding of positive findings 89.0% of time and negative findings 89.5% of time. It agreed with the diagnosis 88.9% of time. Based on these results, Hripcsak et al. (Hripcsak, Knirsch et al., 1997) developed and evaluated different rules for the automated detection of TB. In another example of how artificial laboratory studies do not necessarily apply in the real world, MedLEE was found to have low sensitivity (41%) and very low positive predictive value (3%) for detecting TB. One follow-up analysis, however, did show that MedLEE would increase the proportion of patient identified as candidates for a respiratory isolation protocol (Knirsch, Jain et al., 1998). Effort has been made to move MedLEE outside the CXR report domain. The first new area was mammography reports (Jain and Friedman, 1997). While the original system for CXR reports required 30 semantic categories 4500 words and phrases in the lexicon, and 450 rules in the grammar, extension to mammography reports required about 250 entries in the lexicon and a few new rules for the grammar. An analysis assessed MedLEE’s ability to detect findings suspicious for breast cancer. The system was compared with a departmental logbook containing suspicious findings for 151 patients and 173 findings. These findings were related to those important in the detection of breast cancer, such as masses, calcifications, and architectural distortion. Depending on the section of report processed, MedLEE found the same finding or more information 44 to 45% of the time, less information 13 to 23% of the time, and different information 30 to 42% of the time. A major problem for MedLEE was the lack of explicit location information inferred in the mammography report. MedLEE has also been used to code neuroradiology reports of brain images for a registry of patients suffering from strokes (Elkins, Friedman et al., 2000). It was adapted to 47 clinical findings that included the existence of lesions and their properties (e.g., hemorrhagic), general location (e.g., brainstem), spe-
11. Information Extraction
411
cific location (e.g., cerebellum), and whether they were new or old. MedLEE was compared against the work of manual coders, with the values in the database assumed to represent the gold standard. The rate of accuracy of the manual coders was found to be slightly better than that of the automated MedLEE system (86 vs 84%), but the difference was statistically significant. There has also been effort to move MedLEE outside the realm of radiology reports. Friedman (1997) assessed the challenges in adapting the system to hospital discharge summaries. Unlike the incremental changes required for mammography and neuroradiology reports, extension to discharge summaries required 23 new semantic categories, nearly 300 new grammar rules, and about 6000 new entries to the lexicon. New information representations were required, such as drug dose abbreviations (e.g., 10 mg PO tid) and temporal phrases (e.g., two days after admission). Although evaluation has been incomplete, the general impression from early work in this area has been that unrestricted IE is more difficult than focused detection of explicit findings. MedLEE’s capabilities with discharge summaries were assessed in the area of automation of a severity score guideline of community-acquired pneumonia (Friedman, Knirsch et al., 1999). The goal of this task was to automatically calculate the score of a guideline (classes I–V) for variables from the clinical information system (e.g., age, PO2), CXR report (e.g., pleural effusion), and discharge summary (e.g., abnormal pulse). Compared with a human expert, MedLEE correctly assigned the exact risk class 80% of time and was within one class (i.e., off by 1) 100% of the time. Analysis of individual findings determined that CXR report findings had slightly greater specificity, whereas discharge summary findings had greater sensitivity. Another promising application of MedLEE is in the parsing of progress note text. Barrows et al. (2000) adapted MedLEE to the terse, highly abbreviated text that ophthalmologists enter after an outpatient visit. They found that MedLEE had lower recall but higher precision than a specialized parser that had been developed for glaucoma. For a six specific findings, MedLEE achieved a recall of 80 to 100% and a precision of 100%. A key accomplishment of MedLEE has been its integration into the operational EMR system at Columbia-Presbyterian Medical Center (Friedman, Hripcsak et al., 1995). One challenge was achieving interoperability with all the other components of the EMR system, such as the clinical database for storing all data on the patient (S. Johnson, 1996), the terminology knowledge base for managing encoded concepts (Cimino, Clayton et al., 1994), and the event monitor for detecting events and triggering messages for actions based on those events (Hripcsak, Clayton et al., 1996). This required translating output to the Health Level 7 (HL-7, www.h17.org) format for data transfer into other components of the EMR. It also required reducing the amount of information coming from MedLEE to facilitate subsequent retrieval. Other requirements for adapting to the real-world environment included more robust error handling, since MedLEE output was going to be stored alongside other real data for real patients, and more efficiency in the parsing process, since the system would have to operate on large quanti-
412
III. Research Directions
ties of data. Recent work has focused on maintaining linkage from the structured portion of the processing to its location in the original report, which facilitates editing and querying, as well as expression in eXtensible mark-up Language (XML), which is increasingly used for data tagging and interchange (Friedman, Hripcsak et al., 1999). 11.1.2.3 SymText Another NLP system for processing the clinical narrative used in an operational EMR is SymText (Haug, Koehler et al., 1994, 1995), which enters data into the Health Evaluation through Logical Processing (HELP) system at LDS Hospital (Kuperman, Garnder et al., 1991). Like MedLEE, SymText uses a combination of syntactic and semantic analysis. The syntactic analysis uses an augmented transition network grammar to identify the elements within sentences followed by transformational grammar to group them for processing by a semantic grammar. Its lexicon is augmented with common multiword phrases (e.g., consistent with). In addition, a small list of synonyms is used to normalize terms (e.g., cardiomegaly is equivalent to enlargement of heart). The semantic grammar uses a Bayesian network that maps words and phrases into their appropriate semantic categories with the assignment into the nodes restricted by semantic category and probabilities derived from training data. The goal of this process is to disallow inappropriate assignments such as hazy infiltrate of right heart. Figure 11.2 depicts the Bayesian network, and Table 11.9 lists sample output within the network. SymText has been implemented in a variety of task domains and evaluated. Within the HELP system is an “antibiotic assistant” that helps physicians choose appropriate empirical therapy for infections. This system has been shown to improve patient outcomes as measured by reduced cost of treatment and length
FIGURE 11.2. The Bayesian network for the semantic grammar of SymText. (Haug, Koehler et al., 1994. Courtesy of Hanley & Belfus.)
11. Information Extraction
413
TABLE 11.9. SymText Processing of the Sentence, A hazy opacity is seen in the right upper lobe Finding Event: Localized infiltrate State: Present Presence marker: demonstrates Finding unit: Poorly marginated opacity (infiltrate) Finding: opacity Finding modifier: hazy Severity: null Anatomic unit: Right upper lobe Link unit: involving Anatomic location link: in Anatomic location: lobe Sidedness modifier: right Superior/inferior modifier: upper Change unit: null Source: Haug, Koehler et al. (1994).
of stay in the hospital (Evans, Pestotnik et al., 1998). A goal for SymText is to be able to supply input for the algorithm covering the treatment of acute bacterial pneumonia. To do this, SymText must be able to identify findings in CXR reports to assist in making the diagnosis. Fiszman et al. (Fiszman, Chapman et al., 2000) carried out an evaluation of SymText for this task organized similar to the study of Hripcsak et al. (Hripcsak, Friedman et al., 1995). Physicians, lay persons, SymText, and two keyword algorithms were assessed for their ability to detect three pneumonia-related concepts (pneumonia, infiltrate compatible with acute bacterial pneumonia, and aspiration) as well as an inference that acute bacterial pneumonia was present. Similar to Hripcsak et al., the performance of SymText was within the confidence interval of the physicians. Table 11.10 shows the range of values for each group on the inference that acute bacterial pneumonia was present. A follow-up study compared different classification algorithms for inferring pneumonia and found them all to perform comparably and within the range of performance of the physicians (Chapman, Fiszman et al., 2001). TABLE 11.10. Recall, Precision, and Specificity in Inference of Diagnosis of acute bacterial pneumonia in Evaluation of SymText Group or application Physicians (n 4) SymText Keyword programs (n 2) Lay persons (n 3)
Recall (%)
Precision (%)
Specificity (%)
91–97 95 79–87 38–58
83–92 78 63–70 85–97
88–95 85 71–77 93–99
Source: Fiszman, Chapman et al. (2000).
414
III. Research Directions
SymText also has been used to process ventilation/perfusion (V/Q) nuclear medicine scans of the lung for diagnosing pulmonary embolism. The system was augmented to detect the four classification categories from the Prospective Investigation of Pulmonary Embolism Detection (PIOPED) classification (Worsley, Alavi et al., 1993). SymText achieved a recall of 92% and precision of 88% for making a diagnosis, compared with a gold standard judgment by a radiologist. Reports that lacked an impression section (they reported only findings) had a recall comparable to that of reports that contained impression sections, but much lower precision. SymText also has been applied in the encoding of hospital admission diagnoses (Gundersen, Haug et al., 1996). Such encoding can be valuable not only for retrospective data analysis but also for prospectively alerting healthcare personnel to complex situations that require increased resources. SymText was augmented to map the free-text diagnosis to a coded form. An evaluation found the system was able to encode about 76% of the diagnoses accurately; most of the incorrect encodings were due to the free-text diagnosis not having been encountered in the training data. 11.1.2.4 Other Active Systems for Processing the Clinical Narrative The CLARIT system described in Chapter 9 has also been applied to processing the clinical narrative. D. Evans et al. (Evans, Brownlow et al., 1996) configured CLARIT to recognize drug names and dosages as reported in texts. They were able to achieve an exact matching rate of 61.7% and to partially match another 17.6% of such phrases. A number of European groups have developed systems to process the clinical narrative as well. Not surprisingly, some of their effort has investigated processing in languages other than English. Baud et al. (1998) have developed a morphosemantic parser that processes text down to the level of morphemes, including bound morphemes, to break text into the finest level of semantic granularity possible. One advantage to this approach is that it allows translation across languages, although it is harder than word-level parsing because of the idiosyncrasies of language (Baud, Rassinoux et al., 1999). Schulz and Hahn (2000) have also taken an approach focused on the morpheme level, with a more concentrated focus on the German language. An intriguing application of the morphosemantic parser has been its application to the parsing of physician orders (Lovis, 2001). This approach was analyzed in a laboratory evaluation and was found to achieve a 7% reduction in time required to enter orders compared to a conventional system. Users rated the parser as easier to learn and use than the conventional system.
11.1.3 Classification of the Clinical Narrative The applications described in Section 11.1.2 focused on extraction of clinical findings, some of which were used in diagnosis (or classification) tasks. The systems described in this section focus on going directly to classification (although
11. Information Extraction
415
the distinction is blurry in some instances). The section covers systems that attempt to classify clinical narratives to formal classification schemes such as SNOMED or the UMLS Metathesaurus, as well as those that classify such documents based on ad hoc schemes, such as fields in a database or observations about the severity of illness of a patient. 11.1.3.1 Controlled Clinical Vocabularies and Their Limitations As noted at the outset of this chapter, there is potentially great value in the ability to extract coded data from the clinical narrative for decision support, outcomes research, and other tasks. One of the limitations of the applications described in Section 11.1.2 is that they could detect only small numbers of clinical findings or diagnoses. While MedLEE has many clinical terms in its database, it has not been assessed to encode for more than 47 findings in a given clinical application. yet as was seen in Chapter 5, the large controlled vocabularies have tends of thousands of diagnoses, clinical findings, treatments, and other clinical information whose extraction might be desired. It was also noted in Chapter 5 that there are many controlled vocabularies in health and biomedicine. Each of these has evolved for a specific purpose and tends to be organized to facilitate that purpose (Chute, Cohn et al., 1996). Initial healthcare vocabulary work was centered around describing objects, such as diagnoses and treatments. In the nineteenth century, the International Classification of Diseases (ICD) was created by epidemiologists to classify causes of death. The current version used in this country is the ICD-9-CM, which is used extensively for diagnostic codes in billing. As a vocabulary for clinicians to describe diseases, however, ICD-9-CM is inadequate (Cimino, Hripcsak et al., 1989). The Current Procedural Terminology (CPT-4) codes describe procedures, and are also used for billing. Many specialties have evolved their own term lists to describe various phenomena that occur in their respective limited domains. For example, the American College of Radiology has created its ACR codes to describe findings that occur on radiology exams. Other vocabularies have also been created for information access. The Medical Subject Headings (MeSH) vocabulary was created by the National Library of Medicine (NLM) for indexing medical literature (Coletti and Bleich, 2001). Many of the early large-scale medical record systems adopted their own vocabularies, such as HELP (PTXT) (Pryor, Gardner et al., 1984) and CoSTAR (Barnett, Justice et al., 1979). Each of the large-scale expert system projects also evolved its own vocabulary, such as DxPlain (Barnett, Cimino et al., 1987), Quick Medical Reference (R. Miller, McNeil et al., 1986), and Iliad (Warner, 1989). One problem noted as the computer era unfolded was that the proliferation of disparate vocabularies made data exchange and integration of applications difficult. For example, an automated system to search the literature based on diagnosis (ICD-9) or procedure (CPT-4) codes had no way of converting those codes to the ones that would be used to search the literature via the MeSH vocabulary (Cimino, Johnson et al., 1992). Likewise, most of the early expert systems had major barriers to their use that were not related to the effectiveness of their advice; for example, they required the user to tediously reenter data already in the
416
III. Research Directions
medical record (Shortliffe, 1987). The wide variety of disparate vocabularies was one of the factors motivating the NLM’s UMLS project, which was an attempt to provide linkage across these vocabularies (Humphreys, Lindberg et al., 1998). Another problem noted with most early vocabularies was their inability to represent descriptive data, such as patient findings. While lists of terms are usually adequate to represent diagnoses and procedures, term lists fare worse in describing clinical findings. For example, medical students learning history taking quickly memorize the mnemonic PQRST, which represents the attributes of symptoms: provocative-palliative factors, quality, radiation, severity, temporal factors. These attributes are not insignificant for research, as it is known, for example, that the attributes of chest pain (e.g., radiation to the back vs radiation down the arm) have significant value in diagnosing myocardial infarction. The first effort to describe multifaceted patient findings was the Systematic Nomenclature of Pathology (SNOP), later SNOMED (Rothwell, Cote et al., 1993). These vocabularies for the first time defined axes for representing clinical findings, such as topography, morphology, and etiology. Since there are no rules on how different axes are to be used to create concepts, however, many possible combinations of terms are needed to state a given concept (Campbell and Musen, 1992). There was also interest in multiaxial vocabularies in the early UMLS work. The MedSORT-II project developed a multiaxial representation of findings for the QMR system (D. Evans, 1987). Masarie devised a frame-based schema for decomposing the complex chest pain findings in QMR terms (Masarie, Miller et al., 1991). Barr and Greenes also developed a small vocabulary based on semantic relationships between vocabulary terms (Barr, Komorowski et al., 1988). At least one of the reasons for the failure of these methods to gain widespread acceptance was the amount of time needed to identify terms and their proper relationships. The UMLS project, as noted in Chapter 10, ultimately decided to adopt a “metathesaurus” approach, linking source terms from different vocabularies (Lindberg, Humphreys et al., 1993). While this decision allowed algorithmic approaches to quickly generate such linkages, it imposed some limitations that made representation of complex clinical findings more difficult. In particular, the UMLS Metathesaurus does not allow modifiers to be attached to terms. Because mapping of other than one-to-one relationships is prohibited, complex concepts like clinical findings cannot be represented. This makes the PQRST of clinical findings impossible to represent. 11.1.3.2 Requirements for Clinical Vocabularies Cimino (1998b) has defined the “desiderata” for clinical vocabularies: • Content: coverage of all possible terms that lie within a vocabulary’s domain. • Concept orientation: each term must have one meaning (nonvagueness) and only one (nonambiguity). Furthermore, meanings must correspond to only one term (nonredundancy).
11. Information Extraction
417
• Concept permanence: each concept must have a permanent meaning, even if its preferred name evolves or the concept becomes obsolete. • Nonsemantic concept identifier: each concept must be represented by one unique identifier that should not of itself have any meaning (i.e., representing levels in a hierarchical tree with single digits in a string will limit that level to no more than 10 concepts). • Polyhierarchy: concepts must be allowed to be classified in multiple hierarchies (e.g., allowing Bacterial Pneumonia to be both a lung disease and an infectious disease). • Formal definitions: explicitly defined relationships to other concepts (e.g., if Bacterial pneumonia has a relationship with Streptococcus pneumoniae, the relationship should be define as “caused by”). • Reject “not elsewhere classified”: no “catch-all” terms (e.g., using Hepatitis NOS to represent forms of hepatitis), since new variants (e.g., Hepatitis C) have been discovered in recent years, and more are likely to emerge in the future. • Multiple granularities: allow concepts to be appropriate to the level of the application, but require the structure for finer granularity. For example, Diabetes mellitus is appropriate for defining all who have altered glucose metabolism but vocabularies should allow the use of the finer grained Type II Diabetes mellitus when appropriate. • Multiple consistent views: allow finer grained concepts to be considered synonyms when the coarser grained level is collapsed. • Represent context: define a context or grammar for allowable expressions with the vocabulary. • Evolve gracefully: tell users what changes are occurring and why. • Recognize redundancy: monitor for redundancy as the vocabulary evolves. D. Evans et al. (1991) defined three additional features essential for concepts in clinical vocabularies. First, they must be lexically decomposable, so that different attributes can be assigned. Second, they must be semantically typed, allowing for restriction of allowable modifiers and grounding of synonyms. Third, they must be compositionally extensible, so that certified terms can be allowed to generate new concepts. Using the attributes of chest pain from the earlier example, a template with slots for allowable modifiers (e.g., PQRST) would meet these requirements. 11.1.3.3 Multiaxial Vocabulary Efforts The vocabulary that comes closest to meeting the foregoing requirements, which also has the broadest coverage of clinical content, is SNOMED. This vocabulary is one of the few multiaxial vocabularies. After a decade-long virtual hiatus in the early 1980s through the early 1990s, work has accelerated substantially in the last decade. Work restarted in the early 1990s with the launching of SNOMED International (Rothwell, Cote et al., 1993). Each record or concept in the vocabulary had a canonical or preferred form, along with synonyms, child (or sub-
418
III. Research Directions
TABLE 11.11. The 11 Modules of SNOMED International (abbreviations used in template in text) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.
Topography: anatomical terms (T) Morphology: structural changes in disease (M) Function: physiological (normal) and pathophysiological (abnormal) functions of the body (F) Living organisms: classification of all organisms (L) Chemicals, drugs, and biological products: all chemicals and plant products used in health care (C) Physical agents, activities, and forces: devices and activities associated with disease and trauma (A) Occupations: international list of occupations Social context: social conditions and relationships of importance in medicine Diseases/diagnoses: names of diseases and diagnostic entities (D) Procedures: administrative, therapeutic, and diagnostic procedures used in health care (P) General linkage/modifiers: linkages, descriptors, and qualifiers used to link or modify terms (G)
Source: Cote, Rothwell et al. (1993)
class) terms, and adjectival forms. Each record belonged to one of 11 modules, formerly called axes, that represent different semantic types, as listed in Table 11.11. SNOMED is designed to capture clinical findings. Thus each finding in a patient is represented by a template that allows terms from the different modules to be filled. The basic template is that a procedure is performed at a site with a result-finding that may have one or more modifiers. The result-finding terms can come from the eight remaining modules. Clinical findings are modeled such that terms from a given module fit into the following template: {(P) performed at (T) makes known (MLCFDA) } G An example finding described by SNOMED is shown in Figure 11.3. There are two major limitations of SNOMED, both stemming from the lack of rules of composition. The first problem is that creation of meaningless terms is allowed. There are no rules that prevent terms being combined inappropriately, such as fractured blood caused by Staphlyococcus aureus. The second problem is the ability to represent concepts in multiple different ways. D. Evans et al. (1994) have noted five different ways for SNOMED to represent acute appendicitis as listed in Table 11.12. These problems have been somewhat rectified in the new SNOMED Reference Terminology (RT) by the incorporation of a description logic that enables equivalency and redundancy of terms to be represented (Spackman, Campbell et al., 1997). This process goes beyond multiaxial representation to incorporate attribute–value pairs for terms (e.g., head is an anatomic site) and a foundational model that defines composition of terms and allows different representations (e.g., those in Table 11.12) to map to the same underlying concept (Spackman and Campbell, 1998). Another vocabulary that has adopted a multiaxial approach is the Read Clinical Terms project, developed in the United Kingdom (Stuart-Buttle, Brown et al., 1997). This vocabulary and SNOMED are in the process of merging into a
11. Information Extraction
419
FIGURE 11.3. The SNOMED finding, SBE of the posterior leaflet of the mitral valve due to Streptococcus viridans. (Rothwell, D., Cote, R., et al. (1993). Developing a standard data structure for medical language—the SNOMED proposal. Proceedings of the 17th Annual Symposium on Computer Applications in Medical Care, Washington, DC. McGraw-Hill. 695–699.)
terminology to be called SNOMED CT (Stearns, Price et al., 2001). Bakken et al. (2000) have noted that few of the many nursing vocabularies meet the desiderata for vocabularies but could reach these standards with efforts at structuring. Even if the structural issues concerning vocabularies are resolved, additional challenges remain. As noted earlier, Hersh et al. (1997) found that many words used in clinical narratives were not present in the major medical vocabularies, although some could be discovered algorithmically. Humphreys et al. (1997) assessed coverage at the concept level, assessing the UMLS Metathesaurus and three additional vocabularies [SMOMED, Read Clinical Terms, and the Logical
TABLE 11.12. The Multiple Ways to Represent acute appendicitis in SNOMED International 1. 2. 3.
D5-46210 D5-46100 G-A231 M-41000 G-C006 T-59200
Acute appendicitis, NOS Appendicitis, NOS Acute Acute inflammation, NOS In Appendix, NOS
Source: D Evans, Cimino et al. (1994).
4. G-A231 M-40000 G-C006 T-59200
Acute Inflammation, NOS In Appendix, NOS
420
III. Research Directions
Observations, Identifiers, Names, and Codes (LOINC) terminology] for over 41,000 terms submitted by 63 individuals from their clinical information systems. About 58% of the terms had exact meaning matches with terms in the collection, 41% had related concepts, and 1% had no match whatsoever. For the 28% of the terms that were narrower in meaning than a concept in the controlled vocabularies, 86% shared lexical items with the broader concept but had additional modifiers. The single vocabularies with the largest coverage were SNOMED and Read Clinical Terms, each having over 60% of the terms suggested and over 50% of the concepts. An earlier analysis had also found SNOMED to have the broadest coverage of clinical content (Chute, Cohn et al., 1996; Campbell, Carpenter et al., 1997). How can clinical vocabularies be augmented beyond the tedious addition of terms? A number of researchers have suggested ways. Hersh et al. (1996) carried out preliminary work using a parser to identify modifiers occurring with 10 NP heads in a corpus of 842 megabytes of clinical narrative text. While many of the individual words in the NPs occurred in the major vocabularies, the NPs themselves did not exist as terms but could be formed compositionally from their words. Another challenge in vocabulary maintenance is identifying synonymy. Hole and Srinivasan (2000) used a variety of approaches to detect potential synonyms; the most fruitful approach was to identify lexical-level synonyms not detected by baseline means and compare terms with some but not all words in common. 11.1.3.4 Vocabulary-Based Classification of the Clinical Narrative Most attempts to classify clinical narratives to controlled vocabularies have focused on four vocabularies: ICD-9-CM, MeSH, SNOMED, and the UMLS Metathesaurus. Because ICD-9-CM and MeSH have explicit uses—diagnosis codes for billing and search terms for accessing the biomedical literature—it is not surprising that applications attempting to provide classification with them have been focused on those activities. Yang and Chute (1994) assessed the ability of a system to map free-text surgery diagnosis codes into ICD-9-CM. As noted in Chapter 9, they applied a similar process to MeSH terms in an ad hoc IR task. This work found similar results by means of a linear-least-squares fit, achieving much better performance than simple TF*IDF matching. This method was very computationally intensive, but a subsequent approach called Expert Network worked just as effectively and with much greater efficiency (Y. Yang, 1994). Larkey and Croft (1996) assessed a variety of known effective approaches to text categorization for the ability to select appropriate ICD-9-CM codes based on the entire discharge summary, including k-nearest-neighbor, Bayesian, and relevance feedback approaches. They found that combinations of the classifiers worked better than individual ones, and an approach combining all three worked best, resulting in the principal diagnosis ranking first 46.5% of the time and in the top ten 91.1% of the time. Another system for mapping ICD-9-CM (as well as CPT-4) codes is LifeCode (Heinze,
11. Information Extraction
421
Morsch et al., 2001). The CPT-4 coding portion of this application has been evaluated, with the level of interobserver agreement as measured by the kappa statistic among expert coders and between those coders and the system found to be comparable (Morris, Heinze et al., 2000). MedLEE has also been configured to perform automated ICD-9-CM coding. An analysis of the approach with 10 discharge summaries found that while the system achieved 93% recall in identifying the correct diagnosis, it achieved only 69% recall in assigning the correct ICD-9-CM code due to mismatching between phrases in the document and the codes (Lussier, Shagina et al., 2000). Systems attempting to map the clinical narrative have usually had the goal of connecting the narrative to the literature citations. Early work by Masarie and Miller (1987) assessed the proportion of words in clinical charts that were also in MeSH terms. They found that 50% of the words in clinical narratives also occurred in MeSH and 40% comprised actual MeSH headings or entry terms. Powsner et al. developed a system for extracting MeSH terms from clinical narratives for suggestion as starting points for online searching in the domains of hepatology (Powsner, Riely et al., 1989) and psychiatry (Powsner and Miller, 1992), though they did not evaluate the utility of their approach. Moore et al. (1987) likewise developed a system that suggested MeSH terms from autopsy reports. As noted in Chapter 9, Cooper and Miller (1998) developed two methods for suggesting MeSH terms from the clinical narrative, one based on lexical matching (Postdoc) and the other based on a probabilistic algorithm that nominated MeSH terms based on the probability of terms occurring in association with words in MEDLINE records (Pindex). Blinded to the method by which the terms were suggested, the authors designated which terms might be used in a literature search based on the content in each of 6 radiology reports, 6 pathology reports, and 6 discharge summaries. As seen in Table 11.13, the systems had comparable recall, though a union of the two achieved higher recall. Postdoc had better precision than either Pindex or the union approach. Not surprisingly, a larger number of researchers have investigated automated SNOMED coding from the clinical narrative, as SNOMED by design contains the concepts most likely to be present in such narratives, namely clinical findings and diagnoses. One limitation of all these studies is that SNOMED itself is a moving target: it has been evolving to its RT and CT incarnations. Moore and Berman (1994) developed two systems for mapping to SNOMED codes from pathology reports, one that matched codes against all words in the text and the
TABLE 11.13. Recall and Precision of Pindex and Postdoc for Selecting Relevant MeSH Terms from the Clinical Narrative Method
Recall (%)
Precision (%)
Postdoc Pindex Union
40 45 66
44 17 20
Source: G. Cooper and Miller (1998).
422
III. Research Directions
other that matched SNOMED terms against phrases obtained by means of barrier method. They did not calculate the overall accuracy of their system but did find that the latter approach had a much smaller false-positive matching rate and missed only 0.5% of key diagnoses on surgical pathology reports. Both the LSP and MedLEE systems have been configured to identify SMOMED codes. The LSP was augmented to recognize SNOMED encoding of 10 case reports (Sager, Lyman et al., 1995). Although an overall evaluation was not performed, impediments to accuracy were found to come from the ambiguity of SNOMED terms (as exemplified in Table 11.12) as well as the lack of complete synonymy (e.g., the general phrase difficulty with vision not exactly mapping conceptually to the SNOMED term Decreased Vision NOS). MedLEE was augmented using a single 589-word discharge summary that had been coded by an expert coder into 285 distinct codes (Lussier, Shagina et al., 2001). The authors of this study noted similar challenges in mapping NLP system output into SNOMED codes but were able to achieve a recall of 88% and a precision of 83%. Clearly additional work is necessary to map from the rich structures output by these systems into coding systems such as SNOMED. Some investigators have assessed the ability to map clinical narrative text to UMLS Metathesaurus terms. Two efforts have come from systems developed initially to function as IR systems, MetaMap and SAPHIRE, and have focused on IR-like capabilities. Sneiderman et al. (1998) attempted to identify coronary artery anatomical terms from cardiac catheterization reports with the goal of linking such terms to online anatomical images. They ran into a number of common problems in NLP, such as lack of comprehensive terms in the lexicon (e.g., missing acronyms) and the inability of the parser to map modifiers across conjunctions. Nonetheless, they were able to achieve a recall of 83% and precision of 88% with a test collection of 199 terms marked up in 213 reports. Also part of this effort, Bean et al. (1998) assessed the ability of MetaMap to identify spatial relationships among coronary artery anatomic terms, which was followed up by more detailed work focusing on identifying branching relationships among the arteries (Rindflesch, Bean et al., 2000). Evaluation of the latter with a test collection of 71 branching relationships from catheterization reports found a recall of 83% and a precision of 100%. Hersh et al. (2002) adapted SAPHIRE (algorithm in Figure 9.4) for selective automated indexing of radiology reports in an effort to identify “important” terms that might be used in queries of image collections. Users who might use reports indexed in this fashion would include clinicians seeking to find other images with findings or diagnoses currently being considered or educators seeking to build teaching examples. D. Johnson et al. (1999) also have investigated the processing of radiology reports for building teaching collections. A key challenge of this selective-indexing task was that no attempt was made to identify all Metathesaurus terms present in the text, but rather only those considered to be important as indexing terms. These terms were selected in a collection of 50 reports from a variety of radiological exams, such as CXRs and
11. Information Extraction
423
TABLE 11.14. Results of SAPHIRE Selective Indexing Applied to Radiology Reports Run Baseline Precision enhancing Recall enhancing
Found
Possible
Returned
Precision (%)
Recall (%)
228 203 227
361 361 361
1660 898 765
14 23 30
63 56 63
Source: Hersh, Mailhot et al. (2002).
magnetic resonance imaging, by a medical librarian not involved in the development of the algorithm. Three iterations of algorithms were developed and evaluated, with results shown in Table 11.14. The first iteration consisted of the baseline SAPHIRE algorithm with the addition of a barrier word approach to generating phrases for analysis. This approach yielded a very low precision. A failure analysis showed that recall errors most frequently came from correct concepts ranked lower than incorrect ones. Precision errors most often came from terms that were appropriately identified but not designated by the indexer as important. In the next iteration, effort was made to improve precision by making three modifications: • Filtering out concepts from semantic types unlikely to contain indexing terms, such as Classification or Substance • Recognizing and eliminating terms that were negated • Skipping (i.e., not processing) the History portion of the report The second iteration improved precision but at the expense of recall. Further efforts were made to enhance recall by the addition of stemming the letter “s” and relaxing the requirement that the top-ranking concept be an exact match of the input phrase. This increased recall back to the baseline, and some additional precision-enhancing modifications allowed it to be raised further. The authors noted that more sophisticated NLP techniques might improve performance, but the real challenge was to determine the value of such a system to real users and how much incomplete recall and precision would be acceptable. Nadkarni et al. (2001) also developed a program aiming to identify Metathesaurus terms in narrative text. Their algorithm involved some preprocessing of the Metathesaurus to apply the Porter stemming algorithm to all words and eliminate certain terms or components of terms that would be unlikely to match a clinical narrative. Entities eliminated included the following: • The words NOS and NEC from terms • Concepts of semantic types unlikely to be important for clinical narratives (e.g., Alga, Governmental or Regulatory Activity) • Chemical substance names (e.g., 1,2-Dipalmitoylphosphatidylcholine) • Concepts that were highly compound terms from laboratory tests (mostly from LOINC, e.g., TRYPSIN:CATALYTIC CONCENTRATION:POINT IN
424
III. Research Directions
TIME) or other observations (e.g., Supine and quarter turn to right from the Read Clinical Terms) • Some concepts with embedded acronyms (e.g., O/E Dorsalis Pedis) The first step in the algorithm applied the IBM Feature Extraction tool for generation of phrases, which identifies important phrases in text. This tool is designed to recognize proper nouns, such as names and places, so when these were identified, they were removed from further processing. The remaining phrases were processed to match the longest possible Metathesaurus term(s). If one term could not subsume the phrase, attempts would be made to match multiple terms, with precedence always given to terms subsuming the most words. Terms that were ambiguous (or polysemous, i.e., more than one term matched the phrase or a part of it) were flagged as such. Term-matching proceeded without any requirements for word order (e.g., the phrase Type II Diabetes Mellitus could match the term Diabetes Mellitus, Type II). Nadkarni et al. evaluated their algorithm on 12 discharge summaries and 12 surgical notes manually coded by a physician not involved in development of the algorithm. They found that 76.3% of Metathesaurus terms were matched correctly. About 7.0% of terms were ambiguous, with a nearly equal proportion of false negatives (i.e., concepts not matched that should have been). They conclude that this rate of success would be insufficient for operational systems, and note particularly that polysemous terms will present a major challenge. The ability to detect negation was added to this system and was found to operate with a sensitivity of 95.7% and a specificity of 91.8% (Mutalik, Deshpande et al., 2001). 11.1.3.5 Ad Hoc Classification of the Clinical Narrative A number of researchers have attempted to process clinical narratives aiming to classify documents into ad hoc categories. One approach has aimed to detect deleterious situations, such as an exacerbation of a chronic condition or the presence of an abnormality on a radiological examination requiring follow-up. Aronow et al. used the INQUERY retrieval system (see Chapter 8) to categorize documents as positive, uncertain, or negative for requiring attention of a clinician in a deleterious situation. Such a system, if effective, could reduce the time required for manual chart review (unless the system performed very poorly, flagging too many inappropriate documents). Initial work focused on the detection of asthma exacerbations in clinic progress notes of an urban health maintenance organization (Aronow, Soderland et al., 1995). The most effective training process for classification was found to consist of first querying the document set for the word asthma and then applying both positive and negative relevance feedback to previously classified notes. It was estimated that if reviewers accepted the results of the positive and negative classifications and could omit them from review, the number of notes to be assessed could be reduced by as much as 65% (Aronow, Cooley et al., 1995). Subsequent work looked at the detection of abnormalities on mammography reports noted to require nonroutine follow-up (Aronow, Fangfang et al., 1999).
11. Information Extraction
425
INQUERY was again used to classify reports into positive, uncertain, and negative categories in a test collection with 421 positive and 256 negative reports. A number of variations to the basic algorithm were assessed, with several features found to affect performance: • The addition of words in the relevance feedback process improved performance until an asymptote was reached at about 200 words. • The addition of weight from words of positive documents improved performance more than the subtraction of weight from words of negative documents. • Iterations of reweighting improved performance when done up to four times, but after that overtraining caused performance to degrade. • Increasing the size of the training collection improved performance. • Best performance was seen when the collection contained equal numbers of positive and negative reports. • Detecting negation led to a modest increase in performance. Among the features that did not improve performance was the addition of phrases, whether ordered or unordered, or generated from an NP parser. Another task that has been assessed is the assignment of fields in a database based on classification of the document. Hersh et al. (1998) assessed the ability of machine learning methods to assign codes in a trauma registry. The registry used in this study contained all trauma cases in the state of Oregon and consisted of 119 elements, including medical procedures occurring during the care episode as well as patient outcomes (Mullins, Veum-Stone et al., 1994). Coding of this registry is a labor-intensive task, and the ultimate goal of automated coding would be to improve its efficiency. Another benefit of automated coding, especially from data early in the course of the hospitalization (e.g., the emergency room dictation), would be to identify patients potentially at high risk for complications or mortality. The initial study focused on 64 procedure codes, which included whether the patient required supplemental oxygen, a computerized tomography scan of the abdomen, or admission to the intensive care unit. A collection of 360 emergency room dictations was obtained that had been coded (manually) into the database. One-third of the dictations were used to fit the machine learning model, one-third were used to select the model, and one-third were used for evaluation. Similar to the work of Aronow et al., dictations were broken down into a “bag of words” model, with no effort to identify specific terms or portions of the document. A variety of machine learning methods were assessed, with logistic regression found to be the most effective. In general, the code assignments exceeded chance but did not come close to levels that would be acceptable in an operational system, indicating a need for additional features. One other approach to processing the narrative looked at Puya, a system for identifying abnormal findings in the physical examination portion by screening out sentences which report normal results (de Estrada, Murphy et al., 1997). An analysis of 100 progress notes containing a physical examination were assessed.
426
III. Research Directions
Preliminary work showed that four physicians had an agreement rate of 96% for classifying sentences as normal or abnormal. A total of 610 sentences were analyzed by Puya to match normal findings, and 23% had abnormal findings. Puya operated with perfect recall (100%) but a precision of only 44%.
11.1.4 Alternatives to Natural Language Input of Medical Data Given all the problems and limited capabilities of the systems described in this chapter, one might be tempted to look for approaches that avoid the use of ambiguous natural language in the first place. A number of investigators have attempted to develop structured data entry systems featuring forms that allow direct input of coded data. Early paper-based systems often required scanning and/or transcription, but recent systems have focused on using computerized methods. Form-based input is a trade-off, sacrificing clinician “freedom of expression” for the unambiguous structure of coded data. Systems have been implemented for a variety of small domains, including the cardiovascular exam (Cimino and Barnett, 1987), gastrointenstinal endoscopy (Kuhn, Gus et al., 1992), and obstetric ultrasound (Greenes, 1982). Larger-scale systems have also implemented in the past, without long-lasting usage (Greenes, Barnett et al., 1970; Fischer, Stratmann et al., 1980). More recent comprehensive efforts have attempted to utilize the pointing devices and graphical displays of modern microcomputers as well as more sophisticated coding structures (Campbell, Wieckert et al., 1993; Bell and Greenes, 1994). The proliferation of handheld computers, currently used mainly for data retrieval, may play a role in facilitating coded data entry (Carroll, Tarczy-Hornoch et al., 2001; Ruland, 2002). While structured data entry systems eliminate the need for NLP, they do not overcome the problems in clinical vocabularies. They still require a vocabulary that contains the findings designated by physicians and all the modifiers they may choose to use. The IVORY system utilizes SNOMED but requires extensions to handle progress note findings unambiguously (Campbell, Wieckert et al., 1993). The PEN & PAD system is based on a comprehensive European vocabulary effort known as GALEN (Rector, Nowlan et al., 1991).
11.1.5 Future Directions for Clinical Data Capture and Analysis This chapter has described a wide variety of approaches to capture clinical findings. Narrative processing systems were discussed that achieved high but not complete accuracy. For the narrative processing systems, each one described, from simplest to complex, has been shown to extract appropriate findings at a fairly high rate of accuracy (e.g., 80–95%). This leads to a number of larger questions. Will these systems be able to process that remaining 5 to 20% accurately?
11. Information Extraction
427
If so, how much work will be required to get them to the level of accuracy of a human reader? If not, will systems still be useful for research and quality assurance purposes if they have an inherent level of inaccuracy? For clinical vocabularies and structured data entry systems, there are larger questions as well. For the former, will comprehensive clinical nomenclatures be developed that can scale up to all the types of information desired for capture? For the latter, will clinicians accept structured data entry and provide the comprehensiveness and quality currently entered into the clinical narrative? Despite the problems noted with all the foregoing data extraction and capture systems, these are not questions that can be ignored. As patients, clinicians, managed care organizations, researchers, and other continue to demand more information about the quality of health care, impetus will exist to tap the data in clinical findings, whether in narrative or structured data entry form.
11.2 Knowledge-Based Information Less work has been done in applying IE techniques to knowledge-based information than to patient-specific information, but with the growing mining of genomics text documents, efforts in the latter area have been increasing. As noted already, one early effort was an attempt to apply the LSP system to lipid metabolism articles (Mattick, 1987). Another approach applied the Postdoc system to medical school curriculum documents but found low concordance between the indexing terms selected by document authors and the concept-matching algorithm (Kanter, Miller et al., 1994). Sneiderman et al. (1996) have assessed MetaMap for the ability to extract clinical findings in Parkinson’s Disease from the literature. A major challenge for IE from knowledge-based information documents, present to a lesser extent in the clinical narratives, is text not focused on the core concepts of the document. A scientific report in a journal may, for example, contain background material, methods, or speculation in the discussion that is only loosely related. A review document may oversimplify the knowledge of a field or use elements of language (e.g., conjunctions, negation) that are challenging to process correctly. So any IE process applied to literature must have a specific task in mind and the ability to identify and process appropriate parts of documents. One line of work has looked at processing the medical literature to facilitate scientific discovery. In particular, the literature might contain disconnected threads. For example, there might be studies that identify diseases or syndromes that result in certain manifestations and others that identify treatments that improve such manifestations, yet the treatment has never been assessed for this particular disease or syndrome. Swanson showed this to be the case in two instances: 1. Articles on Raynaud’s Disease found blood viscosity to be abnormally high, while others on the fish oil eicosapentaenoic acid
428
III. Research Directions
found this substance to reduce blood viscosity, yet the latter had never been considered as a treatment for the former (Swanson, 1986). 2. Articles on migraine found the disease to be implicated by spreading depression in the cortex of the brain, while others found magnesium to be effective in inhibiting such depression (usually in the context of treating a different condition, epilepsy) (Swanson, 1988b). When first identified, the two literatures in each instance were completely disconnected, and no researchers had thought of the potential for treatment. After identification of these potential linkages, clinical trials found effectiveness for both treatments (Swanson and Smalheiser, 1997). Additional disconnected complementary literatures have also been discovered, and a system called ARROWSMITH has been developed to facilitate further exploration (Smalheiser and Swanson, 1998). Additional work to expand potential discovery has been developed by Weeber et al. (2000). Genomics is another area in which a great deal of IE from knowledge-based literature is taking place. This effort represents a change in biological research away from iterative hypothesis testing to mining of the literature to generate hypotheses for further exploration. This work has been facilitated by the wide variety of publicly available resources, from MEDLINE to gene and protein sequence and structure databases. Further impetus comes from the development of gene expression microarrays, which provide data on the functional expression of tens of thousands of genes in a single experiment (Duggan, Bittner et al., 1999). The major challenges to biological researchers are grappling with all the data to determine causes and effects on health and disease processes and determining their proper role in healthcare (Collins and McKusick, 2001). The majority of early work focused on the use of IE and NLP techniques to extract terms and relationships from biomedical text, primarily MEDLINE. Textual elements extracted include the following: • Protein names and interactions (Fukuda, Tamura et al., 1998; Blaschke, Andrade et al., 1999; Humphreys, Demetriou et al., 2000; Thomas, Milward et al., 2000; Park, Kim et al., 2001; Pustejovsky, Castano et al., 2002) • Drug–gene interactions (Rindflesch, Tanabe et al., 2000) • Molecular binding information (Rindflesch, Hunter et al., 1999) • Cellular events (Friedman, Kra et al., 2001; Yakushiji, Tateisi et al., 2001; Leroy and Chen, 2002) • Gene names (Krauthammer, Rzhetsky et al., 2000) Some of these approaches involved adaptation of systems developed for IE in other domains, such as MedLEE (Friedman, Kra et al., 2001) and MetaMap (Rindflesch, Hunter et al., 1999; Rindflesch, Tanabe et al., 2000). Krauthammer et al. (2000) actually used the sequence alignment tool Basic Local Alignment Search Tool (BLAST), usually used to map nucleotide sequences into genes, to match sequences of text characters into gene names. Other work has explored the appropriate units of text for analysis (Ding, Berleant et al., 2002)
11. Information Extraction
429
and the disambiguation of the elements listed above (Hatzivassiloglou, Duboue et al., 2001). Additional efforts have focused on modeling of the domain in anticipation of knowledge extraction from textual documents. One effort looked at how UMLS resources can be adopted (Baclawski, Cigna et al., 2000), while another focused on cellular regulatory networks (Rzhetsky, Koike et al., 2000). The goal of the latter is to provide a knowledge representation structure to which facts from literature might be mapped. It includes structures (e.g., cells, organs), substances (e.g., proteins, nucleotides), and processes (e.g., transcription, uptake), organized in a hierarchical fashion and encoding explicit relationships. Another effort to model the genomics domain is Gene Ontology (GO, www.geneontology.org), which contains three ontologies that describe biological processes (e.g., DNA metabolism), molecular functions (e.g., nucleic acid binding), and cellular components (e.g., nucleoplasm) (Ashburner, Ball et al., 2000). Similar to work in IR, investigators have begun to evaluate the optimal approaches for GO codes to MEDLINE references, which would in turn facilitate their linkage with gene products and functions. Raychaudhuri et al. (2002) assessed three document classification techniques and found the following rate of accuracy: 1. Maximum entropy modeling, 72.1% 2. Naïve Bayes classification, 59.6% 3. Nearest-neighbor classification, 61.5% One limitation of this study was that the abstracts came from explicit subsets of MEDLINE and not the database in general. The authors also did not attempt to combine classifiers, which has been effective in other domains (Larkey and Croft, 1996). Other research has investigated gene–gene interactions. Two efforts investigated co-occurrence of gene names in MEDLINE abstracts (Stapley and Benoit, 2000; Jenssen, Laegreid et al., 2001). The latter led to the creation of PubGene (www.pubgene.org), which links gene names to nongene MeSH terms. While this process was shown to yield a number of pairings that had no biological basis, it did detect gene coexpressions subsequently discovered in published experiments. Related to this is work that links genes to MeSH terms with the goal of identifying additional meaning to the output of gene expression microarrays (Masys, Welsh et al., 2001). As additional data become available concerning gene expression (or lack thereof) in disease states, environmental exposures, and so forth, the ability to uncover similar relationships in the literature and in genomics databases could significantly accelerate the growth of scientific knowledge. Additional work has investigated means for improving interaction with the literature. Analyses by Iliopoulos et al. (2001) and Wilbur (2002) have shown that various methods for clustering the literature can identify important “themes” in the clusters. Other efforts have investigated improving BLAST searching by incorporating literature links in the process (Chang, Raychaudhuri et al., 2001).
430
III. Research Directions
11.3 Summary Relative to IR, the field of IE is still young. While IR applications have become “mainstream,” used by millions who utilize the Web, there are few IE applications that have had significant impact. Nonetheless, the potential for IE, whether in the processing of clinical narratives to improve the care of patients or in the mining of information from the scientific literature to augment research, is enormous. Researchers and developers of IE applications will need to continue developing basic techniques as well as practical applications, with their efforts justified by realistic evaluation studies.
Appendix 1 Ten Sample Documents to Illustrate Indexing and Retrieval
Document 1 Title: Definition of Hypertension Text: Hypertension is defined as abnormally elevated blood pressure. The cut-off for abnormal is usually drawn at the level which increased risk of disease occurs. This is typically a value of 140/90, though varies for different age groups. MeSH: Hypertension/diagnosis
Document 2 Title: Complications of Hypertension Text: Long-standing high blood pressure can lead to a number of complications, based on its acceleration of atherosclerosis. These complications include angina pectoris, myocardial infarction, congestive heart failure, cerebrovascular accident, and renal failure. MeSH: Hypertension/complications Atherosclerosis
Document 3 Title: Treatment Goals in Hypertension 431
432
Appendix 1. Ten Sample Documents to Illustrate Indexing and Retrieval
Text: The major goal of therapy in hypertension is to reduce the blood pressure to normal levels. This has been shown to reduce the complications. MeSH: Hypertension/therapy Hypertension/complications
Document 4 Title: Therapies for Hypertension Text: The first line of therapy for hypertension is usually life-style modification, including weight loss, moderate sodium restriction, and an age-appropriate exercise regimen. If these measures are not successful, then pharmacologic therapy is usually needed. MeSH: Hypertension/diet therapy Exercise therapy
Document 5 Title: Pharmacologic Agents for Elevated Blood Pressure Text: An ever-increasing array of drugs is available for treatment of hypertension. The major categories of these drugs include diuretics, beta-adrenergic blockers, calcium-channel blockers, and angiotensin-converting enzyme inhibitors. MeSH: Hypertension/drug therapy Diuretics Adrenergic Beta Receptor Blockaders Calcium Channel Blockers Angiotensin-Converting Enzyme inhibitors
Document 6 Title: Long-Term Benefits of Antihypertensive Therapy
Appendix 1. Ten Sample Documents to Illustrate Indexing and Retrieval
433
Text: While many drugs have known antihypertensive effects, only the beta blockers atenolol and metoprolol have been shown to prolong life. Nifedipine, a calcium-channel blocker, may actually have adverse long-term outcomes. MeSH: Antihypertensive Agents Atenolol/therapeutic use Metoprolol/therapeutic use Nifedipine/therapeutic use
Document 7 Title: Adverse Effects of Antihypertensive Agents Text: Antihypertensive drugs are generally safe but do have a wide spectrum of potential adverse effects. Diuretics, for example, are known to raise glucose and lower potassium levels in the blood, both of which may lead to increased cardiovascular risk. Beta blockers have been shown to raise blood cholesterol level, which also increases cardiovascular risk. MeSH: Antihypertensive Agents/adverse effects Diuretics Adrenergic Beta Receptor Blockaders Hypercholesterolemia
Document 8 Title: Treatment of Glaucoma Text: Glaucoma occurs from elevated intraocular pressure. The intraocular hypertension is treated with a variety of eye drops, including beta blockers. MeSH: Glaucoma Intraocular Hypertension Adrenergic Beta Receptor Blockaders
434
Appendix 1 Ten Sample Documents to Illustrate Indexing and Retrieval
Document 9 Title: Congestive Heart Failure Text: Congestive heart failure occurs when the heart can no longer pump an adequate amount of blood. MeSH: Congestive Heart Failure/etiology
Document 10 Title: Complications of Congestive Heart Failure Text: As congestive heart failure progresses, the heart muscle is increasingly unable to keep up with demands, resulting in a low blood pressure and high heart rate. The first line of treatment is usually diuretic therapy. MeSH: Congestive Heart Failure/complications Diuretics/therapeutic use
Appendix 2 Inverted File of Words from Documents of Appendix 1
Each word in the Appendix 1 document collection is listed in all capital letters, followed on the same line by the document frequency (DF), collection frequency (CF), and inverse document frequency (IDF). This is followed by postings, with one line for each document containing the document number (DN) and term frequency (TF). (Stop words have been filtered and a simple stemming algorithm applied: see Chapter 8.) Word
DF Postings DN TF ABNORMAL 1 1 1 ABNORMALLY 1 1 1 ACCELERATION 1 2 1 ACCIDENT 1 2 1 ACTUALLY 1 6 1 ADEQUATE 1 9 1 ADVERSE 2 6 1 7 2 AGE 2 1 1 4 1
CF
IDF
1
2.00
1
2.00
1
2.00
1
2.00
1
2.00
1
2.00
3
1.70
2
1.70
435
436
Appendix 2. Inverted File of Words from Documents of Appendix 1
AGENT 2 5 1 7 1 AMOUNT 1 9 1 ANGINA 1 2 1 ANGIOTENSIN 1 5 1 ANTIHYPERTENSIVE 2 6 2 7 2 APPROPRIATE 1 4 1 ARRAY 1 5 1 ATENOLOL 1 6 1 ATHEROSCLEROSIS 1 2 1 AVAILABLE 1 5 1 BASED 1 2 1 BENEFIT 1 6 1 BETA 4 5 1 6 1 7 1 8 1 BLOCKER 4 5 2 6 2 7 1 8 1 BLOOD 7 1 1 2 1 3 1 5 1 7 2 9 1 10 1
2
1.70
1
2.00
1
2.00
1
2.00
4
1.70
1
2.00
1
2.00
1
2.00
1
2.00
1
2.00
1
2.00
1
2.00
4
1.40
6
1.40
8
1.15
Appendix 2. Inverted File of Words from Documents of Appendix 1
CALCIUM 2 5 1 6 1 CARDIOVASCULAR 1 7 2 CATEGORY 1 5 1 CEREBROVASCULAR1 2 1 CHANNEL 2 5 1 6 1 CHOLESTEROL 1 7 1 COMPLICATION 3 2 3 3 1 10 1 CONGESTIVE 3 2 1 9 2 10 2 CONVERTING 1 5 1 CUT 1 1 1 DEFINED 1 1 1 DEFINITION 1 1 1 DEMAND 1 10 1 DIFFERENT 1 1 1 DISEASE 1 1 1 DIURETIC 3 5 1 7 1 10 1 DO 1 7 1 DRAWN 1 1 1
2
1.70
2
2.00
1
2.00
1
2.00
2
1.70
1
2.00
5
1.52
5
1.52
1
2.00
1
2.00
1
2.00
1
2.00
1
2.00
1
2.00
1
2.00
3
1.52
1
2.00
1
2.00
437
438
Appendix 2. Inverted File of Words from Documents of Appendix 1
DROP 8 DRUG 5 6 7 EFFECT 6 7 ELEVATED 1 5 8 ENZYME 5 EXAMPLE 7 EXERCISE 4 EYE 8 FAILURE 2 9 10 FIRST 10 GENERALLY 7 GLAUCOMA 8 GLUCOSE 7 GOAL 3 GROUP 1 HEART 2 9 10 HIGH 2 10
1
1
2.00
3
4
1.52
2
3
1.70
3
3
1.52
1
1
2.00
1
1
2.00
1
1
2.00
1
1
2.00
3
6
1.52
1
1
2.00
1
1
2.00
1
2
2.00
1
1
2.00
1
2
2.00
1
1
2.00
3
8
1.52
2
2
1.70
1 2 1 1 1 2 1 1 1 1 1 1 1 2 2 2 1 1 2 1 2 1 1 3 4 1 1
Appendix 2. Inverted File of Words from Documents of Appendix 1
HYPERTENSION 1 2 2 1 3 2 4 2 5 1 8 1 INCLUDE 2 1 5 1 INCLUDING 4 1 8 1 INCREASE 7 1 INCREASED 1 1 7 1 INCREASING 5 1 INCREASINGLY 10 1 INFARCTION 2 1 INHIBITOR 5 1 INTRAOCULAR 8 2 KEEP 10 1 KNOWN 6 1 7 1 LEAD 2 1 7 1 LEVEL 1 1 3 1 7 2 LIFE 4 1 6 1
6
9
1.22
2
2
1.70
2
2
1.70
1
1
2.00
2
2
1.70
1
1
2.00
1
1
2.00
1
1
2.00
1
1
2.00
1
2
2.00
1
1
2.00
2
2
1.70
2
2
1.70
3
4
1.52
2
2
1.70
439
440
Appendix 2. Inverted File of Words from Documents of Appendix 1
LINE 4 1 10 1 LONG 2 1 6 2 LONGER 9 1 LOSS 4 1 LOW 10 1 LOWER 7 1 MAJOR 3 1 5 1 MEASURE 4 1 METOPROLOL 6 1 MODERATE 4 1 MODIFICATION 4 1 MUSCLE 10 1 MYOCARDIAL 2 1 NEEDED 4 1 NIFEDIPINE 6 1 NORMAL 3 1 NUMBER 2 1 OCCUR 1 1 8 1 9 1 OUTCOME 6 1 PECTORIS 2 1
2
2
1.70
2
3
1.70
1
1
2.00
1
1
2.00
1
1
2.00
1
1
2.00
2
2
1.70
1
1
2.00
1
1
2.00
1
1
2.00
1
1
2.00
1
1
2.00
1
1
2.00
1
1
2.00
1
1
2.00
1
1
2.00
1
1
2.00
3
3
1.52
1
1
2.00
1
1
2.00
Appendix 2. Inverted File of Words from Documents of Appendix 1
PHARMACOLOGIC 4 1 5 1 POTASSIUM 7 1 POTENTIAL 7 1 PRESSURE 1 1 2 1 3 1 5 1 8 1 10 1 PROGRESSES 10 1 PROLONG 6 1 PUMP 9 1 RAISE 7 2 RATE 10 1 REDUCE 3 2 REGIMEN 4 1 RENAL 2 1 RESTRICTION 4 1 RESULTING 10 1 RISK 1 1 7 2 SAFE 7 1 SHOWN 3 1 6 1 7 1 SODIUM 4 1
2
2
1.70
1
1
2.00
1
1
2.00
6
6
1.22
1
1
2.00
1
1
2.00
1
1
2.00
1
2
2.00
1
1
2.00
1
2
2.00
1
1
2.00
1
1
2.00
1
1
1.00
1
1
2.00
2
3
1.70
1
1
2.00
3
3
1.52
1
1
2.00
441
442
Appendix 2. Inverted File of Words from Documents of Appendix 1
SPECTRUM 7 STANDING 2 STYLE 4 SUCCESSFUL 4 TERM 6 THERAPY 3 4 6 10 TREATED 8 TREATMENT 3 5 8 10 TYPICALLY 1 UNABLE 10 USUALLY 1 4 10 VALUE 1 VARIETY 8 VARY 1 WEIGHT 4 WHICH 1 7 WHILE 6 WIDE 7
1
1
2.00
1
1
2.00
1
1
2.00
1
1
2.00
1
2
2.00
4
6
1.40
1
1
2.00
4
4
1.40
1
1
2.00
1
1
2.00
3
4
1.52
1
1
2.00
1
1
2.00
1
1
2.00
1
1
2.00
2
3
1.70
1
1
2.00
1
1
2.00
1 1 1 1 2 1 3 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 2 1 1
Appendix 3 Document Vectors for Words from Documents of Appendix 1 with Weighting Measures
Term Document 1 ABNORMAL ABNORMALLY AGE BLOOD CUT DEFINED DEFINITION DIFFERENT DISEASE DRAWN ELEVATED GROUP HYPERTENSION INCREASED LEVEL OCCUR PRESSURE RISK TYPICALLY USUALLY VALUE VARY WHICH Document 2 ACCELERATION ACCIDENT ANGINA ATHEROSCLEROSIS BASED BLOOD CEREBROVASCULAR
TF
IDF
IDF*TF
1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1
2 2 1.7 1.15 2 2 2 2 2 2 1.52 2 1.22 1.7 1.52 1.52 1.22 1.7 2 1.7 2 2 1.7
2 2 1.7 1.15 2 2 2 2 2 2 1.52 2 2.44 1.7 1.52 1.52 1.22 1.7 2 1.7 2 2 1.7
1 1 1 1 1 1 1
2 2 2 2 2 1.15 2
2 2 2 2 2 1.15 2
443
444
Appendix 3. Document Vectors for Words from Appendix 1
Term COMPLICATION CONGESTIVE FAILURE HEART HIGH HYPERTENSION INCLUDE INFARCTION LEAD LONG MYOCARDIAL NUMBER PECTORIS PRESSURE RENAL STANDING Document 3 BLOOD COMPLICATION GOAL HYPERTENSION LEVEL MAJOR NORMAL PRESSURE REDUCE SHOWN THERAPY TREATMENT Document 4 AGE APPROPRIATE EXERCISE HYPERTENSION INCLUDING LIFE LINE LOSS MEASURE MODERATE MODIFICATION NEEDED PHARMACOLOGIC REGIMEN RESTRICTION SODIUM STYLE SUCCESSFUL THERAPY USUALLY WEIGHT
TF
IDF
IDF*TF
2 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1
1.52 1.52 1.52 1.52 1.7 1.22 1.7 2 1.7 1.7 2 2 2 1.22 2 2
3.04 1.52 3.04 1.52 1.7 1.22 1.7 2 1.7 1.7 2 2 2 1.22 2 2
1 1 2 2 1 1 1 1 2 1 1 1
1.15 1.52 2 1.22 1.52 1.7 2 1.22 2 1.52 1.52 1.52
1.15 1.52 4 2.44 1.52 1.7 2 1.22 4 1.52 1.52 1.52
1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 1
1.7 2 2 1.22 1.7 1.7 2 2 2 2 2 2 1.7 2 2 2 2 2 1.52 1.7 2
1.7 2 2 2.44 1.7 1.7 2 2 2 2 2 2 1.7 2 2 2 2 2 3.04 3.4 2
Appendix 3. Document Vectors for Words from Appendix 1 Term Document 5 AGENT ANGIOTENSIN ARRAY AVAILABLE BETA BLOCKER BLOOD CALCIUM CATEGORY CHANNEL CONVERTING DIURETIC DRUG ELEVATED ENZYME HYPERTENSION INCLUDE INCREASING INHIBITOR MAJOR PHARMACOLOGIC PRESSURE TREATMENT Document 6 ACTUALLY ADVERSE ANTIHYPERTENSIVE ATENOLOL BENEFIT BETA BLOCKER CALCIUM CHANNEL DRUG EFFECT KNOWN LIFE LONG METOPROLOL NIFEDIPINE OUTCOME PROLONG SHOWN TERM THERAPY WHILE Document 7 ADVERSE AGENT ANTIHYPERTENSIVE
445
TF
IDF
IDF*TF
1 1 1 1 1 2 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1
1.7 2 2 2 1.4 1.4 1.15 1.7 2 1.7 2 1.7 1.52 1.52 2 1.22 1.7 2 2 1.7 1.7 1.22 1.52
1.7 2 2 2 1.4 2.8 1.15 1.7 2 1.7 2 1.7 3.04 1.52 2 1.22 1.7 2 2 1.7 1.7 1.22 1.52
1 1 2 1 1 1 2 1 1 1 1 1 1 2 1 1 1 1 1 2 1 1
2 1.7 1.7 2 2 1.4 1.4 1.7 1.7 1.52 1.7 1.7 1.7 1.7 2 2 2 2 1.52 2 1.52 2
2 1.7 3.4 2 2 1.4 1.4 1.7 1.7 1.52 1.7 1.7 1.7 2.21 2 2 2 2 1.52 4 1.52 2
2 1 2
1.7 1.7 1.7
3.4 1.7 3.4
446
Appendix 3. Document Vectors for Words from Appendix 1
Term BETA BLOCKER BLOOD CARDIOVASCULAR CHOLESTEROL DIURETIC DO DRUG EFFECT EXAMPLE GENERALLY GLUCOSE INCREASE INCREASED KNOWN LEAD LEVEL LOWER POTASSIUM POTENTIAL RAISE RISK SAFE SHOWN SPECTRUM WHICH WIDE Document 8 BETA BLOCKER DROP ELEVATED EYE GLAUCOMA HYPERTENSION INCLUDING INTRAOCULAR OCCUR PRESSURE TREATED TREATMENT VARIETY Document 9 ADEQUATE AMOUNT BLOOD CONGESTIVE FAILURE HEART LONGER OCCUR PUMP
TF
IDF
IDF*TF
1 1 2 2 1 1 1 1 2 1 1 1 1 1 1 1 2 1 1 1 2 2 1 1 1 2 1
1.4 1.4 1.15 2 2 1.7 2 1.52 1.7 2 2 2 2 1.7 1.7 1.7 1.52 2 2 2 2 1.7 2 1.52 2 1.7 2
1.4 1.4 2.3 4 2 1.7 2 1.52 3.4 2 2 2 2 1.7 1.7 1.7 3.04 2 2 2 4 3.4 2 1.52 2 3.4 2
1 1 1 1 1 2 1 1 2 1 1 1 1 1
1.4 1.4 2 1.52 2 2 1.22 1.7 2 1.52 1.22 2 1.52 2
1.4 1.4 2 1.52 2 4 1.22 1.7 4 1.52 1.22 2 1.52 2
1 1 1 2 2 3 1 1 1
2 2 1.15 1.52 1.52 1.52 2 1.52 2
2 2 1.15 3.04 3.04 4.56 2 1.52 2
Appendix 3. Document Vectors for Words from Appendix 1 Term Document 10 BLOOD COMPLICATION CONGESTIVE DEMAND DIURETIC FAILURE FIRST HEART HIGH INCREASINGLY KEEP LINE LOW MUSCLE PRESSURE PROGRESSES RATE RESULTING THERAPY TREATMENT UNABLE USUALLY
447
TF
IDF
IDF*TF
1 1 2 1 1 2 1 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1.15 1.52 1.52 2 1.52 1.52 2 1.52 1.7 2 2 2 2 2 1.22 2 2 2 1.4 1.4 2 1.52
1.15 1.52 1.98 2 1.52 3.04 2 6.08 1.7 2 2 2 2 2 1.22 2 2 2 1.4 1.4 2 1.52
Appendix 4 Inverted File of MeSH Terms from Documents of Appendix 1 with Term Frequency and Postings
Each MeSH term in the Appendix 1 document collection is listed, followed on the same line by the document frequency (DF). This is followed by postings, with one line for each document containing the document number (DN). MeSH term Postings DN Adrenergic Beta Receptor Blockaders 5 7 8 Angiotensin-Converting Enzyme Inhibitors 5 Antihypertensive Agents 6 Antihypertensive Agents/adverse effects 7 Atenolol/therapeutic use 6 Atherosclerosis 2 Calcium Channel Blockers 5 Congestive Heart Failure/complications 10 Congestive Heart Failure/etiology 9
448
DF 3
1 1 1 1 1 1 1 1
Appendix 4. Inverted File of MeSH Terms from Appendix 1
Diuretics 5 7 Diuretics/therapeutic use 10 Exercise therapy 4 Glaucoma 8 Hypercholesterolemia 7 Hypertension/complications 2 3 Hypertension/diagnosis 1 Hypertension/diet therapy 4 Hypertension/drug therapy 5 Hypertension/therapy 3 Intraocular Hypertension 8 Metoprolol/therapeutic use 6 Nifedipine/therapeutic use 6
2
1 1 1 1 2
1 1 1 1 1 1 1
449
References
Abate, M., Shumway, J., et al. (1992). Use of two online services as drug information sources for health professionals. Methods of Information in Medicine, 31: 153–158. Abraham, V., Friedman, C., et al. (1999). Student and faculty performance in clinical simulations with access to a searchable information resource. Proceedings of the AMIA 1999 Annual Symposium. Washington, DC. Philadelphia, Hanley & Belfus, 648–652. Adam, D. (2002). The counting house. Nature, 415: 726–728. Agger, B. (1990). The Decline of Discourse: Reading, Writing, and Resistance in Postmodern Capitalism. London, Taylor & Francis. Agosti, M., Crestani, F., et al. (1997). On the use of information retrieval techniques for the automatic construction of hypertext. Information Processing and Management, 33: 133–144. Agosti, M., Melucci, M., et al. (1995). Automatic authoring and construction of hypertext for information retrieval. ACM Multimedia Systems, 3: 15–24. Albert, R., Jeong, H., et al. (1999). Diameter of the World Wide Web. Nature, 401: 130–131. Alfred, J. (2001). Mining the bibliome. Nature Reviews—Genetics, 2: 401. Allan, J. (1997). Building hypertext using information retrieval. Information Processing and Management, 33: 145–160. Allen, B. (1992). Cognitive differences in end-user searching of a CD-ROM index. Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Copenhagen, Denmark. ACM Press. 298–309. Allen, J. (1995). Natural Language Understanding. Redwood City, CA, BenjaminCummings. Almind, T. and Ingwersen, P. (1998). Informetric analyses on the World Wide Web: methodological approaches to ‘webometrics.’ Journal of Documentation, 53: 404–426. Altman, L. (1996). The Ingelfinger rule, embargoes, and journal peer review—Part 1. Lancet, 347: 1382–1386. Angell, M. and Kassirer, J. (1991). The Ingelfinger rule revisited. New England Journal of Medicine, 325: 1371–1373. Anonymous (1989). International Publishers Association (IPA) resolution on use of permanent papers in published works. International Publishers’ Bulletin, 5: 9–10. 450
References
451
Anonymous (1992). Evidence-based medicine: a new approach to teaching the practice of medicine. Evidence-Based Medicine Working Group. Journal of the American Medical Association, 268: 2420–2425. Anonymous (1997a). Policies for posting biomedical journal information on the Internet. Journal of the American Medical Association, 277: 808. Anonymous (1997b). Search engine road test: top three returns. Medicine on the Net, 3(4): 3. Anonymous (1997c). North American Health Claim Surf Day Targets Internet Ads Hundreds of E-Mail Messages Sent. Federal Trade Commission. http://www.ftc.gov/opa/ 1997/9711/hlthsurf.htm. Accessed: July 1, 2002. Anonymous (1998). Quality First: Better Health Care for All Americans. Dept. of Health & Human Services. http://www.hcqualitycommission.gov/final/. Accessed: July 1, 2002. Anonymous (1999). Dublin Core Metadata Element Set, Version 1.1: Reference Description. Dublin Core Metadata Initiative. http://www.dublincore.org/documents/dces/. Accessed: July 1, 2002. Anonymous (2000a). Cataloging Practices. National Library of Medicine. www.nlm.nih. gov/mesh/catpractices.html. Accessed: July 1, 2002. Anonymous (2000b). DCMI Type Vocabulary. Dublin Core Metadata Initiative. http://dublincore.org/documents/dcmi-type-vocabulary/. Accessed: July 1, 2002. Anonymous (2000c). Dublin Core Qualifiers. Dublin Core Metadata Initiative. http:// dublincore.org/documents/dcmes-qualifiers/. Accessed: July 1, 2002. Anonymous (2000d). Features of the MeSH Vocabulary. National Library of Medicine. www.nlm.nih.gov/mesh/features.html. Accessed: July 1, 2002. Anonymous (2000e). Gene Ontology: tool for the unification of biology. Nature Genetics, 25: 25–29. Anonymous (2000f). Organization of National Library of Medicine Bibliographic Databases. National Library of Medicine. www.nlm.nih.gov/pubs/techbull/mj00/mj00_buckets. html. Accessed: July 1, 2002. Anonymous (2000g). Programs and Services Annual Report, FY2000. National Library of Medicine. http://www.nlm.nih.gov/ocpl/anreports/fy2000.pdf. Accessed: July 1, 2002. Anonymous (2001a). AMA survey finds upsurge in physician usage and regard for Internet. American Medical Association. http://www.ama-assn.org/ama/pub/article/16164692.html. Accessed: July 1, 2002. Anonymous (2001b). Consumer watchdog accuses search engines of deception. Associated Press. July 16, 2001. Anonymous (2001c). Creating the Gene Ontology resource: design and implementation. Genome Research, 11: 1425–1433. Anonymous (2001d). Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC, National Academy Press. Anonymous (2001e). Digital Libraries: Universal Access to Human Knowledge. President’s Information Technology Advisory Committee. http://www.ccic.gov/pubs/pitac/ pitac-dl-9feb01.pdf. Accessed: July 1, 2002. Anonymous (2001f). The Dublin Core Metadata Element Set. Dublin Core Metadata Initiative. http://www.niso.org/standards/resources/Z39-85.pdf. Accessed: July 1, 2002. Anonymous (2001g). The future of the electronic scientific literature. Nature, 413: 1–3. Anonymous (2001h). Initial sequencing and analysis of the human genome. Nature, 409: 860–921. Anonymous (2001i). Linking to PubMed and other Entrez databases. National Library
452
References
of Medicine. http://www.ncbi.nlm.nih.gov/entrez/query/static/linking.html. Accessed: July 1, 2002. Anonymous (2001j). Longman Dictionary of Contemporary English. New York, Longman Publishing. Anonymous (2001k). Medical Subject Headings—Annotated Alphabetic List. Springfield, VA, National Technical Information Service. Anonymous (2001l). Medical Subject Headings—Tree Structures. Springfield, VA, National Technical Information Service. Anonymous (2001m). Permuted Medical Subject Headings. Springfield, VA, National Technical Information Service. Anonymous (2001n). PubMed Help. National Library of Medicine. http://www.ncbi. nlm.nih.gov/entrez/query/static/help/pmhelp.html. Accessed: July 1, 2002. Anonymous (2001o). Thesaurus of Psychological Index Terms, 9th ed. Washington, DC, American Psychological Association. Anonymous (2001p). Transforming Health Care Through Information Technology. President’s Information Technology Advisory Committee. http://www.ccic.gov/pubs/pitac/ pitac-hc-9feb01.pdf. Accessed: July 1, 2002. Anonymous (2001q). Search and Retrieval. Infoconomy. http://www.infoconomy.com. Accessed: July 1, 2002. Anonymous (2001r). Uniform Requirements for Manuscripts Submitted to Biomedical Journals. International Committee of Medical Journal Editors. http://www.icmje.org. Accessed: July 1, 2002. Anonymous (2001s). Using Information Technology to Transform the Way We Learn. President’s Information Technology Advisory Committee. http://www.ccic.gov/pubs/ pitac/pitac-tl-9feb01.pdf. Accessed: July 1, 2002. Anonymous (2001t). West Coast Cities Hit 70 Percent Internet Penetration, According to Nielsen//NetRatings; Portland, Seattle and San Francisco are the Most Wired Cities in the U.S. Nielsen//NetRatings. http://www.nielsennetratings.com/pr/pr 010403.pdf. Accessed: July 1, 2002. Anonymous (2001u). What’s the Difference Between MEDLINE and PubMed? National Library of Medicine. http://www.nlm.nih.gov/pubs/factsheets/dif_med_pub.html. Accessed: July 1, 2002. Anonymous (2001v). XML in 10 points. World Wide Web Consortium. http://www.w3. org/XML/1999/XML-in-10-points. Accessed: July 1, 2002. Anonymous (2001w). Virtual “Treatments” Can Be Real-World Deceptions. Federal Trade Commission. http://www.ftc.gov/bcp/conline/pubs/alerts/mrclalrt.htm. Accessed: July 1, 2002. Anonymous (2002a). ACP-ASIM Survey Finds Nearly Half of U.S. Members Use Handheld Computers. American College of Physicians. http://www.acponline.org/college/ pressroom/handheld_survey.htm. Accessed: July 1, 2002. Anonymous (2002b). Information Interfaces. Xerox PARC User Interface Research Group. http://www2.parc.com/ist1/projects/uir/projects/ii.html. Accessed: July 1, 2002. Anonymous (2002c). The Handle System. Corporation for National Research Initiatives. http://www.handle.net/documentation.html. Accessed: July 1, 2002. Anonymous (2002d). Invisible Web: What it is, Why it exists, How to find it, and its inherent ambiguity. University of California Berkeley Library. http://www.lib.berkeley. edu/TeachingLib/Guides/Internet/InvisibleWeb.html. Accessed: July 1, 2002. Anonymous (2002e). 2002 UMLS Documentation. National Library of Medicine. http:// www.nlm.nih.gov/research/umls/UMLSDOC.HTML. Accessed: July 1, 2002.
References
453
Anonymous (2002f). UMLS Applications. National Library of Medicine. www.nlm.nih. gov/research/umls/umlsapps.html. Accessed: July 1, 2002. Antman, E., Lau, J., et al. (1992). A comparison of results of meta-analyses of randomized controlled trials and recommendations of clinical experts: treatments for myocardial infarction. Journal of the American Medical Association, 268: 240– 248. Archbold, A. and Evans, D. (1989). On the topical structure of medical charts. Proceedings of the 13th Annual Symposium on Computer Applications in Medical Care. Washington, DC. Piscataway, NJ, IEEE Press, 543–547. Aridor, Y., Carmel, D., et al. (2002). Knowledge encapsulation for focused search from prevasive devices. ACM Transactions on Information Systems, 20: 25–46. Arms, W. (2000). Digital Libraries. Cambridge, MA, MIT Press. Arms, W., Hillmann, D., et al. (2002). A spectrum of interoperability: the Site for Science prototype for the NSDL. D-Lib Magazine, 8. http://www.dlib.org/dlib/january02/ arms/01arms.html. Aronow, D., Cooley, J., et al. (1995). Automated identification of episodes of asthma exacerbation for quality measurement in a computer-based medical record. Proceedings of the 18th Annual Symposium on Computer Applications in Medical Care, New Orleans, LA. Hanley & Belfus. 309–313. Aronow, D., Fangfang, F., et al. (1999). Ad hoc classification of radiology reports. Journal of the American Medical Informatics Association, 6: 393–341. Aronow, D., Soderland, S., et al. (1995). Automated classification of encounter notes in a computer based medical record. MEDINFO 95, Vancouver, BC. Healthcare Computing & Communications Canada, 8–12. Aronson, A. (1996). MetaMap: Mapping Text to the UMLS Metathesaurus. National Library of Medicine. http://ii.nlm.nih.gov/resources/metamap.pdf. Accessed: July 1, 2002. Aronson, A. (2001). Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. Proceedings of the 2001 AMIA Annual Symposium, Washington, DC. Hanley & Belfus. 17–21. Aronson, A., Bodenreider, O., et al. (2000). The NLM Indexing Initiative. Proceedings of the AMIA 2000 Annual Symposium, Los Angeles, CA. Hanley & Belfus. 17–21. Aronson, A. and Rindflesch, T. (1997). Query expansion using the UMLS Metathesaurus. Proceedings of the 1997 AMIA Annual Fall Symposium, Nashville, TN. Hanley & Belfus. 485–489. Ashburner, M., Ball, C., et al. (2000). Gene Ontology: tool for the unification of biology. Nature Genetics, 25: 25–29. Avenell, A., Handoll, H., et al. (2001). Lessons for search strategies from a systematic review, in The Cochrane Library, of nutritional supplementation trials in patients after hip fraction. American Journal of Clinical Nutrition, 73: 505–510. Bachrach, C. and Charen, T. (1978). Selection of MEDLINE contents, the development of its thesaurus, and the indexing process. Medical Informatics, 3: 237–254. Baclawski, K., Cigna, J., et al. (2000). Knowledge representation and indexing using the unified medical language system. Pacific Symposium on Biocomputing, Honolulu, HI. World Scientific. 493–504. Badgett, R., Paukert, J., et al. (2001). Teaching clinical informatics to third-year medical students: negative results from two controlled trials. Biomed Central Education, 1: 3. http://www.biomedcentral.com/1472-6920/1/3.
454
References
Baeza-Yates, R. and Ribeiro-Neto, B., eds. (1999). Modern Information Retrieval. New York, McGraw-Hill. Bahl, L., Jelineck, F., et al. (1983). A maximum-likelihood approach to continued speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI5: 179–190. Bailar, J. (1986). Science, statistics, and deception. Annals of Internal Medicine, 104: 259–260. Bakken, S., Cashen, M., et al. (2000). Representing nursing activities within a conceptoriented terminological system: evaluation of a type definition. Journal of the American Medical Informatics Association, 7: 81–90. Balk, E., Bonis, P., et al. (2002). Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. Journal of the American Medicial Association, 287: 2973–2982. Ballesteros, L. and Croft, W. (1997). Phrasal translation and query expansion techniques for cross-language information retrieval. Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Philadelphia, PA. ACM Press. 84–91. Bandemer, J. and Tannery, N. (1998). A comparison of four current awareness services. Medical Reference Services Quarterly, 17(2): 29–36. Baorto, D. and Cimino, J. (2000). An “infobutton” for enabling patients to interpret online Pap Smear reports. Proceedings of the AMIA 2000 Annual Symposium, Los Angeles, CA. Hanley & Belfus. 47–51. Bar-Hillel, Y. and Carnap, R. (1953). Semantic information. British Journal for the Philosophy of Science, 4: 147–157. Barnett, G., Cimino, J., et al. (1987). DXplain: an evolving diagnostic decision-support system. Journal of the American Medical Association, 258: 67–74. Barnett, G., Justice, N., et al. (1979). COSTAR—A computer-based medical information system for ambulatory care. Proceedings of the IEEE, 67: 1226–1237. Barr, C., Komorowski, H., et al. (1988). Conceptual modeling for the Unified Medical Language System. Proceedings of the 12th Annual Symposium on Computer Applications in Medical Care, Washington, DC. IEEE. 148–151. Barrows, R., Busuioc, M., et al. (2000). Limiting parsing of notational text visit notes: ad-hoc vs. NLP approaches. Proceedings of the AMIA 2000 Annual Symposium, Los Angeles, CA. Hanley & Belfus. 51–55. Barry, C. (1994). User-defined relevance criteria: an exploratory study. Journal of the American Society for Information Science, 45: 149–159. Bartell, B., Cottrell, G., et al. (1994). Automatic combination of multiple ranked retrieval systems. Proceedings of the 17th Annual International SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland. Springer-Verlag. 173–181. Bates, D., Leape, L., et al. (1998). Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. Journal of the American Medical Association, 280: 1311–1316. Bates, M. (2002). After the dot-bomb: getting Web information right this time. First Monday, 7: 7. http://firstmonday.org/issues/issue7_7/bates Baud, R., Lovins, C., et al. (1998). Morpho-semantic parsing of medical expressions. Proceedings of the AMIA 1998 Annual Symposium, Orlando, FL. Hanley & Belfus. 760–764. Baud, R., Rassinoux, A., et al. (1999). The power and limits of a rule-based morphosemantic parser. Proceedings of the AMIA 1999 Annual Symposium, Washington, DC. Hanley & Belfus. 22–26.
References
455
Baxevanis, A. (2002). The Molecular Biology Database Collection: 2002 update. Nucleic Acids Research, 30: 1–12. Beagrie, N. (2002). An update on the Digital Preservation Coalition. D-Lib Magazine, 8. http://www.dlib.org/dlib/april02/beagrie/04beagrie.html. Bean, C., Sneiderman, C., et al. (1998). Automatic semantic interpretation of anatomic spatial relationships in clinical text. Proceedings of the AMIA 1998 Annual Symposium, Orlando, FL. Hanley & Belfus. 897–901. Beaulieu, M., Robertson, S., et al. (1996). Evaluating interactive systems in TREC. Journal of the American Society for Information Science, 47: 85–94. Beckett, D., Miller, E., et al. (2000). Using Dublin Core in XML. Dublin Core Metadata Initiative. http://dublincore.org/documents/dcmes-xml/. Accessed: July 1, 2002. Belew, R. (2000). Finding Out About—A Cognitive Perspective on Search Engine Technology and the WWW. Cambridge, U.K., Cambridge University Press. Belis, M. and Guiasu, S. (1968). A quantitative–qualitative measure of information in cybernetic systems. IEEE Transactions on Information Theory, IT-14: 593–594. Belkin, N. (2000). Helping people find what they don’t know. Communications of the ACM, 43: 58–61. Belkin, N., Cool, C., et al. (2000). Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval. Information Processing and Management, 37: 403–434. Belkin, N. and Croft, W. (1992). Information filtering and information retrieval: two sides of the same coin? Communications of the ACM, 35: 29–38. Belkin, N., Keller, A., et al. (2000). Support for question-answering in interactive information retrieval: Rutger’s TREC-9 interactive track experience. The Ninth Text REtrieval Conference (TREC-9), Gaithersburg, MD. National Institute of Standards and Technology. 463–474. Belkin, N., Oddy, R., et al. (1978). ASK for information retrieval. I. Background and theory. Journal of Documentation, 38: 61–71. Belkin, N. and Vickery, A. (1985). Interaction in the Information System: A Review of Research from Document Retrieval to Knowledge-Based System. London, The British Library. Bell, D. and Greenes, R. (1994). Evaluation of UltraSTAR: performance of a collaborative structured data entry system. Proceedings of the 18th Annual Symposium on Computer Applications in Medical Care, Washington, DC. Hanley & Belfus. 216–222. Benson, D., Karsch-Mizrachi, I., et al. (2002). GenBank. Nucleic Acids Research, 30: 17–20. Benson, K. and Hartz, A. (2000). A comparison of observational studies and randomized, controlled trials. New England Journal of Medicine, 342: 1878–1886. Berger, A. and Lafferty, J. (1999). Information retrieval as statistical translation. Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Berkeley, CA. ACM Press. 222–229. Berger, A. and Mittal, V. (2000). OCELOT: a system for summarizing Web pages. Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Athens, Greece. ACM Press. 144–151. Bergus, G., Sinift, S., et al. (1998). Use of an e-mail curbside consultation service by family physicians. Journal of Family Practice, 47: 357–360. Berland, G., Elliott, M., et al. (2001). Health information on the Internet: accessibility, quality, and readability in English and Spanish. Journal of the American Medical Association, 285: 2612–2621.
456
References
Berman, H., Westbrook, J., et al. (2000). The protein data bank. Nucleic Acids Research, 28: 235–242. Bernstein, L., Barriere, S., et al. (1982). Utilization of antibiotics: analysis of appropriateness of use. Annals of Emergency Medicine, 11: 21–24. Bernstein, L., Siegel, E., et al. (1980). The hepatitis knowledge base: a prototype information transfer system. Annals of Internal Medicine, 93: 169–181. Bernstein, L. and Williamson, R. (1984). Testing of a natural language retrieval system for a full text knowledge base. Journal of the American Society for Information Science, 35: 235–247. Berry, E., Kelly, S., et al. (2000). Identifying studies for systematic reviews. An example from medical imaging. International Journal of Technology Assessment in Health Care, 16: 668–672. Besser, H. (2002). The next stage: moving from isolated digital collections to interoperable digital libraries. First Monday, 7: 6. http://www.firstmonday.dk/issues/issue7_6/besser/. Bickmore, T. and Schilit, B. (1997). Digestor: device-independent access to the World Wide Web. Stanford University. http://www.scope.gmd.de/info/www6/technical/paper177/ paper177.html. Accessed: July 1, 2002. Biermann, J., Golladay, G., et al. (1999). Evaluation of cancer information on the Internet. Cancer, 86: 381–390. Black, N., van Rooyen, S., et al. (1998). What makes a good reviewer and a good review for a general medical journal? Journal of the American Medical Association, 280: 231–233. Blair, D. (1980). Searching biases in large interactive document retrieval systems. Journal of the American Society for Information Science, 31: 271–277. Blair, D. (2002). Some thoughts on the reported results of TREC. Information Processing and Management, 38: 445–451. Blair, D. and Maron, M. (1985). An evaluation of retrieval effectiveness for a full-text document-retrieval system. Communications of the ACM, 28: 289–299. Blanchi, C. and Petrone, J. (2001). Distributed interoperable metadata registry. D-Lib Magazine, 7. http://www.dlib.org/dlib/december01/blanchi/12blanchi.html. Bland, J. and Altman, D. (2000). The odds ratio. British Medical Journal, 320: 1468. Blaschke, C., Andrade, M., et al. (1999). Automatic extraction of biological information from scientific text: protein-protein interactions. Proceedings of the International Conference on Intelligent Systems in Molecular Biology. 60–67. Blum, B. (1984). Information Systems for Patient Care. New York, Springer-Verlag. Blythe, J. and Royle, J. (1993). Assessing nurses’ information needs in the work environment. Bulletin of the Medical Library Association, 8: 433–435. Bodenreider, O. and McCray, A. (1998). From French vocabulary to the Unified Medical Language System: a preliminary study. MEDINFO 98—Proceedings of the Ninth World Congress on Medical Informatics, Seoul, South Korea. IOS Press. 670–674. Bohannon, R. (1990). Information accessing behaviour of physical therapists. Physiotherapy Theory and Practice, 6: 212–225. Bordage, G. (2001). Reasons reviewers reject and accept manuscripts: the strengths and weaknesses in medical education reports. Academic Medicine, 76: 889–896. Borgman, C. (1986). Why are online catalogs hard to use? Lessons learned from information retrieval studies. Journal of the American Society for Information Science, 37: 387–400. Borgman, C. (1999). What are digital libraries? Competing visions. Information Processing and Management, 35: 227–244. Boyd, S. and Herkovic, A. (1999). Crisis in Scholarly Publishing: Executive Summary.
References
457
Stanford Academic Council Committee on Libraries. www.stanford.edu/boyd/schol_ pub_crisis.html. Accessed: July 1, 2002. Boynton, J., Glanville, J., et al. (1998). Identifying systematic reviews in MEDLINE: developing an objective approach to search strategy design. Journal of Information Science, 24: 137–157. Bradford, S. (1948). Documentation. London, Crosby Lockwood. Bradigan, P. and Mularski, C. (1989). End-user searching in a medical school curriculum: an evaluated modular approach. Bulletin of the Medical Library Association, 77: 348–353. Braschler, M., Harman, D. et al. (2000). The evaluation of systems for cross-language information retrieval. Proceedings of the Second International Conference on Language Resources and Evaluation, Athens, Greece. 1469–1474. Braschler, M. and Schauble, P. (2000). Using corpus-based approaches in a system for multilingual information retrieval. Information Retrieval, 3: 273–284. Brenner, S. and McKinin, E. (1989). CINAHL and MEDLINE: a comparison of indexing practices. Bulletin of the Medical Library Association, 77: 366–371. Brill, E. (1992). A simple rule-based part-of-speech tagger. Proceedings of ANLP-92, Third Conference on Applied Natural Language Processing, Trento, Italy. 152–155. Brill, E. and Mooney, R. (1997). An overview of empirical natural language processing. AI Magazine, 18: 13–24. Brin, S. and Page, L. (1998). The anatomy of a large-scale hypertextual Web search engine. Computer Networks, 30: 107–117. Broder, A., Kumar, R., et al. (2000). Graph structure in the Web. Computer Networks, 33: 309–320. Broglio, J., Callan, J., et al. (1994). Document retrieval and routing using the INQUERY system. Overview of the Third Text REtrieval Conference (TREC-3), Gaithersburg, MD. National Institute of Standards and Technology. 29–38. Buckley, C., Allan, J., et al. (1993). Automatic routing and ad-hoc retrieval using SMART: TREC-2. The Second Text REtrieval Conference (TREC-2), Gaithersburg, MD. National Institute of Standards and Technology. 45–56. Buckley, C., Mitra, M., et al. (2000). Using clustering and superconcepts within SMART: TREC 6. Information Processing and Management, 36: 109–131. Buckley, C. and Salton, G. (1995). Optimization of relevance feedback weights. Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA. ACM Press. 351–356. Buckley, C., Salton, G., et al. (1992). Automatic retrieval with locality information using SMART. The First Text REtrieval Conference (TREC-1), Gaithersburg, MD. National Institute of Standards and Technology. 59–72. Buckley, C., Salton, G., et al. (1994a). The effect of adding relevance information in a relevance feedback environment. Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland. Springer-Verlag. 292–300. Buckley, C., Salton, G., et al. (1994b). Automatic query expansion using SMART: TREC 3. Overview of the Third Text REtrieval Conference (TREC-3), Gaithersburg, MD. National Institute of Standards and Technology. 69–80. Buckley, C., Singhal, A., et al. (1996). Using query zoning and correlation with SMART: TREC 5. The Fifth Text REtrieval Conference (TREC-5), Gaithersburg, MD. National Institute of Standards and Technology. 105–118. Bunyan, L. and Lutz, E. (1991). Marketing the hospital library to nurses. Bulletin of the Medical Library Association, 79: 2131–2133.
458
References
Burger, J., Cardie, C., et al. (2000). Issues, Tasks and Program Structures to Roadmap Research in Question & Answering (Q&A). http://www-nlpir.nist.gov/projects/duc/papers/qa.Roadmap-paper_v2.doc. Accessed: July 1, 2002. Burgin, R. and Dillon, M. (1992). Improving disambiguation in FASIT. Journal of the American Society for Information Science, 43: 101–114. Burnham, J. (1990). The evolution of editorial peer review. Journal of the American Medical Association, 263: 1323–1329. Burnham, J. and Shearer, B. (1993). Comparison of CINAHL, EMBASE, and MEDLINE databases for the nurse researcher. Medical Reference Services Quarterly, 12(3): 45–57. Bush, V. (1945). As we may think. Atlantic Monthly, 176: 101–108. Bush, V. (1967). Science is Not Enough. New York, Morrow. Buyukkokten, O., Kaljuvee, O., et al. (2002). Efficient Web browsing on handheld devices using page and form summarization. ACM Transactions on Information Systems, 20: 82–115. Bykowski, J., Alora, M., et al. (2000). Accessibility and reliability of cutaneous laser surgery information on the World Wide Web. Journal of the American Academy of Dermatology, 42: 784–786. Cabana, M., Rand, C., et al. (1999). Why don’t physicians follow clinical practice guidelines? A framework for improvement. Journal of the American Medical Association, 282: 1458–1465. Callaham, M., Baxt, W., et al. (1998). Reliability of editors’ subjective quality ratings of peer reviews of manuscripts. Journal of the American Medical Association, 280: 229–231. Callaham, M., Wears, R., et al. (2002). Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. Journal of the American Medical Association, 287: 2847–2850. Callaham, M., Wears, R., et al. (1998). Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. Journal of the American Medical Association, 280: 254–257. Callan, J. (1994). Passage level evidence in document retrieval. Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland. Springer-Verlag. 302–310. Callan, J., Croft, W., et al. (1995). TREC and TIPSTER experiments with INQUERY. Information Processing and Management, 31: 315–326. Campbell, J., Carpenter, P., et al. (1997). Phase II evaluation of clinical coding schemes: completeness, taxonomy, mapping, definitions, and clarity. Journal of the American Medical Informatics Association, 4: 238–250. Campbell, K. and Musen, M. (1992). Representation of clinical data using SNOMED III and conceptual graphs. Proceedings of the 16th Annual Symposium on Computer Applications in Medical Care, Baltimore, MD. McGraw-Hill. 354–358. Campbell, K., Wieckert, K., et al. (1993). A computer-based tool for generation of progress notes. Proceedings of the 17th Annual Symposium on Computer Applications in Medical Care, Washington, DC. McGraw-Hill. 284–288. Cardie, C. (1997). Empirical methods in information extraction. AI Magazine, 18: 65–80. Carroll, A., Tarczy-Hornoch, P., et al. (2001). Development of a personal digital assistant (PDA) based client/server NICU patient data and charting system. Proceedings of the 2001 AMIA Annual Symposium, Washington, DC. Hanley & Belfus. 100–104. Carroll, J., Davies, P., et al., eds. (1971). The American Heritage Word-Frequency Book. New York, American Heritage Publishing.
References
459
Caruso, D. (2000). Digital Commerce; If the AOL–Time Warner deal is about proprietary content, where does that leave a noncommercial directory it will own? New York Times. January 17, 2000. Chalmers, I. (1990). Under-reporting scientific research is scientific misconduct. Journal of the American Medical Association, 263: 1405–1408. Chang, J., Raychaudhuri, S., et al. (2001). Including biological literature improves homology search. Pacific Symposium on Biocomputing, Kauai, HI. World Scientific. 374–383. Chapman, W., Fiszman, M., et al. (2001). A comparison of classification algorithms to automatically identify chest x-ray reports that support pneumonia. Journal of Biomedical Informatics, 34: 4–14. Charen, T. (1976). MEDLARS Indexing Manual, Part I, Bibliographic Principles and Descriptive Indexing, 1977. Springfield, VA, National Technical Information Service. Charen, T. (1983). MEDLARS Indexing Manual, Part II. Springfield, VA, National Technical Information Service. Charniak, E. (1997). Statistical techniques for natural language parsing. AI Magazine, 18: 33–44. Chen, E. and Cimino, J. (2001). Use of wireless technology for reducing medical errors. Proceedings of the 2001 AMIA Annual Symposium, Washington, DC. Hanley & Belfus. 877. Chen, H., Houston, A., et al. (1998). Internet browsing and searching: user evaluations of category map and concept space techniques. Journal of the American Society for Information Science, 49: 582–608. Cheng, D., Melnikow, J., et al. (2001). Reporting number needed to treat and number needed to harm in randomized controlled trials. Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain. American Medical Association. http://www.ama-assn.org/public/peer/peerhome.htm. Chin, J., Diehl, V., et al. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. Proceedings of CHI ‘88—Human Factors in Computing Systems, New York. ACM Press. 213–218. Chomsky, N. (1965). Aspects of the Theory of Syntax. Cambridge, MA, MIT Press. Choudhry, N., Stelfox, H., et al. (2002). Relationships between authors of clinical practice guidelines and the pharmaceutical industry. Journal of the American Medical Association, 287: 612–617. Chu, H. and Rosenthal, M. (1996). Search engines for the World Wide Web: a comparative study and evaluation methodology. Proceedings of the ASIS Annual Meeting 1996, Baltimore, MD. 127–135. Chu, W., Johnson, D., et al. (2000). A medical digital library to support scenario and usertailored information retrieval. IEEE Transactions on Information Technology in Biomedicine, ITB-4: 97–107. Chueh, H. and Barnett, G. (1997). “Just-in-time” clinical information. Academic Medicine, 72: 512–517. Chute, C., Cohn, S., et al. (1996). The content coverage of clinical classifications. For the Computer-Based Patient Record Institute’s Work Group on Codes & Structures. Journal of the American Medical Informatics Association, 3: 224–233. Cimino, J. (1996). Linking patient information systems to bibliographics resources. Methods of Information in Medicine, 35: 122–126. Cimino, J. (1998a). Auditing the UMLS with semantic methods. Journal of the American Medical Informatics Association, 5: 41–51. Cimino, J. (1998b). Desiderata for controlled medical vocabularies in the twenty-first century. Methods of Information in Medicine, 37: 394–403.
460
References
Cimino, J., Aguirre, A., et al. (1993). Generic queries for meeting clinical information needs. Bulletin of the Medical Library Association, 81: 195–206. Cimino, J. and Barnett, G. (1987). The physician’s workstation: recording a physical examination using a controlled vocabulary. Proceedings of the 11th Annual Symposium on Computer Applications in Medical Care, Washington, DC. IEEE. 287–291. Cimino, J., Clayton, P., et al. (1994). Knowledge-based approaches to the maintenance of a large controlled medical terminology. Journal of the American Medical Informatics Association, 1: 35–50. Cimino, J., Hripcsak, G., et al. (1989). Designing an introspective, multipurpose, controlled medical vocabulary. Proceedings of the 13th Annual Symposium on Computer Applications in Medical Care. IEEE. 513–517. Cimino, J., Johnson, S., et al. (1992). The MEDLINE button. Proceedings of the 16th Annual Symposium on Computer Applications in Medical Care, Baltimore, MD. McGrawHill. 81–85. Cimino, J., Socratorus, S., et al. (1995). Internet as clinical information system: application development using the World Wide Web. Journal of the American Medical Informatics Association, 2: 273–284. Clarke, M., Alderson, P., et al. (2002). Discussion sections in reports of controlled trials published in general medical journals. Journal of the American Medical Association, 287: 2799–2801. Cleverdon, C. and Keen, E. (1966). Factors determining the performance of indexing systems (Vol. 1: Design, Vol. 2: Results). Cranfield, England, Aslib Cranfield Research Project. Clifford, T., Moher, D., et al. (2001). Absence of associations between funding source, trial outcome, and quality score: a benefit of financial disclosure. Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain. American Medical Association. http://www.ama-assn.org/public/peer/peerhome.htm. Clyman, J., Powsner, S., et al. (1993). Using a network menu and the UMLS Information Sources Map to facilitate access to online reference materials. Bulletin of the Medical Library Association, 81: 207–216. Cochrane, A. (1972). Effectiveness and Efficiency: Random Reflections on Health Services. London, Nuffield Provincial Hospital Trust. Cohen, A. (2001). You pay, you play. Time: 50–51. July 23, 2001. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20: 37–46. Cohen, J. (2001). Dose discrepancies between the Physicians’ Desk Reference and the medical literature, and their possible role in the high incidence of dose-related adverse drug events. Archives of Internal Medicine, 161: 957–964. Colaianni, L. (1992). Retraction, comment, and errata policies of the U.S. National Library of Medicine. Lancet, 340: 536–537. Coletti, M. and Bleich, H. (2001). Medical subject headings used to search the biomedical literature. Journal of the American Medical Informatics Association, 8: 317–323. Collen, M. and Flagle, C. (1985). Full-text medical literature retrieval by computer: a pilot test. Journal of the American Medical Association, 254: 2768–2774. Collins, F. and McKusick, V. (2001). Implications of the Human Genome Project for medical science. Journal of the American Medical Association, 285: 540–544. Concato, J., Shah, N., et al. (2000). Randomized, controlled trials, observational studies, and the hierarchy of research designs. New England Journal of Medicine, 342: 1887–1892. Conklin, J. (1987). Hypertext: an introduction and survey. IEEE Computer, 20: 17–41.
References
461
Connelly, D., Rich, E., et al. (1990). Knowledge resource preferences of family physicians. Journal of Family Practice, 30: 353–359. Cook, D., Mulrow, C., et al. (1997). Systematic reviews: synthesis of best evidence for clinical decisions. Annals of Internal Medicine, 126: 389–391. Cooper, B. (2000). Searching the Internet. London, Dorling, Kindersley. Cooper, G. and Miller, R. (1998). An experiment comparing lexical and statistical methods for extracting MeSH terms from clinical free text. Journal of the American Medical Informatics Association, 5: 62–75. Cooper, W. (1973). On selecting a measure of retrieval effectiveness. Journal of the American Society for Information Science, 24: 87–100. Cooper, W. (1994). The formalism of probability theory in IR: a foundation or an encumbrance? Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland. Springer-Verlag. 242–247. Cooper, W., Chen, A., et al. (1994). Experiments in the probabilistic retrieval of documents. Overview of the Third Text REtrieval Conference (TREC-3), Gaithersburg, MD. National Institute of Standards and Technology. 127–134. Corcoran-Perry, S. and Graves, J. (1990). Supplemental information-seeking behavior of cardiovascular nurses. Research in Health and Nursing, 13: 119–127. Cote, R., Rothwell, D., et al., eds. (1993). SNOMED International—Introduction. Northfield, IL, College of American Pathologists. Covell, D., Uman, G., et al. (1985). Information needs in office practice: are they being met? Annals of Internal Medicine, 103: 596–599. Cowie, J. and Lehnert, W. (1996). Information extraction. Communications of the ACM, 39: 80–91. Coyle, F. (2002). XML, Web Services, and the Data Revolution. Boston. Addison-Wesley. Crain, C. (1987). Appendix A—Protocol study of indexers at the National Library of Medicine. In Carbonell, J., Evans, D., Scott, D. and Thomason, R. Final Report on the Automated Classification Retrieval Project, Grant N01-LM-4-3529. Bethesda, MD. National Library of Medicine. Criswell, D. and Parchman, M. (2002). Handheld computer use in U.S. family practice residency programs. Journal of the American Medical Informatics Association, 9: 80–86. Crocco, A., Villasis-Keever, M., et al. (2002). Analysis of cases of harm associated with use of health information on the Internet. Journal of the American Medical Association, 287: 2869–2871. Croft, W. (1986). Boolean queries and term dependencies in probabilistic retrieval models. Journal of the American Society for Information Science, 37: 71–77. Croft, W. and Harper, D. (1979). Using probabilistic models of document retrieval without relevance ranking. Journal of Documentation, 35: 285–295. Cuadra, C. and Katter, R. (1967). Experimental Studies of Relevance Judgments. Santa Monica, CA, Systems Development Corp. Cummings, A., Witte, M., et al. (1992). University Libraries and Scholarly Communication: A Study Prepared for the Andrew W. Mellon Foundation. Association of Research Libraries. http://etext.lib.virginia.edu/subjects/mellon/. Accessed: July 1, 2002. Curley, S., Connelly, D., et al. (1990). Physicians use of medical knowledge resources: preliminary theoretical framework and findings. Medical Decision Making, 10: 231–241. Cutting, D., Pedersen, J., et al. (1992). Scatter/Gather: a cluster-based approach to browsing large document collections. Proceedings of the 15th Annual International ACM
462
References
SIGIR Conference on Research and Development in Information Retrieval, Copenhagen, Denmark. ACM Press. 318–329. D’Agostino, R. (2000). Debate: the slippery slope of surrogate outcomes. Controlled Clinical Trials in Cardiovascular Medicine, 1: 76–78. Darmoni, S., Leroy, J., et al. (2000). CISMeF: a structured health resource guide. Methods of Information in Medicine, 9: 30–35. Davidoff, F., DeAngelis, C., et al. (2001). Sponsorship, authorship, and accountability. Annals of Internal Medicine, 135: 463–466. Davidoff, F. and Florance, V. (2000). The informationist: a new health profession? Annals of Internal Medicine, 132: 996–998. Davis, D., Thompson, M., et al. (1995). Changing physician performance: a systematic review of the effect of continuing medical education strategies. Journal of the American Medical Association, 274: 700–705. Davis, M. and Ogden, W. (1997). Free resources and advanced alignment for crosslanguage text retrieval. The Sixth Text REtrieval Conference (TREC-6), Gaithersburg, MD. National Institute of Standards and Technology. 385–402. Davison, K. (1997). The quality of dietary information on the World Wide Web. Clinical Performance and Quality Health Care, 5: 64–66. DeBakey, M. (1991). The National Library of Medicine: evolution of a premier information center. Journal of the American Medical Association, 266: 1252–1258. de Bliek, R., Friedman, C., et al. (1994). Information retrieved from a database and the augmentation of personal knowledge. Journal of the American Medical Informatics Association, 1: 328–338. Dee, C. and Blazek, R. (1993). Information needs of the rural physician: a descriptive study. Bulletin of the Medical Library Association, 81: 259–264. Deerwester, S., Dumais, S., et al. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41: 391–407. de Estrada, W., Murphy, S., et al. (1997). Puya: a method of attracting attention to relevant physical findings. Proceedings of the 1997 AMIA Annual Fall Symposium, Nashville, TN. Hanley & Belfus. 509–513. De Jong, G. (1979). Prediction and substantiation: a new approach to natural language processing. Cognitive Science, 3: 251–273. De Rose, S. (1989). Expanding the notion of links. Hypertext ‘89 Proceedings, Pittsburgh. New York, ACM Press, 249–257. Der Simonian, R., Charette, L., et al. (1982). Reporting on methods in clinical trials. New England Journal of Medicine, 306: 1332–1337. Detlefsen, E. (2002). The education of informationists, from the perspective of a library and information sciences educator. Journal of the Medical Library Association, 90: 59–67. Detmer, W., Barnett, G., et al. (1997). MedWeaver: integrating decision support, literature searching, and Web exploration using the UMLS Metathesaurus. Proceedings of the 1997 AMIA Annual Fall Symposium, Nashville, TN. Hanley & Belfus. 490–494. Devereaux, P., Manns, B., et al. (2001). Reviewing the reviewers: the quality of reporting in three secondary journals. Canadian Medical Association Journal, 614: 1573–1576. Dickersin, K. (1990). The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association, 263: 1385–1389. Dickersin, K. and Min, Y. (1993). Publication bias: a problem that won’t go away. Annals of the New York Academy of Sciences, 703: 135–148. Dickersin, K., Scherer, R., et al. (1994). Identifying relevant studies for systematic reviews. British Medical Journal, 309: 1286–1291.
References
463
Dillon, A., Richardson, J., et al. (1990). The effect of display size and text splitting on reading lengthy text from the screen. Behaviour and Information Technology, 9: 215–227. Dillon M. and Gray, A. (1983). FASIT: a fully automatic syntactically based indexing system. Journal of the American Society for Information Science, 34: 99–108. Ding, J., Berleant, D., et al. (2002). Mining MEDLINE: abstracts, sentences, or phrases? Pacific Symposium on Biocomputing, Kauai, Hawaii. World Scientific. 326–337. Ding, W. and Marchionini, G. (1996). A comparative study of Web search service performance. Proceedings of the ASIS Annual Meeting 1996, Baltimore, MD. Information Today. 136–142. Dolin, R., Alschuler, L., et al. (2001). The HL7 Clinical Document Architecture. Journal of the American Medical Informatics Association, 8: 552–569. Dorland, W., ed. (2000). Dorland’s Illustrated Medical Dictionary. Philadelphia, Saunders. Doupi, P. and van der Lei, J. (1999). Rx medication information for the public and the WWW: quality issues. Medical Informatics and the Internet in Medicine, 24: 171–179. Duchnicky, R. and Kolers, P. (1983). Readability of text scrolled on visual display terminals as a function of window size. Human Factors, 25: 683–692. Duggan, D., Bittner, M., et al. (1999). Expression profiling using cDNA microarrays. Nature Genetics, 21 (suppl): 10–14. Dumais, S. (1994). Latent semantic indexing (LSI): TREC-3 Report. Overview of the Third Text REtrieval Conference (TREC-3), Gaithersburg, MD. National Institute of Standards and Technology. 219–230. Dumais, S. and Schmitt, D. (1991). Iterative searching in an online database. Proceedings of the Human Factors Society 35th Annual Meeting. 398–403. Duncan, R. and Shabot, M. (2000). Secure remote access to a clinical data repository using a wireless personal digital assistant (PDA). Proceedings of the AMIA 2000 Annual Symposium, Los Angeles, CA. Hanley & Belfus. 210–214. Duval, E., Hodgins, W., et al (2002). Metadata principles and practicalities. D-Lib Magazine, 8. http://www.dlib.org/dlib/april02/weibel/04weibel.html. Earley, J. (1970). An efficient context-free parsing algorithm. Communications of the ACM, 13: 94–102. Egan, D., Remde, J., et al. (1989). Formative design-evaluation of Superbook. ACM Transactions on Information Systems, 7: 30–57. Egger, M., Smith, G., et al. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315: 629–634. Egger, M., Zellweger-Zahner, T., et al. (1997). Language bias in randomised controlled trials published in English and German. Lancet, 350: 326–329. Eichmann, D., Ruiz, M., et al. (1998). Cross-language information retrieval with the UMLS Metathesaurus. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia. ACM Press. 72–80. Eisenberg, M. (1988). Measuring relevance judgments. Information Processing and Management, 24: 373–389. Eisenberg, M. and Barry, C. (1988). Order effects: a study of the possible influence of presentation order on user judgments of document relevance. Journal of the American Society for Information Science, 39: 293–300. Ekstrom, R., French, J., et al. (1976). Manual for Kit of Factor-Referenced Cognitive Tests. Princeton, NJ, Educational Testing Service. Elkins, J., Friedman, C., et al. (2000). Coding neuroradiology reports for the Northern Manhattan Stroke Study: a comparison of natural language processing and manual review. Computers and Biomedical Research, 33: 1–10.
464
References
Ellerbeck, E., Jencks, S., et al. (1995). Quality of care for Medicare patients with acute myocardial infarction: a four-state pilot study from the Cooperative Cardiovascular Project. Journal of the American Medical Association, 273: 1509–1514. Elstein, A., Shulman, L., et al. (1978). Effects of Hypothesis Generation and Thinking Aloud, 228–251, in Elstein, A., Shulman, L. and Sprafka, S., eds. Medical Problem Solving: An Analysis of Clinical Reasoning. Cambridge, MA. Harvard University Press. Elstein, A., Shulman, L., et al. (1978b). Medical Problem Solving: An Analysis of Clinical Reasoning. Cambridge, MA, Harvard University Press. Ely, J., Burch, R., et al. (1992). The information needs of family physicians: case-specific clinical questions. Journal of Family Practice, 35: 265–269. Ely, J., Levy, B., et al. (1999). What clinical information resources are available in family physicians’ offices? Journal of Family Practice, 48: 135–139. Ely, J., Osheroff, J., et al. (1999). Analysis of questions asked by family doctors regarding patient care. British Medical Journal, 319: 358–361. Ely, J., Osheroff, J., et al. (2000). A taxonomy of generic clinical questions: classification study. British Medical Journal, 321: 429–432. Erickson, S. and Warner, E. (1998). The impact of an individual tutorial session on MEDLINE use among obstetrics and gynaecology residents in an academic training programme: a randomized trial. Medical Education, 32: 269–273. Ericsson, K. and Simon, H. (1993). Protocol Analysis: Verbal Reports as Data, rev. ed, Cambridge, MA, MIT Press. Evans, A., McNutt, R., et al. (1993). The characteristics of peer reviewers who produce good quality reviews. Journal of General Internal Medicine, 8: 422–428. Evans, D. (1987). Final Report on the MedSORT-II Project: Developing and Managing Medical Thesauri. Laboratory for Computational Linguistics, Carnegie Mellon University, Pittsburgh. Evans, D. (1988). Pragmatically-structured, lexical-semantic knowledge bases for unified medical language systems. Proceedings of the 12th Annual Symposium on Computer Applications in Medical Care, Washington, DC. IEEE. 169–173. Evans, D., Brownlow, N., et al. (1996). Automating concept identification in the electronic medical record: an experiment in extracting dosage information. Proceedings of the 1996 AMIA Annual Fall Symposium, Washington, DC. Hanley & Belfus. 388–392. Evans, D., Cimino, J., et al. (1994). Toward a medical concept representation language. Journal of the American Medical Informatics Association, 1: 207–217. Evans, D. and Lefferts, R. (1993). Design and evaluation of the CLARIT TREC-2 system. The Second Text REtrieval Conference (TREC-2), Gaithersburg, MD. National Institute of Standards and Technology. 137–150. Evans, D., Lefferts, R., et al. (1992). CLARIT TREC design, experiments, and results. The First Test REtrieval Conference (TREC-1), Gaithersburg, MD. National Institute of Standards and Technology. 251–286. Evans, D., Rothwell, D., et al. (1991). Toward representation for medical concepts. Medical Decision Making, 11: S102–S107. Evans, D. and Zhai, C. (1996). Noun-phrase analysis in unrestricted text for information retrieval. Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics, Santa Cruz, CA. Association for Computational Linguistics. 17–24. Evans, J., Nadjari, H., et al. (1990). Quotational and reference accuracy in surgical journals: a continuing peer review problem. Journal of the American Medical Association, 263: 1353–1354. Evans, R., Pestotnik, S., et al. (1998). A computer-assisted management program for antibiotics and other antiinfective agents. New England Journal of Medicine, 338: 232–238.
References
465
Eysenbach, G. (2000a). Consumer Health Informatics. British Medical Journal, 320: 1713–1716. Eysenbach, G. (2000b). Report of a case of cyberplagiarism and reflections on detecting and preventing academic misconduct using the Internet. Journal of Medical Internet Research, 2: E4. Eysenbach, G. and Diepgen, T. (1998). Towards quality management of medical information on the Internet: evaluation, labelling, and filtering of information. British Medical Journal, 317: 1496–1502. Eysenbach, G. and Kohler, C. (2002). How do consumers search for and appraise health information on the World Wide Web? Qualitative study using focus groups, usability tests, and in-depth interviews. British Medical Journal, 324: 573–577. Eysenbach, G., Powell, J., et al. (2002). Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. Journal of the American Medical Association, 287: 2691–2700. Eysenbach, G., Su, E., et al. (1999). Shopping around the Internet today and tomorrow: towards the millennium of cybermedicine. British Medical Journal, 319: 1294–1298. Eysenbach, G., Tuische, J., et al. (2001). Evaluation of the usefulness of Internet searches to identify unpublished clinical trials for systematic reviews. Medical Informatics and the Internet in Medicine, 26: 203–218. Eysenbach, G., Yihune, G., et al. (2000). MedCERTAIN: quality management, certification, and rating of health information on the net. Proceedings of the AMIA 2000 Annual Symposium, Los Angeles, CA. Hanley & Belfus. 230–234. Fagan, J. (1987). Experiments in Automatic Phrase Indexing Document Retrieval: A Comparison of Syntactic and Non-Syntactic Methods. Department of Computer Science. Ph.D. Thesis. Cornell University. Fagan, J. (1989). The effectiveness of a non-syntactic approach to automatic phrase indexing for document retrieval. Journal of the American Society for Information Science, 40: 115–132. Fallis, D. and Fricke, M. (2002). Indicators of accuracy of consumer health information on the Internet: a study of indicators relating to information for managing fever in children in the home. Journal of the American Medical Informatics Association, 9: 73–79. Federiuk, C. (1999). The effect of abbreviations on MEDLINE searching. Academic Emergency Medicine, 6: 292–296. Feinstein, A. (1995). Meta-analysis: statistical alchemy for the 21st century. Journal of Clinical Epidemiology, 48: 71–79. Feinstein, A. and Horwitz, R. (1997). Problems in the “evidence” of “evidence-based medicine.” American Journal of Medicine, 103: 529–535. Fellbaum, C., ed. (1998). WordNet: An Electronic Lexical Database. Cambridge, MA, MIT Press. Fenichel, C. (1980). The process of searching online bibliographic databases: a review of research. Library Research, 2: 107–127. Ferguson, T. (2002). From patients to end users: quality of online patient networks needs more attention than quality of online health information. British Medical Journal, 324: 555–556. Fidel, R., Davies, R., et al. (1999). A visit to the information mall: Web searching behavior of high school students. Journal of the American Society for Information Science, 50: 24–37. Fidel, R. and Soergel, D. (1983). Factors affecting online bibliographic retrieval: a conceptual framework for research. Journal of the American Society for Information Science, 34: 163–180.
466
References
Field, M. and Lohr, K., eds. (1990). Clinical Practice Guidelines: Directions for a New Program. Washington, DC, National Academy Press. Fischer, P., Stratmann, W., et al. (1980). User reaction to PROMIS: issues related to acceptability of medical innovations. Proceedings of the 4th Annual Symposium on Computer Applications in Medical Care. IEEE. 1722–1730. Fiszman, M., Chapman, W., et al. (2000). Automatic detection of acute bacterial pneumonia from chest x-ray reports. Journal of the American Medical Informatics Association, 7: 593–604. Flanagin, A., Carey, L., et al. (1998). Prevalence of articles with honorary authors and ghost authors in peer-reviewed medical journals. Journal of the American Medical Association, 280: 222–224. Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32: 221–233. Fletcher, R. and Fletcher, S. (1979). Clinical research in general medical journals: a 30year perspective. New England Journal of Medicine, 301: 180–183. Fletcher, R., Fletcher, S., et al. (1988). Clinical Epidemiology: The Essentials. Baltimore, Williams & Wilkins. Florance, V. (1992). Medical knowledge for clinical problem solving: a structural analysis of clinical questions. Bulletin of the Medical Library Association, 80: 140–149. Florance, V. and Marchionini, G. (1995). Information processing in the context of medical care. Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle. ACM Press. 158–163. Foltz, A. and Sullivan, J. (1996). Reading level, learning presentation preference, and desire for information among cancer patients. Journal of Cancer Education, 11: 32–38. Fox, C. (1992). Lexical Analysis and Stop Lists, 102–130, in Frakes, W. and Baeza-Yates, R., eds. Information Retrieval: Data Structures and Algorithms. Englewood Cliffs, NJ. Prentice-Hall. Fox, M. (2001). Seal of Approval Issued to 13 Health Web Sites. Reuters. http://www. cancerpage.com/cancernews/cancernews3743.htm. Accessed: July 1, 2002. Frakes, W. (1992). Stemming Algorithms, 131–160, in Frankes, W. and Baeza-Yates, R., eds. Information Retrieval: Data Structures and Algorithms. Englewood Cliffs, NJ. Prentice-Hall. Frakes, W. and Baeza-Yates, R., eds. (1992). Information Retrieval: Data Structures and Algorithms. Englewood Cliffs, NJ, Prentice-Hall. Frankel, M., Elliot, R., et al. (2000). Defining and Certifying Electronic Publication in Science. American Association for the Advancement of Science. http://www.aaas.org/ spp.dspp/sfrl/projects/epub/define.htm. Accessed: July 1, 2002. Freiman, J., Chalmers, T., et al. (1978). The importance of beta, the type II error and sample size in the design and interpretation of the randomized controlled trial. New England Journal of Medicine, 299: 690–694. French, J., Powell, A., et al. (2001). A controlled vocabulary to improve collection selection and retrieval effectiveness. Proceedings of the 2001 ACM CIKM International Conference on Information and Knowledge Management, Atlanta, GA. ACM Press. 199–206. Friedlander, A. (2002). The National Digital Information Infrastructure Preservation Program: expectations, realities, choices, and progress to date. D-Lib Magazine, 8. http:// www.dlib.org/dlib/april02/friedlander/04friedlander.html. Friedman, C. (1987). A Sublanguage Narrative Processor, 81–111, in Sager, N., Friedman, C. and Lyman, M., eds. Medical Language Processing: Computer Management of Narrative Data. Reading, MA. Addison-Wesley.
References
467
Friedman, C. (1997). Towards a comprehensive medical language processing system: methods and issues. Proceedings of the 1997 AMIA Annual Fall Symposium, Nashville, TN. Hanley & Belfus. 595–599. Friedman, C., Alderson, P., et al. (1994). A general natural-language text processor for clinical radiology. Journal of the American Medical Informatics Association, 1: 161–174. Friedman, C. and Hripcsak, G. (1998). Evaluating natural language processors in the clinical domain. Methods of Information in Medicine, 37: 334–344. Friedman, C., Hripcsak, G., et al. (1995). Natural language processing in an operational clinical system. Natural Language Engineering, 1: 83–108. Friedman, C., Hripcsak, G., et al. (1998). An evaluation of natural language processing methodologies. Proceedings of the AMIA 1998 Annual Symposium, Orlando, FL. Hanley & Belfus. 855–859. Friedman, C., Hripcsak, G., et al. (1999). Representing information in patient reports using natural language processing and the extensible markup language. Journal of the American Medical Informatics Association, 6: 76–87. Friedman, C., Knirsch, C., et al. (1999). Automating a severity score guideline for community-acquired pneumonia employing medical language processing of discharge summaries. Proceedings of the AMIA 1999 Annual Symposium, Washington, DC. Hanley & Belfus. 256–260. Friedman, C., Kra, P., et al. (2001). GENIES: a natural-language processing system for the extraction of molecular pathways from journal articles. Bioinformatics, 17: S74–S82. Friedman, C., Owens, D., et al. (2000). Evaluation and Technology Assessment, 282–323, in Shortliffe, E. and Perreault, L., eds. Medical Informatics: Computer Applications in Health Care. New York. Springer-Verlag. Friedman, C., Wildemuth, B., et al. (1996). A comparison of hypertext and Boolean access to biomedical information. Proceedings of the 1996 AMIA Annual Fall Symposium, Washington, DC. Hanley & Belfus. 2–6. Friedman, C. and Wyatt, J. (1997). Evaluation Methods in Medical Informatics. New York, Springer-Verlag. Friedman, C. and Wyatt, J. (2001). Publication bias in medical informatics. Journal of the American Medical Informatics Association, 8: 189–191. Friedman, P. (1990). Correcting the literature following fraudulent publication. Journal of the American Medical Association, 263: 1416–1419. Frisse, M. (1988). Searching for information in a hypertext medical handbook. Communications of the ACM, 31: 880–886. Frisse, M. and Cousins, S. (1989). Information retrieval from hypertext: update on the dynamic medical handbook project. Hypertext 89, Pittsburgh, New York, ACM Press. Fry, E. (1977). Fry’s readability graph: clarifications, validity, and extension to level 17. Journal of Reading, 21: 242–252. Fuhr, N. and Knorz, G. (1984). Retrieval test evaluation of a rule-based automatic indexing (AIR/PHYS), 391–408, in vanRijsbergen, C., ed. Research and Development in Information Retrieval. Cambridge. Cambridge University Press. Fukuda, K., Tamura, A., et al. (1998). Toward information extraction: identifying protein names from biological papers. Pacific Symposium on Biocomputing, Maui, Hawaii. World Scientific. 707–718. Fuller, M., Kaszkiel, M., et al. (1997). MDS TREC-6 report. The Sixth Test REtrieval Conference (TREC-6), Gaithersburg, MD. National Institute of Standards and Technology. 241–257.
468
References
Funk, M. and Reid, C. (1983). Indexing consistency in MEDLINE. Bulletin of the Medical Library Association, 71: 176–183. Gagliardi, A. and Jadad, A. (2002). Examination of instruments used to rate quality of health information on the Internet: chronicle of a voyage with an unclear destination. British Medical Journal, 324: 569–573. Gallagher, K., Hulbert, L., et al. (1990). Full-text and bibliographic database searching in the health sciences: an exploratory study comparing CCML and MEDLINE. Medical Reference Services Quarterly, 9(4): 17–25. Gardner, M. and Bond, J. (1990). An exploratory study of statistical assessment of papers published in the British Medical Journal. Journal of the American Medical Association, 263: 1355–1357. Garfield, E. (1964). “Science Citation Index”—a new dimension in indexing. Science, 144: 649–654. Garfield, E. (1979a). Bradford’s Law and Related Statistical Patterns, 476–483, in Garfield, E., ed. Essays of an Information Scientist: 1979–1980. Philadelphia. Institute for Scientific Information. 4. Garfield, E. (1979b). The Citation Index as a Search Tool, 41–61, in Garfield, E., ed. Citation Indexing—Its Theory and Application in Science, Technology, and Humanities. New York. Wiley. Garfield, E. (1994). The impact factor. Current Contents, 25: 3–7. Garfield, E. and Welljams-Dorof, A. (1990). The impact of fraudulent research on the scientific literature: the Stephen Breuning case. Journal of the American Medical Association, 1990: 1424–1426. Garfunkel, J., Lawson, E., et al. (1990). Effect of acceptance or rejection on the author’s evaluation of peer review of medical manuscripts. Journal of the American Medical Association, 263: 1376–1378. Garfunkel, J., Ulshen, M., et al. (1990). Problems identified by secondary review of accepted manuscripts. Journal of the American Medical Association, 263: 1369–1371. Garfunkel, J., Ulshen, M., et al. (1994). Effect of institutional prestige on reviewers’ recommendations and editorial decisions. Journal of the American Medical Association, 272: 137–138. Garrow, J., Butterfield, M., et al. (1998). The reported training and experience of editors in chief of specialist clinical medical journals. Journal of the American Medical Association, 280: 286–287. Gauch, S. and Smith, J. (1991). Search improvement via automatic query reformulation. ACM Transactions on Information Systems, 9: 249–280. Gay, L. and Croft, W. (1990). Interpreting nominal compounds for information retrieval. Information Processing and Management, 26: 21–38. Gazdar, G. and Mellish, C. (1989). Natural Language Processing in LISP: An Introduction to Computational Linguistics. Wokingham, U.K., Addison-Wesley. Gazmararian, J., Baker, D., et al. (1999). Health literacy among Medicare enrollees in a managed care organization. Journal of the American Medical Association, 281: 545–551. Gehanno, J., Paris, C., et al. (1998). Assessment of bibliographic databases performance in information retrieval for occupational and environmental toxicology. Occupational and Environmental Medicine, 55: 562–566. Gilarranz, J., Gonzalo, J., et al. (1997). An approach to conceptual text retrieval using the EuroWordNet multilingual semantic database. AAAI Spring Symposium on CrossLanguage Text and Speech Retrieval, Palo Alto, CA. http://www.ee.umd.edu/medlab/ filter/sss/papers/gilarranz2.ps.
References
469
Gillois, P., Colombet, I., et al. (1999). A critical appraisal of the use of Internet for calculating cardiovascular risk. Proceedings of the AMIA 1999 Annual Symposium, Washington, DC. Hanley & Belfus. 775–779. Giuse, N., Huber, J., et al. (1994). Information needs of health care professionals in an AIDS outpatient clinic as determined by chart review. Journal of the American Medical Informatics Association, 1: 395–403. Gladney, H. and Cantu, A. (2001). Authorization management for digital libraries. Communications of the ACM, 44(5): 63–65. Glantz, S. (1980). Biostatistics: how to detect, correct, and prevent errors in the medical literature. Circulation, 61: 1–7. Glass, G. (1976). Primary, secondary, and meta-analysis of research. Educational Research, 10: 3–8. Glasziou, P. and Irwig, L. (1998). Meta-Analysis of Diagnostic Tests, 2579–2585, in Armitage, P. and Colton, T., eds. Encyclopaedia of Biostatistics, Vol. 4. Chichester. Wiley. Glossbrenner, A. and Glossbrenner, E. (2001). Search Engines for the World Wide Web: Visual QuickStart Guide, 3rd ed. Berkeley, CA, Peachpit Press. Godlee, F., Gale, C., et al. (1998). Effect on the quality of peer review of blinding reviewers and asking them to sign their reports. Journal of the American Medical Association, 280: 237–240. Golovchinsky, G., Price, M., et al. (1999). From reading to retrieval: freeform ink annotations as queries. Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Berkeley, CA. ACM Press. 19–25. Gomez, L., Egan, D., et al. (1986). Learning to use a text editor: some learner characteristics that predict success. Human–Computer Interaction, 2: 1–23. Gonzales, R., Bartlett, J., et al. (2001). Principles of appropriate antibiotic use for treatment of acute respiratory infections in adults: background, specific aims, and methods. Annals of Internal Medicine, 134: 479–486. Gordon, M. and Pathak, P. (1999). Finding information on the World Wide Web: the retrieval effectiveness of search engines. Information Processing and Management, 35: 141–180. Gorman, P. (1993). Does the medical literature contain the evidence to answer the questions of primary care physicians? Preliminary findings of a study. Proceedings of the 17th Annual Symposium on Computer Applications in Medical Care, Washington, DC. McGraw-Hill. 571–575. Gorman, P. (1995). Information needs of physicians. Journal of the American Society for Information Science, 46: 729–736. Gorman, P., Ash, J., et al. (1994). Can primary care physicians’ questions be answered using the medical literature? Bulletin of the Medical Library Association, 82: 140–146. Gorman, P. and Helfand, M. (1995). Information seeking in primary care: how physicians choose which clinical questions to pursue and which to leave unanswered. Medical Decision Making, 15: 113–119. Gotzsche, P. and Olson, O. (2001). Misleading publications of major mammography screening trials in major medical journals. Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain. American Medical Association. http://www.ama-assn.org/public/peer/peerhome.htm. Gove, P., ed. (1976). Webster’s Third International Dictionary. Springfield, MA, Merriam-Webster. Graber, M., Bergus, G., et al. (1999). Using the World Wide Web to answer clinical ques-
470
References
tions: how efficient are different methods of information retrieval? Journal of Family Practice, 49: 520–524. Graber, M., Roller, C., et al. (1999). Readability levels of patient education material on the World Wide Web. Journal of Family Practice, 48: 58–61. Graesser, A., Byrne, P., et al. (1992). Answering questions about information in databases, 229–252, in Lauer, T., Peacock, E. and Graesser, A., eds. Questions and Information Systems. Mahwah, NJ. Erlbaum. Gravois, S., Fisher, W., et al. (1995). Information-seeking practices of dental hygienists. Bulletin of the Medical Library Association, 83: 446–452. Greenes, R. (1982). OBUS: a microcomputer system for measurement, calculation, reporting, and retrieval of obstetric ultrasound examinations. Radiology, 144: 879–883. Greenes, R., Barnett, G., et al. (1970). Recording, retrieval, and review of medical data by physician–computer interaction. New England Journal of Medicine, 282: 307–315. Greenes, R. and Deibel, S. (1991). The DeSyGNER knowledge management architecture: a building block approach based on an extensible kernel. Artificial Intelligence in Medicine, 3: 95–111. Greenes, R., McClure, R., et al. (1992). The findings-diagnosis continuum: implications for image descriptions and clinical databases. Proceedings of the 16th Annual Symposium on Computer Applications in Medical Care, Baltimore, MD. McGraw-Hill. 383–387. Greenes, R., Tarabar, D., et al. (1989). Knowledge management as a decision support method: a diagnostic workup strategy application. Computers and Biomedical Research, 22: 113–135. Gregoire, G., Derderian, F., et al. (1995). Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel bias? Journal of Clinical Epidemiology, 48: 159–163. Griffith, B., White, H., et al. (1986). Test of methods of evaluating bibliographic databases: an analysis of the National Library of Medicine’s handling of literatures in the medical behavioral sciences. Journal of the American Society for Information Science, 37(4): 261–270. Grishman, R. and Hirschman, L. (1978). Question answering from natural language medical data bases. Artificial Intelligence, 11: 25–43. Grossman, D. and Frieder, O. (1998). Information Retrieval Algorithms and Heuristics. Boston, Kluwer. Gruninger, M. and Lee, J. (2002). Ontology: applications and design. Communications of the ACM, 45(2): 39–41. Guard, R., Fredericka, T., et al. (2000). Health care, information needs, and outreach: reaching Ohio’s rural citizens. Bulletin of the Medical Library Association, 88: 374–381. Gundersen, M., Haug, P., et al. (1996). Development and evaluation of a computerized admissions diagnoses encoding system. Computers and Biomedical Research, 29: 351–372. Gupta, A., Gross, C., et al. (2001). Disclosure of financial conflict of interest in published research: a study of adherence to uniform requirements. Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain. American Medical Association. http://www.ama-assn.org/public/peer/peerhome.htm. Guyatt, G. and Rennie, D. (2001). Users’ Guide to the Medical Literature: Essentials of Evidence-Based Clinical Practice. Chicago. American Medical Association. Guyatt, G., Rennie, D., et al. (2001). Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice. Chicago. American Medical Association.
References
471
Hainer, M., Tsai, N., et al. (2000). Fatal hepatorenal failure associated with hydrazine. Annals of Internal Medicine, 133: 877–880. Hallett, K. (1998). Separate but equal? A system comparison study of MEDLINE’s controlled vocabulary MeSH. Bulletin of the Medical Library Association, 86: 491–495. Halpern, S., Karlawish, J., et al. (2002). The continuing unethical conduct of underpowered clinical trials. Journal of the American Medical Association, 288: 358–362. Hanley, J. and McNeil, B. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 148: 839–843. Hargens, L. (1990). Variation in journal peer review systems: possible causes and consequences. Journal of the American Medical Association, 263: 1348–1352. Harman, D. (1991). How effective is suffixing? Journal of the American Society for Information Science, 42: 7–15. Harnad, S. (1998). On-line journals and financial firewalls. Nature, 395: 127–128. Harter, S. (1992). Psychological relevance and information science. Journal of the American Society for Information Science, 43: 602–615. Hatch, C. and Goodman, S. (1998). Perceived value of providing peer reviewers with abstracts and preprints of related published and unpublished papers. Journal of the American Medical Association, 280: 273–274. Hatfield, C., May, S., et al. (1999). Quality of consumer drug information provided by four Web sites. American Journal of Health Systems Pharmacies, 56: 2308–2311. Hatzivassiloglou, V., Duboue, P., et al. (2001). Disambiguating proteins, genes, and RNA in text: a machine learning approach. Bioinformatics, 17: S97–S106. Haug, P., Koehler, S., et al. (1994). A natural language understanding system combining syntactic and semantic techniques. Proceedings of the 18th Annual Symposium on Computer Applications in Medical Care, Washington, DC. Hanley & Belfus. 247–251. Haug, P., Koehler, S., et al. (1995). Experience with a mixed semantic/syntactic parser. Proceedings of the 19th Annual Symposium on Computer Applications in Medical Care, New Orleans, LA. 284–288. Haux, R., Grothe, W., et al. (1996). Knowledge retrieval as one type of knowledge-based decision support in medicine: results of an evaluation study. International Journal of Biomedical Computing, 41: 69–85. Hawking, D. (2000). Overview of the TREC-9 Web Track. The Ninth Text REtrieval Conference (TREC-9), Gaithersburg, MD. National Institute of Standards and Technology. 87–102. Hawking, D. and Craswell, N. (2001). Overview of the TREC-2001 Web Track. The Tenth Text REtrieval Conference (TREC 2001), Gaithersburg, MD. National Institute of Standards and Technology. 61–67. Hawking, D., Craswell, N., et al. (2001a). Measuring search engine quality. Information Retrieval, 4: 33–60. Hawking, D., Craswell, N., et al. (2001b). Which search engine is best at finding online services? The Tenth International World Wide Web Conference, Hong Kong. International World Wide Web Conference Committee. http://www.www10.org/cdrom/posters/1089.pdf. Haynes, R. (2001). Of studies, syntheses, synopses, and systems: the “4S” evolution of services for finding current best evidence. ACP Journal Club, 134: A11–A13. Haynes, R., Johnston, M., et al. (1992). A randomized controlled trial of a program to enhance clinical use of MEDLINE. Online Journal of Controlled Clinical Trials, Doc No 56. Haynes, R. and McKibbon, K. (1987). Grateful Med. M.D. Computing, 4: 47–57. Haynes, R., McKibbon, K., et al. (1985). Computer searching of the medical literature:
472
References
an evaluation of MEDLINE searching systems. Annals of Internal Medicine, 103: 812–816. Haynes, R., McKibbon, K., et al. (1990). Online access to MEDLINE in clinical settings. Annals of Internal Medicine, 112: 78–84. Haynes, R., Mulrow, C., et al. (1990). More informative abstracts revisited. Annals of Internal Medicine, 113: 69–76. Haynes, R., Ramsden, M., et al. (1991). Online access to MEDLINE in clinical settings: impact of user fees. Bulletin of the Medical Library Association, 79: 377–381. Haynes, R., Walker, C., et al. (1994). Performance of 27 MEDLINE systems tested by searches with clinical questions. Journal of the American Medical Informatics Association, 1: 285–295. Haynes, R., Wilczynski, N., et al. (1994). Developing optimal search strategies for detecting clinically sound studies in MEDLINE. Journal of the American Medical Informatics Association, 1: 447–458. Hearst, M. (1996). Improving full-text precision on short queries using simple constraints. Proceedings of the 5th Annual Symposium on Document Analysis and Information Retrieval (SDAIR), Las Vegas, NV. University of Nevada, Las Vegas. 217–232. Hearst, M. (1999). User Interfaces and Visualization, 257–323, in Baeza-Yates, R. and Ribeiro-Neto, B., eds. Modern Information Retrieval. New York. ACM Press. Hearst, M. and Karadi, C. (1997). Cat-a-Cone: an interactive interface for specifying searches and viewing retrieval results using a large category hierarchy. Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Philadelphia, PA. ACM Press. 246–255. Hearst, M. and Plaunt, C. (1993). Subtopic structuring for full-length document access. Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Pittsburgh, PA. ACM Press. 59–68. Heckerling, P. (1990). Information content of diagnostic tests in the medical literature. Methods of Information in Medicine, 29: 61–66. Heinze, D., Morsch, M., et al. (2001). LifeCode: a deployed application for automated medical coding. AI Magazine, 22: 76–88. Hemila, H. (1997). Vitamin C intake and susceptibility to the common cold. British Journal of Nutrition, 77: 59–72. Hendrix, G., Sacerdoti, E., et al. (1978). Developing a natural language interface to complex data. ACM Transactions on Database Systems, 3: 105–147. Hernandez-Borges, A., Macias-Cervi, P., et al. (1999). Can examination of WWW usage statistics and other indirect quality indicators help to distinguish the relative quality of medical websites? Journal of Medical Internet Research, 1. http://www.jmir.org/1999/ 1/e1/index.htm. Hersh, W. (1991). Evaluation of Meta-1 for a concept-based approach to the automated indexing and retrieval of bibliographic and full-text databases. Medical Decision Making, 11: S120–S124. Hersh, W. (1994). Relevance and retrieval evaluation: perspectives from medicine. Journal of the American Society for Information Science, 45: 201–206. Hersh, W. (1999). “A world of knowledge at your fingertips”: the promise, reality, and future directions of on-line information retrieval. Academic Medicine, 74: 240–243. Hersh, W. (2001). Interactivity at the Text Retrieval Conference (TREC). Information Processing and Management, 37: 365–366. Hersh, W. (2002). Medical informatics education: an alternative pathway for training informationists. Journal of the Medical Library Association, 90: 76–79.
References
473
Hersh, W., Ball, A., et al. (1999). Maintaining a catalog of manually-indexed, clinicallyoriented World Wide Web content. Proceedings of the AMIA 1999 Annual Symposium, Washington, DC. Hanley & Belfus. 790–794. Hersh, W., Brown, K., et al. (1996). CliniWeb: managing clinical information on the World Wide Web. Journal of the American Medical Informatics Association, 3: 273–280. Hersh, W., Buckley, C., et al. (1994). OHSUMED: an interactive retrieval evaluation and new large test collection for research. Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland. Springer-Verlag. 192–201. Hersh, W., Campbell, E., et al. (1996). Empirical, automated vocabulary discovery using large text corpora and advanced natural language processing tools. Proceedings of the 1996 AMIA Annual Fall Symposium, Washington, DC. Hanley & Belfus. 159–163. Hersh, W., Campbell, E., et al. (1997). Assessing the feasibility of large-scale natural language processing in a corpus of ordinary medical records: a lexical analysis. Proceedings of the 1997 AMIA Annual Fall Symposium, Nashville, TN. Hanley & Belfus. 580–584. Hersh, W., Crabtree, M., et al. (2000). Factors associated with successful answering of clinical questions using an information retrieval system. Bulletin of the Medical Library Association, 88: 323–331. Hersh, W., Crabtree, M., et al. (2002). Factors associated with success for searching MEDLINE and applying evidence to answer clinical questions. Journal of the American Medical Informatics Association, 9: 283–293. Hersh, W. and Donohoe, L. (1998). SAPHIRE International: a tool for cross-language information retrieval. Proceedings of the AMIA 1998 Annual Symposium, Orlando, FL. Hanley & Belfus. 673–677. Hersh, W., Elliot, D., et al. (1994). Towards new measures of information retrieval evaluation. Proceedings of the 18th Annual Symposium on Computer Applications in Medical Care, Washington, DC. Hanley & Belfus. 895–899. Hersh, W., Gorman, P., et al. (1998). Applicability and quality of information for answering clinical questions on the Web. Journal of the American Medical Association, 280: 1307–1308. Hersh, W. and Greenes, R. (1990). Information retrieval in medicine: state of the art. M.D. Computing, 7(5): 302–311. Hersh, W., Helfand, M., et al. (2001). Telemedicine for the Medicare Population. Rockville, MD, Agency for Healthcare Research and Quality. http://www.ahrq.gov/ clinic/telemedsum.htm. Hersh, W. and Hickam, D. (1992). A comparison of retrieval effectiveness for three methods of indexing medical literature. American Journal of the Medical Sciences, 303: 292–300. Hersh, W. and Hickam, D. (1993). A comparison of two methods for indexing and retrieval from a full-text medical database. Medical Decision Making, 13: 220–226. Hersh, W. and Hickam, D. (1994). The use of a multi-application computer workstation in a clinical setting. Bulletin of the Medical Library Association, 82: 382–389. Hersh, W. and Hickam, D. (1995). An evaluation of interactive Boolean and natural language searching with an on-line medical textbook. Journal of the American Society for Information Science, 46: 478–489. Hersh, W. and Hickam, D. (1998). How well do physicians use electronic information retrieval systems? A framework for investigation and review of the literature. Journal of the American Medical Association, 280: 1347–1352.
474
References
Hersh, W., Hickam, D., et al. (1992). Word, concepts, or both: optimal indexing units for automated information retrieval. Proceedings of the 16th Annual Symposium on Computer Applications in Medical Care, Baltimore, MD. McGraw-Hill. 644–648. Hersh, W., Hickam, D., et al. (1994). A performance and failure analysis of SAPHIRE with a MEDLINE test collection. Journal of the American Medical Informatics Association, 1: 51–60. Hersh, W., Leen, T., et al. (1998). Automatic prediction of trauma registry procedure codes from emergency room dictations. MEDINFO 98—Proceedings of the Ninth World Congress on Medical Informatics, Seoul, South Korea. IOS Press. 665–669. Hersh, W. and Leone, T. (1995). The SAPHIRE server: a new algorithm and implementation. Proceedings of the 18th Annual Symposium on Computer Applications in Medical Care, New Orleans, LA. Hanley & Belfus. 858–862. Hersh, W., Mailhot, M., et al. (2002). Selective automated indexing of findings and diagnoses in radiology reports. Journal of Biomedical Informatics, 34: 262–273. Hersh, W., Miller, R., et al. (2002). Professional’s Information Link (PiL): A Web-based asynchronous consultation service. Proceedings of the 2002 Annual AMIA Symposium, San Antonio, TX. Hanley & Belfus. in press. Hersh, W. and Over, P. (2000). TREC-9 Interactive Track report. The Ninth Text REtrieval Conference (TREC-9), Gaithersburg, MD. National Institute of Standards and Technology. 41–50. Hersh, W., Pentecost, J., et al. (1996). A task-oriented approach to information retrieval evaluation. Journal of the American Society for Information Science, 47: 50–56. Hersh, W. and Price, S. (1998). Identifying randomized controlled trials in conference proceedings abstracts. Proceedings of the Sixth Annual Cochrane Colloquium, Baltimore, MD. 52. Hersh, W., Price, S., et al. (1998). A large-scale comparison of Boolean vs. natural language searching for the TREC-7 interactive track. The Seventh Text REtrieval Conference (TREC-7), Gaithersburg, MD. National Institute of Standards and Technology. 491–500. Hersh, W., Price, S., et al. (2000). Assessing thesaurus-based query expansion using the UMLS Metathesaurus. Proceedings of the AMIA 2000 Annual Symposium, Los Angeles, CA. Hanley & Belfus. 344–348. Hersh, W. and Rindfleisch, T. (2000). Electronic publishing of scholarly communication in the biomedical sciences. Journal of the American Medical Informatics Association, 7: 324–325. Hersh, W., Turpin, A., et al. (2000). Do batch and user evaluations give the same results? Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Athens, Greece. ACM Press. 17–24. Hersh, W., Turpin, A., et al. (2000). Further analysis of whether batch and user evaluations give the same results with a question-answering task. The Ninth Text REtrieval Conference (TREC-9), Gaithersburg, MD. National Institute of Standards and Technology. 407–416. Hersh, W., Turpin, A., et al. (2001). Challenging conventional assumptions of automated information retrieval with real users: Boolean searching and batch retrieval evaluations. Information Processing and Management, 37: 383–402. Hersh, W. and Zhang, L. (1999). Teaching English medical terminology using the UMLS Metathesaurus and World Wide Web. Proceedings of the AMIA 1999 Annual Symposium, Washington, DC. Hanley & Belfus. 1078. Hibble, A., Kanka, D., et al. (1998). Guidelines in general practice: the new Tower of Babel? British Medical Journal, 317: 862–863.
References
475
Hildreth, C. (1989). The Online Catalogue: Developments and Directions. London, The Library Association. Hirakawa, M. (2002). HOWDY: an integrated database system for human genome research. Nucleic Acids Research, 30: 152–157. Hiruki, T., Olson, D., et al. (2001). An automated MeSH mapping tool to facilitate searching of the English-language Web by Japanese users. Technology and Health Care, 9: 495–496. Hodge, J., Gostin, L., et al. (1999). Legal issues concerning electronic health information: privacy, quality, and liability. Journal of the American Medical Association, 282: 1466–1471. Hoffrage, U., Lindsey, S., et al. (2000). Communicating statistical information. Science, 290: 2261–2262. Hole, W. and Srinivasan, S. (2000). Discovering missed synonymy in a large conceptoriented metathesaurus. Proceedings of the AMIA 2000 Annual Symposium, Los Angeles, CA. Hanley & Belfus, 354–358. Hopayian, K. (2001). The need for caution in interpreting high quality systematic reviews. British Medical Journal, 323: 681–684. Horowitz, G., Jackson, J., et al. (1983). PaperChase: self-service bibliographic retrieval. Journal of the American Medical Association, 328: 2495–2500. Horton, R. (2002). Postpublication criticism and the shaping of clinical knowledge. Journal of the American Medical Association, 287: 2843–2847. Hotopf, M., Lewis, G., et al. (1997). Putting trials on trial: the costs and consequences of small trials in depression: a systematic review of methodology. Journal of Epidemiology and Community Health, 51: 354–358. Hripcsak, G., Clayton, P., et al. (1996). Design of a clinical event monitor. Computers and Biomedical Research, 29: 194–221. Hripcsak, G., Friedman, C., et al. (1995). Unlocking clinical data from narrative reports: a study of natural language processing. Annals of Internal Medicine, 122: 681–688. Hripcsak, G., Knirsch, C., et al. (1997). Automated tuberculosis detection. Journal of the American Medical Informatics Association, 4: 376–381. Hripcsak, G., Kuperman, G., et al. (1998). Extracting findings from narrative reports: software transferrability and sources of physician disagreement. Methods of Information in Medicine, 37: 1–7. Hsu, P. (1991). MEDLINE end-user survey: the University of Florida experience. Medical Reference Services Quarterly, 10(1): 49–58. Hu, F., Goldberg, J., et al. (1998). Comparison of population-averaged and subjectspecific approaches for analyzing repeated binary outcomes. American Journal of Epidemiology, 147: 694–703. Hughes, C. (1998). Academic medical libraries’ policies and procedures for notifying library users of retracted scientific publications. Medical Reference Services Quarterly, 17(2): 37–40. Hull, D. (1994). Improving text retrieval for the routing problem using latent semantic indexing. Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland. Springer-Verlag. 282–291. Hull, D. (1996). Stemming algorithms: a case study for detailed evaluation. Journal of the American Society for Information Science, 47: 70–84. Hull, D. and Greffenstette, G. (1996). Querying across languages: a dictionary-based approach to multilingual information retrieval. Proceedings of the 19th Annual Interna-
476
References
tional ACM SIGIR Conference on Research and Development in Information Retrieval, Zurich, Switzerland. ACM Press. 49–57. Humphrey, S. (1988). Medindex system: medical indexing expert system. Information Processing and Management, 25: 73–88. Humphrey, S. (1992). Indexing biomedical documents: from thesaural to knowledge-based systems. Artificial Intelligence in Medicine, 4: 343–371. Humphreys, B. (2000). Electronic health record meets digital library: a new environment for achieving an old goal. Journal of the American Medical Informatics Association, 7: 444–452. Humphreys, B., Lindberg, D., et al. (1998). The Unified Medical Language System: an informatics research collaboration. Journal of the American Medical Informatics Association, 5: 1–11. Humphreys, B., McCray, A., et al. (1997). Evaluating the coverage of controlled health data terminologies: report on the results of the NLM/AHCPR large scale vocabulary test. Journal of the American Medical Informatics Association, 6: 484–500. Humphreys, K., Demetriou, G., et al. (2000). Two applications of information extraction to biological science journal articles: enzyme interactions and protein structures. Pacific Symposium on Biocomputing, Honolulu, HI. World Scientific. 505–516. Hutchinson, D. (1998). Medline for Health Professionals: How to Search PubMed on the Internet. Sacramento, CA, New Wind Publishing. Huth, E. (1989). The underused medical literature. Annals of Internal Medicine, 110: 99–100. Huwiler-Muntener, K., Juni, P., et al. (2002). Quality of reporting of randomized trials as a measure of methodologic quality. Journal of the American Medical Association, 287: 2801–2804. Iliopoulos, I., Enright, A., et al. (2001). Textquest: document clustering of Medicine abstracts for concept discovery in molecular biology. Pacific Symposium on Biocomputing, Kauai, HI. World Scientific. 384–395. Impicciatore, P., Pandolfini, C., et al. (1997). Reliability of health information for the public on the World Wide Web: systematic survey of advice on managing fever in children at home. British Medical Journal, 314: 1879–1881. Ingelfinger, F. (1969). Annual discourse: swinging copy and sober science. New England Journal of Medicine, 281: 526–532. Ingelfinger, F. (1974). Peer review in biomedical publication. American Journal of Medicine, 56: 686–692. Ingwersen, P. (1998). The calculation of Web impact factors. Journal of Documentation, 54: 236–243. Ioannidis, J. (1998). Effect of statistical significance of results on the time to completion and publication of randomized efficacy trials. Journal of the American Medical Association, 279: 281–286. Izumi, Y. and Isozumi, K. (2001). Modern Japanese medical history and the European influence. Keio Journal of Medicine, 50: 91–99. Jacobs, M., Edwards, A., et al. (1998). Criteria for evaluating alternative MEDLINE search engines. Medical Reference Services Quarterly, 17(3): 1–12. Jadad, A. (1999). Promoting partnerships: challenges for the Internet age. British Medical Journal, 319: 761–764. Jadad, A., Cook, D., et al. (1998). Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. Journal of the American Medical Association, 280: 278–280.
References
477
Jadad, A. and Gagliardi, A. (1998). Rating health information on the Internet: navigating to knowledge or to Babel? Journal of the American Medical Association, 279: 611–614. Jain, N. and Friedman, C. (1997). Identification of findings suspicious for breast cancer based on natural language processing of mammogram reports. Proceedings of the 1997 AMIA Annual Fall Symposium, Nashville, TN. Hanley & Belfus. 829–833. Jain, N., Knirsch, C., et al. (1996). Identification of suspected tuberculosis patients based on natural language processing of chest radiograph reports. Proceedings of the 1996 AMIA Annual Fall Symposium, Washington, DC. Hanley & Belfus. 542–546. Jarvelin, K. and Kekalainen, J. (2000). IR evaluation methods for retrieving highly relevant documents. Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Athens, Greece. ACM Press. 41–48. Jefferson, T., Wager, E., et al. (2002). Measuring the quality of editorial peer review. Journal of the American Medical Association, 287: 2786–2790. Jensen, R., Asbury, H., et al. (1980). Costs and benefits to industry of online literature searches. Special Libraries, 71: 291–299. Jenssen, T., Laegreid, A., et al. (2001). A literature network of human genes for highthroughput analysis of gene expression. Nature Genetics, 28: 21–28. Jerant, A. and Hill, D. (2000). Does the use of electronic medical records improve surrogate patient outcomes in outpatient settings? Journal of Family Practice, 49: 349–357. Jiang, Y. (2000). Quality evaluation of orthodontic information on the World Wide Web. American Journal of Orthodontics and Dentofacial Orthopedics, 118: 4–9. Jimison, H. and Sher, P. (1995). Consumer health informatics: health information technology for consumers. Journal of the American Society for Information Science, 46: 783–790. Jing, Y. and Croft, W. (1994). An association thesaurus for information retrieval. RIAO 4 Conference Proceedings, New York. Recherche d’Information Assistee par Ordinateur. 146–160. Johnson, D., Chu, W., et al. (1999). Creating and indexing teaching files from free-text patient reports. Proceedings of the AMIA 1999 Annual Symposium, Washington, DC. Hanley & Belfus. 814–818. Johnson, E., Sievert, M., et al. (1995). Retrieving research studies: a comparison of bibliographic and full-text versions of the New England Journal of Medicine. Proceedings of the 19th Annual Symposium on Computer Applications in Medical Care, New Orleans, LA. Hanley & Belfus. 846–850. Johnson, S. (1996). Generic data modeling for clinical repositories. Journal of the American Medical Informatics Association, 3: 328–339. Johnston, M., Langton, K., et al. (1994). Effects of computer-based clinical decision support systems on clinical performance and patient outcome. Annals of Internal Medicine, 120: 135–142. Jollis, J., Ancukiewicz, M., et al. (1993). Discordance of databases designed for claims payment versus clinical information systems: implications for outcomes research. Annals of Internal Medicine, 119: 844–850. Jones, L., Gassie, E., et al. (1990). INDEX: the statistical basis for an automatic conceptual phrase-indexing system. Journal of the American Society for Information Science, 41: 87–97. Jones, M., Marsden, G., et al. (1999). Improving Web ineraction on small displays. Proceedings of the 8th International World Wide Web Conference, Toronto, Canada. http://www8.org/w8-papers/1b-multimedia/improving/improving.html.
478
References
Joyce, J., Rabe-Hesketh, S., et al. (1998). Reviewing the reviews: the example of chronic fatigue syndrome. Journal of the American Medical Association, 280: 264–266. Justice, A., Cho, M., et al. (1998). Does masking author identity improve peer review quality? A randomized controlled trial. Journal of the American Medical Association, 280: 240–242. Jydstrup, R. and Gross, M. (1966). Cost of information handling in hospitals. Health Services Research, 1: 235–271. Kahle, B. (1997). Preserving the Internet. Scientific American, 276(3): 82–83. Kalsman, M. and Acosta, D. (2000). Use of the Internet as a medical resource by rural physicians. Journal of the American Board of Family Practice, 13: 349–352. Kamel Boulos, M. (2001). HealthCyberMap: Mapping the Health Cyberspace Using Hypermedia GIS and Clinical Codes. http://healthcybermap.semanticweb.org/proposal. htm. Accessed: July 1, 2002. Kanter, S., Miller, R., et al. (1994). Using POSTDOC to recognize biomedical concepts in medical school curricular documents. Bulletin of the Medical Library Association, 82: 283–287. Kassirer, J. (1992). Too many books, too few journals. New England Journal of Medicine, 326: 1427–1428. Katcher, B. (1999). MEDLINE: A Guide to Effective Searching. San Francisco, Ashbury Press. Kent, A., Berry, M., et al. (1955). Machine literature searching. VIII. Operational criteria for designing information retrieval systems. American Documentation, 6: 93–101. Kessler, M. (1963). Bibliographic coupling between scientific papers. American Documentation, 14: 10–25. Kessler, M. (1965). Comparison of the results of bibliographic coupling and analytic subject indexing. American Documentation, 16(3): 223–233. Kim, H. (2002). Predicting how ontologies for the Semantic Web will develop. Communications of the ACM, 45(2): 48–54. Kim, P., Eng, T., et al. (1999). Published criteria for evaluating health related Web sites: review. British Medical Journal, 318: 647–649. King, D. (1987). The contribution of hospital library information services to clinical care: a study of eight hospitals. Bulletin of the Medical Library Association, 75: 291–301. Kingsland, L., Harbourt, A., et al. (1993). COACH: applying UMLS knowledge sources in an expert searcher environment. Bulletin of the Medical Library Association, 81: 178–183. Kingsland, L., Syed, E., et al. (1992). COACH: an expert searcher program to assist Grateful Med users searching MEDLINE. MEDINFO 92, 382–386. Kirby, M. and Miller, N. (1986). MEDLINE searching on Colleague: reasons for failure or success of untrained users. Medical Reference Services Quarterly, 5(3): 17–34. Kjaergard, L., Villumsen, J., et al. (2001). Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Annals of Internal Medicine, 135: 982–989. Klein, M., Ross, F., et al. (1994). Effect of online literature searching on length of stay and patient care costs. Academic Medicine, 69: 489–495. Kleinberg, J. (1998). Authoritative sources in a hyperlinked environment. Journal of the ACM, 46: 604–632. Kleinberg, J. and Lawrence, S. (2001). The structure of the Web. Science, 294: 1849–1850. Klemencic, E. and Todorovski, L. (1999). Analysis of free MEDLINE on the WWW. Studies in Health Technology and Informatics, 68: 553–556.
References
479
Knaus, D., Mittendorf, E., et al. (1994). Improving a basic retrieval method by links and passage level evidence. Overview of the Third Text REtrieval Conference (TREC-3), Gaithersburg, MD. National Institute of Standards and Technology. 241–246. Knight, E., Glynn, R., et al. (2000). Failure of evidence-based medicine in the treatment of hypertension in older patients. Journal of General Internal Medicine, 15: 702–709. Knirsch, C., Jain, N., et al. (1998). Respiratory isolation of tuberculosis patients using clinical guidelines and an automated clinical decision support system. Infection Control and Hospital Epidemiology, 19: 94–100. Kochen, C. and Budd, J. (1992). The persistence of fraud in the literature: The Darsee case. Journal of the American Society for Information Science, 43: 488–493. Kock, N. (1999). A case of academic plagiarism. Communications of the ACM, 42(7): 96–104. Koenemann, J., Quatrain, R., et al. (1994). New tools and old habits: the interactive searching behavior of expert online searchers using INQUERY. Overview of the Third Text REtrieval Conference (TREC-3), Gaithersburg, MD. National Institute of Standards and Technology. 145–177. Korn, F. and Shneiderman, B. (1996). MeSHBROWSE: A Tool for Browsing Medical Terms. ftp://ftp.cs.umd.edu/pub/hcil/Reports-Abstracts-Bibliography/96-01html/9601.html. Accessed: July 1, 2002. Kramer, M. and Feinstein, A. (1981). Clinical biostatistics. LIV. The biostatistics of concordance. Clinical Pharmacology and Therapeutics, 29: 111–123. Krauthammer, M., Rzhetsky, A., et al. (2000). Using BLAST for identifying gene and protein names in journal articles. Gene, 259: 245–252. Krovetz, R. (1993). Viewing morphology as an inference process. Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Pittsburgh, PA. ACM Press. 191–202. Krovetz, R. and Croft, W. (1992). Lexical ambiguity and information retrieval. ACM Transactions on Information Systems, 10: 115–141. Kucera, H. and Francis, W. (1967). Computational Analysis of Present-Day American English. Providence, RI, Brown University Press. Kuhn, K., Gaus, W., et al. (1992). Structured reporting of medical findings: evaluation of a system in gastroenterology. Methods of Information in Medicine, 31: 268–274. Kuhn, T. (1962). The Structure of Scientific Revolutions. Chicago, University of Chicago Press. Kunin, C., Tupasi, T., et al. (1973). Use of antibiotics—a brief exposition of the problem and some tentative solutions. Annals of Internal Medicine, 79: 555–560. Kunst, H., Groot, D., et al. (2002). Accuracy of information on apparently credible websites: survey of five common health topics. British Medical Journal, 324: 581–582. Kuperman, G., Garnder, R., et al. (1991). HELP: A Dynamic Hospital Information System. New York, Springer-Verlag. Kupiec, J., Pedersen, J., et al. (1995). A trainable document summarizer. Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA. ACM Press. 68–73. Kwok, K. (1999). Employing multiple representations for Chinese information retrieval. Journal of the American Society for Information Science, 99: 709–723. Kwok, K., Grunfeld, L., et al. (1994). TREC-3 ad-hoc, routing retrieval, and thresholding experiments using PIRCS. Overview of the Third Text REtrieval Conference (TREC3), Gaithersburg, MD. National Institute of Standards and Technology. 247–255. Lacher, D., Nelson, E., et al. (2000). Computer use and needs of internists: a survey of mem-
480
References
bers of the American College of Physicians—American Society of Internal Medicine. Proceedings of the AMIA 2000 Annual Symposium, Los Angeles, CA. Hanley & Belfus, 453–456.. Lafferty, J. and Zhai, C. (2001). Document language models, query models, and risk minimization for information retrieval. Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, LA. ACM Press. 111–119. Lagergren, E. and Over, P. (1998). Comparing interactive information retrieval systems across sites: the TREC-6 interactive track matrix experiment. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia. ACM Press. 162–172. Lagoze, C. and Van de Sompel, H. (2001). The Open Archives Initiative: building a lowbarrier interoperability framework. Proceedings of the First ACM/IEEE-CS Joint Conference on Digital Libraries, Roanoke, VA. ACM Press. 54–62. Laine, C., Proto, A., et al. (2001). A multi-journal authorship database: variation in author contributions by byline position. Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain. American Medical Association. http://www.ama-assn.org/public/peer/peerhome.htm. Lancaster, F. (1968). Evaluation of the MEDLARS Demand Search Service, Bethesda, MD, National Library of Medicine. Lancaster, F. (1978). Information Retrieval Systems: Characteristics, Testing, and Evaluation. New York, Wiley. Lancaster, F. and Warner, A. (1993). Information Retrieval Today. Arlington, VA, Information Resources Press. Landauer, T. and Dumais, S. (1997). A solution to Plato’s problem: the latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review, 104: 211–240. Landauer, T. and Littman, M. (1990). Fully automatic cross-language document retrieval using latent semantic indexing. Proceedings of the Sixth Annual Conference of the UW Centre for the New Oxford English Dictionary and Text Research, Waterloo, Ontario. UW Centre for the New OED and Text Research. 31–38. Langham, J., Thompson, E., et al. (1999). Identification of randomized controlled trials from the emergency medicine literature: comparison of hand searching versus MEDLINE searching. Annals of Emergency Medicine, 34: 25–34. Larkey, L. and Croft, W. (1996). Combining classifiers in text categorization. Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Zurich, Switzerland. ACM Press. 289–297. Larkey, L., Ogilvie, P., et al. (2000). Acrophile: an automated acronym extractor and server. Proceedings of the Fifth ACM Conference on Digital Libraries, San Antonio, TX. ACM Press. 205–214. Lassila, O., Hendler, J., et al. (2001). The Semantic Web. Scientific American, 284(5): 34–43. Latthe, P., Latthe, M., et al. (2000a). Quality of information about menorrhagia on the World Wide Web. British Journal of Obstetrics and Gynaecology, 107: 39–43. Latthe, M., Latthe, P., et al. (2000b). Quality of information on emergency contraception on the Internet. British Journal of Family Planning, 26: 39–43. Lavrenko, V. and Croft, W. (2001). Relevance-based language models. Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, LA. ACM Press. 120–127.
References
481
Lawrence, S. (2001). Online or invisible? Nature, 411: 521. Lawrence, S. and Giles, C. (1999). Accessibility and distribution of information on the Web. Nature, 400: 107–109. Lawrence, S., Giles, C., et al. (1999). Digital libraries and autonomous citation indexing. Computer, 32: 67–71. Lebowitz, M. (1983). Memory-based parsing. Artificial Intelligence, 21: 363–404. Lee, K., Schotland, M., et al. (2002). Association of journal quality indicators with methodological quality of clinical research articles. Journal of the American Medical Association, 287: 2805–2808. Lehmann, H. and Shortliffe, E. (1991). THOMAS: building Bayesian statistical expert systems to aid in clinical decision making. Computer Methods and Programs in Biomedicine, 35: 251–260. Leigh, T., Young, P., et al. (1993). Performances of family practice diplomates on successive mandatory recertification examinations. Academic Medicine, 68: 912–921. Leighton, H. and Srivastava, J. (1999). First 20 precision results among World Wide Web search services (search engines). Journal of the American Society for Information Science. 50: 870–881. Lenat, D. and Guha, R. (1990). Building Large Knowledge Bases. Reading, MA, AddisonWesley. Lenert, L. and Tovar, M. (1993). Automated linkage of free-text descriptions of patients with a practice guideline. Proceedings of the 17th Annual Symposium on Computer Applications in Medical Care, Washington, DC. McGraw-Hill. 274–278. Leonard, L. (1975). Inter-indexer Consistency and Retrieval Effectiveness: Measurement of Relationships, Champaign, University of Illinois. Leroy, G. and Chen, H. (2002). Filling preposition-based templates to capture information from medical abstracts. Pacific Symposium on Biocomputing, Kauai, Hawaii. World Scientific. 350–361. Lesk, M. (1997). Practical Digital Libraries—Books, Bytes, & Bucks. San Francisco, Morgan Kaufmann. Lesk, M. and Salton, G. (1968). Relevance assessments and retrieval system evaluation. Information Storage and Retrieval, 4: 343–359. Levin, A. (2001). The Cochrane Collaboration. Annals of Internal Medicine, 135: 309–312. Lewis, D. (1995). Evaluating and optimizing autonomous text classification systems. Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA. ACM Press. 246–254. Lewis, D. (1996). The TREC-5 filtering track. The Fifth Text REtrieval Conference (TREC5), Gaithersburg, MD. National Institute of Standards and Technology. 75–96. Liddy, E. (2000). Text mining. Bulletin of the American Society for Information Science, 27(1): 13–14. Lijmer, J., Mol, B., et al. (1999). Empirical evidence of design-related bias in studies of diagnostic tests. Journal of the American Medical Association, 282: 1061–1066. Lin, R., Lenert, L., et al. (1991). A free-text processing system to capture physical findings: canonical phrase identification system (CAPIS). Proceedings of the 15th Annual Symposium on Computer Applications in Medical Care, Washington, DC. McGrawHill. 843–847. Lindberg, D., Humphreys, B., et al. (1993). The Unified Medical Language System project. Methods of Information in Medicine, 32: 281–291. Lindberg, D., Siegel, E., et al. (1993). Use of MEDLINE by physicians for clinical problem solving. Journal of the American Medical Association, 269: 3124–3129.
482
References
Link, A. (1998). US and non-US submissions: An analysis of reviewer bias. Journal of the American Medical Association, 280: 246–247. Lissman, T. and Boehnlein, J. (2001). A critical review of Internet information about depression. Psychiatric Services, 52: 1046–1050. Littenberg, B. (1992). Technology assessment in medicine. Academic Medicine, 67: 424–428. Liu, H., Johnson, S., et al. (2002). Automatic resolution of ambiguous terms based on machine learning and conceptual relations in the UMLS. Journal of the American Medical Informatics Association. In press. Liu, H., Lussier, Y., et al. (2001). Disambiguating ambiguous biomedical terms in biomedical narrative text: an unsupervised method. Journal of Biomedical Informatics, 34: 249–261. Losee, R. (1990). The Science of Information. San Diego, CA, Academic Press. Lovins, J. (1968). Development of a stemming algorithm. Mechanical Translation and Computational Linguistics, 11: 11–31. Lovis, C., Chapko, M., et al. (2001). Evaluation of a command-line parser-based order entry pathway for the Department of Veterans Affairs electronic patient record. Journal of the American Medical Informatics Association, 8: 486–498. Ludwig, L., Mixter, J., et al. (1988). User attitudes toward end-user literature searching. Bulletin of the Medical Library Association, 76: 7–13. Luhn, H. (1957). A statistical approach to mechanized encoding and searching of literary information. IBM Journal of Research and Development, 1: 309–317. Lundberg, G., Paul, M., et al. (1998). A comparison of the opinions of experts and readers as to what topics a general medical journal (JAMA) should address. Journal of the American Medical Association, 280: 288–290. Lundeen, G., Tenopir, C., et al. (1994). Information needs of rural health care practitioners in Hawaii. Bulletin of the Medical Library Association, 82: 197–205. Lussier, Y., Shagina, L., et al. (2000). Automating ICD-9-CM encoding using medical language processing: a feasibility study. Proceedings of the AMIA 2000 Annual Symposium, Los Angeles, CA. Hanley & Belfus. 1072. Lussier, Y., Shagina, L., et al. (2001). Automating SNOMED coding using medical language understanding: a feasibility study. Proceedings of the 2001 AMIA Annual Symposium, Washington, DC. Hanley & Belfus. 418–422. Macleod, C., Chen, S., et al. (1987). Parsing Unedited Medical Narrative, 163–173, in Sager, N., Friedman, C. and Lyman, M., eds. Medical Language Processing: Computer Management of Narrative Data. Reading, MA. Addison-Wesley. Malet, G., Munoz, F., et al. (1999). A model for enhancing Internet medical document retrieval with “medical core metadata.” Journal of the American Medical Informatics Association, 6: 183–208. Mamlin, J., and Baker, D. (1973). Combined time-motion and work sampling study in a general medicine clinic. Medical Care, 11: 449–456. Mandel, C. (1988). Trade-offs: quantifying quality in library technical services. Journal of Academic Librarianship, 14: 214–220. Mani, I. and Bloedorn, E. (1998). Machine learning of generic and user-focused summarization. Proceedings of the 15th National Conference on Artificial Intelligence and 10th Innovative Applications of Artificial Intelligence Conference, Madison, WI. AAAI Press. 821–826. Mani, I. and Maybury, M. (1999). Advances in Automatic Text Summarization. Cambridge, MA, MIT Press.
References
483
Marchionini, G. (1992). Interfaces for end-user information seeking. Journal of the American Society for Information Science, 43: 156–163. Marchionini, G. (1994). Designing Hypertexts: Start with an Index, 77–89, in Fidel, R., Hahn, T., Rasmussen, E. and Smith, P., eds. Challenges in Electronic Text and Images. Medford, NJ. Learned Information. Marcus, M., Santorini, B., et al. (1994). Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19: 313–330. Marcus, R. (1983). An experimental comparison of the effectiveness of computer and humans as search intermediaries. Journal of the American Society for Information Science, 34: 381–404. Markert, R., Parisi, A., et al. (1989). Medical student, resident, and faculty use of a computerized literature searching system. Bulletin of the Medical Library Association, 77: 133–137. Marsh, E. and Sager, N. (1982). Analysis and processing of compact texts. COLING 82: Proceedings of the Ninth International Conference on Computational Linguistics. North-Holland. 201–206. Marshall, E. (1999). E-biomed morphs to E-biosci, focus shifts to reviewed papers. Science, 285: 810–811. Marshall, J. (1990). The continuation of end-user online searching by health professionals: preliminary survey results. Proceedings of the Medical Library Association Annual Meeting, Detroit, MI. Medical Library Association. Marshall, J. (1992). The impact of the hospital library on decision making: the Rochester study. Bulletin of the Medical Library Association, 80: 169–178. Masarie, F. and Miller, R. (1987). Medical Subject Headings and medical terminology: an analysis of terminology used in hospital charts. Bulletin of the Medical Library Association, 75: 89–94. Masarie, F., Miller, R., et al. (1991). An interlingua for electronic exchange of medical information: using frames to map between clinical vocabularies. Computers and Biomedical Research, 24: 379–400. Masuda, D., Tarczy-Hornoch, P., et al. (2001). The disparity between demand and use of electronic library resources in rural clinical practice. Unpublished manuscript. Masys, D. (1992). An evaluation of the source selection elements of the prototype UMLS information sources map. Proceedings of the 16 Annual Symposium on Computer Applications in Medical Care, Baltimore, MD. McGraw-Hill. 295–298. Masys, D., Welsh, J., et al. (2001). Use of keyword hierarchies to interpret gene expression patterns. Bioinformatics, 7: 319–326. Mathis, Y., Huisman, L., et al. (1994). Medical literature searches. Bulletin of the Medical Library Association, 69: 360. Mattick, P. (1987). Sublanguage Analysis of Articles in Lipid Metabolism, 221–251, in Sager, N., ed. Medical Language Processing: Computer Management of Narrative Data. Reading, MA. Addison-Wesley. McAlister, F., Clark, H., et al. (1999). The medical review article revisited: has the science improved? Annals of Internal Medicine, 131: 947–951. McAuley, L., Tugwell, P., et al. (2000). Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet, 356: 1228–1231. McCain, K., White, H., et al. (1987). Comparing retrieval performance in online databases. Information Processing and Management, 23: 539–553. McClung, H., Murray, R., et al. (1998). The Internet as a source for current patient information. Pediatrics, 101: E2.
484
References
McCray, A. (1998). The nature of lexical knowledge. Methods of Information in Medicine, 37: 353–360. McCray, A. and Gallagher, M. (2001). Principles for digital library development. Communications of the ACM, 44: 49–54. McCray, A., Srinivasan, S., et al. (1994). Lexical methods for managing variation in biomedical terminologies. Proceedings of the 18th Annual Symposium on Computer Applications in Medical Care, Washington, DC. Hanley & Belfus. 235–239. McDonald, C. (1996). Medical heuristics: the silent adjudicators of clinical practice. Annals of Internal Medicine, 124: 56–62. McGowan, J. and Richwine, M. (2000). Electronic information access in support of clinical decision making: a comparative study of the impact on rural health outcomes. Proceedings of the AMIA 2000 Annual Symposium, Los Angeles, CA. Hanley & Belfus. 565–569. McKibbon, K., Haynes, R., et al. (1990). How good are clinical MEDLINE searches? A comparative study of clinical end-user and librarian searches. Computers and Biomedical Research, 23(6): 583–593. McKibbon, K., Wilczynski, N., et al. (1995). The medical literature as a resource for health care practice. Journal of the American Society for Information Science, 46: 737–742. McKinin, E., Sievert, M., et al. (1991). The MEDLINE/full-text research project. Journal of the American Society for Information Science, 42: 297–307. McKnight, M. and Peet, M. (2000). Health care providers’ information seeking: recent research. Medical Reference Services Quarterly, 19: 27–51. McLellan, F. (2001). 1966 and all that—when is a literature search done? Lancet, 358: 646. McNutt, R., Evans, A., et al. (1990). The effect of blinding on the quality of peer review. Journal of the American Medical Association, 263: 1371–1376. Meadow, C. (1985). Relevance? Journal of the American Society for Information Science, 36: 354–355. Meadow, C. (1992). Text Information Retrieval Systems. San Diego, CA, Academic Press. Meek, J. (2001). Science world in revolt at power of the journal owners. The Guardian. United Kingdom. www.guardian.co.uk/Archive/Article/0,4273,4193292,00.html. Melucci, M. (1999). An evaluation of automatically constructed hypertexts for information retrieval. Information Retrieval, 1: 89–112. Mendonca, E. and Cimino, J. (1999). Evaluation of the Information Sources Map. Proceedings of the AMIA 1999 Annual Symposium, Washington, DC. Hanley & Belfus. 873–877. Mendonca, E., Cimino, J., et al. (2001a). Accessing heterogeneous sources of evidence to answer clinical questions. Journal of Biomedical Informatics, 34: 85–98. Mendonca, E., Cimino, J., et al. (2001b). Using narrative reports to support a digital library. Proceedings of the 2001 AMIA Annual Symposium, Washington, DC. Hanley & Belfus. 458–462. Miles, A., Bentley, P., et al. (1997). Evidence-based medicine? Why all the fuss? This is why. Journal of Evaluation in Clinical Practice, 3: 83–86. Miles, J., Petrie, C., et al. (2000). Slimming on the Internet. Journal of the Royal Society of Medicine, 93: 254–257. Miles, W. (1982). A History of the National Library of Medicine: The Nation’s Treasury of Medical Knowledge. Bethesda, MD, U.S. Department of Health and Human Services. Miller, D., Leek, T., et al. (1999). A Hidden Markov Model information retrieval system.
References
485
Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Berkeley, CA. ACM Press. 214–221. Miller, E. (1998). An introduction to the Resource Description Framework. D-Lib Magazine. 4. http://www.dlib.org/dlib/may98/miller/05miller.html. Miller, N., Kirby, M., et al. (1988). MEDLINE on CD-ROM: end user searching in a medical school library. Medical Reference Services Quarterly, 7(3): 1–13. Miller, N., Lacroix, E., et al. (2000). MEDLINEplus: building and maintaining the National Library of Medicine’s consumer health Web service. Bulletin of the Medical Library Association, 88: 11–17. Miller, P. (1999). Z39.50 for all. Ariadne, 21. http://www.ariadne.ac.uk/issue21/z3950/. Miller, P., Frawley, S., et al. (1995). Lessons learned from a pilot implementation of the UMLS information sources map. Journal of the American Medical Informatics Association, 2: 102–115. Miller, R., Gieszczykiewicz, F., et al. (1992). CHARTLINE: providing bibliographic references relevant to patient charts using the UMLS Metathesaurus knowledge sources. Proceedings of the 16th Annual Symposium on Computer Applications in Medical Care, Baltimore, MD. McGraw-Hill. 86–90. Miller, R., Masarie, F., et al. (1986). Quick Medical Reference (QMR) for diagnostic assistance. M.D. Computing, 3(5): 34–48. Miller, R., McNeil, M., et al. (1986). The Internist-1/Quick Medical Reference project— status report. Western Journal of Medicine, 145: 816–822. Mitchell, E. and Sullivan, F. (2001). A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980–97. British Medical Journal, 322: 279–282. Mitchell, J., Johnson, E., et al. (1992). Medical students using Grateful Med: analysis of failed searches and a six-month follow-up study. Computers and Biomedical Research, 25: 43–55. Mitchell, T. (1997). Machine Learning. New York, McGraw-Hill. Mitra, M., Singhal, A., et al. (1998). Improving automatic query expansion. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia. ACM Press. 206–214. Moher, D., Dulberg, C., et al. (1994). Statistical power, sample size, and their reporting in randomized controlled trials. Journal of the American Medical Association, 272: 122–124. Moher, D., Jones, A., et al. (2001). Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. Journal of the American Medical Association, 285: 2006–2007. Moher, D., Pham, B., et al. (1998). Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet, 352: 609–613. Moher, D., Schulz, K., et al. (2001). The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. Annals of Internal Medicine, 134: 657–662. Molyneux, R. (1989). ACRL University Library Statistics. Chicago, Association of Research Libraries. Mooers, C. (1951). Zatocoding applied to mechanical organisation of knowledge. American Documentation, 2: 20–32. Moore, G. and Berman, J. (1994). Performance analysis of manual and automated systemized nomenclature of medicine (SNOMED) coding. American Journal of Clinical Pathology, 101: 253–256. Moore, G., Hutchins, G., et al. (1987). Word root translation of 45,564 autopsy reports
486
References
into MeSH titles. Proceedings of the 11th Annual Symposium on Computer Applications in Medical Care, Washington, DC. IEEE. 128–132. Moorman, P., Branger, P., et al. (2001). Electronic messaging between primary and secondary care: a four-year case report. Journal of the American Medical Informatics Association, 8: 372–378. Morris, W., Heinze, D., et al. (2000). Assessing the accuracy of an automated coding system in emergency medicine. Proceedings of the AMIA 2000 Annual Symposium, Los Angeles, CA. Hanley & Belfus. 595–599. Mullaly-Quijas, P., Ward, D., et al. (1994). Using focus groups to discover health professionals’ information needs: a regional marketing study. Bulletin of the Medical Library Association, 82: 305–311. Mullins, R., Veum-Stone, J., et al. (1994). Outcome of hospitalized injured patients after institution of a trauma system in an urban area. Journal of the American Medical Association, 271: 1919–1924. Mulrow, C. (1987). The medical review article: state of the science. Annals of Internal Medicine, 106: 485–488. Mulrow, C. and Cook, D., eds. (1998). Systematic Reviews: Synthesis of Best Evidence for Health Care Decisions. Philadelphia, American College of Physicians. Mulrow, C., Cook, D., et al. (1997). Systematic reviews: critical links in the great chain of evidence. Annals of Internal Medicine, 126: 389–391. Murphy, P. (1994). Reading ability of parents compared with reading level of pediatric patient education materials. Pediatrics, 93: 460–468. Musen, M. (1992). Dimensions of knowledge sharing and reuse. Computers and Biomedical Research, 25: 435–467. Musen, M., Shahar, Y., et al. (2000). Clinical Decision-Support Systems, 573–609, in Shortliffe, E. and Perreault, L., eds. Medical Informatics: Computer Applications in Health Care and Biomedicine. New York. Springer-Verlag. Mutalik, P., Deshpande, A., et al. (2001). Use of general-purpose negation detection to augment concept indexing of medical documents: a quantitative study using the UMLS. Journal of the American Medical Informatics Association, 8: 598–609. Mynatt, B., Leventhal, L., et al. (1992). Hypertext or book: which is better for answering questions? Proceedings of Computer–Human Interface 92, 19–25. Nadkarni, P., Chen, R., et al. (2001). UMLS concept indexing for production databases: a feasibility study. Journal of the American Medical Informatics Association, 8: 80–91. Nankivell, C., Wallis, P., et al. (2001). Networked information and clinical decision making: the experience of Birmingham Heartlands and Solihull National Health Service Trust (Teaching). Medical Education, 35: 167–172. Nelson, J. (1992). An analysis of transaction logs to evaluate the educational needs of end users. Medical Reference Services Quarterly, 11(4): 11–21. Nelson, S., Schopen, M., et al. (2000). An Interlingual Database of MeSH Translations. National Library of Medicine. http://www.nlm.nih.gov/mesh/intlmesh.html. Accessed: July 1, 2002. Nelson, T. (1987). Computer Lib, 2nd ed. Redmond, WA, Microsoft Press. Ng, H. and Zelle, J. (1997). Corpus-based approaches to semantic interpretation in NLP. AI Magazine, 18: 45–64. Nielsen, J. (1993). Usability Engineering. San Diego, CA, Academic Press. Nielsen, J. and Levy, J. (1994). Measuring usability: preference vs. performance. Communications of the ACM, 37: 66–75.
References
487
Nigg, H. and Radulsecu, G. (1994). Scientific misconduct in environmental science and toxicology. Journal of the American Medical Association, 272: 168–170. Norman, F. (1996). Organizing medical networked information: OMNI. Medical Informatics, 23: 43–51. Norman, G. (1999). Examining the assumptions of evidence-based medicine. Journal of Evaluation in Clinical Practice, 5: 139–147. Norman, G. and Shannon, S. (1998). Effectiveness of instruction in critical appraisal (evidence-based medicine): a critical appraisal. Canadian Medical Association Journal, 158: 177–181. Noy, N., Sintek, M., et al. (2001). Semantic Web contents with Protégé-2000. IEEE Intelligent Systems, 16: 60–71. Nylenna, M., Riis, P., et al. (1994). Multiple blinded reviews of the same two manuscripts: effect of referee characteristics and publication language. Journal of the American Medical Association, 272: 149–151. Oard, D. (1997). Alternative approaches for cross-language text retrieval. AAAI Spring Symposium on Cross-Language Text and Speech Retrieval, Palo Alto, CA. AAAI. http://www.ee.umd.edu/medlab/filter/sss/papers/oard/paper.html. Oliver, N. (1992). A Sublanguage-Based Medical Language Processing System for German. Department of Computer Science, Ph.D. Thesis. New York University. Olson, C., Rennie, D., et al. (2001). Speed of publication for submitted manuscripts by direction of study results. Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain. American Medical Association. http://www.amaassn.org/pubic/peer/peerhome.htm. Olson, C., Rennie, D., et al. (2002). Publication bias in editorial decision making. Journal of the American Medical Association, 287: 2825–2828. O’Mahony, B. (1999). Irish health care Web sites: a review. Irish Medical Journal, 92: 334–337. Osheroff, J. and Bankowitz, R. (1993). Physicians’ use of computer software in answering clinical questions. Bulletin of the Medical Library Association, 81: 11–19. Osheroff, J., Forsythe, D., et al. (1991). Physicians’ information needs: analysis of questions posed during clinical teaching. Annals of Internal Medicine, 114: 576–581. Over, P. (2000). The TREC Interactive Track: an annotated bibliography. Information Processing and Management, 369–382. Overland, J., Hoskins, P., et al. (1993). Low literacy: a problem in diabetes education. Diabetes Medicine, 10: 847–850. Paepcke, A., Brandriff, R., et al. (2000). Search middleware and the Simple Digital Library Interoperability Protocol. D-Lib Magazine, 6. http://www.dlib.org/dlib/march00/ paepcke/03paepcke.html. Pandolfini, C., Impiccatore, P., et al. (2000). Parents on the Web: risks for quality management of cough in children. Pediatrics, 105: 1–8. Pao, M. (1986). An empirical examination of Lotka’s law. Journal of the American Society for Information Science, 37: 26–33. Pao, M. (1989). Concepts of Information Retrieval. Englewood, CO, Libraries Unlimited. Pao, M. (1993). Perusing the literature via citation links. Computers and Biomedical Research, 26: 143–156. Pao, M., Grefsheim, S., et al. (1993). Factors affecting students’ use of MEDLINE. Computers and Biomedical Research, 26: 541–555. Pao, M., Grefsheim, S., et al. (1994). Effect of search experience on sustained MEDLINE usage by students. Academic Medicine, 69: 914–920.
488
References
Pao, M. and Worthen, D. (1989). Retrieval effectiveness by semantic and citation searching. Journal of the American Society for Information Science, 40: 226–235. Paris, L. and Tibbo, H. (1998). Freestyle vs. Boolean: a comparison of partial and exact match retrieval systems. Information Processing and Management, 34: 175–190. Park, J., Kim, H., et al. (2001). Bidirectional incremental parsing for automatic pathway identification with combinatory categorial grammar. Pacific Symposium on Biocomputing, Kauai, HI. World Scientific. 396–407. Parker, L. and Johnson, R. (1990). Does order of presentation affect users’ judgment of documents? Journal of the American Society for Information Science, 41: 493–494. Pasca, M. and Harabagiu, S. (2001). High performance question/answering. Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, LA. ACM Press. 366–374. Paskin, N. (1999). DOI: current status and outlook. D-Lib Magazine, 5. http://www.dlib. org/dlib/may99/05paskin.html Patel, V., Evans, D., et al. (1989). Biomedical Knowledge and Clinical Reasoning, 53–112, in Evans, D. and Patel, V., eds. Cognitive Science in Medicine: Biomedical Modeling. Cambridge, MA. MIT Press. Peters, D. and Ceci, S. (1982). Peer-review practices of psychological journals: the fate of published articles, submitted again. Behavioral and Brain Science, 5: 187–255. Petersdorf, R. (1989). General internal medicine: fad or future. Journal of General Internal Medicine, 2: 527–532. Peterson, J. (1980). Computer programs for detecting and correcting spelling errors. Communications of the ACM, 23: 676–687. Petticrew, M. (2001). Systematic reviews from astronomy to zoology: myths and misconceptions. British Medical Journal, 322: 98–101. Pitkin, R., Branagan, M., et al. (1999). Accuracy of data in abstracts of published research articles. Journal of the American Medical Association, 281: 1110–1111. Poisson, E. (1986). End-user searching in medicine. Bulletin of the Medical Library Association. 74: 293–299. Pollitt, A. (1987). CANSEARCH: an expert systems approach to document retrieval. Information Processing and Management, 23: 119–136. Ponte, J. and Croft, W. (1998). A language modeling approach to information retrieval. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia. ACM Press. 275–281. Poon, A., Fagan, L., et al. (1996). The PEN–Ivory project: exploring user-interface design for the selection of items from large controlled vocabularies of medicine. Journal of the American Medical Informatics Association, 3: 168–183. Porter, M. (1980). An algorithm for suffix stripping. Program, 14: 130–137. Powsner, S. and Miller, P. (1989). Linking bibliographic retrieval to clinical reports: Psychtopix. Proceedings of the 13th Annual Symposium on Computer Applications in Medical Care, Washington, DC. IEEE. 431–435. Powsner, S. and Miller, P. (1992). Automated online transition from the medical record to the psychiatric literature. Methods of Information in Medicine, 31: 169–174. Powsner, S., Riely, C., et al. (1989). Automated bibliographic retrieval based on current topics in hepatology: hepatopix. Computers and Biomedical Research, 22: 552–564. Pratt, W. and Fagan, L. (2000). The usefulness of dynamically categorizing search results. Journal of the American Medical Informatics Association, 7: 605–617. Pratt, W., Hearst, M., et al. (1999). A knowledge-based approach to organizing retrieved documents. Proceedings of the 16th National Conference on Artificial Intelligence, Orlando, FL. AAAI. 80–85.
References
489
Pratt, W. and Wasserman, H. (2001). QueryCat: automatic categorization of MEDLINE queries. Proceedings of the 2001 AMIA Annual Symposium, Washington, DC. Hanley & Belfus. 655–659. Price, D. (1963). Little Science, Big Science. New York, Columbia University Press. Price, D. (1965). Networks of scientific papers. Science, 149: 510–515. Price, S. and Hersh, W. (1999). Filtering Web pages for quality indicators: an empirical approach to finding high quality consumer health information on the World Wide Web. Proceedings of the AMIA 1999 Annual Symposium, Washington, DC. Hanley & Belfus. 911–915. Price, S., Hersh, W., et al. (2002). SmartQuery: context-sensitive links to medical knowledge sources from the electronic patient record. Proceedings of the 2002 Annual AMIA Symposium, San Antonio, TX. Hanley & Belfus. in press. Pruitt, K. and Maglott, D. (2002). RefSeq and LocusLink: NCBI gene-centered resources. Nucleic Acids Research, 29: 137–140. Pryor, T., Gardner, R., et al. (1984). The HELP System, 109–128, in Blum, B., ed. Information Systems for Patient Care. New York. Springer-Verlag. Pugh, G. and Tan, J. (1994). Computerized databases for emergency care: what impact on patient care? Methods of Information in Medicine, 33: 507–513. Purcell, G., Donovan, S., et al. (1998). Changes to manuscripts during the editorial process: characterizing the evolution of a clinical paper. Journal of the American Medical Association, 280: 227–228. Pustejovsky, J., Castano, J., et al. (2001). Automatic extraction of acronym–meaning pairs from MEDLINE databases. MEDINFO 2001, London. Pustejovsky, J., Castano, J., et al. (2002). Robust relational parsing over biomedical literature: extracting inhibit relations. Pacific Symposium on Biocomputing, Kauai, Hawaii. World Scientific. 362–373. Quint, B. (1992). The last librarians: end of a millennium. Canadian Journal of Information Science, 17: 33–40. Rabenback, L., Viscoli, C., et al. (1992). Problems in the conduct and analysis of randomized controlled trials. Archives of Internal Medicine, 152: 507–512. Ramsey, P., Carline, J., et al. (1991). Changes over time in the knowledge base of practicing internists. Journal of the American Medical Association, 266: 1103–1108. Rauch, S., Holt, M., et al. (1994). Community hospitals and the Internet: lessons from pilot connections. Bulletin of the Medical Library Association, 82: 401–406. Raychaudhuri, S., Chang, J., et al. (2002). Associating genes with Gene Ontology codes using a maximum entropy analysis of biomedical literature. Genome Research, 12: 203–214. Readings, B. (1994). Caught in the net: notes from the electronic underground. Surfaces, 4: 9–10. Rector, A., Nowlan, W., et al. (1991). Foundations for an electronic medical record. Methods of Information in Medicine, 30: 179–186. Redman, P., Kelly, J., et al. (1997). Common ground: the HealthWeb project as a model for Internet collaboration. Bulletin of the Medical Library Association, 85: 325–330. Rees, A. (1966). The relevance of relevance to the testing and evaluation of document retrieval systems. Aslib Proceedings, 18: 316–324. Rees, A. and Schultz, D. (1967). A Field Experimental Approach to the Study of Relevance Assessments in Relation to Document Searching. Cleveland, OH, Center for Documentation and Communication Research, Case Western Reserve University. Regazzi, J. (1988). Performance measures for information retrieval systems—an experimental approach. Journal of the American Society for Information Science, 39: 235–251.
490
References
Reich, V. and Rosenthal, D. (2001). LOCKSS: a permanent Web publishing and access system. D-Lib Magazine, 7. http://www.dlib.org/dlib/june01/reich/06reich.html. Reid, M., Lachs, M., et al. (1995). Use of methodologic standards in diagnostic test research: getting better but still not good. Journal of the American Medical Association, 274: 645–651. Relman, A. (1999). The NIH “PubMed Central” proposal: a potential threat to the evaluation and orderly dissemination of new clinical studies. New England Journal of Medicine, 340: 1828–1829. Rennels, G., Shortliffe, E., et al. (1987). A computational model of reasoning from the clinical literature. Computer Methods and Programs in Biomedicine, 24: 139–149. Rennie, D. (1997). Thyroid storm. Journal of the American Medical Association, 277: 1238–1243. Resiel, J. and Schneiderman, B. (1987). Is Bigger Better? The Effects of Display Size on Program Reading, 113–122, in Salvendy, G., ed. Social, Ergonomic and Stress Aspects of Work with Computers. Amsterdam. Elsevier Science. Richardson, R. and Smeaton, A. (1995). Automatic word sense disambiguation in a KBIR application. New Review of Document and Text Management, 1: 299–319. Richart, R. (1970). Evaluation of a medical data system. Computers and Biomedical Research, 3: 415–425. Riesenberg, L. and Dontineni, S. (2001). Review of reference inaccuracies. Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain. American Medical Association. http://www.ama-assn.org/public/peer/peerhome.htm. Rindflesch, T., Bean, C., et al. (2000). Argument identification for arterial branching predications asserted in cardiac catheterization reports. Proceedings of the AMIA 2000 Annual Symposium, Los Angeles, CA. Hanley & Belfus. 704–708. Rindflesch, T., Hunter, L., et al. (1999). Mining molecular binding terminology from biomedical text. Proceedings of the AMIA 1999 Annual Symposium, Washington, DC. Hanley & Belfus. 127–131. Rindflesch, T., Tanabe, L., et al. (2000). EDGAR: extraction of drugs, genes and relations from the biomedical literature. Pacific Symposium on Biocomputing, Honolulu, HI. World Scientific. 517–528. Roberts, D. (1995). AMED: a bibliographic database for complementary medicine and allied health. Complementary Therapies in Medicine, 3: 255–258. Roberts, D. and Souter, C. (2000). The automation of controlled vocabulary subject indexing of medical journal articles. Aslib Proceedings, 52: 384–401. Roberts, P. (1999). Scholarly publishing, peer review, and the Internet. First Monday, 4. firstmonday.org/issues/issue4_4/proberts/. Roberts, R. (2001). PubMed Central: the GenBank of the published literature. Proceedings of the National Academy of Sciences, 98: 381–382. Robertson, S. and Hull, D. (2000). The TREC-9 Filtering Track final report. The Ninth Text REtrieval Conference (TREC-9), Gaithersburg, MD. National Institute of Standards and Technology. 25–40. Robertson, S. and Soboroff, I. (2001). The TREC 2001 Filtering Track report. The Tenth Text REtrieval Conference (TREC 2001), Gaithersburg, MD. National Institute of Standards and Technology, 26–37. Robertson, S. and Sparck-Jones, K. (1976). Relevance weighting of search terms. Journal of the American Society for Information Science, 27: 129–146. Robertson, S. and Thompson, C. (1990). Weighted searching: the CIRT experiment. Informatics 10: Prospects for Intelligent Retrieval, York. ASLIB. 153–166.
References
491
Robertson, S. and Walker, S. (1994). Some simple effective approximations to the 2Poisson model for probabilistic weighted retrieval. Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland. Springer-Verlag. 232–241. Robertson, S., Walker, S., et al. (1998). Okapi at TREC-7: automatic ad hoc, filtering, VLC, and interactive track. The Seventh Text REtrieval Conference (TREC-7), Gaithersburg, MD. National Institute of Standards and Technology. 253–264. Robertson, S., Walker, S., et al. (1994). Okapi at TREC-3. Overview of the Third Text REtrieval Conference (TREC-3), Gaithersburg, MD. National Institute of Standards and Technology. 109–126. Robinson, G., Tagare, H., et al. (1996). Medical image collection indexing. Computerized Medical Imaging and Graphics, 20: 209–217. Rochon, P., Bero, L., et al. (2002). Comparison of review articles published in peerreviewed and throwaway journals. Journal of the American Medical Association, 287: 2853–2856. Rodgers, R. (1995). Automated retrieval from multiple disparate information sources: the World Wide Web and the NLM’s Sourcerer project. Journal of the American Society for Information Science, 46: 755–764. Roethlisberger, F. and Dickson, W. (1939). Management and the Worker. Cambridge, MA, Harvard University Press. Rose, L. (1998). Factors Influencing Successful Use of Information Retrieval Systems by Nurse Practitioner Students. School of Nursing. M.S. Thesis. Oregon Health Sciences University. Rose, L., Crabtree, K., et al. (1998). Factors influencing successful use of information retrieval systems by nurse practitioner students. Proceedings of the AMIA 1998 Annual Symposium, Orlando, FL. Hanley & Belfus. 1067. Rosenberg, W., Deeks, J., et al. (1998). Improving searching skills and evidence retrieval. Journal of the Royal College of Physicians of London, 32: 557–563. Rothenberg, J. (1999). Ensuring the Longevity of Digital Information. RAND Corporation. Rothwell, D., Cote, R., et al. (1993). Developing a standard data structure for medical language—the SNOMED proposal. Proceedings of the 17th Annual Symposium on Computer Applications in Medical Care, Washington, DC. McGraw-Hill. 695–699. Rowlands, J., Forrester, W., et al. (1996). British Medical Association Library free MEDLINE service: survey of members taking part in an initial pilot project. Bulletin of the Medical Library Association, 84: 116–121. Royle, J., Blythe, J., et al. (1995). Literature search and retrieval in the workplace. Computers in Nursing, 13: 25–31. Rubin, J. (1994). Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. New York, Wiley. Ruland, C. (2002). Handheld technology to improve patient care: evaluating a support system for preference-based care planning at the bedside. Journal of the American Medical Informatics Association, 9: 192–201. Rzhetsky, A., Koike, T., et al. (2000). A knowledge model for analysis and simulation of regulatory networks. Bioinformatics, 16: 1120–1128. Sackett, D., Richardson, W., et al. (2000). Evidence-Based Medicine: How to Practice and Teach EBM. New York, Churchhill Livingstone. Sadowski, Z., Alexander, J., et al. (1999). Multicenter randomized trial and a systematic overview of lidocaine in acute myocardial infarction. American Heart Journal, 137: 792–798.
492
References
Sager, N. (1981). Natural Language Information Processing: A Computer Grammar of English and Its Applications. Reading, MA, Addison-Wesley. Sager, N., Friedman, C., et al. (1987). Medical Language Processing: Computer Management of Narrative Data. Reading, MA, Addison-Wesley. Sager, N., Lyman, M., et al. (1989). Adapting a medical language processor from English to French. MEDINFO 89, Singapore. Amsterdam, Elsevier Science. Sager, N., Lyman, M., et al. (1994a). Natural language processing and the representation of clinical data. Journal of the American Medical Informatics Association, 1: 142–160. Sager, N., Lyman, M., et al. (1994b). Automatic encoding into SNOMED III: a preliminary investigation. Proceedings of the 18th Annual Symposium on Computer Applications in Medical Care, Washington, DC. Hanley & Belfus. 230–234. Sager, N., Lyman, M., et al. (1995). Medical language processing: applications to patient data representation and automatic encoding. Methods of Information in Medicine, 34: 140–146. Salton, G. (1963). Associative document retrieval techniques using bibliographic information. Journal of the ACM, 10: 440–457. Salton, G. (1970). Automatic processing of foreign language documents. Journal of the American Society for Information Science, 21: 187–194. Salton, G. (1971). Automatic indexing using bibliographic citations. Journal of Documentation, 27: 98–110. Salton, G. (1983). Introduction to Modern Information Retrieval. New York, McGraw-Hill. Salton, G. (1991). Developments in automatic test retrieval. Science, 253: 974–980. Salton, G. and Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing and Management, 24: 513–523. Salton, G. and Buckley, C. (1990). Improving retrieval performance by relevance feedback. Journal of the American Society for Information Science, 41: 288–297. Salton, G. and Buckley, C. (1991). Global text matching for information retrieval. Science, 253: 1012–1015. Salton, G., Buckley, C., et al. (1990). On the application of syntactic methodologies in automatic text analysis. Information Processing and Management, 26: 73–92. Salton, G., Fox, E., et al. (1983). Extended Boolean information retrieval. Communications of the ACM, 26: 1022–1036. Salton, G. and Lesk, M. (1965). The SMART automatic document retrieval system: an illustration. Communications of the ACM, 8: 391–398. Sanderson, M. (1994). Word sense disambiguation and information retrieval. Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland. Springer-Verlag. 142–151. Saracevic, T. (1975). Relevance: a review of and a framework for the thinking on the notion in information science. Journal of the American Society for Information Science, 26: 321–343. Saracevic, T. and Kantor, P. (1988a). A study in information seeking and retrieving. II. Users, questions, and effectiveness. Journal of the American Society for Information Science, 39: 177–196. Saracevic, T. and Kantor, P. (1988b). A study of information seeking and retrieving. III. Searchers, searches, and overlap. Journal of the American Society for Information Science, 39: 197–216. Saracevic, T., Kantor, P., et al. (1988). A study of information seeking and retrieving. I. Background and methodology. Journal of the American Society for Information Science, 39: 161–176.
References
493
Schacher, L. (2001). Clinical librarianship: its value in medical care. Annals of Internal Medicine, 134: 717–720. Schamber, L., Eisenberg, M., et al. (1990). A re-examination of relevance: toward a dynamic, situational definition. Information Processing and Management, 26: 755–776. Scherer, R. and Langenberg, P. (2001). Full publication of results initially presented in abstracts: revisited. Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain. American Medical Association. http://www.ama-assn.org/ public/peer/peerhome.htm. Schilit, B., Price, M., et al. (1999). As we may read: the reading appliance revolution. Computer, 32: 65–73. Schmidt, H., Norman, G., et al. (1990). A cognitive perspective on medical expertise: theory and implications. Academic Medicine, 65: 611–621. Schulz, K., Chalmers, I., et al. (1994). Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. Journal of the American Medical Association, 272: 125–128. Schulz, K., Chalmers, I., et al. (1995). Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Journal of the American Medical Association, 273: 408–412. Schulz, S. and Hahn, U. (2000). Morpheme-based, cross-lingual indexing for medical document retrieval. International Journal of Medical Informatics, 58: 87–99. Schutze, H. and Pedersen, J. (1995). Information retrieval based on word senses. Proceedings of the 4th Annual Symposium on Document Analysis and Information Retrieval, Las Vegas, NV. University of Nevada, Las Vegas. 161–175. Schwartz, L., Woloshin, S., et al. (2002). Media coverage of scientific meetings: too much, too soon? Journal of the American Medical Association, 287: 2859–2863. Self, P., Filardo, T., et al. (1989). Acquired immunodeficiency syndrome (AIDS) and the epidemic growth of its literature. Scientometrics, 17: 49–60. Sewell, W. and Teitelbaum, S. (1986). Observations of end-user online searching behavior over eleven years. Journal of the American Society for Information Science, 37(4): 234–245. Shannon, C. and Weaver, W. (1949). The Mathematical Theory of Communication. Urbana, University of Illinois Press. Shaughnessy, A., Slawson, D., et al. (1994). Becoming an information master: a guidebook to the medical information jungle. Journal of Family Practice, 39: 489–499. Shearer, B., Seymour, A., et al. (2001). Bringing the best of medical librarianship to the patient team. Journal of the Medical Library Association, 90: 22–31. Shekelle, P., Ortiz, E., et al. (2001). Validity of the Agency for Healthcare Research and Quality clinical practice guidelines: how quickly do guidelines become outdated? Journal of the American Medical Association, 286: 1461–1467. Shelstad, K. and Clevenger, F. (1994). On-line search strategies of third year medical students: perception vs. fact. Journal of Surgical Research, 56: 338–344. Shelstad, K. and Clevenger, F. (1996). Information retrieval patterns and needs among practicing general surgeons: a statewide experience. Bulletin of the Medical Library Association, 84: 490–497. Shenk, D. (1998). Data Smog: Surviving the Information Glut. San Francisco, Harper Collins. Sherman, C. and Price, G. (2001). The Invisible Web: Uncovering Information Sources Search Engines Can’t See. Chicago. Independent Publishers Group. Shiffman, R., Brandt, C., et al. (1999). A design model for computer-based guideline im-
494
References
plementation based on information management services. Journal of the American Medical Informatics Association, 6: 99–103. Shiffman, R., Liaw, Y., et al. (1999). Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. Journal of the American Medical Informatics Association, 6: 104–114. Schneiderman, B. (1987). User Interface Design and Evaluation for an Electronic Encyclopedia, 207–223, in Slavendy, G., ed. Cognitive Engineering in the Design of Human-Computer Interaction and Expert Systems. Amsterdam. Elsevier Science. Shneiderman, B. (1998). Designing the User Interface: Strategies for Effective Human– Computer Interaction, 3rd ed. Reading, MA, Addison-Wesley. Shon, J. and Musen, M. (1999). The low availability of metadata elements for evaluating the quality of medical information on the World Wide Web. Proceedings of the AMIA 1999 Annual Symposium, Washington, DC. Hanley & Belfus. 945–949. Shortliffe, E. (1987). Computer programs to support clinical decision-making. Journal of the American Medical Association, 258: 61–66. Shuchman, M. and Wilkes, M. (1997). Medical scientists and health news reporting: a case of miscommunication. Annals of Internal Medicine, 126: 976–982. Silberg, W., Lundberg, G., et al. (1997). Assessing, controlling, and assuring the quality of medical information on the Internet: caveat lector et viewor—let the reader and viewer beware. Journal of the American Medical Association, 277: 1244–1245. Silver, H. and Dennis, S. (1990). Monitoring patron use of CD-ROM databases using SignIn-Stat. Bulletin of the Medical Library Association, 78: 252–257. Simon, F. (1988). A comparison of two computer programs for searching the medical literature. Journal of Medical Education, 63: 331–333. Singhal, A., Buckley, C., et al. (1996). Pivoted document length normalization. Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Zurich, Switzerland. ACM Press. 21–29. Singhal, A. and Kaszkiel, M. (2001). A case study in Web search using TREC algorithms. Proceedings of the 10th International World Wide Web Conference, Hong Kong. 708–716. http://www.www10.org/cdrom/papers/317/index.html. Slingluff, D., Lev, Y., et al. (1985). An end-user search service in an academic health sciences library. Medical Reference Services Quarterly, 4(1): 11–21. Smalheiser, N. and Swanson, D. (1998). Using ARROWSMITH: a computer-assisted approach to formulating and assessing scientific hypotheses. Computer Methods and Programs in Biomedicine, 57: 149–53. Smeaton, A. (1986). Incorporating syntactic information into a document retrieval strategy: an investigation. Proceedings of the Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Pisa, Italy. ACM Press. 103–113. Smith, A. (1999). A tale of two Web spaces: comparing sites using Web impact factors. Journal of Documentation, 55: 577–592. Smith, C., Ganschow, P., et al. (2000). Teaching residents evidence-based medicine skills: a controlled trial of effectiveness and assessment of durability. Journal of General Internal Medicine, 15: 710–715. Smith, R. and Roberts, I. (1997). An amnesty for unpublished trials. British Medical Journal, 315: 622. Sneiderman, C., Rindflesch, T., et al. (1996). Finding the findings: identification of findings in medical literature using restricted natural language processing. Proceedings of the 1996 AMIA Annual Fall Symposium, Washington, DC. Hanley & Belfus. 239–243.
References
495
Sneiderman, C., Rindflesch, T., et al. (1998). Identification of anatomical terminology in medical text. Proceedings of the AMIA 1998 Annual Symposium, Orlando, FL. Hanley & Belfus. 428–432. Soergel, D. (1997). Multilingual thesauri in cross-language text and speech retrieval. AAAI Spring Symposium on Cross-Language Text and Speech Retrieval, Palo Alto, CA. AAAI. http://www.ee.umd.edu/medlab/filter/sss/papers/soergel.ps. Sollins, K. and Masinter, L. (1994). Functional Requirements for Uniform Resource Names. Internet Engineering Task Force. http://www.w3.org/Addressing/rfc1737.txt. Accessed: July 1, 2002. Sonnenreich, W. and MacInta, T. (1998). Web Developer.com(r) Guide to Search Engines. New York, Wiley. Soot, L., Moneta, G., et al. (1999). Vascular surgery and the Internet: a poor source of patient-oriented information. Journal of Vascular Surgery, 30: 84–91. Sowa, J. (1984). Conceptual Structures: Information Processing in the Mind and Machine. Reading, MA, Addison-Wesley. Sowa, J., ed. (1991). Principles of Semantic Networks: Explorations in the Representation of Knowledge. San Mateo, CA, Morgan Kaufmann. Sox, H., Blatt, M., et al. (1988). Medical Decision Making. Boston, Butterworths. Spackman, K. and Campbell, K. (1998). Compositional concept representation using SNOMED: towards further convergence of clinical terminologies. Proceedings of the AMIA 1998 Annual Symposium, Orlando, FL. Hanley & Belfus. 740–744. Spackman, K., Campbell, K., et al. (1997). SNOMED RT: a reference terminology for health care. Proceedings of the 1997 AMIA Annual Fall Symposium, Nashville, TN. Hanley & Belfus. 640–644. Sparck Jones, K. (1999). What is the Role of NLP in Text Retrieval?, 1–24, in Strzalkowski, T., ed. Natural Language Information Retrieval. Dodrecht, The Netherlands. Kluwer. Spath, M. and Buttlar, L. (1996). Information and research needs of acute-care clinical nurses. Bulletin of the Medical Library Association, 84: 112–116. Spink, A., Jansen, B., et al. (2002). From e-sex to e-commerce: Web search changes. Computer, 35: 107–109. Spitzer, V., Ackerman, M., et al. (1996). The visible human male: a technical report. Journal of the American Medical Informatics Association, 3: 118–130. Spring, T. (2001). Prototype tablet PCs take spotlight. PC World. Srinivasan, P. (1992). Thesaurus Construction, 161–218, in Frakes, W. and Baeza-Yates, R., eds. Information Retrieval: Data Structures and Algorithms. Englewood Cliffs, NJ. Prentice Hall. Srinivasan, P. (1996a). Optimal document-indexing vocabulary for MEDLINE. Information Processing and Management, 32: 503–514. Srinivasan, P. (1996b). Query expansion and MEDLINE. Information Processing and Management, 32: 431–444. Srinivasan, P. (1996c). Retrieval feedback in MEDLINE. Journal of the American Medical Informatics Association, 3: 157–168. Staggers, N. and Mills, M. (1994). Nurse–computer interaction: staff performance outcomes. Nursing Research, 43: 144–150. Stapley, B. and Benoit, G. (2000). Biobibliometrics: information retrieval and visualization from co-occurrences of gene names in Medline abstracts. Pacific Symposium on Biocomputing, Honolulu, HI. World Scientific. 529–540. Stearns, M., Price, C., et al. (2001). SNOMED clinical terms: overview of the develop-
496
References
ment process and project status. Proceedings of the 2001 AMIA Annual Symposium, Washington, DC. Hanley & Belfus. 662–666. Stein, H., Nadkarni, P., et al. (2000). Exploring the degree of concordance of coded and textual data in answering clinical queries from a clinical data repository. Journal of the American Medical Informatics Association, 7: 42–54. Stensaas, S. (1994). Animating the curriculum: integrating multimedia into teaching. Bulletin of the Medical Library Association, 82: 133–139. Sterling, T. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association, 54: 30–34. Stern, J. and Simes, R. (1997). Publication bias: evidence of delayed publication in a cohort study of clinical research projects. British Medical Journal, 315: 640–645. Stevens, R., Goble, C., et al. (2001). A classification of tasks in bioinformatics. Bioinformatics, 17: 180–188. Stossel, T. (1985). Reviewer status and review quality: experience of the Journal of Clinical Investigation. New England Journal of Medicine, 312: 658–659. Stross, J. and Harlan, W. (1979). The dissemination of new medical information. Journal of the American Medical Association, 241: 2622–2624. Stross, J. and Harlan, W. (1981). Dissemination of relevant information on hypertension. Journal of the American Medical Association, 246: 360–362. Stroup, D., Berlin, J., et al. (2000). Meta-analysis of observational studies in epidemiology: a proposal for reporting. Journal of the American Medical Association, 283: 2008–2012. Strzalkowski, T. (1992). TTP: a fast and robust parser for natural language. Proceedings of the 14th International Conference on Computational Linguistics, Nantes, France. 198–204. Strzalkowski, T., Lin, F., et al. (1999). Evaluating Natural Language Processing Techniques in Information Retrieval, 113–146, in Strzalkowski, T., ed. Natural Language Information Retrieval. Dordrecht, The Netherlands. Kluwer. Stuart-Buttle, C., Brown, P., et al. (1997). The Read Thesaurus: creation and beyond. Studies in Health Technology and Informatics, 43: 416–420. Stumpf, W. (1980). “Peer” review. Science, 207: 822–823. Su, L. (1994). The relevance of recall and precision in user evaluation. Journal of the American Society for Information Science, 45: 207–217. Suarez, H., Hao, X., et al. (1997). Searching for information on the Internet using the UMLS and Medical World Search. Proceedings of the 1997 AMIA Annual Fall Symposium, Nashville, TN. Hanley & Belfus. 824–828. Suarez-Almazor, M., Belseck, E., et al. (2000). Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Controlled Clinical Trials, 21: 476–487. Sun, S. and Lannom, L. (2002). Handle System Overview. Corporation for National Research Initiatives. http://hdl.handle.net/4263537/4069. Accessed: July 1, 2002. Swan, R. and Allan, J. (1998). Aspect windows, 3-D visualization, and indirect comparisons of information retrieval systems. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia. ACM Press. 173–181. Swanson, D. (1977). Information retrieval as a trial-and-error process. Library Quarterly, 47: 128–148. Swanson, D. (1986). Two medical literatures that are logically but not bibliographically connected. Perspectives in Biology and Medicine, 30: 7–18.
References
497
Swanson, D. (1988a). Historical note: Information retrieval and the future of an illusion. Journal of the American Society for Information Science, 39: 92–98. Swanson, D. (1988b). Migraine and magnesium: eleven neglected connections. Perspectives in Biology and Medicine, 31: 526–557. Swanson, D. and Smalheiser, N. (1997). An interactive system for finding complementary literatures: a stimulus to scientific discovery. Artificial Intelligence, 91: 183–203. Tamm, E., Raval, B., et al. (2000). Evaluation of the quality of self-education mammography material available for patients on the Internet. Academic Radiology, 7: 137–141. Tanenbaum, S. (1994). Knowing and acting in medical practice: the epistemological politics of outcomes research. Journal of Health Politics, Policy and Law, 19: 27–44. Tang, P., Newcomb, C., et al. (1997). Meeting the information needs of patients: results from a patient focus group. Proceedings of the 1997 AMIA Annual Fall Symposium, Nashville, TN. Hanley & Belfus. 672–676. Taylor, H. (2002). Cyberchondriacs Update. Harris Interactive. http://www.harrisinteractive. com/harris_poll/index.asp?PID=299. Accessed: July 1, 2002. Taylor, H. and Leitman, R. (2001). The Increasing Impact of eHealth on Physician Behavior. Harris Interactive. http://www.harrisinteractive.com/news/newsletters/healthnews/ HI_HealthCareNews2001Vol1_iss31.pdf. Accessed: July 1, 2002. Tenopir, C. (1985). Full text database retrieval performance. Online Review, 9(2): 149–164. Thelwall, M. (2000). Extracting macroscopic information from Web links. Journal of the American Society for Information Science, 52: 1157–1168. Thomas, J., Milward, D., et al. (2000). Automatic extraction of protein interactions from scientific abstracts. Pacific Symposium on Biocomputing, Honolulu, HI. World Scientific. 541–552. Thomas, O. and Willett, P. (2000). Webometric analysis of departments of librarianship and information science. Journal of Information Science, 26: 421–428. Tibbo, H. (2001). Archival perspectives on the emerging digital library. Communications of the ACM, 44(5): 69–70. Tierney, W., Miller, M., et al. (1993). Physician inpatient order writing on microcomputer workstations: effects on resource utilization. Journal of the American Medical Association, 269: 379–383. Timpka, T. and Arborelius, E. (1990). The GP’s dilemmas: a study of knowledge need and use during health care consultations. Methods of Information in Medicine, 29: 23–29. Tobia, R., Bierschenk, N., et al. (1990). Improving access to computer-based library and drug information services in patient-care areas. American Journal of Hospital Pharmacy, 47: 137–142. Tolle, K. and Chen, H. (2000). Comparing noun phrasing techniques for use with medical library digital tools. Journal of the American Society for Information Science, 51: 352–370. Tomaiuolo, N. and Packer, J. (1996). An analysis of Internet search engines: assessment of over 200 search queries. Computers in Libraries, 16: 58–62. Tonelli, M. (1998). The philosophical limits of evidence-based medicine. Academic Medicine, 73: 1234–1240. Travis, B. and Broder, A. (2001). The need behind the query: Web search vs classic information retrieval. Presentation at the Sixth Search Engine Meeting, April 9, 2001. Boston. Infornotics, Inc. Trivison, D. (1987). Term co-occurrence in cited/citing journal articles as a measure of document similarity. Information Processing and Management, 23(3): 183–194.
498
References
Trueswell, R. (1969). Some behavioral patterns of library users: the 80/20 rule. Wilson Library Bulletin, 43: 458–461. Turney, P. (2001). Mining the Web for synonyms: PMI-IR versus LSA on TOEFL. Proceedings of the 12th European Conference on Machine Learning (ECML-2001), Freiburg, Germany. 491–502. Turpin, A. and Hersh, W. (2001). Why batch and user evaluations do not give the same results. Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, LA. ACM Press. 225–231. Turtle, H. (1994). Natural language vs. Boolean query evaluation: a comparison of retrieval performance. Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland. Springer-Verlag. 212–220. Turtle, H. and Croft, W. (1991). Evaluation of an inference network-based retrieval model. ACM Transactions on Information Systems, 9: 187–222. Udvarhelyi, I., Colditz, G., et al. (1992). Cost-effectiveness and cost–benefit analyses in medical literature. Annals of Internal Medicine, 116: 238–244. Urquhart, J. and Bunn, R. (1959). A national loan policy for science serials. Journal of Documentation, 15: 21–25. Uruqhart, C. and Crane, S. (1994). Nurses’ information-seeking skills and perceptions of information sources: assessment using vignettes. Journal of Information Science, 20: 237–246. Vandenbroucke, J. (2001). In defense of case reports and case series. Annals of Internal Medicine, 134: 330–334. Van de Sompel, H. and Lagoze, C. (1999). The Santa Fe Convention of the Open Archives Initiative. D-Lib Magazine, 5. http://www.dlib.org/dlib/february00/vandesompeloai/02vandesompel-oai.html. van Nieuwenhoven, C., Buskens, E., et al. (2001). Relationship between methodological trial quality and the effects of selective digestive decontamination on pneumonia and mortality in critically ill patients. Journal of the American Medical Association, 286: 335–340. van Rijsbergen, C. (1979). Information Retrieval. London, Butterworths. van Rooyen, S., Godlee, F., et al. (1998). Effect of blinding and unmasking on the quality of peer review: a randomized trial. Journal of the American Medical Association, 280: 234–237. van Rooyen, S., Goldbeck-Wood, S., et al. (2001). What makes an author? A comparison between what authors say they did and what editorial guidelines require. Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain. American Medical Association. http://www.ama-assn.org/public/peer/peerhome.htm. Varmus, H. (1999). PubMed Central: A Proposal for Electronic Publication in the Biomedical Sciences. National Institutes of Health. www.nih.gov/welcome/director/ebiomed/ebi.htm. Accessed: July 1, 2002. Veenstra, R. (1992). Clinical medical librarian impact on patient care: a one-year analysis. Bulletin of the Medical Library Association, 80: 19–22. Venter, J., Adams, M., et al. (2001). The sequence of the human genome. Science, 291: 1304–1351. Verhoeven, A., Boendermaker, P., et al. (1999). A model for the evaluation of search queries in medical bibliographic sources. Health Libraries Review, 16: 141–150.
References
499
Voorhees, E. (1986). Implementing agglomerative hierarchic clustering algorithms for use in document retrieval. Information Processing and Management, 22: 465–476. Voorhees, E. (1993). Using WordNet to disambiguate word senses for text retrieval. Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Pittsburgh, PA. ACM Press. 171–180. Voorhees, E. (1994). Query expansion using lexical-semantic relations. Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland. Springer-Verlag. 61–69. Voorhees, E. (1998). Variations in relevance judgments and the measurement of retrieval effectiveness. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia. ACM Press. 315–323. Voorhees, E. (2000). Overview of the TREC-9 Question Answering Track. The Ninth Text REtrieval Conference (TREC-9), Gaithersburg, MD. National Institute of Standards and Technology. 71–79. Voorhees, E. (2001). Overview of the TREC 2001 Question Answering Track. The Tenth Text REtrieval Conference (TREC 2001), Gaithersburg, MD. National Institute of Standards and Technology. in press. Voorhees, E. and Harman, D. (2000). Overview of the Sixth Text REtrieval Conference (TREC). Information Processing and Management, 36: 3–36. Voorhees, E. and Tice, D. (2000). Building a question answering test collection. Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Athens, Greece. ACM Press. 200–207. Wager, E. and Middleton, P. (2002). Effects of technical editing in biomedical journals: a systematic review. Journal of the American Medical Association, 287: 2821– 2824. Walczuch, N., Fuhr, N., et al. (1994). Routing and ad-hoc retrieval with the TREC-3 collection in a distributed loosely federated environment. Overview of the Third Text REtrieval Conference (TREC-3), Gaithersburg, MD. National Institute of Standards and Technology. 135–144. Walker, C., McKibbon, K., et al. (1991). Problems encountered by clinical end users of MEDLINE and Grateful Med. Bulletin of the Medical Library Association, 79: 67–69. Wallingford, K., Humphreys, B., et al. (1990). Bibliographic retrieval: a survey of individual users of MEDLINE. M.D. Computing, 7: 166–171. Wang, P. (1994). A Cognitive Model of Document Selection of Real Users of Information Retrieval Systems. College of Library and Information Services. Ph.D. Thesis. University of Maryland. Wang, Y., Addess, K., et al. (2000). MMDB: 3D structure data in Entrez. Nucleic Acids Research, 28: 243–245. Wanke, L. and Hewison, N. (1988). Comparative usefulness of MEDLINE searches performed by a drug information pharmacist and by medical librarians. American Journal of Hospital Pharmacy, 45: 2507–2510. Warner, A. and Wenzel, P. (1991). A linguistic analysis and categorization of nominal expressions. Proceedings of the 54th Annual Meeting of the American Society for Information Science, Washington, DC. Learned Information. 186–195. Warner, H. (1989). Iliad: moving medical decision-making into new frontiers. Methods of Information in Medicine, 28: 370–372. Wartik, S., Fox, E., et al. (1992). Hashing Algorithms, 293–362, in Frakes, W. and Baeza-
500
References
Yates, R., eds. Information Retrieval: Data Structures and Algorithms. Englewood Cliffs, NJ. Prentice-Hall. Watson, R. and Richardson, P. (1999). Identifying randomized controlled trials of cognitive therapy for depression: comparing the efficiency of Embase, Medline, and PsycINFO bibliographic databases. British Journal of Medical Psychology, 72: 535–542. Weeber, M., Klein, H., et al. (2000). Text-based discovery in biomedicine: the architecture of the DAD-system. Proceedings of the AMIA 2000 Annual Symposium, Los Angeles, CA. Hanley & Belfus. 903–907. Weeber, M., Mork, J., et al. (2001). Developing a test collection for biomedical word sense disambiguation. Proceedings of the 2001 AMIA Annual Symposium, Washington, DC. Hanley & Belfus. 746–750. Weibel, S. (1996). The Dublin Core: a simple content description model for electronic resources. ASIS Bulletin, 24(1): 9–11. Weibel, S., Jul, E., et al. (2002). PURLs: Persistent Uniform Resource Locators. OCLC. http://purl.oclc.org/OCLC/PURL/SUMMARY. Accessed: July 1, 2002. Weibel, S. and Koch, T. (2000). The Dublin Core Metadata Initiative: mission, current activities, and future directions. D-Lib Magazine, 6. http://www.dlib.org/dlib/december00/weibel/12weibel.html. Weiner, J., Parente, S., et al. (1995). Variation in office-based quality—a claims-based profile of care provided to Medicare patients with diabetes. Journal of the American Medical Association, 273: 1503–1508. Weisbord, S., Soule, J., et al. (1996). Poison on line: acute renal failure caused by oil of wormwood purchased through the Internet. New England Journal of Medicine, 337: 825. Weiss, S. and Kulikowski, C. (1991). Computer Systems That Learn: Classification and Prediction Methods from Statistics, Neural Nets, Machine Learning, and Expert Systems. San Mateo, CA, Morgan Kaufmann. Welsh, J. (1989). Evaluation of CD-ROM use in a government research library. Laserdisk Professional, 2: 55–61. West, R. (1996). Impact factors need to be improved. British Medical Journal, 313: 1400. Westbrook, J. (1960). Identifying significant research. Science, 132: 1229–1234. Wheeler, D., Chappey, C., et al. (2000). Database resources of the National Center for Biotechnology Information. Nucleic Acids Research, 28: 10–14. Wheeler, D., Church, D., et al. (2002). Database resources of the National Center for Biotechnology Information: 2002 update. Nucleic Acids Research, 30: 13–16. Whitely, W., Rennie, D., et al. (1994). The scientific community’s response to evidence of fraudulent publication: the Robert Slutsky case. Journal of the American Medical Association, 272: 170–173. Wilbur, W. (2002). A thematic analysis of the AIDS literature. Pacific Symposium on Biocomputing, Kauai, Hawaii. World Scientific. 386–397. Wilbur, W. and Yang, Y. (1996). An analysis of statistical term strength and its use in the indexing and retrieval of molecular biology texts. Computers in Biology and Medicine, 26: 209–222. Wilczynski, N., Walker, C., et al. (1995). Reasons for the loss of sensitivity and specificity of methodologic MeSH terms and text words in MEDLINE. Proceedings of the 19th Annual Symposium on Computer Applications in Medical Care, New Orleans, LA. Hanley & Belfus. 436–440. Wildemuth, B., de Bliek, R., et al. (1995). Medical students’ personal knowledge, searching proficiency, and database use in problem solving. Journal of the American Society for Information Science, 46: 590–607.
References
501
Wildemuth, B. and Moore, M. (1995). End-user search behaviors and their relationship to search effectiveness. Bulletin of the Medical Library Association, 83: 294–304. Wilkinson, R. and Fuller, M. (1996). Integration of Information Retrieval and Hypertext via Structure, 257–271, in Agosti, M. and Smeaton, A., eds. Information Retrieval and Hypertext. Norwell, MA. Kluwer. Williams, D., Counselman, F., et al. (1996). Emergency department discharge instructions and patient literacy: a problem of disparity. American Journal of Emergency Medicine, 14: 19–22. Williamson, J. (1990). Information Needs of Qualified Nurses in Bloomsbury Health Authority. M. Phil. Thesis. University of London. Williamson, J., German, P., et al. (1989). Health science information management and continuing education of physicians. Annals of Internal Medicine, 110: 151–160. Winker, M., Flanagin, A., et al. (2000). Guidelines for medical and health information sites on the Internet: principles governing AMA Web sites. Journal of the American Medical Association, 283: 1600–1606. Witten, I., Bainbridge, D., et al. (2001). Greenstone: open-source digital library software. D-Lib Magazine, 7. http://www.dlib.org/dlib/october01/witten/10witten.html. Witten, I. and Bainbridge, D. (2003). How to Build a Digital Library. San Francisco, Morgan Kaufmann. Wolf, M. and Wicksteed, C. (1997). Date and Time Formats. World Wide Web Consortium. http://www.w3.org/TR/NOTE-datetime. Accessed: July 1, 2002. Wood, F., Wallingford, K., et al. (1997). Transitioning to the Internet: results of a National Library of Medicine user survey. Bulletin of the Medical Library Association, 85: 331–340. Woods, W. (1970). Transition network grammars for natural language analysis. Communications of the ACM, 13: 591–602. Worsley, D., Alavi, A., et al. (1993). Chest radiographic findings in patients with acute pulmonary embolism: observations from the PIOPED study. Radiology, 189: 133–136. Wright, B., Williams, C., et al. (1999). Management advice for children with chronic fatigue syndrome: a systematic study of information from the Internet. Irish Journal of Psychological Medicine, 16: 67–71. Wright, L., Nardini, H., et al. (1999). Hierarchical concept indexing of full-text documents in the United Medical Language System (UMLS) Information Sources Map. Journal of the American Society for Information Science, 50: 514–523. Wu, M., Fuller, M., et al. (2000). Using clustering and classification approaches in interactive retrieval. Information Processing and Management, 37: 459–484. Wu, M., Fuller, M., et al. (2001). Searcher performance in question answering. Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, LA. ACM Press. 375–381. Wyatt, J. and Spiegalhalter, D. (1991). Field trials of medical decision-aids: potential problems and solutions. Proceedings of the 15th Annual Symposium on Computer Applications in Medical Care, Washington, DC. McGraw-Hill. 3–7. Xu, J. and Croft, W. (1996). Query expansion using local and global document analysis. Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Zurich, Switzerland. ACM Press. 4–11. Yakushiji, A., Tateisi, Y., et al. (2001). Event extraction from biomedical papers using a full parser. Pacific Symposium on Biocomputing, Kauai, HI. World Scientific. 408–419. Yang, K., Maglaughlin, K., et al. (2000). Passage feedback with IRIS. Information Processing and Management, 37: 521–541.
502
References
Yang, Y. (1994). Expert Network: effective and efficient learning from human decisions in text categorization and retrieval. Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland. Springer-Verlag. 11–21. Yang, Y. (1999). An evaluation of statistical approaches to text categorization. Information Retrieval, 1: 67–88. Yang, Y. and Chute, C. (1994). An example-based mapping method for text categorization and retrieval. ACM Transactions on Information Systems, 12: 252–277. Yasnoff, W. and Bacus, J. (1984). Scene-segmentation algorithm development using error measures. Analytical and Quantitative Cytology, 6: 45–58. Yu, H., Hripcsak, G., et al. (2002). Mapping abbreviations to full forms in biomedical articles. Journal of the American Medical Informatics Association, 9: 262–272. Zeng, Q. and Cimino, J. (1997). Linking a clinical system to heterogenous information resources. Proceedings of the 1997 AMIA Annual Fall Symposium, Nashville, TN. Hanley & Belfus, 553–557. Zhai, C., Tong, X., et al. (1996). Evaluation of syntactic phase indexing: CLARIT NLP Track report. The Fifth Text REtrieval Conference (TREC-5), Gaithersburg, MD. National Institute of Standards and Technology. 347–358. Zhao, J. and Resh, V. (2001). Internet publishing and transformation of knowledge processes. Communications of the ACM, 44(12): 103–110. Ziman, J. (1969). Information, communication, knowledge. Nature, 224: 318–324. Zingmond, D. and Lenert, L. (1993). Monitoring free-text data using medical language processing. Computers and Biomedical Research, 26: 467–481. Zobel, J. (1998). How reliable are the results of large-scale information retrieval experiments? Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia. ACM Press. 307–314. Zobel, J. and Moffat, A. (1998). Exploring the similarity space. SIGIR Forum, 32: 18–34.
Index
A Absolute rate increase, 79 Absolute risk reduction, 78 Access, collections, 392 Access, individual items, 390–391 Access, metadata, 392–394 ACI. See Autonomous citation indexing Acid paper, 389 ACM-SIGIR. See Association for Computing Machinery, Special Interest Group on Information Retrieval ACR codes. See American College of Radiology codes ACROMED, 323 Adaptive filtering, 287 Agreement, measures of, 113 AI. See Artificial intelligence AIDSLINE, 124 ALA. See American Library Association AltaVista, 192, 207–209, 289–290 Ambiguity, 311, 324–325 American Accreditation Healthcare Commission, 63–64 American College of Radiology codes, 415 American Library Association, 12, 386–387 American Medical Informatics Association, 12 American Society for Information Science and Technology , 12 American Society of Indexers, 12
AMIA. See American Medical Informatics Association Anaphoric references, 320 Anatline, 181 AND, 184–187 Anomalous state of knowledge, 5 ARI. See Absolute rate increase ARR. See Absolute risk reduction ARROWSMITH, 428 Artificial intelligence, 371, 397 ASI. See American Society of Indexers ASIST. See American Society for Information Science and Technology ASK. See Anomalous state of knowledge AskJeeves, 208–209 Association for Computing Machinery, Special Interest Group on Information Retrieval, 12 ATN. See Augmented transition network Augmented transition network, 316 Author productivity, 29 Autonomous citation indexing, 392 AutoSummarize feature, Microsoft Word, 297 B Barrier word, 423 Basic Local Alignment Search Tool, 211, 428–429 Batch filtering, 287 Bayes’ Theorem, 282 503
504
Index
Bias, research, 86 Bibliometrics, 28–32 Bioinformatics, 12 Biomed Central, 53, 131 BioMedNet, 201 BIOSIS Previews, 125, 230 BLAST. See Basic Local Alignment Search Tool BMJ. See British Medical Journal Books in Print, 126 Boolean algebra, 185 Boolean operators, 265–266 Bradford’s Law of Scattering, 29–30 BrighamRad, 210 British Medical Journal, 13 Brown Corpus, 174 Browsing, 15 B-tree, 182 C CANCERLIT, 124–125 CancerNet, 142–143 Canonical form, 149 Canonical Phrase Identification System, 405–406 CANSEARCH, 375 CAPIS. See Canonical Phrase Identification System Card catalog, 387 Case frames, 318–319 Cat-a-Cone, 378, 380 Catalog of U.S. Government Publications, 128 Catalogue et Index des Sites Medicaux Francophones, 166–167 CATS. See Critically appraised topics CBS Healthwatch, 135 CCC. See Copyright Clearance Center CDSR. See Cochrane Database of Systematic Reviews Celera Genomics, 138 Cellular regulatory networks, 429 Centroid vector, 272 Chance, research, 86–87 Checklist effect, 86 Chomsky Hierarchy, 315 CINAHL. See Cumulative Index to Nursing and Allied Health Literature CIRT, 298–299
CISMeF. See Catalogue et Index des Sites Medicaux Francophones Citation networks, 28–29 CiteSeer, 392 CLARIT, 331–333, 414 CLEF. See Cross-Language Evaluation Forum Clinical Document Architecture, 168 Clinical Evidence, 141 Clinical finding, 406 Clinical narrative, 397 alternatives to natural language input, 426 classification, 414–426 ad hoc, 424–426 vocabulary-based, 420–424 syntactic incompleteness, 399 Clinical practice guidelines, 52, 259 Clinical vocabularies, multiaxial, 417–420 Clinical vocabularies, requirements, 416–417 ClinicalTrials.gov, 142, 212 CliniWeb, 127, 150, 163, 199–200, 201, 214, 361, 363 CLIR. See Cross-language information retrieval Clustering documents, 373–374 CME. See Continuing medical education COACH, 195, 376 Cochrane Collaboration, 48, 81 Cochrane Database of Systematic Reviews, 201, 242, 374 Cochrane Library, 141 Cohesion measure, 278 Commercial Alert, 210 Complement. See NOT Computer Retrieval of Information on Scientific Projects, 142 Computers in Biology and Medicine, 13 Concept, 149 Conceptual graph, 311 Cone tree, 378, 380 Conference proceedings, 34, 124–125 Confidence interval, 47 CONIT, 375 Consolidated Standards of Reporting Trials statement, 41–42 CONSORT statement. See Consolidated Standards of Reporting Trials statement
Index Content, 4 aggregations, 118, 142–145 bibliographic, 117–129 classification of, 117–118 full-text, 118, 129–135 Context, 377–382 Context deficit, 60 Contextual interpretation, 314, 320, 399, 401 Continuing medical education, 259 Controlled vocabulary, 5, 265–266, 149–162 clinical, 415–420 relationships, 149 hierarchical, 149 related, 149 synonymous, 149 Copernic, 209 Copyright, 54, 388 Copyright Clearance Center, 388 Cosine measure, 270–272 Cosine normalization, 270 CoSTAR, 415 Cost-benefit, 88–89 Cost-effectiveness, 88–89 CPT. See Current Procedural Terminology Cranfield studies, 228 CRISP. See Computer Retrieval of Information on Scientific Projects Critically appraised topics, 81 Crossing the Quality Chasm, 396 Cross-Language Evaluation Forum, 348 Cross-language information retrieval, 347–354 Chinese, 351–352 controlled vocabulary, 348–349 free text, 348–349 corpus-based, 348–349 knowledge-based, 348–349 French, 349, 351, 353–354 German, 349–351 Japanese, 352–353 Spanish, 349, 353–354 CUI. See Unified Medical Language System, Metathesaurus, concept unique identifier Cumulative gain, 102–103 Cumulative Index to Nursing and Allied Health Literature, 125, 156, 229
505
Cumulative Index to Nursing and Allied Health Literature Subject Headings, 156 Current Contents, 125 Current Procedural Terminology, 158, 415, 420–421 D Database management system, 6 Database-assisted scores, 243 Databases, bibliographic. See Databases, literature reference Databases, citation, 140–141 Databases, evidence-based medicine, 118, 141–142 Databases, genomic, 118, 138–140 Databases, genomic, integration, 140 Databases, image, 118, 136–137 Databases, literature reference, 117–126 Databases/Collections, 118, 136–142 DCMI. See Dublin Core Metadata Initiative Decision Systems Group Network of Extensible Resources, 358–359 Definitive publication, 52–53 Description logic, 418 DeSyGNER. See Decision Systems Group Network of Extensible Resources Devices, 383–386 handheld, 133, 384–385, 426 tablet, 385–386 Dewey Decimal System, 164–165 DFO. See Dynamic feedback optimization Diagnosis-related groups, 259 Diagnostic test research, 41 Diagnostic tests, 78 Digestor, 385 Digital Libraries Initiative, National Science Foundation, 387 Digital library, 3, 386–396 definitions, 389 functions, 389–390 Digital Library Magazine, 387 Digital Object Architecture, 393–394 Digital Object Identifier, 391, 394–395 Digital Preservation Coalition, 396 Discounted cumulative gain, 102–103 Discourse analysis, 320 Dissertation Abstracts, 126
506
Index
Distance metric, 358, 407, 409 DLI. See Digital Libraries Initiative, National Science Foundation DNA chips. See Gene expression microarrays DOCLINE, 215 Document delivery, 214–215 Document surrogate, 98 Document weight, 188–189 DogPile, 209 DOI. See Digital Object Identifier DRG. See Diagnosis-related groups DTD. See eXtensible Mark-up Language, Document Type Definition Dublin Core Metadata Initiative, 163–169, 372, 393 Qualifiers, 165–166 tags in Web pages, 167 Type Vocabulary, 166 DxPlain, 143–144, 415 Dynacat, 378–379 Dynamic feedback optimization, 281 Dynamic Medical Handbook, 376–377 E E measure, 99 Early adopters, 227 EBM Solutions, 142 EBM. See Evidence-based medicine eBook. See Electronic book Educational Resources Information Center, 125 Educational Testing Service Kit of Cognitive Factors, 248 Effectiveness, 83, 88–89 Efficacy, 83 Electronic book, 386 Electronic medical record, 359–363, 402 linking to knowledge, 359–363 Electronic publishing, 52–65, 388 models, 54 economic and political challenges, 53–56 Electronic-long, paper-short, 130 Elevators, impact on patient care, 261 ELHILL, 195 ELPS. See Electronic-long, paper-short EMBASE, 125–126, 156–157, 229–230 eMedicine, 135
EMR. See Electronic medical record EMTREE, 157 facets, 157 link terms, 157 End queries, 250 Entrez, 138, 211 Epocrates, 384–385 E-Prints Initiative, 392 ERIC. See Educational Resources Information Center Error, alpha, 86–87 Error, beta, 87 Ethnography, 87, 111 Evaluation, 219–261 Evaluation, abbreviations, 231 assessment of impact, 257–260 batch mode, 93, 266 Boolean searching, 299 classifications of, 88–95 comparing batch and interactive experiments, 304–308 comparing Boolean and natural language searching, 302–303, 305 database, 91 effect of training novice searchers, 238 experienced searchers, 236–241 medical librarians, 236–241 model of factors associated with success, 246–248 natural language searching, 299 novice searchers, 236–241 overlap of relevant articles, 237, 239 predictors of success, 244–254 presentation of search results, 308–309 reasons for using system, 226 relevance feedback, 307–308 relevance-based, 95–110 alternatives to, 110 limitations, 108–110 magnitude of significant difference, 109 ranked output, 99–103 request, 90–91 search outcome, 93 search process, 92–93 search system, 92 searcher, 92 searching quality, 227–244
Index bibliographic systems, 229–232 consumers on Web, 244 full-text systems, 232–234 system-oriented, 228–234 task-oriented measures, 241–244 user relevance-based measures, 236–241 user search critique, 235–236 user-oriented, 234–244 Web searching systems, 234 setting, 89–90 system impact, 94–95 system use, 93–94 types of usage, 225–226 usage frequency, 220–225 directly measured, 220–224 indirectly measured, 224–225 use of simulation, 95 user, 90 user satisfaction, 94, 226–227 effect of user fees, 227 rural physicians, 227 Evidence-based medicine, 8–9, 47, 75–82, 103, 198, 201, 231, 242, 243, 249, 251, 261 best studies, 76–80 first generation, 76–81 limitations, 80–81 limitations, 81–82 next generation, 80–81 resources, 75 Excerpta Medica, 125 Excite, 207–209, 234, 289–290 Expert Network, 420 Expert system, 7 Explorer-1, 358 Explosions. See MEDLINE explosions eXtensible Mark-up Language, 168–169, 372, 412 Schema, 168 Document Type Definition, 168 F Failure analysis, 94, 254–257 Fair use, 388 FASIT, 330–331 FAST, 291 Files, inverted, 181–182 dictionary, 182
507
position, 182 postings, 182 Findings-diagnosis continuum, 180 Flesch-Kinkaid score, 60 Floating subheadings. See Medical subject headings subheadings, floating Focus groups, 87, 111 4S model, 81–82 Fraudulent science, 45–46 Fry Readability Graph, 60 Funnel plots, 49–50 G GALEN, 426 GEE. See Generalized estimating equations GenBank, 138–139, 359 Gene expression microarrays, 139, 428 Gene Ontology, 157, 429 Generalized estimating equations, 250 Generative capacity, 315 Generic queries, 360 Generic questions, taxonomy, 69–70 Genomics, 54, 428–429 GenPept, 138 GO. See Gene Ontology Google, 179, 207–209, 290–293, 324, 364 Graesser’s Taxonomy of Questions, 296–297 Grammar, 314–318 rewrite rules, 315 symbols, 315 Grateful Med, 195, 256–257 Gray literature, 49 Greenstone, 394 H Handle System, 391 HardinMD, 234 Harrison’s Plus, 131 Hashing, 182 Hawthorne effect, 86 HCI. See Human-computer interaction Health decision making, 9–10 Health Evaluation through Logical Processing, 412–413, 415 Health informatics, 12 Health Level 7, 411 Health on the Net, 61–63
508
Index
HealthCyberMap, 372 HealthFinder, 128, 201 HealthSTAR, 123–124 HealthWeb, 128, 201 HELP. See Health Evaluation through Logical Processing Hepatitis Knowledge Base, 356–357 Hidden Markov Model, 321 Hi-Ethics Initiative, 63–64 Highwire Press, 130, 204, 378–379, 381 HL-7. See Health Level 7 HMM. See Hidden Markov Model HON Select, 128 HON. See Health on the Net HotBot, 207–209, 234, 289–290 HTTP. See Hypertext Transfer Protocol Human Genome Project, 138 Human-computer interaction, 373 Hypermedia, 15–17 Hypertext, 15–17 anchor, 16 anchor text, 16 automated generation, 293–294 evaluation, 294 types of links, 293–294 hot spot, 16 link, 16–17 node, 16 Hypertext Transfer Protocol, 390 Hypothesis testing, 84–85 Hypotheticodeductive model, 66 I IBM Feature Extraction Tool, 424 ICD. See International Classification of Diseases Ide dec-hi formula, 281 IDF. See Inverse document frequency IE. See Information extraction IJCME. See International Committee of Medical Journal Editors Iliad, 415 ILL. See Interlibrary loan Illness scripts, 66 Impact factor, 30–32 as determinant of journal quality, 31–32 IND. See Indexing Initiative, National Library of Medicine
INDECS. See Interoperability of Data in e-Commerce Systems Index Medicus, 119, 146–147, 149–150 Indexing, 5, 146–182, 364–372 attributes, 4 automated, 146–147, 172–180 limitations, 173–174 link-based, 179 TF*IDF weighting, 174, 178–179 word, 172–174 word weighting, 174–179 consistency, 147–148, 266 data structures, 146 exhaustivity, 147–148 factors influencing, 147–149 image, 180–181 language, 5, 147 manual, 146–147, 160–172 bibliographic, 162–163 consistency, 171–172 full-text, 163 limitations, 171–172 Web, 163–171 postcoordinate, 147 precoordinate, 147 specificity, 147–148 terms, 4 Indexing Initiative, National Library of Medicine, 367–369 Inference model, 283 Inferential statistics, 86–87 Infobutton, 360 InfoPOEMS, 141 Information extraction, 397–430 knowledge-based information, 427–430 patient-specific information, 397–427 Information filtering, 215–216 Information fragmentation, 28 Information growth, 26 Information linkage, 28–32 Information needs, 5, 65–75, 184 of biological researchers, 75 of nurses, 74–75 of physicians, 67–73 pursued, 70–72 recognized, 69–70 satisfied, 72–73 unrecognized, 67–69 other health care professionals, 73–75
Index Information obsolescence, 26–27 Information on Web, harm from, 60–61 Information overload, 4 Information problem, 184 Information Processing and Management, 12 Information quality on the Web, 57–65 accreditation, 63–64 automatic detection, 63 government regulation, 63 guidelines, 61 predictive attributes, 63 Information resources in physician offices, 71 Information Retrieval (journal), 12 Information retrieval journals, 10–13 Information retrieval organizations, 10–12 Information retrieval system, 4 Information retrieval textbooks, 13 Information retrieval tools, 11–14 Information science, 10, 25 Information seeking, 8 Information theory, 23–25 Information, classification of textual health information, 32–33 Information, definition of, 22–23 Information, knowledge-based, 32–33 Information, patient-specific, 32–33 Information, production of, 33–52 Information, properties of, 26–32 Information, quantity, 24 Information, time handling and using, 23 Information, usefulness, 72 Informationist, 389–390 InfoSeek, 289–290 Infotrieve, 201 Inglefinger Rule, 39–40, 55 INQUERY, 299–300, 345–346, 424–425 INQUIRER, 243 Inquirus, 290–291 Instance recall, 300–301, 306 Intelihealth, 135 Intellectual property, 388, 394–395 Interlibrary loan, 388 International Classification of Diseases, 158, 288, 361–362, 415, 420–421 International Committee of Medical Journal Editors, 35–36, 43, 55 Internet, 14–20
509
Internet Grateful Med, 376 Internet Wayback Machine, 396 Internet, use of, 17–18 Interoperability, 392 Interoperability of Data in e-Commerce Systems, 394–395 Interoperability, with electronic medical record, MedLEE, 411–412 Intersection. See AND Inverse document frequency, 178, 188, 283 Inverse square law of scientific productivity. See Lotka’s Law Invisible Web, 20 IPM. See Information Processing and Management ISM. See Unified Medical Language System Information Sources Map IVORY, 426 J JAMIA. See Journal of the American Medical Informatics Association JASIST. See Journal of the American Society for Information Science and Technology JBI. See Journal of Biomedical Informatics JCDL. See Joint Conference on Digital Libraries JIT. See Just in time information model JMLA. See Journal of the Medical Library Association Joint Conference on Digital Libraries, 387 Journal impact factor. See Impact factor Journal of Biomedical Informatics, 13 Journal of the American Medical Informatics Association, 13 Journal of the American Society for Information Science and Technology, 12 Journal of the Medical Library Association, 13 Journals, throw-away, 49 Just in time information model, 73, 76 K Kappa statistic, 103–104, 113 KDD. See Knowledge discovery from databases
510
Index
k-Nearest Neighbor, 287 kNN. See k-Nearest Neighbor Knowledge discovery from databases, 397 Knowledge entities, 358–359 Knowledge Finder, 123, 200–201, 241 Knowledge representation, 371 Knowledge-based information, use of, 65–75 L Language modeling, 284, 321 Latent semantic indexing, 276–277, 281, 299 Latin squares design, 301 Law of Zipf, 174 Leaf nodes. See Terminal terms Lemur Toolkit, 286 Length of Stay, 259 Lexemes, 313 Lexical atoms, 333 Lexical knowledge, 322 Lexical mapping, 336–344 Lexical-statistical systems, 265–309 assumptions, 266 evaluation, 266–268 filtering, 286–288 implementations, 285–286 routing, 286–288 user evaluation, 298–309 Lexicon, 315–318, 399–400 LHH. See Likelihood of help vs harm Library, functions, 386–389 Library and information science, 11 Library of the Surgeon General’s Office, 119, 146 Library services, value of, 258 LifeCode, 420–421 Likelihood of help vs harm, 78 Linear-Least Squares Fit, 287, 420 Linguistic String Project, 403–405, 422, 427 information formats, 403–405 Linguistic systems, 310–355 rationale, 310–311 semantic approaches, 336–347 Linguistics, 311–312 computational, 310, 312 applications, 312 theoretical, 312 Linkage to human knowledge, 363–365
Literature, primary, 33, 39–46 low relevance, 45 problematic writing, 43–44 reporting of weak methods, 40–41 Literature, secondary, 33, 49–52 problems, 51–52 LLSF. See Linear-Least Squares Fit LOCATORplus, 124 LOCKSS. See Lots of Copies Keep Stuff Safe Logical Observations, Identifiers, Names, and Codes, 419–420 Logical reasoning, 248 Logical relevance. See Relevance, situational Logistic regression, 425 LOINC. See Logical Observations, Identifiers, Names, and Codes Longman Dictionary of Contemporary English, 317, 328 LookSmart, 209, 291 LOS. See Length of Stay Lotka’s Law, 29 Lots of Copies Keep Stuff Safe, 396 LSI. See Latent semantic indexing LSP. See Linguistic String Project LUI. See Unified Medical Language System, Metathesaurus, term unique identifier LWW Medicine, 131 Lycos, 207–209, 289–290 M Machine learning, 286–287, 425 Macroevaluation, 83 Map Viewer, 139 MAP. See Precision, mean average MCM. See Medical Core Metadata Project MDConsult, 131, 145, 163, 234 Mean reciprocal rank, 112, 289–290, 294–295, 297 Medem, 145 Medical Core Metadata Project, 166, 168–169 Medical diagnostic test evaluation, 97–98 Medical informatics, 12 Medical Knowledge Self-Assessment Program, 242
Index Medical Language Extraction and Encoding, 406–412, 421–422, 428 semantic categories, 407–408 Medical Library Association, 12, 387 Medical Matrix, 127, 163, 201 Medical Subject Headings, 149–156, 158, 163, 336, 346–348, 361, 365, 367–370, 382, 415, 420–421 alphabetic index, 150 annotation, 156 browser, 150–151 check tag, 152,154 headings, 150 permuted index, 150 publication type, 152–153, 155, 194–198 related concept, 156 consider also, 156 see related, 156 scope note, 156 subheading, 151–152, 154, 193 allowable qualifiers, 151–152 tree address, 153–154, 156 tree structure, 150 tree, 150 Medical World Search, 382–383 Medindex, 365–366 MEDLARS Indexing Manual, 162 MedLEE. See Medical Language Extraction and Encoding MEDLINE, 119–124, 147, 162, 191, 193, 195, 213, 215, 228–244, 249–257, 276, 288, 325, 344, 346–347, 354, 359–370, 375, 421, 428–429 abstracts, 121 explosions, 193–194 fields, 120–121 indexing, 194 author names, 163 central concept heading, 162–163 read/scan method, 162 publication types, 121 MEDLINEplus, 142–143, 163, 200, 213 Medscape, 135, 201 MedSORT-II, 406, 416 MedWeaver, 143–144, 213–214 MedWeb Plus, 128 Merck Medicus, 145 MeSH. See Medical Subject Headings META tag, 165
511
Meta-analysis, 40–41, 45–49, 80–81 MetaCrawler, 209, 234, 290 Metadata, 4, 146 Web-based, 148–149 MetaMap, 342–344, 346, 365, 367–369, 422, 427–428 Methods of Information in Medicine, 13 MG, 286 Microevaluation, 83 Mismatch, word sense, 324 Missed opportunities, 255 MKSAP. See Medical Knowledge SelfAssessment Program MLA. See Medical Library Association Molecular Modeling Database, 138 More informative abstracts, 80 Morphemes, 312–313 bound, 313 Morphological analysis, 327–328 Morphology, derivational, 327–328 Morphology, inflectional, 327–328 Morphosemantic parser, 414 MRR. See Mean reciprocal rank N National Center for Biotechnology Information, 138 National Digital Information Infrastructure Preservation Program, 55, 396 National Guidelines Clearinghouse, 128–129, 150, 157, 201–203, 361, 363, 374 National Information InfrastructureNational Center for Science Information Systems Test Collection for IR Systems, 348 National Library of Medicine, 12, 119, 356, 387 National Network of Libraries of Medicine, 124 Natural language processing, 310, 397 corpus-based, 321–322 phases, 314–320 NCBI. See National Center for Biotechnology Information NDIIPP. See National Digital Information Infrastructure Preservation Program
512
Index
Negative dictionary. See Stop list NetAccess, 360–363 NetMenu, 370 NetWellness, 135, 364 Neural Network, 287 NGC. See National Guidelines Clearinghouse NLM Gateway, 125, 144–145, 195, 214 NLM. See National Library of Medicine NLP. See Natural language processing NNH. See Number needed to harm NNT. See Number needed to treat Nominal compounds, 325–327 Northern Light, 208, 291 NOT, 184–187 Notification. See Information filtering Noun phrase, 313, 322, 326 head, 313 linguistic variation, 326 modifier, 313 Novelty effect, 222, 227 NP. See Noun phrase NTCIR. See National Information Infrastructure-National Center for Science Information Systems Test Collection for IR Systems Number needed to harm, 78 Number needed to treat, 78 Nursing informatics, 12 O OAI. See Open Archives Initiative OCELOT, 385 OCLC. See Online Computer Library Center Odds ratio, 47 OHSUMED, 267, 287, 346–347, 354 Okapi, 299–300 weighting, 273–274, 304, 306–307 OLDMEDLINE, 119 OMIM. See Online Mendelian Inheritance in Man OMNI. See Organising Medical Networked Information Online Computer Library Center, 390–391 Online Mendelian Inheritance in Man, 133, 359 Online public access catalog, 387 Ontology, 372
OPAC. See Online public access catalog Open Archives Initiative, 392–393 Open Directory, 128, 164, 169–171, 201, 203, 209 OpenText, 290 OR, 184–187 Organising Medical Networked Information, 128 Orthography, 322 Ovid Technologies, 121, 131, 200–201, 241, 249, 344 P PageRank, 179 Palm Pilot, 384–385 PaperChase, 195, 230 Parsing, 314–318, 328–334, 399–402, 404–409 Parsing, partial, 331–334 Part of speech. See Syntactic category Passage retrieval, 279–280 discourse passages, 279 semantic passages, 279 window passages, 279–280 Patient-expected event rate, 78 Patient-oriented evidence that matters, 81 PDA. See Devices, handheld PDQ. See Physician Data Query Peer review, 36–39 problems, 37–39 Peer reviewers, 36–39 blinded, 37 Peer reviewers, consistency, 38 PEER. See Patient-expected event rate Peer-reviewed journal editors, 37 Peer-reviewed journals, 34 PEN & PAD, 426 Penn Treebank, 316–317 Perfect binding, 389 Periodicals, 130–131 Persistent URL, 390–391 Personal digital assistant. See Devices, handheld Personal knowledge scores, 243 Phrase construction, 277–279 PhraseFinder, 345–346 Phrases, 311 statistical, 329 syntactic, 329
Index Physician Data Query, 142–143 Physician orders, parsing, 414 Physician thinking, models of, 66–67 Physicians’ Information and Education Resource, 141–142 PICS. See Platform for Internet Content Selection PIER. See Physicians’ Information and Education Resource PiL. See Professional’s Information Link PINDEX, 345, 421 PIOPED. See Prospective Investigation of Pulmonary Embolism Detection Pivoted normalization, 274–275, 291, 304, 306–307 Plagiarism, 46 Platform for Internet Content Selection, 169 PLS. See Public Library of Science PMC. See Pubmed Central POEMS. See Patient-oriented evidence that matters POLYFIND, 323 Polysemy, 310 Population description, 357–358 POS. See Syntactic category Positive predictive value, 79, 410 PostDoc, 342, 421, 427 PP. See Prepositional phrase PPV. See Positive predictive value Pragmatics, 320 Praxis, 135 Precision, 95–110, 403, 407 mean average, 101, 112, 266, 290 Prepositional phrase, 313–314 Preservation, digital materials, 395–396 Preservation, library collections, 389 President’s Information Technology Committee, 386 Primary literature, authorship, 43 Probabilistic information retrieval, 273, 282–285 Professional’s Information Link, 364 Propositional template, 401–402 Prospective Investigation of Pulmonary Embolism Detection, 414 Protégé, 372 Protein Data Bank, 138 Protocol analysis, 87, 111
513
Proximity operator, 192–193 Psycholinguistics, 312 Psychological relevance. See Relevance, situational PsycINFO, 125, 157 Public Library of Science, 55–56 Publication bias, 44–45, 48, 80 Publication types. See Medical Subject Headings, publication types PubMed, 121, 131, 138, 191, 195–200, 205, 211, 214–215, 361, 363, 375 Clinical Queries, 199, 231 History screen, 198 Limits screen, 197 Preview/Index screen, 198 Related Articles, 190, 200, 367–369, 382 PubMed Central, 54–56, 131 PURL. See Persistent URL Puya, 425–426 Q Query expansion, 280–281, 382–383 Query formulation, 375–377 Query reformulation, 375 Query zone, 280 Question answering systems, 294–297, 354–355 Questionnaire for User Interface Satisfaction, 248–250 Questions, clinical, background, 76 Questions, clinical, foreground, 76 Questions, clinical, phrasing, 75–76 Quick Medical Reference, 415–416 QUIS. See Questionnaire for User Interface Satisfaction R Randomized controlled trial, 40–43, 76–79, 231 RCT. See Randomized controlled trial RDF. See Resource Description Framework Read Clinical Terms, 418–420 Readability, 60 Reading grade level, 60 RealNames, 291 Recall, 95–110, 403, 407 Recall, relative, 96
514
Index
Recall-precision analysis, 250–253 Recall-precision table, 99–102, 267 interpolation, 100 Receiver operator characteristic curve, 102 Reference tracing, 379, 381–382 Relative risk reduction, 78 Relevance, 103–108 Relevance feedback, 189–190, 280–281, 382–383 Relevance judgments, 267 factors effecting, 106–107 hedging phenomenon, 107 magnitude estimation, 107 research about, 106–108 Relevance ranking, 99, 187 Relevance, situational, 105–106 Relevance, topical, 104–105 Reporting, health news, 59 Research intervention, 85 Research population, 85 Research sample, 85 Research, classification of, 84–87 Research, comparative, 84–87 Research, objectivist, 84 Research, qualitative. See Research, subjectivist Research, quantitative. See Research, objectivist Research, subjectivist, 84, 111 ResearchIndex, 392 Residual collection method, 280 Resource Description Framework, 168–169, 372 Retrieval, 5, 183–218, 372–383 Reuters test collections, 267, 287 ROC curve. See Receiver operator characteristic curve Rocchio equation, 189–190, 281 Rosetta stone, 395 Roundsman, 357–358 RRR. See Relative risk reduction S SAPHIRE, 337–342, 349–351, 362, 355, 422–423 Scatter/Gather, 373–374 SCI. See Science Citation Index Science Citation Index, 28, 140–141, 229, 382
Scientific American Medicine, 240–241 Scientific discovery, facilitation, 427–428 Scientific literature, doubling time, 26 Scientific literature, growth of, 388 Scientific meeting presentations, 59–60 Scientific method, 35 Scientific revolutions, 34 Screen size, 383–384 SDLIP. See Simple Digital Library Interoperability Protocol Search engine, 5 Boolean operators, 207 gaming, 209 meta engines, 209 paid placement, 210 proximity operators, 207 relevance ranking, 208 stop words, 208 Web, 206–210 Search Engine Showdown, 234 Search process, 183–184 Searching errors, 255 Searching interfaces, 194–214 bibliographic, 194–201 Searching, Aggregations, 213–214 Searching, automated. See Searching, partial match Searching, Boolean. See Searching, exact match Searching, citations, 212 Searching, Databases/Collections, 210 Searching, exact match, 184–187 Searching, full text, 202–210 Searching, general principles, 184–194 Searching, genomics, 210–211 Searching, images, 210 Searching, natural language. See Searching, partial match Searching, partial match, 184, 187–190 Searching, periodicals, 204–205 Searching, ranked. See Searching, partial match Searching, set-based. See Searching, exact match Searching, specialized registries, 201–202 Searching, term expansion, 192 Searching, term lookup, 191 Searching, term selection, 190–194 Searching, textbooks, 205–206
Index Searching, Web catalogs, 201 Selective dissemination of information. See Information filtering Semantic grammar, 319–320, 407–409, 412 Semantic interpretation, 314, 318–320, 399–401 Semantic network, 311 Semantic type, 318–320 Semantic Web, 365, 371–372 Semantics, 318–320 Sensitivity, 78–79, 410 Sensitivity analysis, 47 Serendipity, 109, 184 Set theory, 185 Shannon and Weaver model, 23–25 Sherlock, 209 Simple Digital Library Interoperability Protocol, 393–394 Singular-value decomposition, 276–277 Skip-and-fit strategy, 333 SLA. See Special Library Association Slice of Brain, 136–137 Slice of Life, 136–137, 180 SMART, 286, 329, 334–335, 354 SmartQuery, 360–363 SNOMED. See Systematic Nomenclature of Medicine SNOMED CT. See Systematic Nomenclature of Medicine, Clinical Terms SNOMED RT. See Systematic Nomenclature of Medicine, Reference Terminology SNOP. See Systematic Nomenclature of Pathology Social Science Citation Index, 140–141, 229 Sourcerer, 370 Spatial visualization, 248–253 Special Library Association, 12, 387 SPECIALIST lexicon, 159, 316–318, 323, 342 Specialized registries, 118, 128–129 Specificity, 79 Spelling correction, 323–324 Spelling errors, 399–400 SSCI. See Social Science Citation Index STAIRS, 232–233 Stat!-Ref, 131, 205–206
515
Statistical mapping, 344–345 Statistical significance, 87 Stemming, 178, 285, 327–328, 423 Stop list, 147, 175–178 PubMed, 175–177 Structured abstracts, 43 Structured data entry, 426 Subgrammar, 401 Subheadings. See Medical subject headings subheadings Subject dispersion, 29–30 Sublanguage analysis, 403 Sublanguage selection, 404 SUI. See Unified Medical Language System, Metathesaurus, string unique identifier SUMSearch, 374–375 SuperBook, 377 SWISSPROT, 138 SymText, 412–414 Synonym, 149 Synonymy, 310, 401 Synset, 335–336 Syntactic analysis, 328–334 Syntactic category, 314 Syntactic disambiguation, 329–331 Syntax, 314–318 Systematic Nomenclature of Medicine, 158, 405, 416–422, 426 Clinical Terms, 418–419 modules, 418 Reference Terminology, 418 Systematic Nomenclature of Pathology, 416 Systematic reviews, 46–49, 80–81 T Tagged Text Parser, 333–335 Tagging, 146 Term, 149 Term dependency, 324 Term discrimination value, 175, 273 Term frequency, 178 Term frequency, augmented normalized, 272–273 Term strength, 288 Terminal terms, 194 Text categorization, 286–287, 420 Text mining, 397
516
Index
Text REtrieval Conference, 110–113, 265–268 ad hoc task, 111 criticism of, 113 Cross-Language Track, 111, 347–348 documents, 268 Filtering Track, 111–112, 216 Interactive Track, 111, 300–309 NLP Track, 332–335 passage retrieval, 280 phrase construction, 278–279 queries, 268 Question-Answering Track, 111–112, 289, 294–297, 354–355, 397 relevance judgments, 111 routing task, 111 tasks, 295, 297 tracks, 111 Web Track, 111, 288–289 Text summarization, 297–298 abstract, 297–298 characteristics of summaries, 298 extract, 297–298 features to determine sentence extraction, 298 Textbooks, 131–134 TF. See Term frequency TF*IDF weighting, 188–190, 269–270, 272 Thesaurus, 147. See Controlled vocabulary Thesaurus generation, automatic, 274–276 clique methods, 276 single-link methods, 276 THOMAS, 358 TileBars, 377 TOIS. See Transactions on Information Systems TopicMap, 378–379, 381 ToxNet, 145 Transactions on Information Systems, 13 Transformational tagging, 321–322 TREC. See Text REtrieval Conference TRIP database, 374 Truncation, 192 TTP. See Tagged Text Parser Turnaround time, 183 U UMLS. See Unified Medical Language System
Unified Medical Language System, 149, 157–162, 316, 416, 429 Information Sources Map, 365, 387, 369–371, 374 Metathesaurus, 159–161, 325, 337, 340–344, 346–347, 360, 382–383, 400, 419–424 canonical form, 159 concept unique identifier, 159 data elements, 161 string unique identifier, 159 term unique identifier, 159 Semantic Network, 159 Uniform Resource Identifier, 390 Uniform Resource Locator, 390 Uniform Resource Name, 390 Unigene, 139 Union. See OR Unix spell checker utility, 400 Up-to-Date, 141 URAC. See American Accreditation Healthcare Commission URI. See Uniform Resource Identifier URL. See Uniform Resource Locator URN. See Uniform Resource Name Usability testing, 87, 111, 373 V Validity, external, 87 Validity, internal, 87 Vancouver style, 36 Variable, dependent, 85–86 Variable, independent, 86 Variable, outcome. See Variable, dependent Variable, predictor. See Variable, independent Vector-space model, 187, 269–272 Verb phrase, 313–314 Verbal reasoning, 248 Visible Human Project, 136–137, 180–181, 210 Visible Web, 20 VP. See Verb phrase W Web catalogs, 118, 126–128 Web crawling, 179–180 Web impact factor, 32, 292
Index correlation with research output, 292 external, 292 Web of Science, 140–141, 212 Web searching, 288–294 logs, 224 Web sites, 118, 134–135 U.S. government, 134 Web, analysis of, 292–293 Web, distribution of links, 293 Web, estimated size of, 179–180 WebMD, 135 WebMEDLINE, 201, 230 Westlaw Inference Network, 299 WIF. See Web impact factor Wild-card character, 192 WIN. See Westlaw Inference Network Windows CE, 384–385 WIPO. See World Intellectual Property Organization WordNet, 309, 335–336
517
Word-sense disambiguation, 322, 325, 334–336 World Intellectual Property Organization, 394 World Wide Web, 14–20 amount of health information, 18 hubs and authorities, 19–20 mathematical properties, 18–19 size, 14 WSD. See Word-sense disambiguation WT10g, 288–289 X XML. See eXtensible Mark-up Language Y Yahoo, 290–291, 373 Z Z39.50, 392
