Topics in Cognitive Science 3 (2011) 1–2 Copyright ! 2011 Cognitive Science Society, Inc. All rights reserved. ISSN: 1756-8757 print / 1756-8765 online DOI: 10.1111/j.1756-8765.2010.01129.x
Introduction to Volume 3, Issue 1 of topiCS Wayne D. Gray Executive Editor
With this issue, topiCS begins its third year of high-quality papers with on-time print and electronic publication of each issue. In recognition of our fast entry into the ranks of journal excellence, the ISI Web of Science has announced that topiCS will be included in the Social Science Citation Index as well as the Current Contents: Social and Behavioral Sciences Index. We will receive our first Impact Factor in the 2011 edition of the Journal Citation Reports, which will be released in Summer 2012. This is great news for our past, current, and future authors as well as our topiCS Editors. It means that papers in our first two volumes are already highly cited. Of course, we knew we were doing well; however, it is very gratifying to have our collected labors acknowledged by the folks at Thomson Reuters. This issue of Topics in Cognitive Science introduces a new topic, continues an annual tradition, and includes a commentary and response on a paper published in our January 2009 issue (Volume 1, Issue 1). Computational Methods to Extract Meaning from Text and Advance Theories of Human Cognition has been shepherded by Danielle McNamara (University of Memphis). Counting her excellent introduction, there will be nine papers on this topic, five of which appear in this issue with the rest appearing in 3:2. We also continue our tradition of publishing the Best Of papers that have appeared in recent cognitive science conferences. All Best Ofs have been triply reviewed: once to get into a conference, once as a prize winner for that conference, and once for topiCS. This year’s crop of four Best Ofs include two each from the 2010 Cognitive Science Conference and the 2010 International Conference on Cognitive Modeling. Although there were other Best Ofs that met our criteria, those authors declined our invitation because they were already in the process of including their conference work as part of a longer journal publication. Our final topic is a Commentary by Benjamin Hilbig (University of Mannheim) and Tobias Richter (University of Kassel) and response by Henry Brighton and Gerd Gigerenzer (Max Planck Institute for Human Development) on the paper, ‘‘Homo heuristicus: Why Biased Minds Make Better Inferences’’ (Gigerenzer & Brighton, 2009). We believe this commentary and response serve to deepen the discussion begun in the original paper and we are sure that the cognitive science community will enjoy this exchange. In August, we delivered our first annual report on topiCS to the Cognitive Science Society’s Governing Board. For that report, we assembled a variety of statistics focused on
2
W. D. Gray ⁄ Topics in Cognitive Science 3 (2011)
estimating how well we did in our first year (Volume 1, 2009) in relation to our big sister journal, Cognitive Science. As we could not compare citation counts, our report highlighted the number of downloads of 2009 papers in the 16-month period from January 2009 to April 2010. This is a partial count because it is limited to papers downloaded from the WileyBlackwell journals website. In that period, we had 11,205 downloads of 2009 topiCS papers compared to 9,826 downloads of 2009 Cognitive Science papers. Obviously, with their 30+ years of publication, Cognitive Science had vastly more total downloads than topiCS, but we are very pleased that our first year attracted so much positive attention from the cognitive science community. Note that at that time (April 2010), our number-one download was the paper by Gigerenzer and Brighton (2009) that is the subject of the commentary that appears in this issue. As always, topiCS encourages commentaries and new topics. Send your commentaries directly to me at
[email protected] along with a short note. If you are proposing a topic, please open communications with a short first note (about 300–650 words or fewer) and be sure to consult the topiCS FAQ page, http://csjarchive.cogsci.rpi.edu/topiCS/FAQs.html, for Preparing a Proposal for topiCS.
Reference Gigerenzer, G., & Brighton, H. (2009). Homo heuristicus: Why biased minds make better inferences. Topics in Cognitive Science, 1(1), 107–143.
Topics in Cognitive Science 3 (2011) 3–17 Copyright ! 2010 Cognitive Science Society, Inc. All rights reserved. ISSN: 1756-8757 print / 1756-8765 online DOI: 10.1111/j.1756-8765.2010.01117.x
Computational Methods to Extract Meaning From Text and Advance Theories of Human Cognition Danielle S. McNamara Department of Psychology, University of Memphis
Abstract Over the past two decades, researchers have made great advances in the area of computational methods for extracting meaning from text. This research has to a large extent been spurred by the development of latent semantic analysis (LSA), a method for extracting and representing the meaning of words using statistical computations applied to large corpora of text. Since the advent of LSA, researchers have developed and tested alternative statistical methods designed to detect and analyze meaning in text corpora. This research exemplifies how statistical models of semantics play an important role in our understanding of cognition and contribute to the field of cognitive science. Importantly, these models afford large-scale representations of human knowledge and allow researchers to explore various questions regarding knowledge, discourse processing, text comprehension, and language. This topic includes the latest progress by the leading researchers in the endeavor to go beyond LSA. Keywords: Sematic models; Computational techniques; Meaning extraction; Cognition; Memory; Embodiment; Latent representations; LSA
1. Introduction One method of scientifically investigating human cognition is to examine what we write. This endeavor is facilitated by current technologies that allow scientists to conduct discourse analyses on extremely large scales. Statistical models are applied to large linguistic corpora (i.e., collections of written text) to extract (what we think is) the meaning of the text and, moreover, to enhance our understanding of how the human mind works. A fundamental assumption among this collection of articles is that human cognition manifests itself in our Correspondence should be sent to Danielle S. McNamara, Department of Psychology ⁄ Institute for Intelligent Systems, University of Memphis, 202 Psychology Building Memphis, TN. E-mail: dsmcnamara1@gmail. com
4
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
writings. Large text corpora combined with computational techniques for analyzing these corpora allow scientists to extract meaning from text and, by consequence, to explore various aspects of the human mind and culture that manifest in text. To a large extent, this volume of topiCS is dedicated to showing the value of using corpus analytical techniques to understand cognition. A well known statistical method of extracting meaning from text is latent semantic analysis (LSA). In Landauer, McNamara, Dennis, and Kintsch (2007), we provided basic information about LSA as well as a number of examples of its use in furthering our understanding of cognition and in contributing to applied technologies. LSA was a groundbreaking method in which word meanings are extracted by determining the company the word keeps across large corpora of texts. Collections of word meanings then comprise sentence, paragraph, and document meanings. More specifically, given a large corpus of text with millions of words and thousands of documents, a matrix is created that indicates the context in which each word occurs. The context of a word is the document that it occurs in, which may be the sentence, paragraph, or entire text. Each word or term is defined in terms of a vector (i.e., one dimensional array of values) of the documents in which it occurs. This is a sparse matrix because most terms occur in few documents and it is a large matrix because there are many terms across many documents. Thus, the matrix is reduced to discover its latent properties. In LSA, it is reduced by applying singular value decomposition (SVD) and then truncating the number of dimensions to include hundreds rather than thousands of dimensions in the matrix. This process creates an LSA space that is multidimensional, where each term is located in this multidimensional space. Similarity metrics such as the cosine between words, or collections of words, are indicative of how related they are. A higher cosine indicates that the words are more related. By uncovering these latent relations between words, the meanings of the words, sentences, and texts can be derived. Latent semantic analysis has been extremely successful in helping researchers understand a wide range of theoretical issues concerning meaning and also in scaling up theoretical insights to applied tasks (see e.g., Landauer et al., 2007; Louwerse, 2010, this volume). However, LSA alone is not the focus of this issue of topiCS. Here, we focus on going beyond LSA. In this topic, the featured articles describe new methods of using LSA as well as other mathematical and computational approaches that are comparable and sometimes exceed the capabilities of LSA. Our objective is not to reinforce the multiple uses of LSA, but more so to show how current research goes beyond LSA.
2. Statistical semantic models Statistical models of semantics have been a focus of research for over a half century (e.g., Osgood, Suci, & Tannenbaum, 1957; Smith, Shoben, & Rips, 1974). However, the availability of technologies that could handle large corpora and high-dimensional spaces (e.g., Salton, Wong, & Yang, 1975) helped to change the field of semantics. These technological advances helped to spur LSA, which has been largely responsible for the growth in interest
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
5
in and use of statistical models of semantics over the last two decades. LSA was immensely successful both as a theoretical model (e.g., Landauer, 2007; Landauer & Dumais, 1997) and as a practical tool (e.g., Landauer et al., 2007). Albeit successful and effective, a good deal of research has been devoted to improving LSA and to developing and improving semantic models in general. The majority of the articles in this volume are concerned with this objective. Over the past few decades, numerous semantic models have been developed and thus there is a large scope of models available that are capable of (more or less) successfully extracting meaning from text. Riordan and Jones (2010, this volume) describe nine statistical models of semantics, which they refer to as distributional models: • LSA (Landauer & Dumais, 1997) • Probabilistic Topic Model (Steyvers, Chemudugunta, & Smyth, 2010, this volume; Steyvers & Griffiths, 2007) • Correlated Occurrence Analog to Lexical Semantics (COALS; Rohde, Gonnerman, & Plaut, 2005) • Contextual Similarity Log-Likelihood (CS-LL; Dunning, 1993) • Contextual Similarity Log-Odds (CS-LO); Lowe, 2001) • Hyperspace Analog to Language (HAL; Lund & Burgess, 1996) • High Dimensional Explorer (HIDEx; Shaoul & Westbury, 2006) • Positive Pointwise Mutual Information (PosPMI; Church & Hanks, 1990) • Bound Encoding of the Aggregate Language Environment (BEAGLE; Jones & Mewhort, 2007) Stone, Dennis, and Kwantes (2010, this volume) describe three additional models: • Vectorspace Model (Salton et al., 1975) • Sparse Nonnegative Matrix Factorization (SpNMF: Xu, Liu, & Gong, 2003) • Constructed Semantics Model (CSM; Kwantes, 2005) In addition, Howard, Shankar, and Jagadisan (2010, this volume) describe their transformation of an episodic memory model temporal context model (TCM) into a new approach to semantic modeling: • Predictive Temporal Context Model (pTCM) One means of subdividing semantic models is into two categories, context word and context region (see e.g., Riordan & Jones, 2010, this volume). These two approaches differ in terms of the matrix that comprises the initial representation. Context word models use a word-by-word (term-by-term) matrix that is defined using a moving window approach. Essentially, the document size is reduced and defined only by words surrounding the target word. Words’ co-occurrences within a defined window (e.g., two or more consecutive words) comprise the matrix. These models differ in the size of the window and whether the window considers both prior and subsequent words within the window. In addition, the words’ co-occurrences are differentially weighted (e.g., by multiplying the co-occurrence
6
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
matrix by another matrix that includes weights). They may be weighted by distance from the target words (HAL, COALS), frequency within the corpus (HIDEx), or their conditionalized co-occurrences (COALS, CS-LL, CS-LO, PosPMI). BEAGLE is also a context word model, but it incorporates convolution (e.g., Murdock, 1992) as a means to compress n-gram information, which captures both syntactic (word order) and semantic (contextual) information about each word. Context region models, including the Vectorspace model, LSA, Topic, SpNMF, CSM, and pTCM, use a word-by-context (or term-by-document) matrix. In models such as the Vectorspace model, LSA, and SpNMF, the context of a word is the document in which it occurs; as described earlier, the document may be the sentence, paragraph, or entire text. In the Topic model (see Steyvers et al., 2010, this volume), the context is defined by latent topics identified in the corpus. Each document is represented by multiple topics as a probability distribution, and each topic is represented by a probability distribution of words. LSA and SpNMF are much like the Vectorspace model in that they both weight the vector representations of the terms by multiplying the vector by the log of the terms’ frequency within the document and then dividing by the inverse document frequency (IDF)—these two techniques ensure that words that are important to a document are heavily weighted but the impact is reduced for words that are frequent across documents (and thus contain less discriminatory information). LSA departed from the Vectorspace model by its use of SVD to reduce the matrices to their latent dimensions. SpNMF differs from other models in that it is constrained to contain only nonnegative values. The CSM model (Kwantes, 2005) differs from other context region models by using retrieval mechanisms in lieu of dimension reduction techniques. The retrieval mechanisms in CSM are based on the MINERVA 2 model of episodic memory (Hintzman, 1984). Accordingly, memories for events are feature-based representations stored as individual traces, or episodes. Each event or item is stored as a vector of values associated with features, which in turn have a probability value or learning rate. Retrieval depends on similaritybased matching rules and prior probabilities. Semantic similarity in CSM is a function of resonance between a probe vector and the context vectors. The context vectors with the stronger resonance to the probe contribute more strongly to the ultimate outcome. Different from other distributional models, CSM is an incremental learning model because it updates the representation as the sequence of words is presented. The pTCM introduced by Howard et al. (2010, this volume) similarly updates the representation incrementally. pTCM is an extension of TCM, a model of episodic memory (Howard & Kahana, 2002). TCM is a distributed memory model that specifies the effects of temporal context on episodic memory (usually memory for words embedded in lists). Rather than using co-occurrences in memory to explain memory phenomena such as primacy and recency, TCM provides an account that is based on mechanisms related to contextual drift and contextual retrieval. Thus, TCM has similarities to models such as LSA due to the overlapping assumptions regarding the effects of context on memory and learning. pTCM models semantic memory by using the current state of context (i.e., a weighted sum of the semantic representations of recently presented items) as a cue to generate predictions about what word will be presented next. The semantic representation
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
7
of a word is formed by averaging all of the predictions that precede its presentation. The model forms these representations incrementally over text rather than using the entire text representation as do most semantic memory models (and thus the time to run the model is somewhat cumbersome). Howard et al. show that the pTCM model performs comparably to LSA in the task of identifying semantically related words. Although a disadvantage stems from the time to run the model, its clear advantage is that it is based on a model of episodic memory; thus, this research shows that similar architectures may be used for both episodic memory and semantic representations in memory (see also the BEAGLE model; Jones & Mewhort, 2007). The deep generative model described by Hinton and Salakhutdinov (2010, this volume) is also iteratively trained and inspired by cognitive models, but it is quite different from other semantic models described in this volume. Hinton and Salakhutdinov describe and test a nonlinear, multilayer model that yields binary codes that can serve to describe documents’ meaning. The lower layers of the model are hidden layers with distributed representations that are iteratively trained using a Restricted Boltzman Machine (Hinton, Osindero, & Teh, 2006). After the features of one layer are learned, those vectors are used as data to train the next hidden layer. This process of iterative learning yields an encoder network that converts the word-count vectors to compact codes. Then the process is reversed such that the Restricted Boltzman Machine acts as a decoder network that reconstructs the word-count vectors from the compact code vectors. These are combined to produce a multilayer autoencoder network. Backpropagation is used to iteratively improve the solution. This learning process is relatively slow. By contrast, after learning, retrieving solutions is rapid, much more rapid than techniques such as LSA. This difference would be particularly noticeable with large data sets. In addition, the new data can be added to the network iteratively, so the system can continue to learn. Thus, this type of approach would be particularly suitable for handling extremely large corpora such as the Wikipedia. The initial training would be very slow, but thereafter, new data could be added, and retrieval based on meaning would be extremely fast.
3. Comparing semantic models In the quest to improve computational models of semantics, one current research question regards which models more effectively capture meaning and cognition. This topic includes two studies that compared models in terms of their ability to account for cognitive phenomenon. Stone et al. (2010, this volume) compared six models’ ability to judge similarity between paragraphs (i.e., LSA, SpNMF, Topic, Topic-JS, Vectorspace, and CSM). They found that all of the models fared relatively dismally in comparison to a word overlap model that used the cosine between term frequency vectors. That is, the model that used the nonlatent, overt representation of the text fared best in simulating humans’ judgments of paragraph similarity. The latent models, however, showed improved performance with when smaller, topic-focused corpora were used to train the models (see also Shapiro & McNamara, 2000).
8
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
Constraining the knowledge space helps to avoid contamination from multiple word meanings across contexts. For example, whereas humans can quickly discern the meaning of bank in the context of finances, most statistical models of semantics will retrieve all senses of words (Kintsch & Mangalath, 2010, this volume). Thus, more focused, topic-specific spaces avoid the problem of retrieving multiple senses of words that are likely to be irrelevant. Riordan and Jones (2010, this volume) compared models’ ability to capture semantic clusters in comparison to humans’ judgments of concept features. They found that among the nine models compared, a context word model, COALS, was best at capturing the semantic clusters. The model’s success was attributed to how it conditionalizes the co-occurrences of words. COALS is based on HAL, but it reduces the influence of high-frequency words such as function words (e.g., the, a, that) to reduce the impact of chance co-occurrence. It does so by calculating the conditional rate of co-occurrence (i.e., the frequency of co-occurrence in comparison to its base rate co-occurrence with other words). This is achieved in COALS using a correlation matrix in which the negative values are discarded and the positive values square rooted (to increase their weight). However, this explanation for COALS relative success may not provide an entirely convincing explanation because other models also control for chance co-occurrence of words and virtually all models control for highly frequent words. Thus, why this model faired so well in terms of detecting semantic clusters remains a question (but see also Rohde et al., 2005). The success of COALS in Riordan and Jones (2010, this volume) supports the notion that the meaning of words is in the company they keep (Firth, 1957)—but further implies that the company only includes close neighbors who are not friends with everyone else. Hence, local word co-occurrence (corrected for chance) goes a long distance in extracting meaning. Likewise, Stone et al. (2010, this volume) found that the nonlatent word overlap model faired far better than the six statistical models. One important issue regards whether meaning need be extracted using a latent or second-order representation or by contrast whether that meaning can be extracted with equal success from the words alone.
4. Improving semantic models Comparing semantic models improves our understanding of which aspects of the models contribute most to successful meaning extraction. Another approach to improving methods of extracting meaning from text is to augment statistical models with other techniques. Steyvers, Chemudugunta, and Smyth (2010, this volume) do so by combining the Topic model with human judgments. They propose that models of the learning process can be enhanced by using corpus techniques to represent background knowledge of the learner. As such, by combining information from corpora and human data, statistically derived relationships can complement human-defined concepts by filling in missing information, and vice versa, the combination of the techniques may indicate the relative importance of preexisting and new knowledge extracted from a text. Another means of improvement regards the analyses that are conducted with their output and how the results are interpreted. Louwerse (2010, this volume) uses multidimensional
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
9
scaling (MDS) to extract the representation yielded by the LSA analysis, reducing it from hundreds of dimensions to a two-dimensional visualization. One advantage of MDS (and other similar methods) is that it provides an indication of the distance between concepts. For Louwerse, this technique affords the visualization of spatial and temporal relations present in the text corpora. Similarly, Riordan and Jones (2010, this volume) use MDS to convey the nature of the clusters of concepts. The distance between clusters represents similarity of the clusters; the height represents the internal consistency of the cluster; and the area indicates the number of terms in the cluster. Thus, MDS can provide additional information as well as facilitate interpretation of underlying relationships in the data. Other researchers have also used this technique. For example, O’Rourke and Calvo (2009) have incorporated the use of MDS in combination with LSA to examine the relationships between paragraphs in essay writing. MDS essentially allows the researcher to uncover the clusters of related concepts and the distances between text sections, much like principal component and factor analyses.
5. Embodiment One important and hot topic in computational semantics regards the potential importance of perceptual simulations and embodiment in cognitive processing. It is clear to most that our experiences are constrained by the world and how our body fits into that world. By consequence, cognition is shaped by our experiences in the material world. Similarly, it is clear that the mind often represents the world, and our experiences in that world, in some sort of analog form, such as images. If that is the case, some argue that modeling human cognition using mathematically based or symbolic representations is futile because these models cannot capture a fundamental aspect of cognition, embodiment (e.g., Barsalou, 1999; De Vega, Glenberg, & Graesser, 2008; Glenberg, 1997). Moreover, some argue that meaning as it relates to cognitive processes cannot be extracted from representations such as text because text, being verbal and symbolic, cannot provide a complete picture of human cognition. One potential value of this extremist view is that it has served to fuel the debate. In addition, some researchers seek elegant models or attempt to see how far one can go with a simple model. Pushing the limits of simple models can reveal a good deal about the phenomena. Nonetheless, most recognize that there is value to both viewpoints (e.g., Paivio & Sadoski, in press). Most researchers and theorists recognize that cognition is comprised of symbolic, abstract, and verbal thought and reasoning, in addition to concrete and embodied representations. The notion that we have only one form or the other to represent meaning in the mind is, frankly, absurd. Two articles in this topic address this issue (Louwerse, 2010, this volume; Riordan & Jones, 2010, this volume). Riordan and Jones (2010, this volume) compare symbolic, distributional models (e.g., LSA) and feature-based models (that rely on human judgments of features) in their ability to represent semantic clusters. Feature-based models have been argued to more aptly represent perception and action in cognition than do purely symbolic, distributional models such as LSA because they represent words’ meanings in terms of their descriptive features (e.g., McRae, Cree, Seidenberg, & McNorgan, 2005; see Riordan &
10
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
Jones, 2010, this volume, for additional citations). These features are assumed to be closer to sensorimotor experiences, which in turn are assumed to comprise a fundamental aspect of words’ meanings (e.g., Barsalou, 2003). Riordan and Jones (2010, this volume) compare feature-based models to the nine distributional models listed earlier (i.e., LSA, Topic, COALS, CS-LL, CS-LO, HAL, HIDEx, PosPMI, and BEAGLE). They examine the ability of these nine models to extract semantic clusters in comparison to human generated norms on word features. In the first two studies, the models were trained using the TASA corpus and then compared on their ability to cluster words according to semantic class labels used in WordNet. Their performance was compared to McRae feature norms (McRae et al., 2005) on concrete nouns in the first study and Vinson and Vigliocco (2008) feature norms on nouns and verbs in the second study. In the third study, they compared models using semantic classes from the MacArthur-Bates Communicative Development Inventories (Fenson et al., 1994) and trained the models using the CHILDES database (MacWhinney, 2000) of caregiver speech (adults’ utterances to children, 12–48 months). Across the three studies, Riordan and Jones (2010, this volume) found that several models rivaled the human-based feature norms. However, across datasets, COALS was the most consistently found to be comparable to the feature-based norms in reproducing semantic classes. However, they also found that the distributional and featural information were not redundant. The distributional models tended to pick up on actions, functions, and situations, and less about perceptual attributes such as color or texture. Nonetheless, the distributional models use of linguistic cues in the language rivaled performance by feature norms (produced by humans). One implication of this research is that a model that combines human derived norms within a distributional approach may be more successful (see Steyvers et al., 2010, this volume). On the whole this research indicates that the statistical models are able to pick up on information associated with embodied thought, but not in the same way as do humans. Louwerse (2010, this volume) also argues that both aspects of processing (i.e., symbolic and embodied) are important to describing human cognition. He refers to this as the Symbol Interdependency hypothesis, which supposes that language and language comprehension depend on interdependencies of abstract linguistic symbols as well as the references these symbols make to perception and action. Furthermore, he provides evidence that perceptual and modal aspects of cognition can be extracted from text corpora, which has been considered purely symbolic by the embodied theorists. The underlying assumption of Louwerse’s argument (and this issue of topiCS) is that how we think manifests itself in the language we use. As such, the effects of perception and action, or embodied thought, on cognition can be extracted from language. Louwerse (2010, this volume) shows that a wide array of results that have been used to support the embodied perspective can be replicated using techniques such as LSA. For example, he demonstrated that word pairs from the same sensory modality (motoric, smell, sound, taste, touch, and vision) have higher LSA cosines than do words from different modalities. In addition, concepts have a stronger relationship according to LSA cosine values to their properties or features than to properties descriptive of other concepts. Thus, like
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
11
Riordan and Jones (2010, this volume), Louwerse shows that features can be detected using computational models such as LSA, even if those features might be considered embodied and thus beyond computation by many researchers.
6. Where is meaning in text? We assume that there is meaning in text and that meaning resides in the words, sentences, paragraphs, and so on. However, one question is where the meaning of the text resides. Can the full scope of meaning be extracted solely from the words and their co-occurrences (i.e., the company they keep) or is more context and information needed, such as syntax or human-derived data? Most if not all text comprehension models assume that comprehension occurs at various levels that together produce the reader’s mental representation (see e.g., Kintsch, 1998; McNamara & Magliano, 2009). For example, the Construction-Integration model assumes at least three levels of representation, the surface code (words and syntax), the propositional textbase (deeper meaning of the text), and the situation model (a combination of the text with prior knowledge). Because readers’ comprehension is multidimensional, a more complete picture of it is provided when comprehension is assessed in various ways and at multiple levels (e.g., using both text-based and situation model questions; McNamara & Kintsch, 1996). One cause for multiple levels of comprehension can be attributed to cognitive processing mechanisms. However, another is the signal itself. Language is comprised of multiple levels of information—it is multidimensional. If language comprises different levels of meaning, then statistical models that seek to extract meaning from text should also assume that the meaning should be extracted based on these levels. For example, Jones, Kintsch, and Mewhort (2006) demonstrated that including both semantic and syntactic (word order) information improves the ability of semantic models to account for a wider range of cognitive phenomena (see also, Dennis, 2005; Dennis & Kintsch, 2008). Similarly, different aspects of a representation can be emphasized by varying parameters within the algorithms. For example, McNamara, Cai, and Louwerse (2007) evaluated variations of the LSA algorithm to examine whether the performance of LSA could be improved by varying two factors: emphasis on high- versus low-frequency words, and similarity strictness. Overall, the study indicated that different algorithms may be more apt to detect differences in meaning depending on the level of analysis. Thus, different algorithms may be more or less appropriate and effective depending on the cognitive processes that are targeted in the particular study. Indeed, this is an underlying assumption of Coh-Metrix (Graesser & McNamara, 2010, this volume; McNamara, Louwerse, McCarthy, & Graesser, 2010). Coh-Metrix provides information about various levels of language to support investigations of how and when these levels come into play in various situations. Coh-Metrix provides indices on words (e.g., frequency, concreteness, homonymy, polysemy), sentences (e.g., length, noun phrase density, number of words before the main verb), and cohesion (lexical diversity, referential cohesion, semantic cohesion). Cohesion is the level of connectivity in a
12
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
text—the degree to which clauses, sentences, ideas, and larger sections of a text are explicitly tied together. If text is conceptualized as a network of nodes and links, then the nodes would represent the words (or the parts of words) of the text. The words of a text can be predominately abstract or concrete, familiar or unfamiliar, ambiguous or unambiguous. These are characteristics of the words that result from and are thus signals for the meaning of a text. Indeed, articles in this topic support the notion that a good deal of a text’s meaning can be detected just on the basis of the words in the text. For example, Stone et al.’s (2010, this volume) findings indicate that meaning can be extracted from the nonlatent information available in large text corpora (see also Recchia & Jones, 2009). LSA was a groundbreaking technique because it demonstrated that the text could be reduced to fundamental components comprising a latent representation, which contained the essentials of the text’s semantics. However, dimension reduction is not always crucial to successful meaning extraction. Along these lines, Louwerse (2010, this volume) argues that it is not computational models such as LSA that, first and foremost, afford the ability to extract meaning from text, but rather this ability emerges from the organization of language itself. Cohesion is one aspect of that organization and, in particular, connectivity. Cohesion is the glue that allows the ideas to stick together. At the surface level, words units serve as a glue to connect phonemes and morphemes, and syntax serves as glue to combine the words into a meaningful unit. At the textbase level, verbs serve as glue to connect ideas together to form meaningful clauses. Overlapping words and concepts among sentences serve to tie the ideas together. Likewise, at the situation model level, connectives serve as signals of the relationships between ideas and between the larger concepts that are being expressed across the text. Further, rhetoric, pragmatics, and global cohesion cues tie the text or discourse into a meaningful unit of discourse. These cohesion cues are important to comprehension because when those cues are not present in the text, then the reader must use reasoning and prior knowledge or retrieve prior text to infer the missing connections (e.g., McNamara & Kintsch, 1996; O’Reilly & McNamara, 2007). It is the combined information from the words, sentences, and relations that afford the most complete picture of language. Thus, picking up on the multidimensionality of text and its deeper meaning likely depends on assessing the various dimensions of how those words are connected, beyond their particular characteristics, proximity, and co-occurrence (latent or nonlatent). If meaning is present at multiple levels, then a more complete statistical model of semantics would benefit by taking into account multiple levels of processing. Kintsch and Mangalath (2010, this volume) do just that. They present a model that makes use of both context-word and context-region matrices as well as syntactic information. These sources of information are combined within a modified version of the Construction-Integration model of comprehension (CI-II). The word-document matrix generated using the Topic Model provides relational, gist information (called the gist trace), whereas a word-word matrix provides information representative of a surface level of processing (called the explicit relational trace). Syntactic information is provided by a dependency grammar parser, which provides two explicit sequential traces (one for each side of the dependency unit). Thus, this model
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
13
potentially captures both textbase (gist) and surface (explicit) level representations of text, as well as syntactic information. The CI-II model randomly samples from the explicit sequential trace with the constraint that the sample be both semantically and syntactically relevant as well. Kintsch and Mangalath show that the conditionalized combination of these three sources of information is more powerful across of a range of semantic, lexical, and sentential tasks compared to using only one or two sources or using LSA. The CI-II model (Kintsch & Mangalath, 2010, this volume) also emphasizes the importance of context in deriving meaning from words and sentences. Whereas multiple meanings of words may reside in long-term memory, a generative comprehension model is necessary to weed out these multiple word senses when a word in understood in the context of text and discourse. Whereas a word such as band has multiple meanings in long-term memory, its meaning is constrained by context in sentences such as He played in a rock band and He wore his wrist band. The CI-II model narrows down the meaning of words in context by making use of multiple sources of information and basing activation on the combined conditionalized probabilities. This operates somewhat like the predication model (Kintsch, 2001, 2007, 2008). Such an approach allows the contextualized meaning of words to emerge iteratively in working memory. This is particularly important for accounting for more complex cognition, such as the understanding of metaphor.
7. Conclusion Collectively, the research presented in this topic supports a number of conclusions. First, numerous models and variations on models have been developed over the last two decades, with a dramatic growth in interest and research in the last decade. Semantic models that apply statistical algorithms to large corpora of text are powerful means for extracting meaning from text. The computational power that has emerged over the last three decades has afforded the ability to analyze large corpora that successfully represent a good deal of human knowledge. Some have questioned the ability of such techniques in capturing the true essence of human cognition because ‘‘bag of words’’ techniques potentially miss out on important aspects of cognition. However, we see here that even nonlatent approaches to corpora analysis successfully capture much of what has been deemed well beyond the mere words in text. Indeed, Louwerse (2010, this volume) shows that a good deal of the results reported by embodiment theorists can be simulated using both nonlatent and latent (i.e., LSA) statistical models. Second, semantic models can be augmented by combining them with data generated by humans, by accounting for word order or syntax, and by accounting for multiple levels of cognitive processing. Essentially, it seems that including multiple sources of information and assuming multiple levels of processing will be necessary for models to account for a wide range of cognitive data. For the most part, semantic models simulate human knowledge, as it resides inertly in long-term memory. The contents of a model’s knowledge base can be manipulated by controlling the text that comprises the space. For example, to simulate an
14
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
8-year-old, one would use text and discourse to which a typical 8-year-old might have been exposed. The performance of semantic models improves when the text corpora used to create the semantic space is contextually and developmentally constrained (e.g., Shapiro & McNamara, 2000; Stone et al., 2010, this volume). The importance of the ability of semantic models to simulate human knowledge should not be trivialized—it was a path-breaking achievement both theoretically and practically. Nonetheless, one challenge for these models is to go beyond the fine tuning of extracting semantic similarity based on statistical constraints in corpora, which are in turn aligned with particular mathematical properties. Semantic models must account for complex cognitive phenomena such as humans’ understanding of synonymy, paraphrase, metaphor, and coherence. Indeed, several of the researchers featured in this issue have just done that—or at least paved the road to do so in future research. To go beyond the simulation of knowledge, and account for performance on a wide variety of tasks, it seems that semantic models must use a combination of approaches. First, models that combine assumptions relevant to both episodic memory and semantic processing are successful in accounting for a variety of phenomena, including incremental learning, memory, and semantic processing (Hinton & Salakhutdinov, 2010, this volume; Howard et al., 2010, this volume; Jones & Mewhort, 2007). Second, information from syntax often plays an important role in the processing of text and discourse, and by consequence including sources of information representative of syntax (e.g., word order, grammatical dependencies) improves model performance (Dennis, 2005; Dennis & Kintsch, 2008; Jones et al., 2006; Kintsch & Mangalath, 2010, this volume; Riordan & Jones, 2010, this volume). Third, comprehension comprises multiple levels of processing, including surface, textbase, and situation model levels of understanding (Kintsch, 1998), and thus including multiple sources of information may be necessary to account for the full scope of comprehension, memory, and learning phenomena (Graesser & McNamara, 2010, this volume; Kintsch & Mangalath, 2010, this volume). Indeed, many practical applications use latent representations extracted using statistical algorithms such as LSA in combination with information from the words and syntax (e.g., McNamara, Boonthum, Levinstein, & Millis, 2007). The need for multilevel models may not always be apparent because some levels of processing overwhelm the others, depending on the situation and the targeted dependent variable (McNamara & Magliano, 2009). For example, oftentimes prior knowledge overwhelms other factors to the extent that there are few discernable contributions from the text itself (other than word frequency). Likewise, when the targeted construct is the global text understanding, or a document, then the effects of syntax may be overwhelmed by the meanings of the words and the text as a whole. This is likely an explanation for why statistical models such as LSA that ignore syntax are oftentimes able to successfully extract meaning from text, despite ignoring fundamental levels of text meaning. Nonetheless, extracting the full meaning of text, including the full glory of its multidimensionality, will require using multiple, complementary approaches. The future likely lies more in the combination of techniques, rather than determining one winning model. Better, more complete, models of semantics are likely to emerge by measuring multiple levels of meaning.
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
15
References Barsalou, L. W. (1999). Perceptual symbol systems. Behavior and Brain Sciences, 22, 577–660. Barsalou, L. W. (2003). Abstraction in perceptual symbol systems. Philosophical Transactions of the Royal Society of London: Series B, 358, 1177–1187. Church, K. W., & Hanks, P. (1990). Word association norms, mutual information, and lexicography. Computational Linguistics, 16, 22–29. De Vega, M., Glenberg, A., & Graesser, A. C. (2008). Symbols and embodiment: Debates on meaning and cognition. Oxford, England: Oxford University Press. Dennis, S. (2005). A memory-based theory of verbal cognition. Cognitive Science, 29, 145–193. Dennis, S., & Kintsch, W. (2008). Text mapping and inference rule generation problems in text comprehension: Evaluating a memory-based account. In F. Schmalhofer & C. Perfetti (Eds.), Higher level language processes in the brain: Inference and comprehension processes (pp. 105–132). Mahwah, NJ: Erlbaum. Dunning, T. (1993). Accurate methods for the statistics of surprise and coincidence. Computational Linguistics, 19, 61–74. Fenson, L., Dale, P., Reznick, S., Bates, E., Thal, D., & Pethick, S. (1994). Variability in early communicative development. Monographs of the Society for Research in Child Development, 59, 1–185. Firth, J. R. (1957). A synopsis of linguistic theory, 1930–1955. In J. R. Firth (Ed.), Studies in linguistic analysis (pp. 1–32). Special volume of the Philological Society. Oxford, England: Blackwell. Glenberg, A. M. (1997). What memory is for. Behavioral and Brain Sciences, 20, 1–55. Graesser, A. C., & McNamara, D. S. (2010). Computational analyses of multilevel discourse comprehension. Topics in Cognitive Science, DOI: 10.1111/j.1756-8765.2010.01081.x Hinton, G., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527–1554. Hinton, G., & Salakhutdinov, R. (2010). Discovering binary codes for documents by learning deep generative models. Topics in Cognitive Science, 3, 74–91. DOI: 10.1111/j.1756-8765.2010.01109.x Hintzman, D. L. (1984). MINERVA2: A simulation model of human memory. Behavior, Research Methods, Instruments, and Computers, 16, 96–101. Howard, M. W., & Kahana, M. J. (2002). A distributed representation of temporal context. Journal of Mathematical Psychology, 46, 269–299. Howard, M., Shankar, K., & Jagadisan, U. (2010). Constructing semantic representations from a graduallychanging representation of temporal context. Topics in Cognitive Science, 3, 48–73. DOI: 10.1111/j.17568765.2010.01112.x Jones, M. N., Kintsch, W., & Mewhort, D. J. K. (2006). High-dimensional semantic space accounts of priming. Journal of Memory and Language, 55, 534–552. Jones, M. N., & Mewhort, D. J. K. (2007). Representing word meaning and order information in a composite holographic lexicon. Psychological Review, 114, 1–37. Kintsch, W. (1998). Comprehension: A paradigm for cognition. New York: Cambridge University Press. Kintsch, W. (2001). Predication. Cognitive Science, 25, 173–202. Kintsch, W. (2007). Meaning in context. In T. K. Landauer, D. S. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of latent semantic analysis (pp. 89–105). Mahwah, NJ: Erlbaum. Kintsch, W. (2008). Symbol systems and perceptual representations. In M. de Vega, A. Glenberg, & A. Graesser (Eds.), Symbols and embodiment: Debates on meaning and cognition (pp. 145–163). New York: Oxford University Press. Kintsch, W., & Mangalath, P. (2010). The construction of meaning. Topics in Cognitive Science, DOI: 10.1111/ j.1756-8765.2010.01107.x Kwantes, P. J. (2005). Using context to build semantics. Psychonomic Bulletin & Review, 12, 703–710. Landauer, T. K. (2007) LSA as a theory of meaning. In T. K. Landauer, D. S. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of latent semantic analysis (pp. 3–34). Mahwah, NJ: Erlbaum.
16
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
Landauer, T., & Dumais, S. (1997). A solution to Plato’s problem: The Latent Semantic Analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review, 104, 211– 240. Landauer, T. K., McNamara, D. S., Dennis, S., & Kintsch, W. (Eds.) (2007). Handbook of latent semantic analysis. Mahwah, NJ: Erlbaum. Louwerse, M. M. (2010). Symbol interdependency in symbolic and embodied cognition. Topics in Cognitive Science, DOI: 10.1111/j.1756-8765.2010.01106.x Lowe, W. (2001). Towards a theory of semantic space. In J. Moore & K. Stenning (Eds.), Proceedings of the 23rd Conference of the Cognitive Science Society (pp. 576–581). Mahwah, NJ: Erlbaum. Lund, K., & Burgess, C. (1996). Producing high-dimensional semantic spaces from lexical cooccurrence. Behavioral Research Methods, Instrumentation, and Computers, 28, 203–208. MacWhinney, B. (2000). The CHILDES Project: Tools for analyzing talk, 3rd ed. Mahwah, NJ: Erlbaum. McNamara, D. S., Boonthum, C., Levinstein, I. B., & Millis, K. (2007). Evaluating self-explanations in iSTART: Comparing word-based and LSA algorithms. In T. Landauer, D. S. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of Latent Semantic Analysis (pp. 227–241). Mahwah, NJ: Erlbaum. McNamara, D. S., Cai, Z., & Louwerse, M. M. (2007). Comparing latent and non-latent measures of cohesion. In T. Landauer, D. S. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of latent semantic analysis (pp. 379–400). Mahwah, NJ: Erlbaum. McNamara, D. S., & Kintsch, W. (1996). Learning from text: Effects of prior knowledge and text coherence. Discourse Processes, 22, 247–288. McNamara, D. S., Louwerse, M. M., McCarthy, P. M., & Graesser, A. C. (2010). Coh-Metrix: Capturing linguistic features of cohesion. Discourse Processes, 47, 292–330. McNamara, D. S., & Magliano, J. P. (2009). Towards a comprehensive model of comprehension. In B. Ross (Ed.), The psychology of learning and motivation (pp. 298–372). New York: Elsevier Science. McRae, K., Cree, G. S., Seidenberg, M. S., & McNorgan, C. (2005). Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods, 37, 547–559. Murdock, B. B. (1992). Serial organization in a distributed memory model. In A. F. Healy, S. M. Kosslyn, & R. M. Shiffrin (Eds.), From learning theory to connectionist theory: Essays in honor of William K. Estes (pp. 201–225). Hillsdale, NJ: Erlbaum. O’Reilly, T., & McNamara, D. S. (2007). Reversing the reverse cohesion effect: Good texts can be better for strategic, high-knowledge readers. Discourse Processes, 43, 121–152. O’Rourke, S. T., & Calvo, R. A. (2009). Visualizing paragraph closeness for academic writing support. In S. Murugesan (Ed.), Handbook of research on the web 2.0, 3.0, and X.0 technologies, business, and social applications (Ch. XLVII). Hershey, PA: IGI Global. Osgood, C. E., Suci, G., & Tannenbaum, P. (1957). The measurement of meaning. Urbana, IL: University of Illinois Press. Paivio, A., & Sadoski, M. (in press). Lexicons, contexts, events, and images: Commentary on Elman (2009) from the perspective of dual coding theory. Cognitive Science. Recchia, G. L., & Jones, M. N. (2009). More data trumps smarter algorithms: Comparing pointwise mutual information with latent semantic analysis. Behavior Research Methods, 41, 657–663. Riordan, B., & Jones, M. N. (2010). Redundancy in perceptual and linguistic experience: Comparing feature-based and distributional models of semantic representation. Topics in Cognitive Science, DOI: 10.1111/j.17568765.2010.01111.x Rohde, D. L. T., Gonnerman, L. M., & Plaut, D. C. (2005). An improved method of deriving word meaning from lexical co-occurrence. Unpublished manuscript. Available at: http://tedlab.mit.edu/~dr/. Accessed January 15, 2010. Salton, G., Wong, A., & Yang, C. S. (1975). A vector space model for automatic indexing. Communications of the ACM, 18, 613–620. Shaoul, C., & Westbury, C. (2006). Word frequency effects in high-dimensional co-occurrence models: A new approach. Behavior Research Methods, 38, 190–195.
D. S. McNamara ⁄ Topics in Cognitive Science 3 (2011)
17
Shapiro, A. M., & McNamara, D. S. (2000). The use of latent semantic analysis as a tool for the quantitative assessment of understanding and knowledge. Journal of Educational Computing Research, 22, 1–36. Smith, E. E., Shoben, E. J., & Rips, L. J. (1974). Structure and process in semantic memory: A featural model for semantic decisions. Psychological Review, 1, 214–241. Steyvers, M., Chemudugunta, C., & Smyth, P. (2010). Combining background knowledge and learned topics. Topics in Cognitive Science, 3, 18–47. DOI: 10.1111/j.1756-8765.2010.01097.x Steyvers, M., & Griffiths, T. L. (2007). Probabilistic topic models. In T. K. Landauer, D. S. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of latent semantic analysis (pp. 427–448). Mahwah, NJ: Erlbaum. Stone, B. P., Dennis, S. J., & Kwantes, P. J. (2010). Comparing Methods for Single Paragraph Similarity Analysis. Topics in Cognitive Science, 10.1111/j.1756-8765.2010.01108.x. Vinson, D. P., & Vigliocco, G. (2008). Semantic feature production norms for a large set of objects and events. Behavior Research Methods, 40, 183–190. Xu, W., Liu, X., & Gong, Y. (2003). Document clustering based on non-negative matrix factorization. In C. Clarke and G. Cormack (Eds.), Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval (pp. 267–273). New York: ACM Press.
Topics in Cognitive Science 3 (2011) 18–47 Copyright ! 2010 Cognitive Science Society, Inc. All rights reserved. ISSN: 1756-8757 print / 1756-8765 online DOI: 10.1111/j.1756-8765.2010.01097.x
Combining Background Knowledge and Learned Topics Mark Steyvers,a Padhraic Smyth,b Chaitanya Chemuduganta,b a
Department of Cognitive Sciences, University of California, Irvine Department of Computer Science, University of California, Irvine
b
Received 12 November 2008; received in revised form 19 June 2009; accepted 08 September 2009
Abstract Statistical topic models provide a general data-driven framework for automated discovery of high-level knowledge from large collections of text documents. Although topic models can potentially discover a broad range of themes in a data set, the interpretability of the learned topics is not always ideal. Human-defined concepts, however, tend to be semantically richer due to careful selection of words that define the concepts, but they may not span the themes in a data set exhaustively. In this study, we review a new probabilistic framework for combining a hierarchy of human-defined semantic concepts with a statistical topic model to seek the best of both worlds. Results indicate that this combination leads to systematic improvements in generalization performance as well as enabling new techniques for inferring and visualizing the content of a document. Keywords: Topic model; Concept-topic model; Hierarchical concept-topic model; Concepts; Background knowledge; Human-defined knowledge; Data-driven learning; Bayesian models
1. Introduction Many recent computational approaches to semantic cognition and statistical natural language processing operate on a purely data-driven basis. These models can extract useful information merely on the basis of statistical information contained in large text collections. From a machine-learning perspective, such models are attractive because they allow for a rapid analysis and understanding of new collections of text without significant human coding or annotation effort (e.g., Newman, Chemudugunta, Smyth, & Steyvers, 2006). From a cognitive science perspective, these models are attractive because they show that many findings related to semantic cognition can be explained by simple statistical learning processes. Correspondence should be sent to Mark Steyvers, Department of Cognitive Sciences, University of California, 3151 Social Sciences Plaza, Irvine, CA 92697-5100. E-mail:
[email protected]
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
19
Such learning processes can account for many empirical findings in areas such as language acquisition (Newport & Aslin, 2004; Newport, Hauser, Spaepen, & Aslin, 2004), multimodal language learning (Yu, Ballard, & Aslin, 2005), object perception (Fiser & Aslin, 2005), and eye movements (Najemnik & Geisler, 2005). In this research, we start with the assumption that much of our semantic representations can be acquired from experience in the form of large text collections, given appropriate statistical learning machinery. However, we also assume that building in some structure and prior knowledge might be required to create suitable representations. It has been shown recently how data-driven learning approaches can be combined with structured representations such as hierarchies, graphs, trees, and rules to create powerful new learning models (Chater & Manning, 2006; Kemp & Tenenbaum, 2008). In our research, we show how structured background knowledge and statistical learning processes can be combined. The combination of prior knowledge and novel information gained from experience raises two broad theoretical questions. First, how can prior knowledge facilitate the acquisition of new knowledge? We will investigate the circumstances under which prior knowledge can significantly help in learning semantic representations. Second, how can new knowledge be used to make changes in our background knowledge? We will demonstrate how corpus-driven learning processes can be used to identify gaps in an existing semantic representation. 1.1. Data-driven learning approaches There are a variety of unsupervised approaches for extracting semantic representations from large text collections that do not rely on background knowledge. In the context of a general ‘‘bag-of-words’’ framework, each document is represented by a vector that contains counts of the number of times each term (i.e., word or word combination) appears in the document. One general approach is to apply dimensionality reduction algorithms to represent the high-dimensional term vectors in a low-dimensional space. The dimensionality reduction can involve nonlinear projection methods such as self-organizing maps (Kohonen et al., 2000; Lagus, Honkela, Kaski, & Kohonen, 1999) or linear projection methods such as latent semantic analysis (LSA; Landauer & Dumais, 1997; Landauer, Foltz, & Laham, 1998). As a result of the dimensionality reduction, neighboring points in the semantic space often represent words or documents with similar contextual usages or meaning. These representations have been shown to model human knowledge in a variety of cognitive tasks (Landauer & Dumais, 1997) and educational assessment applications (Foltz, Gilliam, & Kendall, 2000). Other recent models in cognitive science have focused on alternative unsupervised methods to extract semantic representations at the sentence or document level (e.g., Dennis, 2004; Jones & Mewhort, 2007). In a probabilistic framework, a variety of clustering techniques have been developed that characterize each document by a single latent cluster or topic (e.g., Cutting, Karger, Pedersen, & Tukey, 1992; McCallum, Nigam, & Ungar, 2000; Popescul, Ungar, Flake, Lawrence, & Giles, 2000). Through unsupervised learning, these clusters can be learned automatically and give broad information about the content of documents. The drawback of the one-to-one mapping between documents and clusters is that documents that cover a
20
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
diverse set of topics can only be represented by a single cluster leading to problems in interpretation (e.g., Newman et al., 2006). A more flexible unsupervised framework, known as statistical topic modeling, allows each document to be represented by multiple topics (Blei, Ng, & Jordan, 2003; Buntine & Jakulin, 2004; Griffiths & Steyvers, 2004; Griffiths, Steyvers, & Tenenbaum, 2007; Hofmann, 1999; Steyvers & Griffiths, 2007). The basic concept underlying topic modeling is that each document is composed of a probability distribution over topics, where each topic represents a probability distribution over words. The topic–document and topic–word distributions are learned automatically from the data and provide information about the semantic themes covered in each document and the words associated with each semantic theme. The underlying statistical framework of topic modeling enables a variety of interesting extensions to be developed in a systematic manner, such as author-topic models (Steyvers, Smyth, Rosen-Zvi, & Griffiths, 2004), correlated topics (Blei & Lafferty, 2006), hierarchical topic models (Blei, Griffiths, Jordan, & Tenenbaum, 2003; Li, Blei, & McCallum, 2007; Teh, Jordan, Beal, & Blei, 2006), time-dependent topics (Wang, Blei, & Heckerman, 2008) and models that combine topics and syntax (Griffiths, Steyvers, Blei, & Tenenbaum, 2005), as well as image features and text (Blei & Jordan, 2003). Topic models have also been useful as cognitive models to explain human associations, gist extraction, and memory errors (Griffiths et al., 2007). One of the drawbacks of a purely data-driven learning process, such as topic modeling, is that the resulting representations can require some effort to interpret. As an illustrative example of the information learned by topic models, Fig. 1 (top row) shows five examples of topics that were derived from the TASA corpus, a collection of over 37,000 text passages from educational materials (e.g., language and arts, social studies, health, sciences) collected by Touchstone Applied Science Associates (TASA; see Landauer et al., 1998). The figure shows the 15 words that have the highest probability under each topic. Each number corresponds to the probability that a word is generated conditioned on a learned topic. It is often easier to interpret topics relative to other representations such as document clusters in cluster models or latent dimensions in latent semantic analysis (e.g., Newman et al., 2006). The words in the topics in the top row of Fig. 1 appear to relate to colors, gases, and the atmosphere, American presidents, European countries in World War II, and Japan and World War II, respectively. However, because topics are defined by probability distributions over words and have no simple names or definitions that can explain their content, an interpretation of the content of a topic often requires a subjective analysis of the connections between the high-probability words in a topic. This subjective process can lead to different outcomes depending on which individual is doing the analysis. Some progress has been made to automate the labeling of topics (Mei, Shen, & Zhai, 2007), but it remains to be seen how easily accessible such statistical representations are to human users. Even with these techniques, topic interpretability remains an issue when faced with small noisy data sets. Data-driven learning models require large amounts of data in order to obtain accurate and useful representations, and such data might not always be available. In addition, because the model tunes itself to the dominant semantic themes in a corpus, it might not accurately represent outlier documents. Although such models might be able to tell that
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011) word red blue green yellow white color bright colors orange brown pink look black purple cross
prob. 0.202 0.099 0.096 0.073 0.048 0.048 0.030 0.029 0.027 0.027 0.017 0.017 0.016 0.015 0.011
!"**"$(!"/"&, red green blue brown yellow orange pink purple reddish yellowish greenish brownish bluish redness pinkish
word oxygen carbon dioxide air ramona gas nitrogen gases atmosphere hydrogen water respiraion process beezus breathe
prob. 0.136 0.097 0.050 0.046 0.037 0.036 0.030 0.026 0.020 0.020 0.016 0.014 0.014 0.012 0.011
!-+*.!)/(+/+*+$%, oxygen gold iron lead carbon hydrogen copper mercury nitrogen sodium tin aluminum sulfur calcium uranium
word president roosevelt congress johnson office wilson nixon reagan kennedy carter presidents administration presidential white budget
prob. 0.129 0.032 0.030 0.026 0.021 0.021 0.020 0.018 0.018 0.017 0.012 0.012 0.011 0.011 0.010
/+)0+&,("1($)%."$)/()$0( &+2."$)/(2"3+&$*+$%, president governor presidential ruler presidency dynasty sovereign chancellor premier governorship regent dynastic gubernatorial vp mp
word france french europe germany german countries britain italy western european british war germans country nations
prob. 0.071 0.069 0.051 0.043 0.041 0.030 0.024 0.019 0.019 0.019 0.016 0.015 0.013 0.012 0.012
!"#$%&'($)*+, america england france china mexico india spain canada japan germany egypt russia italy alaska australia
21 word war japanese japan II american peace civil end wars treaty fought fighting military ended forces
prob. 0.201 0.035 0.035 0.035 0.030 0.029 0.019 0.016 0.014 0.013 0.012 0.012 0.012 0.011 0.011
4)& war peace pacific campaign peaceful hostile warfare wartime peacefully tactics concord crusade warlike warring peacetime
Fig. 1. Examples of five topics (out of 300) extracted from the TASA corpus (top row). The closest corresponding concepts from the CALD knowledge base (bottom row). The most likely words in each topic along with the matching word in the concept are highlighted. The column ‘‘prob’’ for each topic refers to the probability of each word in that topic.
a document falls outside the scope of representative semantic themes, it is difficult to identify the content that is covered by such documents. Therefore, in the absence of a large repository of relevant background documents to build topic models, it can be difficult to get interpretable and effective representations. 1.2. Human-defined semantic representations An entirely different approach to constructing semantic representations is to rely on human knowledge and judgment. Considerable effort has gone into developing humandefined knowledge databases that characterize commonsense and lexical knowledge in humans. Such knowledge is created by trained experts in projects such as Cyc (Lenat & Guha, 1989; Panton et al., 2006), WordNet (Fellbaum, 1998; Miller, 1990), and Roget’s thesaurus (Roget, 1911) or by untrained volunteers in projects such as ConceptNet (Havasi, Speer, & Alonso, 2007). Similarly, in cognitive science, many behavioral experiments have elicited detailed knowledge from many college students about semantic associations (Nelson, McEvoy, & Schreiber, 1998) and concepts and features (McRae, Cree, Seidenberg, & McNorgan, 2005; Ruts et al., 2004). Such human-defined representations can serve as proxies for mental representations.
22
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
To highlight the difference between learned topics and human-defined knowledge, we will show some examples of human-defined concepts that were created by lexicographers as part of the Cambridge Advanced Learner’s Dictionary (CALD). In contrast to other taxonomies such as WordNet (Fellbaum, 1998; Miller, 1995), CALD groups words primarily according to semantic topics with the topics hierarchically organized. In addition, CALD provides names for each concept that are helpful for visualization purposes. CALD consists of 2,183 semantic concepts with each concept consisting of a set of words and a name that describes the concept. Fig. 1 (bottom row) shows an example of CALD concepts that resemble the meaning of the learned topics in the top row of Fig. 1. Each concept is illustrated with the name of the concept (shown on top) and a subset of 15 words that are part of the concept. Apart from alphabetical order, there is no natural way to order words within a concept. To better summarize the sometimes large number of words in each concept, we ordered the words by word frequency in the TASA corpus. A comparison between learned topics and human-defined CALD concepts in Fig. 1 reveals some interesting differences: The words in each topic are associated with probabilities that indicate how likely each word is to be found in a context of that topic, which is quite useful to get a fine-grained indication about the relevance of a word to a topic. In contrast, CALD concepts do not provide any information about the prominence, frequency, or representativeness of the words in each concept—either a word is present or it is absent in a concept. A clear advantage of concepts is that they are often more interpretable than learned topics by virtue of having a name (or small number of words) that describes the concept, providing concepts more precise coverage compared to topics. For example, the first topic on colors includes words that are not color words (e.g., bright and look), whereas a color concept will restrict itself to just color words. Concepts can also have broader coverage relative to topics because all words are considered as candidates and not just words occurring in a particular corpus. For example, the concept on chemical elements lists all chemical elements (as known by the lexicographer), whereas a learned topic might focus more on the high-frequency chemical elements. Also, a learned topic could omit certain elements altogether because they did not occur in the corpus. Although there are many advantages of human-defined knowledge databases, a major drawback is that they require extensive manual involvement and are time consuming to build and update given new emerging information. For some applications such as analyzing and summarizing text collections, no suitable knowledge database might even be available that has a suitable coverage of the domain. In contrast, data-driven topics can be tuned to themes in a corpus and can easily discover and summarize the dominant semantic themes for a wide variety of text collections. 1.3. Combining human-defined knowledge and data-driven learning approaches Clearly, representations based on either a purely data-driven approach or human-defined knowledge have limitations. In this article, we will review some of our recent work that combines human-defined concepts with statistical data-driven approaches to learning
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
23
semantic representations (Chemudugunta, Holloway, Smyth, & Steyvers, 2008; Chemudugunta, Smyth, & Steyvers, 2008a, 2008b). The objective is to combine these approaches with the goal of taking advantage of the best features of both approaches. When there are few documents to learn from, these hybrid models are primarily driven by humandefined concepts. When trained on large document collections, data-driven topics can fill in gaps in the human-defined concepts. From a machine-learning perspective, automatically identifying such gaps can lead to a variety of useful applications where we update existing representations without requiring extensive human effort in discovering new emerging themes. From a cognitive science perspective, the hybrid model leads to novel ways of thinking about semantic representations. Instead of assuming that such representations are purely the result of data-driven learning processes, they might be a combination of preexisting knowledge and new knowledge extracted from a collection of text. We make no theoretical claims about the source of the prior knowledge. Although it is likely that such prior knowledge is itself acquired by experience, we do not attempt to explain how this is learned from experience. The plan for the rest of the study is as follows. In Section 2, we review the basic principles of topic models and then describe the concept–topic model that combines concepts and topics into a single probabilistic model. We also describe the hierarchical concept–topic model which takes advantage of known hierarchical structure among concepts. Section 3 describes the text corpus and concept data set that we used to conduct our experiments. Section 4 describes a series of experiments that evaluate the predictive performance of a number of different models, showing for example that prior knowledge of concept words and concept relations can lead to better topic-based language models. In Section 5, we discuss a number of examples that illustrate how documents can be tagged at the word level with human-defined concepts. In Section 6, we discuss the type of information that is learned by topics but not captured by concepts. In Section 7, we show how the concept– topic model can automatically find appropriate concepts for novel words. In the final sections, we conclude the study with a brief discussion of related research, future directions, and final comments.
2. Concept–topic models A clear advantage of an unsupervised learning approach such as topic modeling is that the model can be tuned to the themes of the particular document collection it is trained on. In addition, the probabilistic model that underlies the topic model allows one to automatically tag each word in a document with the topic most likely to have generated it. On the contrary, human-defined concepts such as the CALD knowledge base have much broader coverage of English words and include useful names of concepts that clarify the set of words that could be included in the concept, and aid in interpretability. In this section, we will describe concept–topic and hierarchical concept–topic models that combine data-driven topics and human-defined concepts (Chemudugunta et al., 2008b, 2008c). We begin with a brief review of topic models.
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
24
2.1. Topic model The topic model (or latent Dirichlet allocation [LDA] model; Blei et al., 2003) is a statistical learning technique for extracting a set of topics that describe a collection of documents. A topic t is represented by a multinomial distribution over the V unique word types in the ðtÞ ðtÞ ðtÞ corpus, /ðtÞ ¼ ½u1 ; :::; uV %, where uw ¼ pðwjtÞ and 1 £ w £ V. Therefore, a topic can be viewed as a V-sided die and generating n word tokens from a topic is akin to throwing the topic-specific die n times. There are a total of T topics and a document d is represented as a ðdÞ ðdÞ ðdÞ multinomial distribution over those T topics, hðdÞ ¼ ½h1 ; :::; hT %, where ht ¼ pðtjdÞ and 1 £ t £ T. The variables u and h indicate which words are important for which topic and which topics are important for a particular document, respectively. Generating a word token for a document d involves first selecting a topic t from the document–topic distribution h(d) and then selecting a word from the corresponding topic distribution u (t). This process is repeated for each word token in the document. Let z be the random variable that represents the topic indices sampled from h(d). We write p(zi = t|d) as the probability that the tth topic was sampled for the ith word token (in document d) and p(wi|zi = t) as the probability of word wi under topic t. The model specifies the following conditional probability of the ith word token in a document: T X pðwi jzi ¼ tÞpðzi ¼ tjdÞ ð1Þ pðwi jdÞ ¼ t¼1
In the LDA model, Dirichlet priors are placed on both u and h, to smooth the word–topic and topic–document distributions (for a description of Dirichlet priors, see Steyvers & Griffiths, 2007; Gelman, Carlin, Stern, & Rubin, 2003). In many applications, a symmetric Dirichlet density with single hyperparameters a and b are used for h and u, respectively. For all the topic models in this research, we will use a symmetric Dirichlet prior for u using a single hyperparameter b. For the topic–document distributions h, we will use an asymmetric Dirichlet prior h, with a vector a containing hyperparameter values for every topic (and concept for concept–topic models). An asymmetric prior is useful when some concepts (or topics) are expressed in many or just a few documents across the collection. With an asymmetric prior, more skewed marginal distributions over h can be obtained to express rare or frequent topics (or concepts). The sequential process of first picking a topic from a topic distribution, and then picking a word token from a distribution over word types associated with that topic can be formalized as follows: 1. For each topic t 2 f1; :::; Tg, select a word distribution /ðtÞ & DirichletðbÞ 2. For each document d 2 f1; :::; Dg (a) Select a distribution over topics hðdÞ & DirichletðaÞ (b) For each word position i in document ! "d (i) Select a topic zi & Discrete hðdÞ ! " (ii) Generate a word token from topic zi, wi & Discrete /ðzi Þ
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
25
This generative process can be summarized by the graphical model shown in Fig. 2A. In the graphical notation, shaded and unshaded variables indicate observed and latent (i.e., unobserved) variables, respectively, and the arrows indicate the conditional dependencies between variables. The plates (the boxes in the figure) refer to repetitions of sampling steps with the variable in the right corner referring to the number of samples. For example, the inner plate over z and w illustrates the repeated sampling of topics and words until Nd word tokens have been generated for document d. The plate surrounding h illustrates the sampling of a distribution over topics for each document d for a total of D documents. The plate surrounding u illustrates the repeated sampling of distributions over word types for each topic until T topics have been generated. Given the words in a corpus, the inference problem involves estimating the word–topic distributions u, the topic–document distributions h, and the topic assignments z of individual words to topics. These distributions can be learned in a completely unsupervised
!"#
!$#
Į
Į #
ș β
!
ij
"
!%#
βφ
ij
βφ
!
ij
"
%$'
βψ ψ
%$'
Į
γ
τ !)"
ș
ȟ
ȗ !)"
!
(
Ȝ
&
# &
βψ ψ
" '
#
ș
%$&
Fig. 2. Graphical models for the topic model (A), the concept–topic model (B), and the hierarchical concept– topic model (C).
26
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
manner without any prior knowledge about topics or which topics are covered by which documents. One efficient technique for obtaining estimates of these distributions is through collapsed Gibbs sampling (Griffiths & Steyvers, 2004). Steyvers and Griffiths (2007) present a tutorial introduction to topic models that discusses collapsed Gibbs sampling. The main idea of collapsed Gibbs sampling is that inference is performed only on z, the assignments of word tokens to topics. The remaining latent variables h and u are integrated out (‘‘collapsed’’). Words are initially assigned randomly to topics and the algorithm then iterates through each word in the corpus and samples a topic assignment given the topic assignments of all other words in the corpus. This process is repeated until a steady state is reached and the topic assignments are then used to estimate the word–topic and topic– document distributions. The vector a that contains the hyperparameter values for every topic (and concept for concept–topic models, see below) is updated using a process involving fixed-point update equations (Minka, 2000; Wallach, 2006). See Appendix A of Chemudugunta et al. (2008b) for more details. To summarize, the topic model provides several pieces of information that are useful for understanding documents. The topic–document distributions indicate the important topics in each document. The word–topic distributions indicate which words are important for which topic (e.g., the top row of Fig. 1 shows some example word–topic distributions estimated for the TASA corpus). Finally, the probabilistic assignments zi of word tokens to topics are useful for tagging purposes, providing information about the role each word is playing in a specific document context and helping to disambiguate multiple meanings of a word (e.g., Griffiths et al., 2007). 2.2. Concept–topic model The concept–topic model is a simple extension to the topic model where we add C concepts to the T topics of the topic model resulting in an effective set of T + C word distributions for each document. We assume that each of the C concepts (such as the CALD concepts in Fig. 1) are represented as a set of words. Therefore, these human-defined concepts only give us a membership function over words—either a word is a member of the concept or it is not. One straightforward way to incorporate concepts into the topic modeling framework is to convert them to probability distributions over their associated word sets. In the concept–topic model, we will treat each concept c as a multinomial distribution ðcÞ ðcÞ wðcÞ ¼ ½w1 ; :::; wV %, where wðcÞ w ¼ pðwjcÞ and 1 £ w £ V. Importantly, each word type that = c. Of course, is not part of the concept will have zero probability, that is, wðcÞ w = 0 for w 2 there are no direct observations available about the probabilities of word types within a concept, but we can use a model similar to the topic model to estimate these probabilities from corpus data. Therefore, the concept–topic model is simply an extension of the topic model where we have a number of learned topics as well as constrained topics where nonzero probability can only be given to words in human-defined concepts. In the concept–topic model, the conditional probability of the ith word token wi in a document d is
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
pðwi jdÞ ¼
T X t¼1
pðwi jzi ¼ tÞpðzi ¼ tjdÞþ
C þT X
t¼Tþ1
pðwi jzi ¼ tÞpðzi ¼ tjdÞ;
27
ð2Þ
where the indices 1 £ t £ T refer to all topics and indices T + 1 £ t £ T + C refer to all concepts. In this generative process, an index zi is sampled from the distribution over topics and concepts for the particular document. If zi £ T, a word token is sampled from topic zi, and if T + 1 £ zi £ T + C, a word token is sampled from concept zi ) T among word types associated with the concept. The topic model can be viewed as a special case of the concept–topic model when there are no concepts present, that is, when C = 0. At the other extreme of this model where T = 0, the model relies entirely on predefined concepts. ðtÞ ðtÞ ðtÞ To specify the complete generative model, let /ðtÞ ¼ ½u1 ; :::; uV &, where uw ¼ pðwjtÞ and 1 £ w £ V, refer to the multinomial distribution over word types for topic t when ðcÞ ðcÞ 1 £ t £ T, and let WðcÞ ¼ ½w1 ; :::; wV &, where wðcÞ w ¼ pðwjcÞ and 1 £ w £ V refer to the multinomial distribution over word types for concept c = t–T when T + 1 £ t £ T + C. As with the topic model, we place Dirichlet priors on the multinomial variables h, u, and w, with corresponding hyperparameters a, b/, and bw. The complete generative process can be described as follows:
! " 1. For each topic t 2 f1; :::; Tg, select a word distribution /ðtÞ ' Dirichlet b/! " 2. For each concept c 2 f1; :::; Cg, select a word distribution wðcÞ ' Dirichlet bw 3. For each document d 2 f1; :::; Dg (a) Select a distribution over topics and concepts hðdÞ ' DirichletðaÞ (b) For each word position i in document d# $ (i) Select a component zi ' Discrete hðdÞ # $ (ii) If zi £ T, generate a word token from topic zi, wi ' Discrete /ðzi Þ# ; other$ wise, generate a word token from concept ci = zi - T, wi ' Discrete wðci Þ
Note that in Step 2, the sampling of words for a concept is constrained to only the words that are members of the human-defined concept. Fig. 2B shows the corresponding graphical model. All the latent variables in the model can be inferred through collapsed Gibbs sampling in a similar manner to the topic model (see Chemudugunta et al., 2008b for details). We note that even though we are partially relying on humans to define the word–concept memberships, we still apply purely unsupervised algorithms to estimate the latent variables in the model. This is in contrast to a supervised learning approach where the human-defined knowledge is used as a target for prediction. Here, the human-defined knowledge is only used as a constraint on the probability distributions that can be learned for each concept. We also note that the concept–topic model is not the only way to incorporate semantic concepts. For example, we could use the concept–word associations to build informative priors for the topic model and then allow the inference algorithm to learn word probabilities for all words (for each concept), given the prior and the data. We chose the restricted vocabulary approach to exploit the sparsity in the concept–word associations (topics are distributions over all the words in the vocabulary but concepts are restricted to just their sets of associated words, which are much smaller than the full vocabulary). This sparsity at the
28
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
word level allows us to easily perform inference with tens of thousands of concepts on large document collections. A general motivation for the concept–topic approach is that there might be topics present in a corpus that are not represented in the concept set (but that can be learned). Similarly, there may be concepts that are either missing from the text corpus or are rare enough that they are not found in the data-driven topics of the topic model. The marriage of concepts and topics provides a simple way to augment concepts with topics and has the flexibility to mix and match topics and concepts to describe a document. 2.3. Hierarchical concept–topic model Although the concept–topic model provides a simple way to combine concepts and topics, it does not take into account any hierarchical structure the concepts might have. For example, CALD concepts are arranged in a hierarchy that starts with the concept everything which splits into 17 concepts at the second level (e.g., science, society, general ⁄ abstract, communication). The hierarchy has up to seven levels with each level specifying more specific concepts. In this section, we describe a hierarchical concept–topic model that incorporates hierarchical structure of a concept set. Similar to the concept–topic model described in the previous section, there are T topics and C concepts. However, as opposed to the flat organization of the concepts in the concept–topic model, we now utilize the hierarchical organization of concepts when sampling words from concepts. Before we formally describe the model, we illustrate the basic idea in Fig. 3. Each topic and concept is associated with a ‘‘bag of words’’ that represents a multinomial distribution over word types. In the generative process, word tokens can be generated from the concept part of the model by sampling a path from the root of the concept tree to some distribution over word types associated with the concept (left box in Fig. 3). Alternatively, word tokens can be generated through the topic part of the model (right box). The dashed and dotted lines show examples of two word tokens sampled through the hierarchical concept part of the model and the topic part of the model, respectively. For the first word token, the option ‘‘topic’’ is sampled at the root node, Topic 1 is then sampled, and then a word token is sampled from the multinomial over words associated with Topic 1. For the second word token, the option ‘‘concept’’ is sampled at the root node, then the option science is sampled as a child of the concept everything, the word distribution for science is then selected, and a word from this distribution is sampled. Each transition in the hierarchical part of the model has an associated probability and the transition probabilities are document dependent—some paths are more likely in context of some documents. For example, in physics and chemistry documents, one might expect all transitions toward the science concept to be elevated but differentiated between the transitions toward the physics and chemistry concepts. To preview what information is learned by the model, we need to distinguish between variables learned at the word, document, and corpus levels. At the word level, the model learns the assignments of topics or concepts to word tokens. These assignments can be directly used for tagging purposes and word–sense disambiguation. At the document level,
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
29
+/012345/654/3718
!"!#$%&'()
9
%,-'+
:
*,+'!%$
*+'!(+!
-&$*'+*
%,-'+
+&!.'*%#$
Fig. 3. An illustration of the hierarchical concept–topic model.
the model learns both topic probabilities and concept–transition probabilities in the concept tree. The latter information is useful because it allows a hierarchical representation of document content. At the document level, the model also learns the switch probability that a word is generated through the topic or concept route. The adaptive nature of the switch probability allows the model to flexibly adapt to different documents. Documents that contain material that has poor concept coverage will have a high probability of switching to the topic route. At the corpus level, the model learns the probabilities of the word–topic and word–concept distributions. The word–topic distributions are useful to learn which semantic themes beyond those covered in the concepts are needed to explain the content of the whole document collections. The word–concept distributions are useful to learn which words are important for each concept. Finally, at the corpus level, the model also learns the hyperparameters for each transition in the concept tree. The learned hyperparameters allow the model to make certain paths more prominent across all documents. For example, if a document collection includes many documents on science, the path toward the science concept could be made more likely (a priori). Our approach is related to the hierarchical pachinko allocation model 2 (HPAM 2) as described by Mimno, Li, and McCallum (2007). In the HPAM 2 model, topics are arranged
30
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
in a three-level hierarchy with root, super-topics, and subtopics at Levels 1, 2, and 3, respectively, and words are generated by traversing the topic hierarchy and exiting at a specific level and node. In our model, we use a similar mechanism for word generation via the concept route. There is additional machinery in our model to incorporate the data-driven topics (in addition to the hierarchy of concepts) and a switching mechanism to choose the word generation process via the concept route or the topic route. To give a formal description of model, for each document d, we introduce a ‘‘switch’’ distribution p(x|d) that determines if a word should be generated via the topic route or the concept route. Every word token wi in the corpus is associated with a binary switch variable xi. If xi = 0, the previously described standard topic mechanism is used to generate the word. That is, we first select a topic t from a document-specific mixture of topics h(d) and generate a word token from the word distribution associated with topic t. If xi = 1, we generate the word token from one of the C concepts in the concept tree. To do that, we associate with each concept node c in the concept tree a document-specific multinomial distribution with dimensionality equal to Nc + 1, where Nc is the number of children of the concept node c. This distribution allows us to traverse the concept tree and exit at any of the C nodes in the tree—given that we are at a concept node c, there are Nc child concepts to choose from and an additional option to choose an ‘‘exit’’ child to exit the concept tree at concept node c. We start our walk through the concept tree at the root node and select a child node from one of its children. We repeat this process until we reach an exit node and a word token is generated from the parent of the exit node. Note that for a concept tree with C nodes, there are exactly C distinct ways to select a path and exit the tree, as there is only one parent for each concept node, and thus, one path to each of the C concepts. In the hierarchical concept–topic model, a document is represented as a weighted combination of mixtures of T topics and C paths through the concept tree and the conditional probability of the ith word token in document d is given by pðwi jdÞ ¼ pðxi ¼ 0jdÞ þ pðxi ¼ 1jdÞ
T X t¼1
TþC X
c¼Tþ1
pðwi jzi ¼ tÞpðzi ¼ tjdÞ
pðwi jzi ¼ cÞ½pðexitjc; dÞpðcjparentðcÞ; dÞ% % %pðrootjdÞ'
ð3Þ
The sequential process to generate a document collection with D documents under the hierarchical concept–topic model is as follows: ! " 1. For each topic t 2 f1; :::; Tg, select a word distribution /ðtÞ ( Dirichlet b/! " 2. For each concept c 2 f1; :::; Cg, select a word distribution wðcÞ ( Dirichlet bw 3. For each document d 2 f1; :::; Dg (a) Select a switch distribution nðdÞ ( BetaðcÞ (b) Select a distribution over topics hðdÞ ( DirichletðaÞ (c) For each concept c 2 f1; :::; Cg ! " i. Select a distribution over children of c, fðcdÞ ( Dirichlet sðcÞ
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
31
(d) For each word position i in document ! d" (i) Select a binary switch xi ! Bernoulli nðdÞ (ii) If xi = 0 ! " (A) Select a topic zi ! Discrete hðdÞ ! " (B) Generate a word from topic zi, wi ! Discrete /ðzi Þ (iii) Otherwise, create a path starting at the root concept node, k1 ¼ 1 (A) Select a child node kj ; kjþ1 ! Discreteðfðkj dÞ Þ, and increment j. Repeat until kjþ1 is an exit node ! " (B) Generate a word from concept ci ¼ kj , wi ! Discrete wðci Þ . Set zi = ci + T where u(t), w(c), bu, and bw are analogous to the corresponding symbols in the concept–topic model described in the previous section. The variable n(d), where n(d) = p(x|d), represents the switch distribution and h(d), where h(d) = p(t|d) represents the distribution over topics for document d. The variable f(cd)represents the multinomial distribution over children of concept node c for document d (this has dimensionality Nc + 1 to account for the additional ‘‘exit’’ child). The hyperparameters c, a, and s(c) are the parameters of the priors on n(d), h(d), and f(cd), respectively. Note that a, as in the previous topic and concept–topic models, is a vector with hyperparameter values for each topic. Similarly, s(c)is a vector of hyperparameters values, to allow for different a priori probabilities of traversing the concept-tree. This allows the model to tune itself to different corpora and make it more likely to sample a path toward the science concept in a corpus of scientific documents. Fig. 2C shows the corresponding graphical model. The generative process above is quite flexible and can handle any directed-acyclic concept graph (for any nontree, there would be more than one way of reaching each concept, leading to increased complexity in the inference process). The model cannot, however, handle cycles in the concept structure as the walk of the concept graph starting at the root node is not guaranteed to terminate at an exit node. In the hierarchical concept–topic model, the only observed information is the set of words in each document, the word–concept memberships, and the tree structure of the concepts. All remaining variables are latent and are inferred through a collapsed Gibbs sampling procedure. Details about this procedure are described by Chemudugunta et al. (2008b).
3. Text and concept data The experiments for all our simulations are based on the TASA corpus (Landauer & Dumais, 1997) consisting of D = 37,651 documents with passages excerpted from educational texts used in curricula from the first year of school to the first year of college. The documents are divided into nine different educational genres. We focus here on a subset of TASA documents classified as science, consisting of D = 5,356 documents. As mentioned previously, CALD consists of 2,183 semantic concepts. CALD groups words primarily according to semantic concepts with the concepts hierarchically organized. The hierarchy starts with the concept everything which splits into 17 concepts at the second level (e.g.,
32
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
science, society, general ⁄ abstract, communication). The hierarchy has up to seven levels, with each interior node splitting into a median of seven children nodes. The concepts vary in the number of the words with a median of 54 word types and a maximum of 3,074. Each word can be a member of multiple concepts, especially if the word has multiple senses. We created two vocabularies. One is a W = 21,072 word vocabulary based on the intersection between the vocabularies of TASA and CALD. We also created a vocabulary of W = 142,010 words based on the union of TASA and CALD vocabularies. For both vocabularies, all stop words and infrequent words were removed.
4. Tagging documents One application of concept models is to tag unlabeled documents with human-defined concepts. The tagging process involves assigning likely concepts to each word in a document, depending on the context of the document. The document content can then be summarized by the probability distribution over concepts that reveal the dominant semantic themes. Because the concept models assign concepts at the word level, the results can be aggregated in many ways, allowing for document summaries at multiple levels of granularity. For example, tagging can be performed on snippets of text, individual sections of a document, whole documents, or even collections of documents. For all of our tagging examples, we used the intersection vocabulary (the results are qualitatively similar using the union vocabulary). 4.1. Tagging with the concept–topic model As an illustration of how the model can be used to quickly summarize a document, Fig. 4 shows the CALD concept assignments to individual words in a TASA document. We used the concept–topic model with concepts only (T = 0). The four most likely concepts are listed for this document. For each concept, the estimated probability distribution over words is shown next to the concept (note that these estimates are over the whole corpus and are not document specific). For example, for the concept of chemical elements, the word oxygen is more likely than the word chlorine. The probability of words in concepts is not just influenced by number of tokens across the whole corpus but also by the number of concepts that contain the word type and the relative probability between concepts in each document. The model has estimated that in the conceptual context of chemical elements, the word oxygen is more likely than the word chlorine. This conditional salience is useful for evaluating the relative importance of words to specific concepts, going beyond the logical set definitions provided by the human lexicographers who developed the concepts. In the document, words assigned to the four most likely concepts are tagged with letters a–d (and color coded if viewing in color). The words assigned to any other concept are tagged with ‘‘o’’ and words outside the vocabulary are not tagged. In the concept–topic model, the distributions over concepts within a document are highly skewed such that the probability mass is distributed over only a small number of concepts. In the example
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011) !"# "
$%!&"' ./01.2
C
./0;2@
+
./.?@?
<
./.?2B
)
./@.?0
33
()*+,-! $3456(5
$%#&!' ,7,+!8)*9 %./21:1' ,7,+!8)* %./0;:1' 8"<="Ɵ)* %./.>??' -8)!)*9 %./.12;' =)*9 %./.@;2' 8"<=)"+ƟA, %./.B1:' -8)!)* %./.2>2' (3DE6(FG DGDEDHI5 )JK#,* %./;.2;' LK<8)#,* %./0>10' +"8C)* %./.10.' *=!8)#,* %./.:1.' 9)<=MN %./.@:2' 9M7OM8 %./.B0B' +L7)8=*, %./.;?>' FIPE5Q EPGD(RGD5Q FHS "!)N9 %./;..?' N)7,+M7,9 %./2?:@' "!)N %./22?0' N)7,+M7, %./0.>@' =)*9 %./.2:2' =9)!)-,9 %./.0;@' =)* %./.0.@' =9)!)-, %./..:?' 5RTUFIPE6( $FVI6(GD5 DGD(IV6(6I4 FHS DGD(IVPH6(5 ,7,+!8=+=!K %./2B:B' ,7,+!8=+ %./22?0' ,7,+!8=+"7 %./0.>2' +M88,*! %./.>>2' !)W %./.BB>' N"#*,Ɵ9N %./.;2?' PI3DV
IL, LK<8)#,*C =)*9" =NN,<="!,7K) "Ʃ"+L) !L,N9,7A,9 !) W"!,8) N)7,+M7,9+ !) O)8N) +)NC=*"Ɵ)*9) +"77,<) LK<8)*=MN =)*9"/ IL, +L7)8=*,C =)*9" "79) "99)+="!,) W=!L W"!,8) N)7,+M7,9+ "*< C,+)N, LK<8"!, P8<=*"8=7K)Q !L, -)9=ƟA,) LK<8)*=MN =)*9" "*< !L, *,#"ƟA,) +L7)8=*,C =)*9" W"*<,8) "C)M! O8,,7K) =* !L, 9)7MƟ)*) =* "77 <=8,+Ɵ)*9)/ 3)W,A,8Q WL,* !L, ,7,+!8)7KƟ+ +,77) =9 +)**,+!,<) !) " C"Ʃ,8K)Q !L, "*)<,< C,+)N,9 -)9=ƟA,7K) +L"8#,<" "*< !L, +"!L)<,< C,+)N,9 *,#"ƟA,7K) +L"8#,<"/ IL, -)9=ƟA,7K) +L"8#,<" LK<8)*=MN =)*9" "8, !L,* "Ʃ8"+!,<) !)W"8< !L, +"!L)<,< "*< !L, *,#"ƟA,7K) +L"8#,<" +L7)8=*,C =)*9" "8, "Ʃ8"+!,< ) !)W"8< !L, "*)<, IL, !)W< )O +M88,*!< =*9=<,) !L, +,77) !L,8,O)8, +)*9=9!9 )O -)9=ƟA,) LK<8)*=MN =)*9" !)W=*#< =* )*, <=8,+Ɵ)*) "*< *,#"ƟA,) +L7)8=*,C =)*9" !)W=*#< =* !L, )--)9=!,) <=8,+Ɵ)*)/ XL,* !L, LK<8)*=MN =)*9" 8,"+L) !L, +"!L)<,
Fig. 4. Illustrative example of tagging a document excerpt using the concept–topic model with concepts from CALD.
document, the four most likely concepts cover about 50% of all words in the document. Fig. 4 illustrates that the model correctly disambiguates between words that have several conceptual interpretations. For example, the word charged has many different meanings and appears in 20 CALD concepts. In the example document, this word is assigned to the physics concept, which is a reasonable interpretation in this document context (the word charged does not appear in the list of words associated with the concept physics because its probability falls below the threshold for visualization). Similarly, the ambiguous words current and flow are correctly assigned to the electricity concept. 4.2. Tagging with the hierarchical concept–topic model One of the advantages of the hierarchical concept–topic model is that the hierarchical relations between concepts can be used to enhance the visualization of documents. Fig. 5 shows the result of inferring the hierarchical concept mixture for an individual TASA document using CALD concept sets. For the hierarchy visualization, we selected the seven concepts with the highest probability and included all ancestors of these concepts when visualizing the tree (we selected seven concepts to tradeoff informativeness and complexity of the display). The ancestors that were not part of the top seven concepts are visualized with dashed ovals. The CALD subtree highlights the specific semantic themes of birth, breathing, and stopping breathing along with the more general themes of science and medicine. This illustration shows that the model is able to give interpretable results for an individual document at multiple levels of granularity. At a higher level of granularity, the hierarchical concept–topic model can also summarize sets of documents. Across documents, the model learns the hyperparameters associated with
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
34
B?C !"# $%&'()')* $#+,%- %. -#/#*%$0#(' *)&'& .+%0 1,+'" 2(',* -#)'" )(- 3)( 1# -,/,-#- ,('% ) (#%()')* $#+,%-4 ,(.)(354 3",*-"%%-4 )-%*#&3#(3#4 )-2*'"%%-4 )(- (#&3#(3#6 !"# (#%()')* $#+,%-4 7",3" #8'#(-& .+%0 1,+'" '% '"# #(- %. '"# .,+&' .%2+ 7##9&4 1#:,(& /#+5 )1+2$'*5 )' 1,+'"6 ;"5&,%*%:,3)* )-<2&'0#('& 02&' 1# 0)-# =2,39*54 1#3)2 '"# (#71%+( 02&' &2--#(*5 -% .%+ ,'*. '"% '",(:& '")' '"# 0%'"#+ 1%-5 ")& 1##( -%,(: .%+ ,'6 !"2&4 '"# (#71%+( 02&' 3)++5 %( +#&$,+)',%(4 %1'),( (2'+,#('&4 -,:#&' (2'+,#('&4 #83+#'# 7)&'#&4 +#:2*)'# 1%-5 '# 0$#+)'2+#4 )(- &% .%+'"6 >%7#/#+4 ,'& 0%&' ,00#-,)'# (##- ,& '% %1'),( %85:#( )(- #83+#'# 3)+1%( -,%8,-#4 &% ,'& .,+&' 1+#)'" ,& 3+,',3)*6 !"# .,+&' 1+#)'" 02&' 1# $)+',32*)+*5 .%+3#.2*4 1#3)2 '"# (#71%+( *2(:& )+# 3%**)$-4 )(- '"# ),+7)5& )+# &0)** )(- %..#+ 3%(&,-#+)1*# +#&,&')(3# '% ),+ 0%/# 0#('6 ?*&%4 &2+.)3# '#(&,%( '#(-& '% "%*- '"# 0%,&' 0#01+)(#& %. '"# *2(:& '%:#'"#+6 @%+'2()'#*54 '"# *2(:& %. ) .2** '#+0 .#'2& +#'# &2+.)3')('4 7",3" +#-23#& &2+.)3# '#(&,%(4 )(- ).'#+ '"# .,+&' $%7#+.2* 1+#)'" 1#:,(& '% #8$)(- '"# *2(:&4 1+#)'",(: 1#3%0#& #)&,#+6 A' ,& (%' 3*#)+ 7"#'"#+ '"# .,+&' 1+#)'" ,& &',02*)'#- 15 %(# %+ /#+)* .)3'%+&6 !"% '")' 0)5 1# ,(/%*/#- ,(3*2-# )( ,(3+#)&,(: *#/#* %. 3)+1%( -,%8,-#4 ) -#3+#)&,(: $"4 *%7 %85:#( 3%(3#('+)',%(4 ) -+%$ ,( 1%-5 '#0$#+)'2+# )(- 0#3")(,3)* &',02*)',%( '")' %332+& -2+,(: )(- ).'#+ '"# 1,+'" $+%3#&&6 ;+,%+ '% 1,+'"4 '"# .#'2& -#$#(-& $+,0)+,*5 %( :*23% )(- .)''5 )3,-& %1'),(#- .+%0 '"# 0%'"#+ 1*%%- )& #(#+:5 &%2+3#&
BDC .11160 ROOT
.07608 SCIENCE
.00203 ANIMAL AND PLANT BIOLOGY
.00014 ANIMAL PHYSIOLOGY
.03935 BREATHING AND STOPPING BREATHING
.03859 CHEMISTRY
.02433 MOVEMENT AND LOCATION
.00118 MEDICINE
.06017 OBSTETRICS (PREGNANCY AND BIRTH)
.06003 BIRTH
Fig. 5. Example of a single TASA document from the science genre (A). The seven-most probable concepts inferred by the hierarchical concept–topic model for this document using the CALD concepts (B). The dashed concepts are ancestor concepts of the top seven concepts that were included for visualization purposes.
the transitions from concept nodes to the children of concept nodes. These hyperparameters determine how likely it is (a priori) for the model to generate a document along the path to the physics and chemistry concepts (for example). Fig. 6 shows the 20 highest probability concepts for a random subset of 200 TASA documents from the science genre. For each concept, the name of the concept is shown in all caps. The visualization also includes the ancestor nodes (shown in dashed ovals) to complete the path to the root node. The numbers in Fig. 6 represent the marginal probability for the concept. The marginal probability is computed based on the product of probabilities along the path of reaching the node as well as the probability of exiting at the node, marginalized (averaged) across all documents: X ½pðexitjc; dÞpðcjparentðcÞ; dÞ# # #pðrootjdÞ%: ð4Þ pðcÞ / d
Many of the most likely concepts as inferred by the model relate to specific science concepts (e.g., geography, astronomy, chemistry, etc.). These concepts all also fall under the general science concept, which is also one of the most likely concepts for this document collection. Therefore, the model is able to summarize the semantic themes in a set of documents at multiple levels of granularity. In the original CALD concept set, each concept consists of a set of words and no knowledge is provided about the prominence, frequency, or representativeness of words within the concept. In the hierarchical concept–topic model, for each concept, a distribution over words is inferred that is tuned to the specific collection of documents. For example, for the concept astronomy, the word planet receives much higher probability than the word Saturn or equinox, all of which are members of the concept. These differences in word probabilities
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
35
.10427 ROOT
.01149 SCIENCE
.00961 CHEMISTRY
.00571 CHEMICAL ELEMENTS
.00670 THE EARTH AND OUTER SPACE
.00391 CHEMISTRY GENERAL WORDS
.00364 ASTRONOMY
.02236 LIFE, DEATH AND THE LIVING WORLD
.00355 MEASURES AND QUANTITIES
.00128 GEOGRAPHY
.00109 SEAS, RIVERS AND WATER
.01182 USING THE MIND
.00228 PHYSICS
.00382 WEATHER AND CLIMATE
.01061 MOVEMENT AND LOCATION
.00417 TECHNOLOGY
.00635 ATOMS, MOLECULES AND SUB-ATOMIC PARTICLES
.00343 THE STATE OF MATTER
.00492 ELECTRICITY AND ELECTRONICS
.00464 COMMUNICATION
.00216 FARMING AND FORESTRY
.00587 ELECTRICITY
.00109 ANIMAL FARMING
.00330 ANIMAL FOOD
.00511 FLOODS, TIDES AND CURRENTS
Fig. 6. Visualization of the marginal concept distributions from the hierarchical concept–topic model learned on science documents using CALD concepts. The 20 most likely concepts are shown, including the five ancestor nodes (shown in dashed ovals) needed to complete the path to the root node.
highlight the ability of the model to adapt to variations in word usage across document collections.
5. Generalization performance In the previous section, the tagging illustrations provided a qualitative assessment of concept–topic models. To get a more quantitative evaluation, we assess the performance of the topic model, concept–topic model, and the hierarchical concept–topic model by evaluating their capability to explain new documents that the model has not been trained on. The idea is that models that are trained on documents of a certain genre should generalize to new documents from the same genre. One formal way to assess generalization performance is through perplexity. Perplexity is a quantitative measure for comparing language models (Brown, deSouza, Mercer, Della Pietra, & Lai, 1992) and is widely used to compare the predictive performance of topic models (e.g., Blei et al., 2003; Wallach et al., 2009). Although perplexity does not directly measure aspects of a model such as interpretability or coverage, it is nonetheless a useful general predictive metric for assessing the quality of a topic model. Perplexity is equivalent to the inverse of the geometric mean of the likelihood of holdout data. The perplexity of a collection of test documents given the training set is defined as:
36
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
Perpðwtest jDtrain Þ ¼ exp $
! log p ð w jD Þ d train d¼1 PDtest N d d¼1
PDtest
ð5Þ
where wtest is the set of word tokens in the test documents, wd is the set of word tokens in document d of the test set, Dtrain is the training set, and Nd is the number of word tokens in document d. Lower perplexity scores indicate that the model’s predicted distribution of heldout data is closer to the true distribution. The experiments in this section are again based on the TASA data set. We train the models on a random subset of 90% of documents classified as science, creating a training set of D = 4,820 documents. By training the models, we obtain estimates for the word–topic distributions, topic–document distributions, the assignments of word tokens to topics and concepts, as well as the hyperparameters on the topic–document distributions (for all models, asymmetric Dirichlet priors were used for the document-specific topic distributions). Note that in all reported simulations, we used the intersection vocabulary (the results are qualitatively similar using the union vocabulary). We then evaluate generalization performance on the remaining documents in the science genre and also on a subset of documents classified as social studies. By testing on science and social studies documents, we evaluate the models’ ability to generalize either within the same genre or between genres. For each test document, we used a random 50% of words of the document to estimate document-specific distributions and measure perplexity on the remaining 50% of words using the estimated distributions. More details about the perplexity computation are provided in Appendix B of Chemudugunta et al. (2008b). 5.1. Perplexity comparison across models We compare the perplexity of the topic model (TM), concept–topic model (CTM), and the hierarchical concept–topic model (HCTM) trained on document sets from the science genre of the TASA collection and using concepts from CALD. Fig. 7A, B shows the perplexity of TM, CTM, and HCTM as a function of the number of data-driven topics T. Panel (a) shows the results when the model is trained and tested on science documents. Panel (b) shows the results when the model is trained on science documents and tested on social studies documents. The point T = 0 indicates that there are no topics used in the model. The results clearly indicate that incorporating concepts greatly improves the perplexity of the models (lower perplexity indicates better predictive performance). Both CTM and HCTM outperform TM, which does not rely on human-defined concepts. The results also show that human-defined concepts by themselves (i.e., the perplexity obtained when the number of learned topics T = 0) are not sufficient to get the best generalization performance— additional learned topics that are tuned to the specific content of the document collection are needed for optimal performance (around 100–300 learned topics). One important point to note is that the improved performance by the concept models is not due to the high number of word distributions T + C, compared with the topic model that utilizes only T topics. In fact, even with T = 2,000 topics, TM does not improve its perplexity and even shows signs of deterioration in quality.
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
37
!%#
!"# 800
4000 3800
Perplexity
750
Perplexity
4200
Topic Concept Topic Hierarchical Concept Topic
700 650
3600 3400 3200 3000
600
2800
550 0 50 100
200
300
//
//
2000
2600 0 50 100
500 1000 2000
Number of Learned Topics
!$#
1000
300
400
//
//
500 1000 2000
8000
Topic Concept Topic Hierarchical Concept Topic
1500
200
Number of Learned Topics
!
7000
Perplexity
Perplexity
400
6000 5000 4000 3000
500
0
20
40
60
80
100
Percentage of training documents
2000 0
20
40
60
80
Percentage of training documents
100
Fig. 7. Comparing perplexity for topic model, concept–topic model, and the hierarchical concept–topic model as a function of number of topics (A–B) and percentage of training documents (C–D). Panels (A) and (C) show the results when the model is trained and tested on documents from the science genre. Panels (B) and (D) show the results when the model is trained on documents from the science genre, but tested on documents from the social studies genre.
We next look at the effect of varying the amount of training data for all models. Fig. 7C shows the results when the model is trained and tested on science documents. Fig. 7D shows the results when the model is trained on science documents and tested on social studies documents. When there is very little training data (e.g., up to 500 documents), both concept– topic models significantly outperform the topic model. Because learned topics in TM are entirely data driven, there is not enough statistical information to build accurate representations on the basis of just a few hundred documents (in the extreme case where there is no training data available, topics of TM will just be uniform distributions and prediction will be at chance). In the regime of little training data, however, the concept models can leverage the human-defined concepts, providing a priori structure to the learning algorithm.
38
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
Of the two concept models, HCTM outperforms CTM when little training data are available (see Fig. 7A,B) or when the model generalizes to documents from a different genre (see Fig. 7B,D). HCTM and CTM rely on the same set of human-defined concepts, but HCTM also imposes hierarchical constraints on these concepts. For example, the concept model needs no documents to learn that physics and chemistry are related concepts since this knowledge is already built in. Therefore, if a document appears to be about physics, the model predicts with small probability that chemistry words can appear in the document. These a priori concept relations are clearly useful when little data are available but are relatively less beneficial with larger amounts of training data. In this scenario, the concepts and topics can be fine-tuned to the data and the difference in performance between flat concept– topic representations and hierarchical concept representations are less pronounced. When generalizing to new kinds of documents (e.g., when training on science documents and testing on social studies documents), the hierarchical concept–topic model outperforms the concept–topic model regardless of the amount of training data. In this case, the learned knowledge is less useful and the a priori structure in the hierarchical relations between concepts provides necessary constraints on the inference process.
6. Relation between learned topics and concepts Both the concept–topic and hierarchical concept–topic models allow for a combination of concepts and learned topics. These learned topics are useful to identify different gaps in the existing concept sets and capture semantic themes beyond those covered in the concepts. Such learned topics will depend on the background corpus. To get a better understanding of the kind of information captured by these models, we applied the concept–topic model to the TASA documents in the science genre. We set the number of learned topics T = 50. In one simulation, we ran the model using the union vocabulary that combines words from the TASA corpus and CALD concepts. Importantly, the union vocabulary includes words that are not part of CALD. Because the model gives zero probability for such words under any concepts, these words have to be modeled by the learned topics. Fig. 8A shows examples of topics learned by the model. The words not covered by CALD are shown in bold. The learned topics clearly capture many of the words in the corpus that are not part of CALD, including names of people (Darwin), technical words (axon), but also some words such as later and easiest that one would expect to be present in a thesaurus. Note that the reason words such as later and easiest are excluded in CALD is not because CALD lists only root word forms (related word forms are encoded in the database). These words appear to be genuine omissions in CALD that the concept–topic model handled by learned topics. We also ran a simulation with the concept–topic model using the intersection vocabulary that includes words only present in both the TASA corpus and CALD database. Fig. 8B shows some example topics learned by the model. By definition, all words shown in these topics are members of some concepts so the concept–topic model is able to explain these words by first selecting a concept and then a word from a concept. These learned topics focus on word correlations that are not currently captured by the concepts. For example, the
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
39
(A) word darwin charles evolution son galapagos lamarck beagle england flytrap malthus wallace alfred geological jacques later
prob. 0.049 0.024 0.012 0.006 0.006 0.006 0.004 0.003 0.003 0.003 0.003 0.003 0.003 0.003 0.003
word newton galileo isaac later newtons inertial permanganate fleming straight-line huygens italian newtonian alexander tabletop rectilinear
prob. 0.084 0.027 0.018 0.013 0.010 0.008 0.007 0.006 0.006 0.004 0.004 0.004 0.004 0.004 0.003
word axon fulcrum dendrites permanganate acetylcholine axons nadph parasympathetic riverwood easiest inhibitory cutter effectors energized parkman
prob. 0.012 0.011 0.010 0.008 0.007 0.007 0.005 0.005 0.005 0.004 0.004 0.004 0.004 0.004 0.004
word paleozoic mesozoic carbonyl precambrian cenozoic cambrian thermonuclear aldehyde quaternary aldol ketones alvin quats coatings flood-hazard
prob. 0.071 0.069 0.051 0.043 0.041 0.030 0.024 0.019 0.019 0.019 0.016 0.015 0.013 0.012 0.012
word universe galaxy galaxies milky nebula cosmic billions spiral interstellar resembles acquiring nebulae galactic static accustomed
prob. 0.094 0.075 0.062 0.039 0.015 0.013 0.011 0.007 0.007 0.006 0.005 0.005 0.004 0.004 0.003
word hypothesis scientific scientist hypotheses educated suggested outcome suggests verified wrong suggestions duplicate incorrect searches suggest
prob. 0.145 0.063 0.043 0.041 0.011 0.010 0.009 0.008 0.007 0.007 0.005 0.005 0.005 0.005 0.004
word acres farmer swirling cornfield plowing ranchers differed well-preserved energetically feasible interlocking splinter tumbling bewildering buttocks
prob. 0.023 0.019 0.008 0.007 0.007 0.006 0.005 0.005 0.004 0.004 0.004 0.004 0.004 0.003 0.003
word pollution pollutants era chemicals large factories smog polluted automobiles fulcrum sulfur amounts unfit dumping areas
prob. 0.148 0.053 0.034 0.029 0.020 0.018 0.018 0.013 0.013 0.011 0.011 0.011 0.010 0.009 0.009
(B)
Fig. 8. Examples of learned topics for the CTM model. Panel (A) illustrates a simulation using the union vocabulary that includes words that are part of the TASA corpus but are not part of the CALD vocabulary (these words are shown in bold). Panel (B) illustrated topics from a simulation on the intersection vocabulary that only includes words present in both TASA and CALD.
first two words in the left-most topic, universe and galaxy, are clearly related but are not members of the same concept. Similarly, hypothesis and scientific as well as pollution and chemicals are word pairs that are not members of the same concept but often co-occur in documents. The learned topics can be used to capture such correlations.
7. Expanding existing concepts with new words One of the biggest disadvantages of utilizing human-defined knowledge databases is the large amount of manual effort involved to build and update the knowledge database. For
40
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
example, the lexicographers of the CALD database have to continually update the concepts to include words that have changed meaning or to insert entirely new concepts. In addition, the CALD database has to be checked for human errors, which might be difficult to detect manually. One way to test the utility of the concept model is to see whether it can automatically identify omissions within human-defined concepts, that is, words that should be in a concept but have been omitted. In the previous section, we showed how such words can become part of learned topics. In this section, we tested whether a concept–topic model could learn to expand existing concepts with new words that appear to have been omitted from concepts. In our simulation approach, we removed selected words from the CALD concepts and tested how well a concept–topic model could use the TASA corpus to identify which concept they should be associated with. We only evaluate the concept–topic model with no learned topics (i.e., T = 0) on this concept recovery task (we expect that the hierarchical concept–topic model gives similar results). As a baseline method, we could compare the model against a number of existing models such as LDA or LSA. For simplicity, we focus here on LSA. We computed the singular value decomposition of the document word co-occurrence matrix for the TASA corpus and projected the terms onto N-dimensional concept space. Given a test word, we return a ranking of the concepts determined by the average distance from the test word to the m closest words in the concept. Distance is measured using cosine similarity. We experimented with different values of N and m, and report our results for N = 150 and m = 5 (results were relatively insensitive to the exact values used). We compiled a list of 152 terms from the corpus where each term was a member of only one concept. The concept had to be well represented by the corpus, that is, at least 60% of the words in the concept were present in the corpus. Furthermore, the term had to be a significant member of the concept, that is, the term had the highest frequency in the corpus among all the terms in the same human-defined concept. In the concept–topic model, if a word is not included in the set of concepts to begin with, the model will be unaware of it. So for the purposes of this experiment, we ‘‘removed’’ a word by placing it in all 2,183 concept sets—in effect this tells the model that the word exists but gives the model no clue about which concept it belongs to. After training the model, we can simply count how often a word is assigned to a particular concept (via the z assignments) to produce a ranked list of concepts given a word. Fig. 9 shows an example of the rankings returned by the concept–topic model for three test words. For each removed word, the figure shows the top five ranked concepts. We label each concept with one of four letters: M(atch) indicates a match to the target concept, P(arent) indicates a concept on the path from the root concept to the target concept, C(hild) indicates a concept in the subtree rooted at the target concept, and O(ther). In the example, the model is able to rank the target concept as the first or second ranked concept but even the highly ranked mismatching concepts are often quite reasonable target concepts. For example, the word soot has strong associations with the concept cleaning and tidying places and things, as well as dirt untidiness, which are semantically related to the word soot, but which did not originally contain the removed word.
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011) ,$0'/$) .'-)
41
,+&*$) ('&%$#"!
!""#$ 1234(567898:478;4<9;=98:4>57(6?478;4<@98:? %&'$()*+,-./$*0$-*&1,/.2*3 1234;9,<478;4A8<9;986?? 1234BA95;98:?C487D6?478;4<=>6?4 2E 123468F9,28D68<7549??A6? 45!67#! 1(3498?6( %&'$23/8-.$39&8/ 1234?2(96<= 12349D>,2F98:4E6,<959<=478;4>6?<4(28<,25 1>34>578478;4789D75? :4;67#4"5!$ %&'$(<9-8/$93+$<*-9.2*3/ 12346D9<<98:478;4(7?<98:459:@< 1234>29842E4<@64(2D>7?? 1234?>2,7?<9D6? 1234>7=98:47<<68<928478;4B698:4(7,6EA5 1D4H40+"%IJ4(4H4%IKL)J4>4H4#+-$&"J424H4'"I$-3
Fig. 9. Example of rankings by concept–topic model in word recovery task.
Note that the concept–topic model is also able to identify multiple meanings for a given word. For example, the word directions can refer to north, south, east and west (koints of the kompass) as well as a set of instructions (places and locations). Also note that many words in the CALD concepts are classified according to their definition (e.g., soot is a product of combustion) rather than their descriptive qualities (e.g., soot causes dirtiness and soot is an environmental issue). Table 1 shows the overall results for the concept–topic model and latent semantic analysis. The table shows the probability that a concept is ranked in the top K returned concepts (precision), as a target concept, parent concept, or child concept. The results show that the concept–topic model outperforms the latent semantic analysis approach in this conceptrecovery task. The concept–topic model is often able to rank the target concept (out of 2,183 concepts) in the top 10 or 20. For both the concept–topic models and the latent semantic analysis approach, the parent or child of the target concept also often appear (more than expected by chance) in the top 10 or 20 concepts, indicating that these models are able to recover more specific as well as more general concepts related to the novel word.
8. Discussion Although most of the earlier work on topic modeling is purely data driven, in that no human knowledge is used in learning the topic model, there are some exceptions. Boyd-Graber, Blei, and Zhu (2007) develop a topic modeling framework that combines
42
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011) Table 1 Precision results for the concept-recovery task LSA Top 10 concepts Target Parent Child Top 20 concepts Target Parent Child
Concept–Topic
.45 .22 .05
.55 .15 .04
.53 .30 .05
.57 .26 .05
human-derived linguistic knowledge using unsupervised topic models for the purpose of word–sense disambiguation. Andrzejewski, Zhu, and Craven (2009) recently introduced an iterative topic modeling process where a human can inspect the topics and specify which words should have high probability in a topic and which words should not appear together in a topic. By replacing the multinomial distribution over words in a topic with a Dirichlet forest prior, the knowledge expressed by a human can be taken into account in the next iteration of the topic modeling process. Wei and Croft (2007) use manually built topics using documents and categories from the Open Directory Project for information retrieval. The manual topics are built by aggregating documents for selected categories and obtaining probability distributions by normalizing the word counts of the associated documents. Topic modeling has also been used for finding mappings between ontology pairs (Spiliopoulos, Vouros, & Karkaletsis, 2007). The work of Ifrim and Weikum (2006) and Bundschus, Dejori, Yu, Tresp, and Kriegel (2008) combines topics and concepts for the purposes of text classification. Our framework is somewhat more general in that we not only improve the quality of making predictions on text data by using prior human concepts but also are able to make inferences in the reverse direction about concept words and concept hierarchies given data. In addition, our concept–topic models do not require labeled data. Although topic modeling has also been used to semi-automatically build taxonomies from data (e.g., Dietz & Stewart, 2006; Zavitsanos, Paliouras, Vouros, & Petridis, 2007), these approaches do not make use of existing ontologies. There is also a significant amount of prior work on using data to help with ontology construction, evaluation, and document tagging, such as learning ontologies from text data (e.g., Maedche & Staab, 2001), methodologies for evaluating how well ontologies are matched to specific text corpora (Alani & Brewster, 2006; Brewster, Alani, Dasmahapatra, & Wilks, 2004), and systems for tagging documents with semantic concepts using word-level matching techniques (Dill et al., 2003). Our work is broader in scope in that we propose general-purpose probabilistic models that combine concepts and topics within a single framework, allowing us to use the data to make inferences about how documents and concepts are related (for example). It should be noted that in the work reviewed in this paper, we do not explicitly investigate techniques for modifying an ontology in a data-driven manner (e.g., adding ⁄ deleting words from concepts
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
43
or relationships among concepts)—however, the framework we propose could certainly be used as a basis for exploring such ideas. There are several potentially useful directions in which the hierarchical concept–topic model can be extended. One interesting extension to try is to substitute the Dirichlet prior on the concepts with a Dirichlet process prior, where each concept will now have a potentially infinite number of children, a finite number of which are observed at any given instance (e.g., Teh et al., 2006). When we do a random walk through the concept hierarchy to generate a word, we now have an additional option to create a child topic and generate a word from that topic. There would be no need for the switching mechanism as data-driven topics are now part of the concept hierarchy. Such a model would allow us to add new topics to an existing concept set hierarchy and could potentially be useful in building a recommender system for updating concept ontologies. An alternative direction to pursue would be to introduce additional machinery in the generative model to handle different aspects of transitions through the concept hierarchy. In HCTM, we currently learn one set of path correlations for the entire corpus (captured by the Dirichlet parameters s in HCTM). It would be interesting to introduce another latent variable to model multiple path correlations. Under this extension, documents from different genres can learn different path correlations (similar to the work of Boyd-Graber et al., 2007). For example, scientific documents could prefer to utilize paths involving scientific concepts, and humanities concepts could prefer to utilize a different set of path correlations when they are modeled together. A model of this type would also be able to make use of class labels of documents if available.
9. Conclusions We have proposed a probabilistic framework for combining data-driven topics and semantically rich human-defined concepts. We first introduced the concept–topic model, a straightforward extension of the topic model, to utilize human-defined semantic concepts in the topic modeling framework. The model represents documents as a mixture of topics and concepts, thereby allowing us to describe documents using the semantically rich concepts. We further extended this model with the hierarchical concept–topic model where we incorporate the concept hierarchy into the generative model by modeling the parent–child relationship in the concept hierarchy. Our experimental results show that the semantic concepts significantly improve the quality of the resulting models. Modeling concepts and their associated hierarchies appears to be particularly useful when there are limited training data—the hierarchical concept–topic model has the best predictive performance overall in this regime. We view the current set of models as a starting point for exploring more expressive generative models that can potentially have wide-ranging applications, particularly in areas of document modeling and tagging, ontology modeling and refining, and information retrieval. In addition, these models are useful to expand the current cognitive science framework to characterize human learning of semantic information. Many existing models in cognitive
44
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
science explain how a human learner extracts semantic information from only a single source of information: statistical co-occurrence information between words and documents. The current set of models suggests that the learning process in such models can be enhanced when additional background information is available. For example, a human learner might already be familiar with certain concepts (and the relations between concepts) and the exposure to (new) statistical information such as word–document co-occurrences serves to refine existing concepts or perhaps learn new ones.
Acknowledgments This material is based upon work supported in part by the National Science Foundation under Award Number IIS-0083489, and by the Office of Naval Research under Award Number N00014-08-1-1015. We are extremely grateful to three anonymous reviewers, and to Danielle McNamara and Simon Dennis, for their very helpful comments on an earlier version of this article. We thank America Holloway for assistance in providing the results in Section 7.
References Alani, H., & Brewster, C. (2006). Metrics for ranking ontologies. 4th International EON Workshop, 15th International World Wide Web Conference. New York: ACM. Andrzejewski, D., Zhu, X., & Craven, M. (2009). Incorporating domain knowledge into topic modeling via Dirichlet forest priors. The 26th International Conference on Machine Learning (ICML). New York: ACM. Blei, D. M., Griffiths, T. L., Jordan, M. I., & Tenenbaum, J. B. (2003). Hierarchical topic models and the nested Chinese restaurant process. In S. Thrun, L. Saul, & B. Scholkopf (Eds.), Advances in Neural Information Processing Systems 16. Cambridge, MA: MIT Press. Blei, D., & Jordan, M. (2003). Modeling annotated data. Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 127–134). New York: ACM. Blei, D., & Lafferty, J. (2006). Correlated topic models. In Y. Weiss, B. Scho¨lkopf, & J. Platt (Eds.), Advances in Neural Information Processing Systems 18 (pp. 147–154). Cambridge, MA: MIT Press. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022. Boyd-Graber, D., Blei, D., & Zhu, X. (2007). A topic model for word sense disambiguation. Proceedings of the Joint Conference of Empirical Methods in Natural Language Processing and Computational Natural Language Learning (pp. 1024–1033). New York: ACM. Brewster, C., Alani, H., Dasmahapatra, S., & Wilks, Y. (2004). Data driven ontology evaluation. International Conference on Language Resources and Evaluation. Paris, France. Brown, P. F., deSouza, P. V., Mercer, R. L., Della Pietra, V. J., & Lai, J. C. (1992). Class-based n-gram models of natural language. Computational Linguistics, 18(4), 467–479. Bundschus, M., Dejori, M., Yu, S., Tresp, V., & Kriegel, H. (2008). Statistical modeling of medical indexing processes for biomedical knowledge information discovery from text. Proceedings of the 8th International Workshop on Data Mining in Bioinformatics (BIOKDD). New York: ACM. Buntine, W. L., & Jakulin, A. (2004). Applying discrete PCA in data analysis. In: M. Chickering & J. Halpern (Eds.), Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence (pp. 59–66). San Francisco, CA: Morgan Kaufmann Publishers.
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
45
Chater, N., & Manning, C. (2006). Probabilistic models of language processing and acquisition. Trends in Cognitive Sciences, 10(7), 335–344. Chemudugunta, C., Holloway, A., Smyth, P., & Steyvers, M. (2008a). Modeling documents by combining semantic concepts with unsupervised statistical learning. 7th International Semantic Web Conference (pp. 229–244). Berlin: Springer-Verlag. Chemudugunta, C., Smyth, P., & Steyvers, M. (2008b). Combining concept hierarchies and statistical topic models. ACM 17th Conference on Information and Knowledge Management. New York: ACM. Chemudugunta, C., Smyth, P., & Steyvers, M. (2008c). Text modeling using unsupervised topic models and concept hierarchies. Technical Report. Available at: http://arxiv.org/abs/0808.0973. Accessed on May 16, 2010. Cutting, D. R., Karger, D., Pedersen, J. O., & Tukey, J. W. (1992). Scatter ⁄ gather: A cluster-based approach to browsing large document collections. Proceedings of the Fifteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 318–329). New York: ACM Press. Dennis, S. (2004). An unsupervised method for the extraction of propositional information from text. Proceedings of the National Academy of Sciences, 101, 5206–5213. Dietz, L., & Stewart, A. (2006). Utilize probabilistic topic models to enrich knowledge bases. Proceedings of the E SWC 2006 workshop on mastering the gap: From information extraction to semantic representation. Dill, S., Eiron, N., Gibson, D., Gruhl, D., Guha, R., Jhingran, A., Kanungo, T., Rajagopalan, S., Tomkins, A., Tomlin, J., & Zien, J. (2003). SemTag and seeker: Bootstrapping the semantic web via automated semantic annotation. Proceedings of the 12th international conference on World Wide Web (pp. 178–186). New York: ACM. Fellbaum, C. (Ed.) (1998). WordNet, an electronic lexical database. Cambridge, MA: MIT Press. Fiser, J., & Aslin, R. N. (2005). Encoding multi-element scenes: Statistical learning of visual feature hierarchies. Journal of Experimental Psychology: General, 134, 521–537. Foltz, P. W., Gilliam, S., & Kendall, S. (2000). Supporting content-based feedback in online writing evaluation with LSA. Interactive Learning Environments, 8, 111–129. Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003). Bayesian data analysis, 2nd ed. London: Chapman & Hall. Griffiths, T. L., & Steyvers, M. (2004). Finding scientific topics. Proceedings of the National Academy of Science, 101, 5228–5235. Griffiths, T. L., Steyvers, M., Blei, D. M., & Tenenbaum, J. B. (2005) Integrating topics and syntax. In L. K. Saul (Ed.), Advances in neural information processing 17 (pp. 537–544). Cambridge, MA: MIT Press. Griffiths, T. L., Steyvers, M., & Tenenbaum, J. B. T. (2007). Topics in semantic representation. Psychological Review, 114(2), 211–244. Havasi, C., Speer, R., & Alonso, J. (2007). ConceptNet 3: A flexible, multilingual semantic network for common sense knowledge. Proceedings of Recent Advances in Natural Language Processing. Hofmann, T. (1999). Probabilistic latent semantic indexing. Proceedings of the 22nd Annual ACM Conference on Research and Development in Information Retrieval (pp. 50–57). New York: ACM Press. Ifrim, G., & Weikum, G. (2006). Transductive learning for text classification. 10th European conference on principles and practice of knowledge discovery in databases (pp. 223–234). Berlin, Germany. Jones, M. N., & Mewhort, D. J. K. (2007). Representing word meaning and order information in a composite holographic lexicon. Psychological Review, 114, 1–37. Kemp, C., & Tenenbaum, J. B. (2008). The discovery of structural form. Proceedings of the National Academy of Sciences, 105(31), 10687–10692. Kohonen, T., Kaski, S., Lagus, K., Saloja¨rvi, J., Honkela, J., Paatero, V., & Saarela, A. (2000). Self organization of a massive document collection. IEEE Transactions on Neural Networks (Special Issue on Neural Networks for Data Mining and Knowledge Discovery), 11-3, 574–585. Lagus, K., Honkela, T., Kaski, S., & Kohonen, T. (1999). WEBSOM for textual data mining. Artificial Intelligence Review, 13, 5–6. 345–364. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104, 211–240.
46
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
Landauer, T. K., Foltz, P. W., & Laham, D. (1998). Introduction to latent semantic analysis. Discourse Processes, 25, 259–284. Lenat, D. B., & Guha, R. V. (1989). Building large knowledge-based systems: Representation and inference in the Cyc project. Reading, MA: Addison-Wesley. Li, W., Blei, D., & McCallum, A. (2007). Nonparametric Bayes pachinko allocation. Conference on Uncertainty in Artificial Intelligence (UAI). Corvallis, OR: AUAI Press. Maedche, A., & Staab, S. (2001). Ontology learning for the semantic web. IEEE Intelligent Systems, 16(2), 72– 79. McCallum, A., Nigam, K., & Ungar, L. H. (2000). Efficient clustering of high-dimensional data sets with application to reference matching. Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 169–178). New York: ACM Press. McRae, K., Cree, G. S., Seidenberg, M. S., & McNorgan, C. (2005). Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods, 37, 547–559. Mei, Q., Shen, X., & Zhai, C. (2007). Automatic labeling of multinomial topic models. Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 490–499). New York: ACM Press. Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., & Miller, K. J. (1990). Introduction to Word Net: An on line lexical database. International Journal of Lexicography, 3(4), 235–244. Mimno, D. M., Li, W., & McCallum, A. (2007). Mixtures of hierarchical topics with pachinko allocation. International Conference on Machine Learning (ICML) (pp. 633–640). Corvallis, OR. Minka, T. P. (2000). Estimating a Dirichlet distribution. Technical report. Cambridge, MA: Massachusetts Institute of Technology. Najemnik, J., & Geisler, W. S. (2005). Optimal eye movement strategies in visual search. Nature, 434, 387– 391. Nelson, D. L., McEvoy, C. L., & Schreiber, T. A. (1998). The University of South Florida word association, rhyme, and word fragment norms. Available at: http://www.usf.edu/FreeAssociation/. Newman, D., Chemudugunta, C., Smyth, P., & Steyvers, M. (2006). Analyzing entities and topics in news articles using statistical topic models. Springer Lecture Notes in Computer Science (LNCS) series—IEEE international conference on intelligence and security informatics. Berlin: Springer-Verlag. Newport, E. L., & Aslin, R. N. (2004). Learning at a distance: I. Statistical learning of non-adjacent dependencies. Cognitive Psychology, 48, 127–162. Newport, E. L., Hauser, M. D., Spaepen, G., & Aslin, R. N. (2004). Learning at a distance: II. Statistical learning of non-adjacent dependencies in a non-human primate. Cognitive Psychology, 49, 85–117. Panton, K., Mtuszek, C., Lenat, D., Schneider, D., Witbrock, M., Siegel, N., & Shepard, B. (2006). Common sense reasoning—From Cyc to intelligent assistant. Lecture Notes in Computer Science, 3864, 1–31. Popescul, A., Ungar, L. H., Flake, G. W., Lawrence, S., & Giles, C. L. (2000). Clustering and identifying temporal trends in document databases. In Proceedings of the IEEE Advances in Digital Libraries (pp. 173–182). Los Alamitos, CA: IEEE Computer Society. Roget, P. M. (1911). Roget’s thesaurus of English words and phrases. New York: Thomas Y. Crowell. Ruts, W., De Deyne, S., Ameel, E., Vanpaemel, W., Verbeemen, T., & Storms, G. (2004). Flemish norm data for 13 natural concepts and 343 exemplars. Behavior Research Methods, Instruments, and Computers, 36, 506–515. Spiliopoulos, V., Vouros, G., & Karkaletsis, V. (2007). Mapping ontologies elements using features in a latent space. In Proceedings of the IEEE ⁄ WIC ⁄ ACM International Conference on Web Intelligence (pp. 457–460). Washington DC: IEEE Computer Society. Steyvers, M., & Griffiths, T. L. (2007). Probabilistic topic models. In T. Landauer, D. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of latent semantic analysis (pp. 427–448). Mahwah, NJ: Erlbaum. Steyvers, M., & Griffiths, T. L. (2008). Rational analysis as a link between human memory and information retrieval. In N. Chater & M. Oaksford (Eds.), The probabilistic mind: Prospects from rational models of cognition (pp. 327–347). Oxford, England: Oxford University Press.
M. Steyvers et al. ⁄ Topics in Cognitive Science 3 (2011)
47
Steyvers, M., Smyth, P., Rosen-Zvi, M., & Griffiths, T. (2004). Probabilistic author-topic models for information discovery. In W. Kim, R. Kohavi, J. Gehrke, & W. DuMouchel (Eds.), The Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 306–315). New York: ACM. Teh, Y. W., Jordan, M., Beal, M., & Blei, D. (2006). Hierarchical Dirichlet processes. Journal of the American Statistical Association, 101(476), 1566–1581. Wallach, H. (2006). Topic modeling: Beyond bag-of-words. In W. Cohen & A. Moore (Eds.), Proceedings of the 23rd International Conference on Machine Learning (pp. 977–984). Pittsburgh, PA. Wallach, H. M., Murray, I., Salakhutdinov, R., & Mimno, D. (2009). Evaluation methods for topic models. In A. P. Danyluk, L. Bottou, & M. L. Littman (Eds.), Proceeding of the 26th International Conference on Machine Learning (pp. (1105–1112.) New York: ACM. Wang, C., Blei, D., & Heckerman, D. (2008). Continuous time dynamic topic models. In D. McAllester and A. Nicholson (Eds.), Uncertainty in artificial intelligence (pp. 579–586). Corvallis, OR: AUAI Press. Wei, X., & Croft, W. B. (2007). Investigating retrieval performance with manually-built topic models. Proceedings of the 8th Large-Scale Semantic Access to Content (Text, Image, Video and Sound) Conference (RIAO’07). Paris, France. Yu, C., Ballard, D. H., & Aslin, R. N. (2005). The role of embodied intention in early lexical acquisition. Cognitive Science, 29, 961–1005. Zavitsanos, E., Paliouras, G., Vouros, G.A., & Petridis, S. (2007). Discovering subsumption hierarchies of ontology concepts for text corpora. In proceedings of the IEEE ⁄ WIC ⁄ ACM International Conference on Web Intelligence (pp.402–408.) Washington, D.C.: IEEE Computer Society.
Topics in Cognitive Science 3 (2011) 48–73 Copyright ! 2010 Cognitive Science Society, Inc. All rights reserved. ISSN: 1756-8757 print / 1756-8765 online DOI: 10.1111/j.1756-8765.2010.01112.x
Constructing Semantic Representations From a Gradually Changing Representation of Temporal Context Marc W. Howard, Karthik H. Shankar, Udaya K. K. Jagadisan Department of Psychology, Syracuse University Received 26 January 2009; received in revised form 2 November 2009; accepted 5 November 2009
Abstract Computational models of semantic memory exploit information about co-occurrences of words in naturally occurring text to extract information about the meaning of the words that are present in the language. Such models implicitly specify a representation of temporal context. Depending on the model, words are said to have occurred in the same context if they are presented within a moving window, within the same sentence, or within the same document. The temporal context model (TCM), which specifies a particular definition of temporal context, has proved useful in the study of episodic memory. The predictive temporal context model (pTCM) uses the same definition of temporal context to generate semantic memory representations. Taken together pTCM and TCM may prove to be part of a general model of declarative memory. Keywords: Episodic memory; Semantic memory; Mathematical modeling; Computational modeling
1. Introduction The importance of temporal context in learning the meaning of words has long been central to our understanding of the acquisition of word meaning. Contemporary computational models of semantic memory exploit this basic idea. However, the definitions of temporal context they use are contradictory with one another and often not theoretically motivated. For instance, in the BEAGLE model (Jones & Mewhort, 2007), the semantic representation of a word is the weighted average of all other word vectors that were presented in the same sentence as the word. In BEAGLE temporal context is operationalized as being constant within a sentence but changing completely between sentences. That is, words within the same sentence are in the same temporal context, but words in adjacent sentences are in completely different temporal contexts. Similarly, in latent semantic analysis (LSA) Correspondence should be sent to Marc Howard, Department of Psychology, Syracuse University, 430 Huntington Hall, Syracuse, NY 13244. E-mail:
[email protected]
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
49
and the topic model (Griffiths, Steyvers, & Tenenbaum, 2007; Landauer & Dumais, 1997), a word · document matrix is the starting point for the calculations. This implies a representation of temporal context that is constant within a document, but that changes completely between documents.1 Both of these approaches, BEAGLE and LSA and the topic model, share the assumption that temporal context is a categorical variable, but they differ in the time scale associated with the rate of change of temporal context. The fact that temporal context is only implicitly defined by these (and related) models makes the task of comparing the models, which vary on a number of other dimensions as well, considerably more difficult. The basic strategy of the research program described here is to use an explicit representation of temporal context inherited from work on episodic memory as a starting point for developing a computational semantic model. We will first briefly describe temporal context as defined by the temporal context model (TCM, Howard & Kahana, 2002; Howard, Fotedar, Datey, & Hasselmo, 2005; Sederberg, Howard, & Kahana, 2008). We will then describe how retrieval of temporal context can function to efficiently extract relationships between stimuli. Next, we describe the predictive temporal context model (pTCM, Shankar, Jagadisan, & Howard, 2009) as a solution for how to efficiently extract the meanings of words embedded in natural sequences. We then present evidence that pTCM can provide a useful description of information extracted from natural text. We close by describing several significant challenges that remain. 1.1. Temporal context in episodic memory The initial goal of TCM was to account for the recency and contiguity effects observed in episodic recall tasks. The recency effect refers to the finding that, all other things being equal, memory is better for more recently experienced information. The contiguity effect refers to the finding that, all other things being equal, items experienced close together in time become associated such that when one comes to mind it tends to bring the other to mind as well. The contiguity effect has been extensively studied in episodic recall tasks, where it exhibits a characteristic asymmetry (see Kahana, Howard, & Polyn, 2008a, for a review). Somewhat surprisingly, the contiguity effect, like the recency effect, persists over relatively long time scales, extending at least hundreds of seconds (Howard, Youker, & Venkatadass, 2008b). Similarly, the contiguity effect is observed in the very earliest stages of immediate free recall (Howard, Venkatadass, Norman, & Kahana, 2007), a prediction unique to TCM among models of the recency effect. Table 1 summarizes verbally the assumptions that constitute TCM. In TCM episodic recall proceeds by cuing with the current state of a distributed representation of temporal Table 1 Principles of operation of the temporal context model (TCM) 1. Temporal context changes gradually over time 2. Items are cued by a state of context to the extent it overlaps with their encoding contexts 3. Presentation of items causes a change in the state of context 4. Repeated/recalled items can recover the state of context in which they were previously studied
50
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
context. This state of context changes gradually over time. Studied items are activated by a context cue to the extent that it overlaps with the state of context when they were studied. The recency effect results from the combination of these two properties. After study of a list, items presented more recently in time were encoded in states of context that more strongly resemble the probe context. The concept that a gradually changing memory signal contributes to forgetting is not unique to TCM, but it has a long history in the mathematical psychology of learning and memory (e.g., Anderson & Bower, 1972; Estes, 1955, see also Brown, Neath, & Chater, 2007; Mensink & Raaijmakers, 1988; Murdock, 1997). TCM builds on these models, but it makes the additional assumption that contextual drift is caused by the items themselves. This assumption enables the model to account for the contiguity effect in episodic memory (Howard, Kahana, & Wingfield, 2006; Polyn, Norman, & Kahana, 2009a; Sederberg et al., 2008). Because the input to the context vector is caused by items, repetition of an item causes the state of context to change to a state similar to that during study of the neighbors of the original presentation of the repeated item, resulting in a contiguity effect. A further assumption of TCM is that repeated items can recover or retrieve their study context. That is, they can cause the context state to be partially reset to the state prior to the previous presentation of the repeated item. An example may make this more concrete. Suppose that the model is presented with a list of words that includes the sequence … absence, hollow, pupil, river, darling … The temporal context in which pupil is encoded includes input caused by hollow, and, to a lesser extent because it was further in the past, input caused by absence. Similarly, the temporal context in which each of the other items was encoded is composed of the input caused by the preceding items, weighted by their recency. If the context immediately after presentation of this sequence is used as a cue, darling would be most strongly activated because its encoding context is most similar to the cue context. In this way the model accounts for the recency effect. The model accounts for contiguity as well. Suppose that pupil is repeated at some later time, and it successfully recovers its encoding context. Then, the context cue recovered by pupil provides a better cue for river than for darling because the encoding context for river did not drift as far from pupil. Similarly, recovery of pupil’s encoding context makes a more effective cue for hollow than absence for the same reason. In this way, the model accounts for the contiguity effect in both the forward and backward directions.2 The ability of items to recover their prior contextual states endows TCM with a number of important properties. For instance, the backward association manifest in the contiguity effect depends on the ability of items to recover their prior temporal contexts. Similiarly, because the prior state of context includes input caused by the items that preceded the initial presentation of the repeated item, recovering this state results in a mixing of the input patterns caused by items on the basis of their contiguity. This property can be exploited to describe effects in relational memory, as we shall see shortly. A more formal description of TCM follows. Readers who wish to avoid a mathematical description may choose to skip this subsection.
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
51
1.1.1. Formal description of TCM We will deviate from some of the details (and notation) used in previous papers in order to create as much consistency as possible with the development of the semantic memory model used here. In the discussion that follows we will assume, unless otherwise noted, that the subject studies an extremely long list without repetitions of items. The state of temporal context at time step i, ti, is a result of the previous state of context and an input pattern tIN i , caused by the item presented at time step i: ti ¼ qti"1 þ ð1 " qÞtIN i ;
ð1Þ
where q is a scalar less than one. We assume that the input vectors are chosen such that the sum of their components is unity. Under these conditions, the sum of the components of t is also unity. Eq. 1 implements the assumption that context changes gradually over time; all other things being equal, the state of temporal context at time step i resembles the previous state of context more than other states more distant in the past. Temporal context changes gradually as more (unique) items are successively presented. We use an outer product matrix associating contexts to items to enable items to be activated by a contextual cue. During study, items are encoded in their temporal context. The matrix M is updated such that the change in M is given by: DM ¼ fi t0i"1 ;
ð2Þ
where fi is the vector associated with item i and the prime reflects the transpose. In order to recall an item, the matrix M is multiplied from the right with the current state of context. Eq. 2 results in the property that each item fi is activated to the extent that its study context overlaps with the context used as a cue. It remains to describe the properties of the tIN vectors. The input pattern tIN caused by an item is composed of a fixed component that does not change across repetitions of an item over the time scale of a laboratory memory experiment that we will refer to as f and a changing component we will refer to as h. Each fi and each hi are caused by the item presented at time step i and depend only on the identity of that item and its previous history. The f vectors for each item are fixed throughout the simulation. If item a is presented at time step i, then ^ tIN i ¼ ð1 " cÞfa þ cha :
ð3Þ
The hat in the second term indicates that h is normalized prior to entering this expression. We fix the fas to be unit vectors that serve as the basis vectors for the t space. With learning, ha changes from one presentation of item fa to another according to Dha ¼ ti"1 :
ð4Þ
The function of h is to enable items to recover the temporal context in which they were studied. This implements property 4 in Table 1.
52
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
1.2. Relational memory, retrieved context, and the hippocampus Consider the case of a repeated item that recovers its prior study context when it is repeated. This means that the input caused by this item is not consistent across its two presentations. The change in the input patterns with repetitions has wide-reaching implications. The mixing of input patterns creates the ability for the model to describe associations between items that do not actually co-occur. Consider the case in which the subject learns a pair of items a–b and then much later learns b–c. If contextual retrieval takes place (i.e., if c is nonzero), then during learning of a–b, the input pattern caused by b will include the temporal context that preceded it. This state of context includes information contributed by item a. As a consequence, during learning of b–c, the input pattern caused by b includes information about a. This means that the encoding context for c includes ‘‘parts of’’ a, even though a and c were not presented close together in time. In fact, such transitive associations among items that have not been presented close together in time are observed (e.g., Bunsey & Eichenbaum, 1996; Howard, Jing, Rao, Provyn, & Datey, 2009; Slamecka, 1976). For instance, Howard et al. (2009) taught human subjects a long list of paired-associates with overlapping pairs. That is, subjects learned a list of 35 pairs of the form a–b, b–c, c–d … presented in a random order for a total of 12 presentations each. During a final free recall session, subjects were asked to recall all the items from all the pairs in the order they came to mind. If a subject had just recalled a double-function word from the list, the next word the subject recalled would tend to come from a nearby pair, even if it was not from the same pair as the just-recalled word. For example, if the subject had just recalled b, the next word that came to mind would be more likely be d than e. In other words, subjects showed behavioral evidence for transitive associations between items across pairs that fell off as a function of the number of links in the chain between items. It is as though the subjects were able to integrate the pairs into a common global memory structure. Howard et al. (2009) demonstrated that TCM provides a good description of this behavior. Although transitive associations actually make paired associate learning more difficult (see especially Provyn, Sliwinski, & Howard, 2007), they provide an extremely useful computational function by allowing the model to infer relationships between items that are not explicitly instructed. That is, the model does not have to be explicitly instructed that a and c ‘‘go together’’ in order for a to evoke c. A successful model of semantic memory needs to be able to place tens of thousands of symbols in the proper relation to one another. If each of those relations needed to be explicitly instructed, the number of presentations necessary would be extremely large. Moreover, the model does not need to make a priori assumptions about the possible structure present in the learning set (Kemp & Tenenbaum, 2009; Tenenbaum, Griffiths, & Kemp, 2006). This is possible because retrieved context ‘‘spreads’’ along the links in the chain such that the representation at the end of training reflects the topology of the pairs it was trained on. Note that this functionality depends on a training set in which relationships can be directly inferred from contiguity. The function of contextual retrieval in TCM is in some sense analogous to the function of dimensional reduction in LSA and the topic model. To illustrate this, Fig. 1 shows results for TCM, LSA, and the topic model trained on a ‘‘corpus’’ that consisted of a set of
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011) (b)
(a) H G F E D C B A
(c) H G F E D C B A
H G F E D C B A A
B
C
D
E
F
G
H
53
A
B
C
D
E
F
G
H
A
B
C
D
E
F
G
H
Fig. 1. The temporal context model (TCM), a model of episodic recall based on contextual overlap, and computational models of semantic memory both predict transitive associations. In all three panels, the figure shows the similarity of the representation of each item in a double function list of paired associates after training. (a) Retrieved temporal context as defined by TCM shows transitive associations. The shading of each square codes for the similarity of the temporal context vector retrieved by the corresponding pair of items after 10 trials of learning on the corresponding double function list. Vector similarity was assessed using the inner product. High values of the inner product are shaded dark. (b) A representation generated using latent semantic analysis (Landauer & Dumais, 1997) shows transitive associations. A singular value decomposition was computed for an item-context matrix corresponding to training on a double function list of pairs. Two dimensions were retained. Similarity of each pair of vectors was assessed using the cosine of the angle between them. High values of cosine are dark. (c) The topic model (Griffiths et al., 2007) was trained on a set of contexts simulating presentation of a double function list. The simulation used two topics and a ¼ 0.1 and b ¼ 0.1 (see Griffiths et al., 2007 for details). The similarity between each pair of items was estimated by comparing the Kullback–Leibler divergence of the distribution over topics induced by each item. Small values of divergence, corresponding to high similarity, are dark.
‘‘documents’’ each of which contained a single double function pair.3 That is, document 1 consisted of the words a and b, document 2 consisted of the words b and c, and so on. Each panel shows the similarity of the representation of each word to each other word. Transitive associations can be seen by the shading among pairs of items that did not co-occur. TCM, LSA (with two dimensions), and the topic model (with two topics) all build transitive associations that bridge across pairs. Interestingly, LSA only exhibits transitive associations if the number of dimensions retained is less than the number possible. That is, if all seven dimensions were retained for LSA, the model does not exhibit across-pair associations. Rather it only makes words that occur in the same document similar to one another. Similar results are observed for the topic model with seven topics (one for each document). It should be noted that HAL and BEAGLE also illustrate transitive associations, although this is not attributable to dimensional reduction in those models. Contextual retrieval enables the development of a representation of the items that reflects their global co-occurrence structure. For instance, suppose that we train the model on a set of overlapping pairs a–b, b–c … z–a, with the pairs presented in a random order and each pair completely isolated from the others. After training, the input caused by b will resemble the input caused by c more than the input caused by d. Similarly, the input caused by b will resemble the input caused by d more than that caused by e and so on. Retrieved temporal context enables TCM to place the input vectors caused by each item in the proper global relationship to each other. Rao and Howard (2008) showed that TCM with retrieved context
54
(A)
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
(B)
!/!!
34*,%)(*56)&
!/!" !/!# !/!$ !/!2 !/!1 !/!0
!
" # %&'()*+),-.)&
$
Fig. 2. Contextual retrieval enables the extraction of global structure from isolated episodes. (A) Miniature example of a small-world network with connectivity chosen according to the structure of English as estimated by Steyvers and Tenenbaum (2005). (B) The cue strength between pairs chosen from the small-world network are shown as a function of the shortest path between the items. Filled symbols show TCM with contextual retrieval. The open symbol shows the value for TCM without contextual retrieval. Only one point is shown for the model without contextual retrieval because the cue strength is zero for items not presented as part of the same pair.
can not only learn a one-dimensional topology, the ring, but also a two-dimensional topology in which items form a sheet, and more realistic topologies corresponding to naturally occurring language. Fig. 2A shows a miniature version of a small-world network (Strogatz, 2001; Watts & Strogatz, 1998) used to train the model. The network was generated with 10,000 nodes (500 are shown in Fig. 2A) with connectivity comparable to that of the English language, as estimated by the network analysis of WordNet performed by Steyvers and Tenenbaum (2005). We trained TCM on pairs chosen by selecting nodes connected by an edge of the graph. Fig. 2B shows the cue strength between items4 as a function of the length of the shortest path between them in the network. Note that pairs with a value of shortest path greater than one were never presented together during training. Nonetheless, the model correctly describes the distances among items from remote areas of the network. Further, this behavior depends on contextual retrieval—the cue strength is zero for remote items if contextual retrieval does not take place (open symbols). 1.3. pTCM: Learning structure by predicting the future We have seen that contextual retrieval enables TCM to discover latent structure from presentation of isolated stimulus events and integrate them into a global representation. We
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
55
have also seen that the model can learn complex topologies believed to underlie natural language. This seems like it might be sufficient to construct a model of semantic structure. Our initial strategy was to take TCM as just described and present it with a very long sequence of natural language and evaluate the model’s behavior. As it turns out, this is a deeply theoretically unsatisfactory model.5 The reason turns out to be that, unlike the artificial examples explored above, proximity in natural language is not a strong proxy for similarity. Consider the meaning we would learn for a novel word presented in the following sentence ‘‘The baker reached into the oven and pulled out the FLOOB.’’ What is the meaning of FLOOB? In TCM, the representation of FLOOB would be updated to include information from the preceding context; that is, FLOOB would become similar to the representation of ‘‘out,’’ ‘‘pulled,’’ ‘‘oven,’’ ‘‘baker,’’ etc. While it is reasonable to infer that a FLOOB has something to do with those words, it is not at all the case that FLOOB is synonymous with the preceding context. If FLOOB were synonymous with its predecessors, it would be redundant and there would be no purpose to use the word FLOOB in that context. A much more natural way to describe the meaning of FLOOB would be to make FLOOB similar to the words that would have fit into that temporal context, for instance ‘‘cake’’ or ‘‘bread’’ (Dennis, 2005). Fig. 3 illustrates this problem more concretely. We trained TCM with a set of sentences generated by the simple language generator program (SLG, Rohde, 1999) using a simple grammar previously used in a connectionist simulation of language acquisition (Borovsky & Elman, 2006). The SLG generated a set of sentences from words drawn from several categories of nouns (e.g., animals, people) and verbs (e.g., perception, action) subject to both syntactic and semantic constraints (examples can be found in Fig. 3A). Fig. 3B reflects the IN semantic space generated from TCM. More explicitly, we calculated tIN a !tb between different words and aggregated the results according to their category relationships. As shown by Fig. 3, words become similar to the words that precede them; because the sentences all have either a N-V-N or a N-V structure, nouns become similar to verbs and vice versa. pTCM (Shankar et al., 2009) builds on the framework of TCM (Table 2). Just like TCM, it uses a representation of temporal context that changes gradually over time. Similarly, context is used to cue items and the input to context is caused by items. However, in pTCM, the context is used as a cue not only when items are to be recalled, but also at each time step to create a prediction about what will happen next (Property 2, Table 2). The semantic representation of an item is composed of the prediction vectors in which the word is experienced over time. This semantic representation for each word becomes part of the input to the temporal context vector when that word is presented. A more formal definition follows in the next subsection, which can be omitted by the reader not interested in the mathematical details of the model’s operation. Before that, we briefly demonstrate that the adjustments present in pTCM enable us to solve the problem of learning semantic representations from sequentially organized materials. Fig. 3C shows the results of the simulation with the SLG conducted with pTCM. In pTCM, the representations of words become similar to other words from the same category (dark boxes along the diagonal). To the extent that there is residual similarity across categories, it respects the part of speech of the words. For instance, the shaded box in the upper right of Fig. 3C reflects the fact that verbs are more similar to other verbs than they are to
3
N_humans
4
N_objects N_places
VI_CoS VI_Communication VI_Motion
5 6 7 8 Categories VT_Eating
VT_Perception
9 10 11
VT_Action
1
1
lion eats bread. 2
2
N_food
N_food
2
sandwich drops.
1
3
3
1
N_animals
2
N_food
3
N_humans
4
N_objects
N_places
VI_CoS VI_Communication VI_Motion
5 6 7 8 Categories
VT_Eating
VT_Perception
9 10 11 VT_Action
Fig. 3. Predictive temporal context model (pTCM) as a model of semantic learning. (A) Sample sentences generated by the simple language generator. (B–C) Similarity between the representations of words belonging to each category of the simple language. Dark boxes correspond to high similarity. The similarity between each word and itself is excluded from this plot. (B) Category structure for temporal context model (TCM) after being trained on sentences sampled from the simple language. (C) Same as (B), but for pTCM. Unlike TCM, pTCM has learned the category structure underlying this simple language.
N_animals
N_animals
N_humans
N_humans
animal smells juice.
N_food
N_objects
4
N_places
5
6
VI_CoS
7
VI_Communication
VI_Motion
8
VT_Action
9
VT_Eating
10
11
VT_Perception
4
N_animals
C
N_objects
N_places
5
6
VI_CoS
7
VI_Communication
VI_Motion
8
VT_Action
9
VT_Eating
10
11
VT_Perception
lamb sees pizza.
mouse walks.
rabbit walks.
mother drinks juice.
dad gobbles fruit.
rabbit enters room.
sister sits.
B
Categories
A
tiger walks.
Categories
56 M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
57
Table 2 Principles of operation of the predictive temporal context model (pTCM). Compare to Table 1 1. Temporal context changes gradually over time 2. Items are cued by a state of context to the extent it overlaps with their encoding context; cuing at each time step of learning yields a prediction 3. Presentation of items causes a change in the state of context partially driven by the stored semantic representations 4. Repeated/recalled items can recover the state of context in which they were previously studied 5. The semantic representation of an item is composed of the prediction vectors that obtained when it was presented
nouns. This ability to simultaneously capture syntactic and semantic roles is common to the simple recurrent network (Elman, 1990) and the syntagmatic-paradigmatic model (Dennis, 2004, 2005). 1.3.1. Formal description of pTCM Let us describe the process of computing the prediction vector and exploiting this information to develop a semantic representation more formally. The prediction vector at time step i, pi, is calculated using pi ¼ Mti ;
ð5Þ
where M differs from the matrix in Eq. 2 by virtue of being row-normalized. The vector pi can be thought of as the prediction for what item will be presented at time step i + 1. It has been proven that for bigram languages this prediction can be perfect (Shankar et al., 2009). Each word a in the language is associated with a semantic representation sa that is built up from the prediction vectors available when the item is presented. If word a is presented at time step i, then sa is updated such that the change in sa is given by: Dsa ¼ pi$1 :
ð6Þ
Finally, the semantic vector contributes to the input pattern (tIN) to context when the corresponding item is presented. If item a is presented at time step i, then the input pattern tIN i is given by sa : tIN i ¼ ð1 $ /Þfa þ /^
ð7Þ
In principle, we could consider a combination of Eq. 3 with Eq. 7, using both c and / to generate tIN i . Because our focus in this paper is on learning semantic representations and not on episodic recall, we set c = 0 and ignore contextual retrieval. Shankar et al. (2009) demonstrated several useful results regarding the behavior of pTCM using toy languages (even simpler than the SLG) to be able to quantitatively compare the model’s behavior to the generating function of the language. One key finding is that / enables the model to generalize across items that have similar roles in the language in much the same way that c enables TCM to generalize across contiguity relations among items. In addition, Shankar et al. (2009) derived an expression that enables one to calculate the steady
58
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
state behavior of the model in a much more computationally efficient way. In pTCM, calculation of the p vector at each time step requires a matrix multiplication. Hence, pTCM is much more computationally intensive than TCM. The expression for the steady-state behavior of the model exploits the somewhat surprising fact that at steady-state the semantic representations can be calculated just with knowledge of M. Similarly, the steady state M can be calculated directly if the semantic representations are known. Shankar et al. (2009) proved that this steady state approximation precisely describes the simulation model’s behavior with a sufficiently long training set and also closely approximated the simulation model’s behavior with shorter training sequences. An important point to note is that pTCM is history dependent. That is, the effect of being trained on a particular sequence early in training is different from the effect of being trained on the same sequence later in training. If the model is being used to estimate a language with a constant generating function, this property is a nuisance. The approximation can be thought of as the ‘‘historyindependent’’ model that would result from averaging over all possible sequences that could lead to the same initial estimate of M.
2. pTCM as a model of natural language processing The foregoing results suggest that pTCM ought to be able to learn semantic structure from natural language. In order to test this, we trained pTCM on the TASA corpus and examined the model’s behavior on a synonym test and a free association test. Because of the size of the corpus, it was not practical to run the entire simulation model. Instead, we used the history-independent steady state approximation of the simulation model (Shankar et al., 2009). 2.1. Simulation methods In order to test the applicability of the model to real-life language acquisition, we trained pTCM on a widely used corpus of the English language—the Touchstone Applied Science Associates (TASA) college level corpus. The TASA corpus contains about 11 million words across 37,000 documents and 93,000 unique words. To preprocess the corpus, we stripped it of punctuation, numbers, very frequent and commonly occurring words (including function words like ‘‘a,’’ ‘‘the,’’ etc.), and words that occurred in fewer than three documents and fewer than 10 times in the whole corpus. This resulted in a reduced corpus of about 5 million tokens and 48,000 unique words. The context vector was set to zero at the beginning of every paragraph. The sentence separators, on the other hand, were treated as being equivalent to distractor tasks, and q was changed transiently during the transition from one sentence to the next, assuming a value of qD between sentences. The computation time for the steady-state approximation was sped up further by writing a parallel sparse implementation using the message passing interface (MPI) library in C++. Throughout the simulations described here, we set the sparsity threshold to 10)5. However, the amount of time required to run the approximation on a dual Xeon quad core (3.16 GHz)
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
59
machine with 8 Gb of RAM made it impractical to evaluate the model many times with varying parameters. To reduce the processing time further, we collapsed a large number of low-frequency words remaining in the preprocessed tokens into a single token. This token was treated differently from the others in that it did not have an entry in M or a semantic representation s. The effect of our treatment was such that when this special token was encountered in the corpus, the context vector was multiplied by q, but no other action was taken. After reducing the number of words to 10,152 in this way, calculating the model on the corpus with a single set of parameters took about 16 h. We evaluated the model parameters based on the model’s ability to describe the semantic structure of English. For this, we assembled a pool of cue words such that: 1. Each word was normed in the USF free association database (Nelson, McEvoy, & Schreiber, 2004). 2. Each word was present in our reduced corpus. 3. Each word had a synonym as evaluated by WordNet. 4. Each word’s first associate was present in our reduced corpus. 5. Each word’s first listed synonym was present in our reduced corpus. There were 1,040 such words. These had 591 unique synonyms and 583 unique first associates. In order to evaluate the models’ ability to describe performance on the synonym test in a fair manner, it was necessary to find a nonparametric measure of the degree to which the model captures the structure of the space. We arbitrarily separated the synonym pairs into a set of cues and a set of targets. For each cue, we calculated its inner product with the semantic representation of each of the targets, and retained the rank of the correct target relative to the other targets in the set. Ties were addressed by taking the mean rank of the values tied with that of the target. The distribution of ranks was retained and the mean log rank on the synonym test was minimized. The results presented in this paper are based on the parameters for which the mean log rank is minimal. An analogous procedure, wherein the mean log rank on the free association test is minimized, can be adopted to evaluate the model’s parameters based on the models’ performance on the free association test. In this paper, we do not report results from these parameters. We computed ranks for pTCM using two measures of similarity between word a and b. One measure, which we refer to as the pTCM free associate strength, is constructed by taking the input pattern caused by item a, tIN a multiplying it by the matrix M and measuring the degree to which word b is present in the output.6 This is analogous to presenting item a, allowing it to provide input to the state of temporal context, and seeing to what extent b is predicted. The other measures the inner product of IN tIN a , the input pattern caused by item a, with tb , the input pattern caused by item b. We refer to this latter measure as the pTCM vector space model. The time necessary to compute pTCM on the corpus precluded a thorough search of the parameter space. We adopted the strategy of attempting to search the parameter space retaining 3,000 dimensions, then evaluating the best-fitting parameters for the model retaining 10,152 dimensions. We used a downhill simplex algorithm to minimize performance on a variety of synonym tests; these ultimately did not completely converge to a solution and
60
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
we took the most satisfactory solution that was available. We evaluated the simulation model with the same parameters and 3,000 dimensions. However, optimization of the simulation model was not practical and we only report results from the approximation. In order to compare pTCM to a version of LSA trained on the same inputs, we computed an LSA solution on the reduced corpus. This corpus did not include the stop words, short words, and extremely infrequent words omitted at the parsing stage, but did include the words collapsed into a single string. We varied the number of dimensions retained in steps of 50 to find the value that resulted in the best performance (as measured by mean rank) on our synonym test. This value was 800. One might argue that our use of an impoverished version of LSA is somewhat unfair to that method—unlike pTCM, LSA is not subject to computational limitations that make it impractical to run on the entire corpus. For this reason, we also calculated results for LSA calculated on the entire corpus with 300 dimensions. This calculation was done with the same in-house LSA implementation we used to compute the reduced corpus. The results of this calculation were checked against the SEMMOD package (Stone, Dennis, & Kwantes, 2008). 2.2. Results The best-fitting value of q, 0.68 was much greater than zero, indicating that multiple preceding content words contributed to the model’s estimate of temporal context. The value of qD describing the rate of contextual drift across sentence boundaries was also much greater than zero, indicating that information across sentence boundaries contributed positively to model performance. Critically, the best-fitting value of /, 0.41 was greater than zero, indicating that the ability to generalize across experiences was important to the model’s performance. We found that a broad variety of values of / yielded roughly similar ranks on the synonym test as long as the value of / did not approach zero or one, at which point performance fell off dramatically. Fig. 4A shows the cumulative distribution of ranks for the pTCM vector space model (dark blue) and LSA (light red) when both are trained on the reduced corpus. The graph gives the cumulative probability that the similarity of the cue word’s synonym obtained a given rank relative to the other targets. Good performance is reflected as a large number of low ranks, which means that the cumulative probability increases at low ranks. Put another way, good performance is manifest as a higher line in Fig. 4A. As can be seen from Fig. 4A, the distribution of ranks generated by the pTCM vector space model for synonyms was robustly lower than the ranks generated by LSA when they were both trained on the reduced corpus. Fig. 4 compares performance on the synonym test for the pTCM vector space model trained on the reduced corpus (dark blue) to LSA trained on the entire corpus (light red). Although the performance of the two models appears very similar, it can be established statistically that LSA trained on the entire corpus outperforms pTCM trained on the reduced corpus. For instance, the rank on the synonym test was lower for LSA trained on the entire corpus for 551 synonyms, whereas pTCM trained on the reduced corpus only produced a lower rank for 461 synonyms (p < .005 by the binomial distribution; there were 28 ties). The results of the analysis of the synonym test indicate that pTCM trained on the reduced
B
Cumulative Probability
0.0 0.2 0.4 0.6 0.8 1.0
Cumulative Probability
A
0
100
300 Rank
500
61
0.0 0.2 0.4 0.6 0.8 1.0
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
0
100
300
500
Rank
Fig. 4. Predictive temporal context model’s (pTCM’s) vector space model performs comparably to latent semantic analysis (LSA) on a synonym test. A set of 1,040 synonym pairs was assembled. For each model, we calculated the similarity between a word and its synonym and expressed that as a rank relative to all the other words’ synonyms. Low ranks indicate the model is doing well at placing words with similar meanings in similar locations. (A) Cumulative probability distribution of the rank of the synonym for the pTCM vector space model (dark blue) and LSA trained on the same words (light red). The higher curve indicates a larger proportion of low ranks, and thus better model performance. pTCM shows a marked improvement over LSA trained on the same words. (B) Same as (A), but comparing pTCM trained on the reduced corpus (dark blue) to LSA trained on the entire corpus (light red). Despite the fact that pTCM only had detailed information about 10,000 words (as opposed to 93,000 for LSA), there are relatively modest differences between the performance of pTCM and LSA.
corpus (approximately 10,000 unique words) outperforms LSA trained on the same corpus, and comparable, although slightly worse, to LSA when it was trained on the entire corpus (approximately 100,000 unique words). Performance by pTCM on the synonym test was moderately correlated with performance by LSA. The correlation across pairs between the rank assigned to synonyms by pTCM and by LSA trained on the reduced corpus was 0.56. The correlation between pTCM and LSA trained on the entire corpus was 0.69. However, both of these numbers were reliably less than the correlation between LSA trained on the reduced corpus and LSA trained on the entire corpus, 0.74. Note that although this comparison led to the highest correlation, it also corresponded to the largest difference in performance. We obtained comparable results for the free associate test. First, Fig. 5A shows the cumulative probability functions for the distribution of ranks of the first free associates for the pTCM vector space model (light red) and the pTCM free associate model in which the semantic representation of the cue item is used to predict the subsequent item (dark blue). There is a strong advantage for the pTCM free associate model over the pTCM vector space model in modeling free associates. Fig. 5B shows the cumulative probability distribution for the pTCM free associate model trained on the reduced corpus (dark blue), LSA trained on the reduced corpus (light red) and LSA trained on the entire corpus (lighter green). As with the comparison with the synonym test, pTCM produced reliably lower ranks than LSA when they were both trained with the reduced corpus. As with the synonym test, when LSA is trained on the entire corpus, there is a small but reliable advantage over pTCM trained on the
B
Cumulative Probability
Cumulative Probability
A
0
100
300 Rank
500
0.0 0.2 0.4 0.6 0.8 1.0
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011) 0.0 0.2 0.4 0.6 0.8 1.0
62
0
100
300
500
Rank
Fig. 5. Performance on free association norms. Similarity ratings were evaluated for a list of 1,040 words paired with their strongest associates. The strength of the relationship between the prime and its first associate was calculated and turned into a rank relative to the first associates of the other primes. Cumulative probability distributions of ranks are shown. Lower ranks reflect better model performance, meaning that higher curves reflect better model performance. (A) The cumulative probability distribution of ranks of the first associates for the predictive temporal context model (pTCM) ‘‘recall’’ model (dark blue) and the pTCM vector space model (light red). For the pTCM recall model, the semantic representation of the cue item was used as a cue via the contextto-item matrix. The activation of each target was used to generate its rank. The vector space model simply uses the inner product of the semantic representation of items to generate similarity values. (B) Cumulative probability distributions of ranks for the pTCM recall model (dark blue), latent semantic analysis (LSA) trained on the same words (light red) and LSA trained on the entire corpus (lighter green). pTCM trained on the reduced corpus shows dramatic improvement over LSA when it was trained on the same words. LSA trained on the entire corpus shows a modest improvement over pTCM trained on the reduced corpus.
reduced corpus. For instance, the rank of the first free associate was lower for LSA trained on the entire corpus for 551 cues, whereas pTCM trained on the reduced corpus only produced a lower rank for 388 cues (p < .001 by the binomial distribution; there were 101 ties). On the free associate test, the pTCM free associate model was only moderately correlated with LSA and with the pTCM vector space model, and the pTCM vector space model was more strongly correlated with LSA trained on the entire corpus. The correlation across pairs of the rank assigned by the pTCM free associate model to the first free associate to the rank assigned by the pTCM vector space model was 0.48. The correlations between the pTCM free associate model and LSA trained on the reduced and entire corpus were also moderate, both r ¼ .49. Interestingly, the correlation between the ranks assigned by the pTCM vector space model and LSA trained on the entire corpus were reliably higher, 0.68, and also higher than the correlation between LSA trained on the entire corpus and LSA trained on the reduced corpus, 0.64. There are several conclusions that can be reached from these analyses. First, the two measures derived from pTCM, the vector space similarity and free associate strength, produce somewhat different results. In particular, the vector space model was inferior at modeling human free associate performance (Fig. 5A). For both the synonym and free associate test, pTCM produced a dramatic advantage over LSA when both methods were trained on the reduced corpus. When LSA was trained on all the words in the corpus, approximately
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
63
Table 3 Nearest neighbors to the word ‘‘baker’’ using various measures of semantic similarity pTCM Vector Space
pTCM Free Associate
LSA
Quimby Frits Wiggle Liza Roberts Rogers Miyo Mandy Frances Cass Handing Pooh Jed Gregory Nicky Oswald Zaphod Pippi Gran
Helper Tennessee Cindy Peel Shoemaker Cooper Loaf Rotten Lazy Shop Baked Blacksmith Cakes Onion Dough Novels Baking Huddled Batter
Gaslight Pastry Holmes Sherlock Kendrick Passersby Sirhan Richard Cakes Tarts Humphrey Wallace Dough Hubert Irwin Daley Assasinations Begrimed Leavened
Notes. The first column shows the nearest neighbors in the pTCM semantic space. The second column shows the free associates of ‘‘baker’’ using pTCM. The third column shows the LSA nearest neighbors with pseudodoc weighting from lsa.colorado.edu with words that appeared in the corpus less than or equal to three times removed. The highest ranking word for all three measures was baker, which has been removed from this table. pTCM, Predictive temporal context model; LSA, latent semantic analysis.
100,000 unique words, it produced superior results to pTCM trained on about 10,000 unique words. It is tempting to assume that if pTCM were also provided more information by means of training it on the entire corpus it would dramatically outperform LSA. While this is a possibility, it is possible that information about low-frequency words may function as a source of noise that could reduce pTCM’s performance. Nonetheless, there are clearly qualitative differences between what is being responded to by the different pTCM measures and LSA. Table 3 shows the nearest neighbors of the word baker for the pTCM vector space model, the pTCM free associate model and LSA trained on the entire corpus. Several results can be obtained from examination of Table 3. First, the pTCM vector space model has exclusively identified proper names, with an emphasis on last names (e.g., quimby, frits, and wiggle all appear as last names in the TASA corpus). The vector space model ultimately rates as similar words that occur in similar contexts. In the TASA corpus, proper names often occur in similar roles in descriptions of conversations, as well as in the context of first names. The pTCM free associate measure of baker generates words with a variety of origins that can be understood from examination of the TASA corpus. For instance, the presence of
64
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
tennessee in this list is due to several passages that discuss former Senator Howard Baker of Tennessee. The presence of cindy in the list is attributable to a single passage in which a student (Cindy) has a conversation with her teacher (Mr. Baker). A majority of the entries for baker are related to the baking profession (e.g., loaf, shop, baked). In most cases, the pTCM free associate measure produces words that appear in close proximity to baker in the corpus. In contrast, LSA’s responses are grouped according to several broad themes that occur in the corpus. One easily identifiable theme are words related to the profession of baking (e.g., pastry, cakes, tarts, dough). The documents describing former Senator Howard Baker give rise to several near-neighbors that are related to politics and news in the late sixties and early seventies (e.g., sirhan, humphrey, wallace, daley, assasinations). In addition, multiple LSA near-neighbors are related to passages describing the fictional detective Sherlock Holmes (e.g., gaslight, holmes, sherlock, begrimed), who lived on Baker Street in London. Although these measures provide comparable performance on synonym tests (Fig. 4B) and free associate tests (Fig. 5B), Table 3 suggests that the measures accomplish this level of performance by responding to different sources of information. Examination of Table 3 relects the fact that LSA responds to thematic information available at the level of the document. The vector space model of pTCM responds by rating words that are preceded by similar temporal contexts as similar to one another. The pTCM free associate measure rates as similar words that occur in close temporal proximity to one another. The fact that these different sources of information lead to similar performance suggests the possibility that these measures could be combined to provide a metric more useful than any one of the measures taken in isolation.
3. General discussion Previous work on TCM demonstrated that gradually changing temporal context can provide a good account of recency and contiguity effects observed in episodic memory (Howard & Kahana, 2002; Howard et al., 2008b; Polyn et al., 2009a; Sederberg et al., 2008). Here we discuss efforts to construct a model of semantic memory within the same framework. Contextual learning enables generalization across multiple experiences that lack overlapping elements placing stimuli into the correct global arrangement (Howard et al., 2009; Rao & Howard, 2008). pTCM (Shankar et al., 2009) enables generalization to take place in sequentially organized materials, such as natural language. We then showed several demonstrations that pTCM can be applied to natural language by training the model on a reduced version of the TASA corpus. pTCM dramatically outperformed LSA trained on the same reduced corpus and produced results close, although statistically inferior, to those of LSA when LSA trained on the entire corpus. The point of this exercise was not to claim that pTCM is of superior practical utility than LSA or other computational models of semantic memory at this time. At present, the computational demands of pTCM and the necessity of fitting multiple parameters make it
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
65
unwieldy for many applications. However, because of its tight coupling with a successful model of episodic recall, it has theoretical advantages over other existing computational semantic memory models. The fact that a common description of temporal context can be used as a core concept of both a model of episodic memory performance and a model of semantic memory acquisition suggest that these concepts can form the basis of a common account of declarative memory. pTCM may be uniquely suited for describing the process of learning and memory retrieval that combines both semantic and episodic information. 3.1. Where is pTCM in the space of computational models of semantic learning? A natural question to ask is how pTCM relates to extant computational models of semantic memory. Here we briefly discuss the commonalities and differences between pTCM and several widely used models. 3.1.1. HAL The hyperspace analogue to language (HAL, Lund & Burgess, 1996) model uses a semantic representation that codes each word as the vector of counts of other words in the language appeared in a moving context window with that word. There are numerous similarities between HAL and pTCM. These include the fact that more recent words within a document contribute more strongly to the meaning of a word (this property is not shared with LSA or the topic model). In HAL, a word recovers the set of words that occurred nearby during learning—a process similar to retrieval of temporal context. This property enables HAL, along with the other models considered here, to account for transitive associations among items that did not co-occur in the corpus. One of the differences between HAL and pTCM is that the range over which temporal context is defined in pTCM can be quite long in pTCM—many prior words can contribute to the context. Although it remains to be seen how the ‘‘tail’’ of this distribution contributes to the semantic representations obtained from natural language, it is worth noting that the best-fitting parameters obtained here indicate that performance was optimal when many prior items contributed. A quick calculation reveals that in our natural language simulations as many as 27 prior items could have contributed to the context vector before passing under the sparsity threshold with the parameters used.7 This difference between HAL and pTCM is perhaps analogous to the distinction between buffer models of short-term memory (e.g., Atkinson & Shiffrin, 1968; Raaijmakers & Shiffrin, 1980) and TCM in understanding episodic free recall. There, the primary advantage of gradually changing temporal context over buffers with finite range is that temporal context can provide a more natural account of recency and contiguity accounts that extend over multiple time scales (see Howard, Kahana, & Sederberg, 2008a; Kahana, Sederberg, & Howard, 2008b; Sederberg et al., 2008; Usher, Davelaar, Haarmann, & Goshen-Gottstein, 2008, for a thorough discussion of the relationship between TCM and buffer models). The other major point of distinction between HAL and pTCM is that in HAL there is no generalization across word meaning during learning.
66
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
3.1.2. BEAGLE In BEAGLE (Jones, Kintsch, & Mewhort, 2006; Jones & Mewhort, 2007), each word is initially assigned a random vector of some length (Jones & Mewhort, 2007 used vectors of dimension 2,048). During learning, an item, or semantic, representation and an order representation is aggregated for each word. This distinction between item and order information, as well as much of the mathematical machinery of BEAGLE, is inherited from the TODAM model of episodic memory tasks (Murdock, 1982, 1997). In BEAGLE, the semantic representation is formed by summing the vectors of the other words that co-occur in the same sentence. The order representation is used by constructing an N-gram convolution between successive words. The pTCM has many commonalities with BEAGLE. BEAGLE’s contextual representation can be understood as analogous to the average prior context in which an item is presented (i.e., it is similar to h if it were averaged over all prior presentations of the item). If it were possible to set c to zero during learning, but non-zero during retrieval, the h that would result would be very similar to the context representation in BEAGLE. In both models, there is a natural model of free association that emerges—in BEAGLE this is taken from the order representation used to support the cued recall task (Murdock, 1982). There are, however, important differences. As discussed above, context in pTCM changes continuously and persists across sentence boundaries, allowing for long-range contingencies between words.8 In contrast, BEAGLE stops at sentence boundaries. The major difference between the models is that in pTCM the representation of a word that is used to make subsequent predictions changes during learning. BEAGLE relies on statistical averaging of the initial word vectors to build up the representations. In pTCM, the changing semantic representation of a word contributes to the temporal context vector, so that all of the information that has been learned up to that point can be brought to bear in making a prediction and thus updating the representation of the other items in the language. This may result in more robust generalization during learning. 3.1.3. LSA Latent semantic analysis (Landauer & Dumais, 1997) has set a standard for computational models of semantic knowledge for more than a decade. It has been successful in a broad range of applied settings and has shed considerable light on the basis of knowledge formation in a theoretical sense. Although the end-state of learning in pTCM and LSA are similar to some extent (e.g., Fig. 4B), pTCM and LSA are conceptually very different. The pTCM is a learning model in which information is gradually built up over experience. In contrast, the algorithm of LSA requires that all experience be accessible prior to calculating the semantic representation. LSA is consistent with a representation of temporal context that changes abruptly between documents but does not change at all within a document. The parameters of pTCM are sufficiently flexible to approximate a very slowly changing context vector by setting q.1 and qD ¼ 1. The best-fitting parameters were far from these values, suggesting that the there are meaningful changes in temporal context within a document. As mentioned previously, although a vector space can be extracted from
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
67
pTCM, this is not the only, or even necessarily the best, representation of meaning possible within pTCM. The free associate measure is not subject to the constraints of a vector space. For instance, the associative strength between two words is asymmetric and can violate the triangle inequality. With all these differences in mind, it is remarkable that the end-state of pTCM is as similar to that of LSA as it is. 3.1.4. The topic model The probabilistic topic model (Griffiths et al., 2007), like LSA, starts with a word-bydocument co-occurrence model. It makes the assumption that the words in a document are sampled from mixtures across a latent variable referred to as a topic. The model constructs a prior on the degree of mixing of topics in a given document, then estimates the probability of sampling a word given each topic using latent Dirichlet allocation (Blei, Ng, & Jordan, 2003). The distribution of words across topics gives an estimate of their meaning. Many of the points of contrast between pTCM and the topic model are the same as those with LSA: The construction of topics makes the assumption that meaning does not change within a document, the topics calculation is taken after study of the entire corpus. Both pTCM and the topic model have a natural account of retrieval from memory, although in pTCM’s case this is embedded more strongly in a model of episodic retrieval. The primary advantage of the topic model over pTCM is its natural treatment of polysemy, which does not currently have an analog in pTCM. 3.1.5. The syntagmatic-paradigmatic model The syntagmatic-paradigmatic model (SP, Dennis, 2004, 2005) attempts to automatically extract knowledge from naturally occurring text by using training exemplars to mutually satisfy syntagmatic and paradigmatic constraints. Syntagmatic association are formed between words that occur in series in language—for instance, run and fast. In contrast, paradigmatic associations are formed between words that have similar meaning—or that fulfill similar roles in language—for example, run and walk. SP utilizes both types of associations to model the generation of language. In pTCM, paradigmatic associations are analogous to those constructed using the vector space representation. Paradigmatic associates are words that fit into similar contextual roles and thus have similar semantic representations in pTCM. Syntagmatic associates also have an analog in pTCM. Given a semantic representation of an item a, when multiplied by M, this gives the set of items that are predicted to follow a based on experience with the corpus (see Table 3). A single matrix, however, cannot capture the rich syntactic structure of English as may be possible with the SP model. 3.2. Does pTCM provide additional insight into the neural basis of memory? One of the strengths of TCM as a model of episodic memory is the existence of a linking hypothesis between the structures of the model and physical processes that take place in the medial temporal lobe of the brain (Howard et al., 2005). This linking hypothesis has led to predictions about the behavior of brain states that have been confirmed with measurements from neural ensembles (Manns, Howard, & Eichenbaum, 2007). The confirmation of these
68
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
neurophysiological predictions, coupled with the confirmation of numerous behavioral predictions (Howard et al., 2007, 2008b; Howard et al., 2009; Polyn et al., 2009a; Polyn, Norman, & Kahana, 2009b; Schwartz, Howard, Jing, & Kahana, 2005; Unsworth, 2008) make TCM a strong model of episodic recall. Although our understanding of pTCM is at a much earlier stage, it is possible that the extension to pTCM will enhance the set of neural phenomena that can be addressed in a common cognitive framework. Because pTCM is a superset of TCM (compare Table 1 with Table 2), a linking hypothesis between pTCM and the brain shares many of the same contact points—the context vector should reside in extrahippocampal medial temporal lobe (MTL) regions, especially the entorhinal cortex (see Polyn & Kahana, 2008, for a different hypothesis) and the hippocampus should be responsible for the recovery of temporal context. There are two unique predictions of pTCM. One is that the semantic representation of items should come to reflect the temporal contexts in which they are experienced. The second is that the brain uses the current state of temporal context to generate a prediction about what will happen next. There is neurophysiological evidence that suggests both of these predictions hold. Neurons in area TE of the monkey inferotemporal cortex, a region one synapse away from the MTL, respond to high-level visual stimuli during and following their presentation in a way that is not dependent on their coarse physical properties (Miyashita & Chang, 1988). Remarkably, neurons that respond to a particular stimulus are also more likely to respond to other stimuli that are repeatedly experienced close together in time (Miyashita, 1988) or as members of a bidirectionally presented pair of stimuli that predict one another (Sakai & Miyashita, 1991). Because the neurons are responding to an arbitrary temporal pairing of the stimuli rather than any physical property they have, these findings are as one would expect if the neurons were coding a semantic representation constructed from prediction vectors. This pair-coding phenomenon has been observed both in TE and perirhinal cortex (Erickson, Jagadeesh, & Desimone, 2000; Messinger, Squire, Zola, & Albright, 2001; Naya, Yoshida, & Miyashita, 2003), an extrahippocampal medial temporal lobe region one synapse from the entorhinal cortex. The pair-coding phenomenon also depends on feedback from the medial temporal lobe (Higuchi & Miyashita, 1996; Naya et al., 2003). Both of these properties are as one would expect if the change in the neurons’ responsiveness with experience depended on a prediction generated by a temporal context vector residing in extrahippocampal medial temporal lobe, especially the entorhinal cortex. The other large-scale prediction of pTCM, that the brain generates a prediction about subsequent stimuli based on the current state of temporal context, may also have a neurophysiological analog. The N400 is a negative potential observed most prominently when subjects are perceiving words that are semantically incongruous (Kutas & Hillyard, 1980), that is, ‘‘The baker reached into the oven and pulled out the boat.’’ The N400 is observed to the extent that a word is not well-predicted by its preceding semantic context (Bentin, McCarthy, & Wood, 1985; van Berkum, Hagoort, & Brown, 1999; Federmeier, 2007). Notably, the prediction can be generated both by proximate words (Bentin et al., 1985) and more remote semantic context (van Berkum et al., 1999), suggesting that the prediction is generated across multiple time scales at once. These findings suggest that the N400 could reflect a mismatch between a prediction generated from a temporal context vector and a presented stimulus.
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
69
The identification of these ERPs with the mismatch between a prediction vector and the presented stimulus may facilitate development of another strong link between the mathematical framework of pTCM and MTL physiology. The N400 has a large generator in extrahippocampal MTL cortical regions (McCarthy, Nobre, Bentin, & Spencer, 1995). The N400 may be understood, at least in part, as a modulation of ongoing oscillatory activity in the MTL (Fell et al., 2004). While we do not wish to claim that the MTL generator is the sole source of the scalp ERP, presentation of a stimulus that is poorly predicted by its semantic context apparently has a profound effect on human MTL physiology. Moreover, the N400 in the anterior MTL to a studied stimulus predicts whether that stimulus will subsequently be recalled (Fernandez et al., 1999; similar effects are also recorded at the scalp, see Paller & Wagner, 2002 for a review). The involvement of the N400 in the MTL in both integration with semantic context and episodic memory encoding could eventually lead to a number of interesting constraints on a physical model of declarative memory. 3.3. Episodic memory and semantic memory We have shown that it is possible to build a model of semantic memory acquisition in the same framework occupied by a model of episodic memory. This framework leads to predictions about a tight coupling between episodic and semantic memory that we have not yet explored. For instance, in the simulations of natural language using pTCM we did not allow contextual recovery, a process we believe to be an essential aspect of episodic memory (Howard et al., 2005; Sederberg, Miller, Kahana, & Howard, in press), to take place. One challenge of this future work is to specify the conditions under which episodic recovery succeeds. One intriguing possibility is that words that are poorly predicted by their study context are bound effectively to that context such that they can recover the context in the future. Another challenge is to determine which context is recovered by a word that is experienced multiple times. On the one hand, the function of episodic memory as recall of a specific event situated in a specific spatiotemporal context is defeated if all prior contexts in which a word has been experienced contribute to the context it recovers. On the other hand, simulations of learning double-function lists within a particular experiment suggest a gradual change in the temporal context recovered by an item, reflecting multiple study events in the same experiment (Howard et al., 2009). It is possible that these simulations mistake recovery of temporal context for the buildup of a prediction vector such as that utilized here. These distinctions may be teased apart by future experimentation. The specific integration of semantic memory into the TCM framework offered by pTCM potentially places strong constraints on TCM as a model of episodic recall. Since the earliest treatments of TCM, the input caused by an item when it is initially presented is to be understood as reflecting its prior history. In more recent treatments of TCM, a distinction is made between the preexperimental context-to-item matrix and the newly learned part of the context-to-item matrix which encodes information about the study items’ encoding context (Polyn et al., 2009a; Sederberg et al., 2008). Polyn et al. (2009a) used the preexperimental matrix to carry information about semantic relationships among words which was sufficient to account for the existence of semantic clustering in free recall. In the context of modeling
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
70
episodic memory, pTCM may be understood as a method to initialize the values of the preexperimental context-to-item matrix and the input patterns caused by items when they are initially presented. Taken together, the two models reflect a shared hypothesis about the interaction between semantic and episodic factors on memory retrieval.
Notes 1. See Shankar et al., 2009 for a more complete discussion of this point. 2. The asymmetry observed in the contiguity effect is also explained by the model. This is because, unlike this simplified example, the input pattern caused by pupil when it is repeated also includes the input pattern it caused during study (see Eq. 3). Because this overlaps with the encoding context for words that followed pupil but not those that preceded it, this causes an asymmetry. 3. The results for the topic model were generously provided by Mark Steyvers. 4. More explicitly, f0b MtIN a is the cue strength between item a and item b. 5. Actually, one can get practically useful results out of the model if one allows c to be zero during study but nonzero during retrieval. This representation ends up being similar to the semantic representation in BEAGLE or the vectors of the HAL model. Given TCM, though, this account is theoretically unsatisfactory. If retrieved context is useful, why wouldn’t it be used during the thousands of hours of study that are presumably reflected by the corpus? 6. That is, we compute f0b MtIN a . 7. This is an upper limit calculation that assumes that there are no sentence boundaries in this string of words. 8. This can be seen from the fact that the best-fitting value of qD was not zero. Acknowledgment Supported by NIH grant 1-R01 MH069938 to M.W.H. Thanks to Mark Steyvers who calculated the predictions of the topic model used in Fig. 1C. We thank Vinayak Rao for developing the software for presenting pairs chosen from a small-world network and performing early simulations. Aditya Datey, Hongliang Gai and Aditya Udas provided software support. Udaya Jagadisan is now in the Department of Biomedical Engineering, University of Pittsburgh.
References Anderson, J. R., & Bower, G. H. (1972). Recognition and retrieval processes in free recall. Psychological Review, 79(2), 97–123.
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
71
Atkinson, R. C., & Shiffrin, R. M. (1968). Human memory: A proposed system and its control processes. In K. W. Spence & J. T. Spence (Eds.), The psychology of learning and motivation (Vol. 2, pp. 89–105). New York: Academic Press. Bentin, S., McCarthy, G., & Wood, C. C. (1985). Event-related potentials, lexical decision and semantic priming. Electroencephalography and Clinical Neurophysiology, 60(4), 343–355. van Berkum, J. J. Hagoort, P., & Brown, C. M. (1999). Semantic integration in sentences and discourse: Evidence from the N400. Journal of Cognitive Neuroscience, 11(6), 657–671. Blei, D., Ng, A., & Jordan, M. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022. Borovsky, A., & Elman, J. (2006). Language input and semantic categories: A relation between cognition and early word learning. Journal of Child Language, 33(4), 759–790. Brown, G. D., Neath, I., & Chater, N. (2007). A temporal ratio model of memory. Psychological Review, 114(3), 539–576. Bunsey, M., & Eichenbaum, H. B. (1996). Conservation of hippocampal memory function in rats and humans. Nature, 379(6562), 255–257. Dennis, S. (2004). An unsupervised method for the extraction of propositional information from text. Proceedings of the National Academy of Sciences of the United States of America, 101(Suppl. 1), 5206–5213. Dennis, S. (2005). A memory-based theory of verbal cognition. Cognitive Science, 29, 145–193. Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14, 179–211. Erickson, C. A., Jagadeesh, B., & Desimone, R. (2000). Clustering of perirhinal neurons with similar properties following visual experience in adult monkeys. Nature Neuroscience, 3(11), 1143–1148. Estes, W. K. (1955). Statistical theory of spontaneous recovery and regression. Psychological Review, 62, 145– 154. Federmeier, K. D. (2007). Thinking ahead: The role and roots of prediction in language comprehension. Psychophysiology, 44(4), 491–505. Fell, J., Dietl, T., Grunwald, T., Kurthen, M., Klaver, P., Trautner, P., Schaller, C., Elger, C. E., & Ferna´ndez, G. (2004). Neural bases of cognitive ERPs: More than phase reset. Journal of Cognitive Neuroscience, 16(9), 1595–1604. Fernandez, G., Effern, A., Grunwald, T., Pezer, N., Lehnertz, K., Du¨mpelmann, M., Van Roost, D., & Elger, C. E. (1999). Real-time tracking of memory formation in the human rhinal cortex and hippocampus. Science, 285, 1582–1585. Griffiths, T. L., Steyvers, M., & Tenenbaum, J. B. (2007). Topics in semantic representation. Psychological Review, 114(2), 211–244. Higuchi, S., & Miyashita, Y. (1996). Formation of mnemonic neuronal responses to visual paired associates in inferotemporal cortex is impaired by perirhinal and entorhinal lesions. Proceedings of the National Academy of Sciences of the United States of America, 93(2), 739–743. Howard, M. W., & Kahana, M. J. (2002). A distributed representation of temporal context. Journal of Mathematical Psychology, 46(3), 269–299. Howard, M. W., Fotedar, M. S., Datey, A. V., & Hasselmo, M. E. (2005). The temporal context model in spatial navigation and relational learning: Toward a common explanation of medial temporal lobe function across domains. Psychological Review, 112(1), 75–116. Howard, M. W., Kahana, M. J., & Wingfield, A. (2006). Aging and contextual binding: Modeling recency and lag-recency effects with the temporal context model. Psychonomic Bulletin & Review, 13, 439–445. Howard, M. W., Venkatadass, V., Norman, K. A., & Kahana, M. J. (2007). Associative processes in immediate recency. Memory & Cognition, 35, 1700–1711. Howard, M. W., Kahana, M. J., & Sederberg, P. B. (2008a). Postscript: Distinguishing between temporal context and short-term store. Psychological Review, 115, 1125–1126. Howard, M. W., Youker, T. E., & Venkatadass, V. (2008b). The persistence of memory: Contiguity effects across several minutes. Psychonomic Bulletin & Review, 15, 58–63.
72
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
Howard, M. W., Jing, B., Rao, V. A., Provyn, J. P., & Datey, A. V. (2009). Bridging the gap: Transitive associations between items presented in similar temporal contexts. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 391–407. Jones, M. N., & Mewhort, D. J. K. (2007). Representing word meaning and order information composite holographic lexicon. Psychological Review, 114, 1–32. Jones, M. N., Kintsch, W., & Mewhort, D. J. (2006). High-dimensional semantic space accounts of priming. Journal of Memory and Language, 55, 534–552. Kahana, M. J., Howard, M., & Polyn, S. (2008a). Associative processes in episodic memory. In H. L. Roediger III (Ed.), Cognitive psychology of memory, Vol. 2 of learning and memory – a comprehensive reference (J. Byrne, Editor) (pp. 476–490). Oxford, England: Elsevier. Kahana, M. J., Sederberg, P. B., & Howard, M. W. (2008b). Putting short-term memory into context: Reply to Usher, Davelaar, Haarmann and Goshen-Gottstein (2008). Psychological Review, 115, 1119–1126. Kemp, C., & Tenenbaum, J. B. (2009). Structured statistical models of inductive reasoning. Psychological Review, 116(1), 20–58. Kutas, M., & Hillyard, S. A. (1980). Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207(4427), 203–205. Landauer, T. K., & Dumais, S. T. (1997). Solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104, 211–240. Lund, K., & Burgess, C. (1996). Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments & Computers, 28(2), 203–208. Manns, J. R., Howard, M. W., & Eichenbaum, H. B. (2007). Gradual changes in hippocampal activity support remembering the order of events. Neuron, 56, 530–540. McCarthy, G., Nobre, A. C., Bentin, S., & Spencer, D. D. (1995). Language-related field potentials in the anterior-medial temporal lobe: I. Intracranial distribution and neural generators. The Journal of Neuroscience, 15, 1080–1089. Mensink, G.-J. M., & Raaijmakers, J. G. W. (1988). A model for interference and forgetting. Psychological Review, 95, 434–455. Messinger, A., Squire, L. R., Zola, S. M., & Albright, T. D. (2001). Neuronal representations of stimulus associations develop in the temporal lobe during learning. Proceedings of the National Academy of Sciences of the United States of America, 98(21), 12239–12244. Miyashita, Y. (1988). Neuronal correlate of visual associative long-term memory in the primate temporal cortex. Nature, 335(6193), 817–820. Miyashita, Y., & Chang, H. S. (1988). Neuronal correlate of pictorial short-term memory in the primate temporal cortex. Nature, 331(6151), 68–70. Murdock, B. B. (1982). A theory for the storage and retrieval of item and associative information. Psychological Review, 89, 609–626. Murdock, B. B. (1997). Context and mediators in a theory of distributed associative memory (TODAM2). Psychological Review, 104(2), 839–862. Naya, Y., Yoshida, M., & Miyashita, Y. (2003). Forward processing of long-term associative memory in monkey inferotemporal cortex. Journal of Neuroscience, 23(7), 2861–2871. Nelson, D. L., McEvoy, C. L., & Schreiber, T. A. (2004). The University of South Florida free association, rhyme, and word fragment norms. Behavior Research, Methods Instruments and Computers, 36(3), 402–407. Paller, K. A., & Wagner, A. D. (2002). Observing the transformation of experience into memory. Trends in Cognitive Science, 6(2), 93–102. Polyn, S. M., & Kahana, M. J. (2008). Memory search and the neural representation of context. Trends in Cognitive Science, 12(1), 24–30. Polyn, S. M., Norman, K. A., & Kahana, M. J. (2009a). A context maintenance and retrieval model of organizational processes in free recall. Psychological Review, 116, 129–156. Polyn, S. M., Norman, K. A., & Kahana, M. J. (2009b). Task context and organization in free recall. Neuropsychologia, 47(11), 2158–2163.
M. W. Howard, K. H. Shankar, U. K. K. Jagadisan ⁄ Topics in Cognitive Science 3 (2011)
73
Provyn, J. P., Sliwinski, M. J., & Howard, M. W. (2007). Effects of age on contextually mediated associations in paired associate learning. Psychology and Aging, 22, 846–857. Raaijmakers, J. G. W., & Shiffrin, R. M. (1980). SAM: A theory of probabilistic search of associative memory. In G. H. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory (Vol. 14, pp. 207–262). New York: Academic Press. Rao, V. A., & Howard, M. W. (2008). Retrieved context and the discovery of semantic structure. In J. Platt, D. Koller, Y. Singer, & S. Roweis (Eds.), Advances in neural information processing systems 20 (pp. 1193– 1200). Cambridge, MA: MIT Press. Rohde, D. L. T. (1999). The simple language generator: Encoding complex languages with simple grammars. In Technical Report, CMU-CS-99-123. Pittsburgh, PA: Carnegie Mellon, Department of Computer Science. Sakai, K., & Miyashita, Y. (1991). Neural organization for the long-term memory of paired associates. Nature, 354(6349), 152–155. Schwartz, G., Howard, M. W., Jing, B., & Kahana, M. J. (2005). Shadows of the past: Temporal retrieval effects in recognition memory. Psychological Science, 16(11), 898–904. Sederberg, P. B., Howard, M. W., & Kahana, M. J. (2008). A context-based theory of recency and contiguity in free recall. Psychological Review, 115, 893–912. Sederberg, P. B., Miller, J. F., Kahana, M. J., & Howard, M. W. (in press). Temporal contiguity between recalls predicts episodic memory performance. Memory & Cognition. Shankar, K. H., Jagadisan, U. K. K., & Howard, M. W. (2009). Sequential learning using temporal context. Journal of Mathematical Psychology, 53, 474–485. Slamecka, N. J. (1976). An analysis of double-function lists. Memory & Cognition, 4, 581–585. Steyvers, M., & Tenenbaum, J. (2005). The large scale structure of semantic networks: Statistical analyses and a model of semantic growth. Cognitive Science, 29, 41–78. Stone, B., Dennis, S., & Kwantes, P. J. (2008). A systematic comparison of semantic models on human similarity rating data: The effectiveness of subspacing. In B. C. Love, K. McRae, & V. M. Sloutsky (Eds.), The Proceedings of the Thirtieth Conference of the Cognitive Science Society (pp. 1813–1818). Austin, TX: Cognitive Science Society. Strogatz, S. H. (2001). Exploring complex networks. Nature, 410(6825), 268–276. Tenenbaum, J. B., Griffiths, T. L., & Kemp, C. (2006). Theory-based Bayesian models of inductive learning and reasoning. Trends in Cognitive Science, 10(7), 309–318. Unsworth, N. (2008). Exploring the retrieval dynamics of delayed and final free recall: Further evidence for temporal-contextual search. Journal of Memory and Language, 59, 223–236. Usher, M., Davelaar, E. J., Haarmann, H. J., & Goshen-Gottstein, Y. (2008). Short-term memory after all: Comment on Sederberg, Howard, and Kahana (2008). Psychological Review, 115(4), 1108–1118. Watts, D. J., & Strogatz, S. H. (1998). Collective dynamics of ‘small-world’ networks. Nature, 393(6684), 440–442.
Topics in Cognitive Science 3 (2011) 74–91 Copyright ! 2010 Cognitive Science Society, Inc. All rights reserved. ISSN: 1756-8757 print / 1756-8765 online DOI: 10.1111/j.1756-8765.2010.01109.x
Discovering Binary Codes for Documents by Learning Deep Generative Models Geoffrey Hinton,a Ruslan Salakhutdinovb a Department of Computer Science, University of Toronto Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
b
Received 12 March 2009; received in revised form 8 January 2010; accepted 8 January 2010
Abstract We describe a deep generative model in which the lowest layer represents the word-count vector of a document and the top layer represents a learned binary code for that document. The top two layers of the generative model form an undirected associative memory and the remaining layers form a belief net with directed, top-down connections. We present efficient learning and inference procedures for this type of generative model and show that it allows more accurate and much faster retrieval than latent semantic analysis. By using our method as a filter for a much slower method called TF-IDF we achieve higher accuracy than TF-IDF alone and save several orders of magnitude in retrieval time. By using short binary codes as addresses, we can perform retrieval on very large document sets in a time that is independent of the size of the document set using only one word of memory to describe each document. Keywords: Deep learning; Semantic hashing; Auto-encoders; Restricted Boltzmann machines; Document retrieval; Binary codes
1. Introduction Representing the semantic content of a document is an unsolved problem. We think it is very unlikely that a low-dimensional representation containing only a few hundred numbers will ever be capable of capturing more than a tiny fraction of the content of the distributed representation over millions of neurons that is used by the brain. Even if the documents do lie on (or near) a fairly low-dimensional, nonlinear manifold in a high-dimensional space of word sequences, it is unlikely that the best way to capture the structure of this manifold is by trying to learn explicit coordinates on the manifold for each document. The brain is much Correspondence should be sent to Geoffrey Hinton, Department of Computer Science, University of Toronto, 6 King’s College Rd, Toronto, ON, M5S 3G4 Canada. E-mail:
[email protected]
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
75
more likely to capture the manifold structure implicitly by using an extremely high-dimensional space of distributed representations in which all but a tiny fraction of the space has been ruled out by learned interactions between neurons. This type of implicit representation has many advantages over the explicit representation provided by a low-dimensional set of coordinates on the manifold: 1. It can be learned efficiently from data by extracting multiple layers of features to form a ‘‘deep belief net’’ in which the top-level associative memory contains energy ravines. The low energy floor of a ravine is a good representation of a manifold (Hinton, Osindero, & Teh, 2006). 2. Implicit representation of manifolds using learned energy ravines makes it relatively easy to deal with data that contain an unknown number of manifolds each of which has an unknown number of intrinsic dimensions. 3. Each manifold can have a number of intrinsic dimensions that varies along the manifold. 4. If documents are occasionally slightly ill-formed, implicit dimensionality reduction can accommodate them by using energy ravines whose dimensionality increases appropriately as the allowed energy level is raised. The same approach can also allow manifolds to merge at higher energies. In addition to these arguments against explicit representations of manifold coordinates, there is not much evidence for small bottlenecks in the brain. The lowest bandwidth part of the visual system, for example, is the optic nerve with its million or so nerve fibers and there are good physical reasons for that restriction. Despite all these arguments against explicit dimensionality reduction, it is sometimes very useful to have an explicit, low-dimensional representation of a document. One obvious use is visualizing the structure of a large set of documents by displaying them in a two or three-dimensional map. Another use, which we focus on in this paper, is document retrieval. We do not believe that the low-dimensional representations we learn in this paper tell us much about how people represent or retrieve documents. Our main aim is simply to show that our nonlinear, multilayer methods work much better for retrieval than earlier methods that use low-dimensional vectors to represent documents. We find these earlier methods equally implausible as cognitive models, or perhaps even more implausible as they do not work as well. A very unfortunate aspect of our approach to document retrieval is that we initialize deep autoencoders using the very same ‘‘pretraining’’ algorithm as was used in Hinton et al. (2006). When this algorithm is used to learn very large layers, it can be shown to improve a generative model of the data each time an extra layer is added (strictly speaking, it improves a bound on the probability that the model would generate the training data). When the pretraining procedure is used with a central bottleneck, however, all bets are off. Numerous models for capturing low-dimensional latent representations have been proposed and successfully applied in the domain of information retrieval. Latent semantic analysis (LSA; Deerwester, Dumais, Landauer, Furnas, & Harshman, 1990) extracts
76
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
low-dimensional semantic structure using singular value decomposition to get a low-rank approximation of the word-document co-occurrence matrix. This allows document retrieval to be based on ‘‘semantic’’ content rather than just on keywords. Given some desired dimensionality for the codes, LSA finds codes for documents that are optimal in the sense that they minimize the squared error if the word-count vectors are reconstructed from the codes. To achieve this optimality, however, LSA makes the extremely restrictive assumption that the reconstructed counts for each document are a linear function of its code vector. If this assumption is relaxed to allow more complex ways of generating predicted word counts from code vectors, then LSA is far from optimal. As we shall see, nonlinear generative models that use multiple layers of representation and much smaller codes can perform much better than LSA, both for reconstructing word-count vectors and for retrieving semantically similar documents. When LSA was introduced, there were no efficient algorithms for fitting these more complex models, but that has changed. LSA still has the advantages that it does not get trapped at local optima, it is fast on a conventional computer, and it does not require nearly as much training data as methods that fit more complex models with many more parameters. LSA is historically important because it showed that a large document corpus contains a lot of information about meaning that is relatively easy to extract using a sensible statistical method. As a cognitive model, however, LSA has been made rather implausible by the fact that nonlinear, multilayer methods work much better. A probabilistic version of LSA (pLSA) was introduced by Hofmann (1999), using the assumption that each word is modeled as a single sample from a mixture of topics. The mixing proportions of the topics are specific to the document, but the probability distribution over words that is defined by each topic is the same for all documents. For example, a topic such as ‘‘soccer’’ would have a fixed probability of producing the word ‘‘goal’’ and a document containing a lot of soccer-related words would have a high mixing proportion for the topic ‘‘soccer.’’ To make this into a proper generative model of documents, it is necessary to define a prior distribution over the document-specific topic distributions. This gives rise to a model called ‘‘Latent Dirichlet Allocation,’’ which was introduced by Blei, Ng, and Jordan (2003). All these models can be viewed as graphical models (Jordan, 1999) in which hidden topic variables have directed connections to variables that represent word counts. One major drawback is that exact inference is intractable due to explaining away (Pearl, 1988), so they have to resort to slow or inaccurate approximations to compute the posterior distribution over topics. A second major drawback, that is shared by all mixture models, is that these models can never make predictions for words that are sharper than the distributions predicted by any of the individual topics. They are unable to capture an important property of distributed representations, which is that the broad distributions predicted by individual active features get multiplied together (and renormalized) to give the sharp distribution predicted by a whole set of active features. This intersection or ‘‘conjunctive coding’’ property allows individual features to be fairly general but their joint effect to be much more precise. The ‘‘disjunctive coding’’ employed by mixture models cannot achieve precision in this way. For example, distributed representations allow the topics ‘‘torture,’’ ‘‘deregulation,’’
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
77
and ‘‘oil’’ to combine to give very high probability to a few familiar names that are not predicted nearly as strongly by each topic alone. Since the introduction of the term ‘‘distributed representation’’ (Hinton, McClelland, & Rumelhart, 1986), its meaning has evolved beyond the original definition in terms of set intersections, but in this paper the term is being used in its original sense. Welling, Rosen-Zvi, and Hinton (2005) point out that for information retrieval, fast inference is vital and to achieve this they introduce a class of two-layer undirected graphical models that generalize restricted Boltzmann machines (RBMs; see Section 2) to exponential family distributions, thus allowing them to model nonbinary data and to use nonbinary hidden variables. Maximum likelihood learning is intractable in these models because they use nonlinear distributed representations, but learning can be performed efficiently by following an approximation to the gradient of a different objective function called ‘‘contrastive divergence’’ (Hinton, 2002). Several further developments of these undirected models (Gehler, Holub, & Welling, 2006; Xing, Yan, & Hauptmann, 2005) show that they are competitive in terms of retrieval accuracy to their directed counterparts. There are limitations on the types of structure that can be represented efficiently by a single layer of hidden variables and a network with multiple, nonlinear hidden layers should be able to discover representations that work better for retrieval. In this paper, we present a deep generative model whose top two layers form an undirected bipartite graph (see Fig. 1). The lower layers form a multilayer directed belief network, but unlike Latent Dirichlet Allocation this belief net uses distributed representations. The model can be trained efficiently by using an RBM to learn one layer of hidden variables at a time (Hinton et al., 2006; Hinton, 2007a). After learning the features in one hidden layer, the activation vectors of those features when they are being driven by data are used as the ‘‘data’’ for training the next hidden layer.
2000 T
W 1 +ε 6 Top Layer Binary Codes
128
128
500
RBM
T
W 2 +ε 5
W3
500
500 W3
500
500 T
W2
500
2000
The Deep Generative Model
500 W1
2000
Recursive Pretraining
128 Gaussian Noise
W2
500 T W1
T
W 3 +ε 4
RBM
RBM
Code Layer W 3 +ε 3
500 W 2 +ε 2
500 W 1 +ε 1
2000
Fine−tuning
Fig. 1. Left panel: The deep generative model. Middle panel: Pretraining consists of learning a stack of restricted Boltzmann machines (RBMs) in which the feature activations of one RBM are treated as data by the next RBM. Right panel: After pretraining, the RBMs are ‘‘unrolled’’ to create a multilayer autoencoder that is fine-tuned by backpropagation.
78
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
After this greedy ‘‘pretraining’’ is complete, the composition of all of the RBMs yields a feed-forward ‘‘encoder’’ network that converts word-count vectors to compact codes. By composing the RBMs in the opposite order (but with the same weights) we get a ‘‘decoder’’ network that converts compact code vectors into reconstructed word-count vectors. When the encoder and decoder are combined, we get a multilayer autoencoder network that converts word-count vectors into reconstructed word-count vectors via a compact bottleneck. This autoencoder network only works moderately well, but it is an excellent starting point for a fine-tuning phase of the learning which uses back-propagation to greatly improve the reconstructed word counts. In general, the representations produced by greedy unsupervised learning are helpful for regression or classification, but this typically requires large hidden layers that recode the structure in the input as complex sparse features while retaining almost all of the information in the input. When the hidden layers are much smaller than the input layer, a further type of learning is required (Hinton & Salakhutdinov, 2006). After the greedy, layerby-layer training, the deep generative model of documents is not significantly better for document retrieval than a model with only one hidden layer. To take full advantage of the multiple hidden layers, the layer-by-layer learning must be treated as a ‘‘pretraining’’ stage that finds a good region of the parameter space. Starting in this region, back-propagation learning can be used to fine-tune the parameters to produce a much better model. The backpropagation fine-tuning is not responsible for discovering what features to use in the hidden layers of the autoencoder. Instead, it just has to slightly modify the features found by the pretraining in order to improve the reconstructions. This is a much easier job for a myopic, gradient descent procedure like back-propagation than discovering what features to use. After learning, the mapping from a word-count vector to its compact code is very fast, requiring only a matrix multiplication followed by a componentwise nonlinearity for each hidden layer. In Section 2 we introduce the RBM. A longer and gentler introduction to RBMs can be found in Hinton (2007a). In Section 3 we generalize RBMs in two ways to obtain a generative model for word-count vectors. This model can be viewed as a variant of the Rate Adaptive Poisson model (Gehler et al., 2006) that is easier to train and has a better way of dealing with documents of different lengths. In Section 4 we describe both the layer-by-layer pretraining and the fine-tuning of the deep generative model. We also show how ‘‘deterministic noise’’ can be used to force the fine-tuning to discover binary codes in the top layer. In Section 5 we show that 128-bit binary codes are slightly more accurate than 128 real-valued codes produced by LSA, in addition to being faster and more compact. We also show that by using the 128-bit binary codes to restrict the set of documents searched by TF-IDF (Salton & Buckley, 1988), we can slightly improve the accuracy and vastly improve the speed of TF-IDF. Finally, in Section 6 we show that we can use our model to allow retrieval in a time independent of the number of documents. A document is mapped to a memory address in such a way that a small hamming-ball around that memory address contains the semantically similar documents. We call this technique ‘‘semantic hashing’’ (Salakhutdinov & Hinton, 2007).
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
79
2. Learning feature detectors for binary images A Boltzmann machine is a network of symmetrically connected, neuron-like units that make stochastic decisions about whether to be on or off. Boltzmann machines have a simple learning algorithm (Hinton & Sejnowski, 1983) that allows them to discover features that represent complex regularities in the training data. The learning algorithm is very slow in networks with many layers of feature detectors, but it is fast in the RBM—a network with a single layer of feature detectors that are not directly connected to one another. RBMs have been used extensively for modeling binary images and so they will be explained in this context as in Hinton (2007b) before introducing the modifications that are required for modeling documents. If there are no direct interactions between the feature detectors and no direct interactions between the pixels, there is a simple and efficient way to learn a good set of feature detectors from a set of binary training images (Hinton, 2002). We start with zero weights on the symmetric connections between each pixel i and each feature detector j. We then repeatedly update each weight, wij, using the difference between two measured, pairwise correlations Dwij ¼ !ð< vi hj >data # < vi hj >recon Þ
ð1Þ
where ! is a learning rate, < vihj > data is the frequency with which pixel i and feature detector j are on together when the feature detectors are being driven by images from the training set, and < vihj > recon is the corresponding frequency when the feature detectors are being driven by reconstructed images. A similar learning rule can be used for the biases. Given a training image, we set the binary state, hj, of each feature detector to be 1 with a probability given by the logistic sigmoid, r(x) ¼ (1 + exp ()x)))1 ! X ð2Þ pðhj ¼ 1Þ ¼ r bj þ vi wij i2pixels
where bj is the bias of j and vi is the binary state of pixel i. Once binary states have been chosen for the hidden units we produce a ‘‘reconstruction’’ of the training image by setting the state of each pixel to be 1 with probability ! X ð3Þ hj wij pðvi ¼ 1Þ ¼ r aj þ j2features
where ai is the bias of i. The learned weights and biases of the features implicitly define a probability distribution over all possible binary images. Sampling from this distribution is difficult, but it can be done by using ‘‘alternating Gibbs sampling.’’ This starts with a random image and then alternates between updating all of the features in parallel using Eq. 2 and updating all of the pixels in parallel using Eq. 3. After Gibbs sampling for sufficiently long, the network reaches ‘‘thermal equilibrium.’’ The states of pixels and feature detectors still change, but the probability of finding the system in any particular binary configuration does not.
80
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
3. A generative model of word counts The binary stochastic units used in Boltzmann machines can be generalized to ‘‘softmax’’ units that have more than two discrete values (Hinton, 2002). We use an RBM in which the hidden units are binary, but each visible unit has as many different values as there are word types. We represent the mth discrete value of visible unit i by a vector containing a 1 at the mth location and zeros elsewhere. Each hidden unit, j, then has many different weights connecting it to each visible unit, i, and it provides top-down support for the mth value of visible unit i via a weight, wm ij . The activation rule for a softmax unit is !P " m exp h w j j ij !P " pðvm ð4Þ i ¼ 1jhÞ ¼ P k exp h w j ij k j
where the superscript m is used to denote one of the discrete values of i and k is an index over all possible discrete values. Now suppose that for each document we create an RBM with as many softmax units as there are words in the document. Assuming that we are ignoring the order of the words, all of these softmax units can share the same set of weights connecting them to the binary hidden units. The weights can also be shared by the whole family of different-sized RBMs that are required for documents of different lengths. We call this the ‘‘Replicated Softmax’’ model. Using N softmax units with identical weights is equivalent to having one softmax unit which we sample N times. This makes it clear that using N replicated softmax units is equivalent to taking N samples from a multinomial distribution. A pleasing property of softmax units is that the learning rule in Eq. 1 remains the same (Hinton, 2002): m m Dwm ij ¼ !ð< vi hj >data $ < vi hj >recon Þ
ð5Þ
As all of the N softmax units share the same weights, we can drop the subscript on the v and write the learning rule as: m m Dwm j ¼ !ð< Nv hj >data $ < Nv hj >recon Þ
ð6Þ
where vm denotes the count for the mth word divided by N. This model was called the ‘‘constrained Poisson model’’ in Hinton and Salakhutdinov (2006).
4. Pretraining and fine-tuning a deep generative model A single layer of binary features is not the best way to capture the structure in the count data. After learning the first layer of features, a second layer is learned by treating the activation probabilities of the existing features, when they are being driven by real data, as the
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
81
data for the second-level RBM (see Fig. 1). The difference from learning the first layer of features is that the ‘‘visible’’ units of the second-level RBM are also binary, as in a standard RBM. This greedy, layer-by-layer training can be repeated several times to learn a deep, hierarchical model in which each layer of features captures strong high-order correlations between the activities of features in the layer below. Recursive learning of deep generative model: 1. Learn the parameters h1 ¼ (W1,a1,b1) of the replicated softmax model. 2. Freeze the parameters of the replicated softmax model and use the activation probabilities of the binary features, when they are being driven by training data, as the data for training the next layer of binary features. 3. Freeze the parameters h2 that define the second layer of features and use the activation probabilities of those features as data for training the third layer of binary features. 4. Proceed recursively for as many layers as desired. To justify this layer-by-layer approach, it would be good to show that adding an extra layer of feature detectors always increases the probability that the overall generative model would generate the training data. This is almost true: Provided the number of feature detectors does not decrease and their weights are initialized correctly, adding an extra layer is guaranteed to raise a lower bound on the log probability of generating the training data (Hinton et al., 2006). 4.1. Fine-tuning the weights After pretraining, the individual RBMs at each level are ‘‘unrolled’’ as shown in Fig. 1 to create a deep autoencoder. If the stochastic activities of the binary features are replaced by deterministic, real-valued probabilities, we can then backpropagate through the entire network to fine-tune the weights for optimal reconstruction of the count data. For the finetuning, we divide the count vector by the number of words so that it represents a probability distribution across words. Then we use the cross-entropy error function, C, with a ‘‘softmax’’ at the output layer. C¼"
X
m vm data log voutput
m
ð7Þ
The fine-tuning makes the codes in the central layer of the autoencoder work much better for information retrieval. 4.2. Making the codes binary During the fine-tuning, we want backpropagation to find codes that are good at reconstructing the count data but are as close to binary as possible. To make the codes binary, we add Gaussian noise to the bottom-up input received by each code unit. Assuming
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
82 A
B
5
2.5
x 10
5
7
x 10
6 2 5 1.5
4 3
1
2 0.5 1 0 0
0.2
0.4
0.6
0.8
1
0 0
0.2
0.4
0.6
0.8
1
Fig. 2. The distribution of the activities of the 128-code units on the 20 Newsgroup training data before and after adding deterministic noise to the code units. When the fine-tuning of the autoencoder network is performed without adding noise, the code layer learns to transmit a lot of information by using precise real values that lie between 0 and 1. The distribution of values is shown on the left. If these values are thresholded to produce a binary code, the reconstruction will be poor because the fine-tuning did not take the thresholding into account. If additional noise is added to the input to the code units during the fine-tuning, the total input received from the layer below learns to be big and positive or big and negative so that the code unit can still transmit one bit of information despite the noise. This distribution is shown on the right and it makes the codes much more robust to thresholding.
that the decoder network is insensitive to very small differences in the output of a code unit, the best way to communicate information in the presence of added noise is to make the bottom-up input received by a code unit large and negative for some training cases and large and positive for others. Fig. 2 shows that this is what the fine-tuning does. We tried other ways of encouraging the code units to be binary, but Gaussian noise worked better. To prevent the added Gaussian noise from messing up the conjugate gradient fine-tuning, we used ‘‘deterministic noise’’ with mean zero and variance 16. For each training case, the sampled noise values are fixed in advance and do not change during training. With a limited number of training cases, the optimization could tailor the parameters to the fixed noise values, but this is not possible when the total number of sampled noise values is much larger than the number of parameters. 4.3. Details of the training To speedup the pretraining, we subdivided both datasets into small mini-batches, each containing 100 cases,1 and updated the weights after computing the gradient of the reconstruction error on each mini-batch. For large datasets this is much more efficient than using the entire dataset to compute the gradient. Early in the training, the gradient vector computed from a small mini-batch is likely to point in the same general direction as the gradient vector computed from the whole dataset, so progress can be made rapidly by just using one
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
83
small mini-batch per weight update. Of course, there will be sampling noise when the gradient is computed in this way, but that will be corrected on subsequent mini-batches. For both datasets each layer was greedily pretrained for 50 passes (epochs) through the entire training dataset. The weights were updated using a learning rate of 0.1, momentum of 0.9, and a weight decay of 0.0002 · weight · learning rate. The weights were initialized with small random values sampled from a zero-mean normal distribution with variance 0.01. For fine-tuning we used the method of conjugate gradients2 on larger minibatches of 1,000 data vectors, with three line searches performed for each minibatch in each epoch. To determine an adequate number of epochs and to avoid overfitting, we fine-tuned on a fraction of the training data and tested performance on the remaining validation data. We then repeated fine-tuning on the entire training dataset for 50 epochs. Slight overfitting was observed on the 20 Newsgroup corpus but not on the Reuters corpus. After fine-tuning the codes were thresholded to produce binary code vectors. The asymmetry between 0 and 1 in the energy function of an RBM causes the unthresholded codes to have many more values near 0 than near 1, so we used a threshold of s ¼ 0.1. This works well for document retrieval even though it is suboptimal for document reconstruction. We experimented with various values for the noise variance and the threshold, as well as the learning rate, momentum, and weight-decay parameters used in the pretraining. Our results are fairly robust to variations in these parameters and also to variations in the number of layers and the number of units in each layer. The precise weights found by the pretraining do not matter as long as it finds a good region from which to start the fine-tuning (Erhan, Manzagol, Bengio, Bengio, & Vincent, 2009).
5. Experimental results To evaluate performance of our model on an information retrieval task we do the following: Document retrieval procedure: 1. Map all query (test) documents into 128-bit binary codes by performing an up-pass through the model and thresholding top-level activation probabilities at s ¼ 0.1. 2. For each query document: Calculate its similarity to all other test documents in the 128-bit space using Hamming distance. Retrieve the D most similar documents. Measure accuracy by computing the ratio of correctly retrieved documents (which belong to the same category as a query document) to the total number of retrieved documents. 3. Average results across all queries. Results of Gehler et al. (2006) show that pLSA and LDA models do not generally outperform LSA and TF-IDF. Therefore, for comparison, we only used LSA and TF-IDF as
84
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
benchmark methods. For LSA each word count, ci, was replaced by log (1 + ci) before the SVD, which slightly improved performance. TF-IDF computes document similarity directly in the word-count space, which is slow. For both these methods we used the cosine of the angle between two vectors as a measure of their similarity. ‘‘TF-IDF’’ stands for ‘‘Term-Frequency, Inverse-Document-Frequency’’ and it is a very sensible way of deciding how important the counts of a particular word-type are for determining the similarity of two documents. If a particular word has a high count (a high Term Frequency) in both documents this makes them similar. But the similarity is much greater if that word is rare in documents in general (a high Inverse Document Frequency). Generally, the logarithm of the inverse document frequency is used. 5.1. Description of the text corpora In this section we present experimental results for document retrieval on two text datasets: 20-Newsgroups and Reuters Corpus Volume II. The 20 newsgroup corpus contains 18,845 postings taken from the Usenet newsgroup collection. The corpus is partitioned fairly evenly into 20 different newsgroups, each corresponding to a separate topic.3 The data were split by date into 11,314 training and 7,531 test articles, so the training and test sets were separated in time. The training set was further randomly split into 8,314 training and 3,000 validation documents. Some newsgroups are very closely related to each other, for example, soc.religion.christian and talk.religion.misc, while others are very different, for example, rec.sport.hockey and comp.graphics (see Fig. 3). We further preprocessed the data by removing common stopwords, stemming, and then only considering the 2,000 most frequent words in the training dataset. As a result, each posting was represented as a vector containing 2,000 word counts. No other preprocessing was made. The Reuters Corpus Volume II is an archive of 804,414 newswire stories4 that have been manually categorized into 103 topics. The corpus covers four major groups: corporate/industrial, economics, government/social, and markets. Sample topics are displayed in Fig. 3. The data were randomly split into 402,207 training and 402,207 test articles. The training set was further randomly split into 302,207 training and 100,000 validation documents. The available data were already in the preprocessed format, where common stopwords were removed and all documents were stemmed. We again only considered the 2,000 most frequent words in the training dataset. 5.2. Results For both datasets we used the 2000-500-500-128 architecture shown in Fig. 1. To see whether the learned 128-bit codes preserve class information, we used Stochastic Neighbor Embedding (Hinton & Roweis, 2003) to visualize the 128-bit codes of all the documents from five or six separate classes. Fig. 3 shows that for both datasets the 128-bit codes preserve the class structure of the documents.
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011) A
85
20 Newsgroup 2−D Topic Space rec.sport.hockey comp.graphics
talk.politics.guns
sci.cryptography talk.politics.mideast
soc.religion.christian
B
Reuters 2−D Topic Space Disasters and Accidents Government Borrowing
European Community Monetary/Economic
Energy Markets
Accounts/Earnings
Fig. 3. Two-dimensional embedding of 128-bit codes using SNE for 20 Newsgroup data (panel A) and Reuters RCV2 corpus (panel B).
In addition to requiring very little memory, binary codes allow very fast search because fast bit counting routines5 can be used to compute the Hamming distance between two binary codes. On a 3GHz Intel Xeon running C, for example, it only takes 3.6 ms to search through 1 million documents using 128-bit codes. The same search takes 72 ms for 128dimensional LSA and 1.2 s for TF-IDF, though this could be considerably reduced by using a less naive method that uses inverted indexing.
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
86
Newsgroup Dataset
A 0.9 0.8
Fine−tuned 128−bit DGM
0.7
Accuracy
0.6 0.5 0.4
128−bit DGM prior to fine−tuning
0.3 0.2
LSA 128
0.1 0
1
3
7
15
31
63
127
255
511 1023 2047 4095 7531
Number of Retrieved Documents
Newsgroup Dataset
B 0.9 0.8
Hybrid 128−bit DGM using TF−IDF
0.7 TF−IDF
Accuracy
0.6 0.5
LSA 128
0.4 0.3 0.2 0.1 0
1
3
7
15
31
63
127
255
511 1023 2047 4095 7531
Number of Retrieved Documents
Fig. 4. Accuracy curves for 20 Newsgroup dataset, when a query document from the test set is used to retrieve other test set documents, averaged over all 7,531 possible queries.
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011) A
87
Reuters Corpus 0.55
0.5
Fine−tuned 128−bit DGM
0.45
Accuracy
LSA 128 0.4
0.35
128−bit DGM prior to fine−tuning
0.3
0.25
0.2
1
3
7
15
31
63
127
255
Number of Retrieved Documents
B
511
1023
Reuters Corpus 0.55
0.5
Hybrid 128−bit DGM using TF−IDF
Accuracy
0.45
0.4
LSA 128
0.35
0.3
0.25
TF−IDF 0.2
1
3
7
15
31
63
127
255
Number of Retrieved Documents
511
1023
Fig. 5. Accuracy curves for Reuters RCV2 dataset, when a query document from the test set is used to retrieve other test set documents, averaged over all 402,207 possible queries.
Figs. 4 and 5 (panels A) show that our 128-bit codes are better at document retrieval than the 128 real-values produced by LSA. TF-IDF is slightly more accurate than our 128-bit codes6 when retrieving the top 15–30 documents in either dataset. If, however, we use our 128-bit codes to preselect the top 100 documents for the 20 Newsgroup data or the top 1,000 for the Reuters data, and then re-rank these preselected documents using TF-IDF, we get better accuracy than running TF-IDF alone on the whole document set (see Figs. 4 and 5). This shows that the 128-bit codes can correctly reject some documents that TF-IDF would rank very highly. On 400,000 documents, the naive implementation of TF-IDF takes 0.48 s and our more accurate hybrid method takes .0014 + .0012 s—about 200 times faster.
88
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
6. Retrieval in constant time using a semantic address space Using 128-bit codes, we have shown that documents which are semantically similar can be mapped to binary vectors that are close in hamming space. If we could do this for 30-bit codes, we would have the ultimate retrieval method: Given a query document, compute its 30-bit address and retrieve all of the documents stored at similar addresses with no search at all. For a billion documents, a 30-bit address space gives a density of one document per address and a hamming-ball of radius 5 should contain about 175,000 documents. The retrieved documents could then be given to a slower but more precise retrieval method. It is unlikely that the 175,000 documents in the hamming-ball would contain most of the very similar documents, so recall would not be high, but it is possible that precision could still be very good. Using 20-bit codes, we checked whether our learning procedure could discover a way to model similarity of count-vectors by similarity of 20-bit addresses that was good enough to allow high precision retrieval for our set of 402,207 test documents. For example, a hamming ball of radius 4 contains 6,196 of the million addresses so it should contain about 2,500 documents. Fig. 6 shows that accuracy is not lost by restricting TF-IDF to this preselected set. We can also use a two-stage filtering procedure by first retrieving documents using 20-bit addresses in a hamming ball of larger radius 6 (about 25,000 documents), filter these down to 1,000 using 128-bit codes, and then apply TF-IDF. This method is faster and achieves higher accuracy as shown in Fig. 6. Scaling up the learning to a billion training cases would not be particularly difficult. Using mini-batches, the learning time is sublinear in the size of the dataset if there is
TF−IDF TF−IDF using 20 bits TF−IDF using 20 bits and 128 bits Locally Sensitive Hashing
50
Precision (%)
40
30
20
10
0
0.1
0.2
0.4
0.8
1.6
3.2
Recall (%)
6.4
12.8 25.6 51.2 100
Fig. 6. Accuracy curves for Reuters RCV2 dataset, when a query document from the test set is used to retrieve other test set documents, averaged over all 402,207 possible queries. Locality sensitive hashing (LSH) is the fastest current method for finding similar documents. It is less accurate and also much slower than using the Hamming ball around a learned binary code to create a shortlist for TF-IDF.
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
89
redundancy in the data as there surely is. Also, different processors can be used to compute the gradients for different examples within a large mini-batch. To improve recall, we could learn several different ‘‘semantic’’ address spaces on disjoint training sets and then preselect documents that are close to the query document in any of the semantic address spaces.
7. Discussion We have described a rather complicated way of learning efficient, compact, nonlinear codes for documents, and we have shown that these codes are very good for retrieving similar documents. The learning is slow, especially the fine-tuning stage, but it scales very well with the size of the dataset. The use of a simple gradient method as opposed to a complicated convex optimization, means that it is easy to incorporate new data without starting all over again. Even though the learning is quite slow, the inference is very fast. The code for a new document can be extracted by a single, deterministic feed-forward pass through the encoder part of the network and this requires only a few vector-matrix multiplies. To gain further insight into the nature of the codes that are learned, it would be helpful if we could visualize what the individual elements of the codes represent. Unfortunately, this is not nearly as easy to do as with topic models because in our codes, each component is activated by a very large set of documents and the precision comes by intersecting all of these sets. By contrast, in topic models each word must be generated by a single topic so the individual topics are, necessarily, far more precise and therefore far easier to visualize. For generative models of labeled data it is possible to fix the class label and see what typical instances of that class look like as is done in Hinton et al. (2006), but this method cannot be applied when the model is learned on unlabeled data. Our method for learning codes could be applied to the same word document matrix in a different way by treating each word as a training case rather than each document. The codes would then represent word meanings and it would be interesting to visualize them in a two-dimensional map using t-SNE (van der Maaten & Hinton, 2008), which is very good at organizing the layout so that very similar codes are very close to one another (see http:// www.cs.toronto.edu/ hinton/turian.png for an example). While the use of short codes may be interesting for the technology of information retrieval, we think it is largely irrelevant to the issue of how people represent and retrieve documents for the reasons given in the Introduction. However, the same deep learning methods can be used to extract very large, very sparse codes, and we think this is a far more promising direction for future research on how the brain represents the contents of a document. Notes 1. The last minibatch contained more than 100 cases. 2. Code is available at http://www.kyb.tuebingen.mpg.de/bs/people/carl/code/minimize/.
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
90
3. The data are available at http://people.csail.mit.edu/jrennie/20Newsgroups (20newsbydate.tar.gz). It has been preprocessed and organized by date. 4. The Reuter Corpus Volume 2 is available at http://trec.nist.gov/data/reuters/ reuters.html. 5. Code is available at http://www-db.stanford.edu/!manku/bitcount/bitcount.html. 6. 256-bit codes work slightly better that 128-bit ones.
Acknowledgments This research was supported by NSERC, CFI, and Google. G. E. H. is a fellow of the Canadian Institute for Advanced Research.
References Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022. Deerwester, S. C., Dumais, S. T., Landauer, T. K., Furnas, G. W., & Harshman, R. A. (1990). Indexing by latent semantic analysis. Journal of the American Society of Information Science, 41(6), 391–407. Erhan, D., Manzagol, P., Bengio, Y., Bengio, S., & Vincent, P. (2009). The difficulty of training deep architectures and the effect of unsupervised pre-training. AISTATS 2009, 5, 153–160. Gehler, P., Holub, A., & Welling, M. (2006). The rate adapting poisson (RAP) model for information retrieval and object recognition. Proceedings of the 23rd International Conference on Machine Learning. Pittsburgh, PA. Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8), 1711–1800. Hinton, G. E. (2007a). Learning multiple layers of representation. Trends in Cognitive Science, 11, 428–434. Hinton, G. E. (2007b). To recognize shapes, first learn to generate images. In P. Cisek, D. Kalaska, & J. Haran (Eds.), Computational neuroscience: Theoretical insights into brain function (pp. 535–547). Montreal: Elsevier. Hinton, G. E., & Roweis, S. T. (2003). Stochastic neighbor embedding. In S. Thrun, L. K. Saul, & B. Scholkopf (Eds.), Advances in neural information processing systems 15 (pp. 833–840). Cambridge, MA: MIT Press. Hinton, G. E., & Salakhutdinov, R. R. (2006). Non-linear dimensionality reduction using neural networks. Science, 313, 504–507. Hinton, G. E., & Sejnowski, T. J. (1983). Optimal perceptual inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, Washington, DC. Hinton, G. E., McClelland, J. L., & Rumelhart, D. E. (1986). Distributed representations. In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel distributed processing: explorations in the microstructure of cognition. Volume 1: Foundations (pp. 77–109). Cambridge, MA: MIT Press. Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527–1554. Hofmann, T. (1999). Probabilistic latent semantic analysis. In P. Laskey & H. Prade (Eds.), Proceedings of the 15th conference on uncertainty in AI (pp. 286–296). Stockholm, Sweden: Morgan Kaufmann. Jordan, M. (1999). Learning in Graphical Models. Cambridge, MA: MIT Press. van der Maaten, L. J. P., & Hinton, G. E. (2008). Visualizing data using t-sne. Journal of Machine Learning Research, 9, 2579–2605.
G. Hinton, R. Salakhutdinov ⁄ Topics in Cognitive Science 3 (2011)
91
Pearl, J. (1988). Probabilistic inference in intelligent systems: networks of plausible inference. San Mateo, CA: Morgan Kaufmann. Salakhutdinov, R. R., & Hinton, G. E. (2007). Semantic hashing. In J. Fernandez-Luna, B. Piwowarski, & J. F. Huete (Eds.), Proceedings of the SIGIR Workshop on Information Retrieval and Applications of Graphical Models). Amsterdam, The Netherlands. Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing and Management, 24(5), 513–523. Welling, M., Rosen-Zvi, M., & Hinton, G. (2005). Exponential family harmoniums with an application to information retrieval. In L. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural information processing systems 17 (pp. 1481–1488). Cambridge, MA: MIT Press. Xing, E., Yan, R., & Hauptmann, A. G. (2005). Mining associated text and images with dual-wing harmoniums. In F. Baccus, & T. Jaakkola (Eds.), Proceedings of the 21st conference on uncertainty in artificial intelligence (pp. 633–641). Edinburgh, Scotland: AUAI press.
Topics in Cognitive Science 3 (2011) 92–122 Copyright ! 2010 Cognitive Science Society, Inc. All rights reserved. ISSN: 1756-8757 print / 1756-8765 online DOI: 10.1111/j.1756-8765.2010.01108.x
Comparing Methods for Single Paragraph Similarity Analysis Benjamin Stone,a Simon Dennis,b Peter J. Kwantesc a
School of Psychology, The University of Adelaide Department of Psychology, Ohio State University c Defence Research and Development Canada (Toronto) b
Received 27 February 2009; received in revised form 6 July 2009; accepted 8 September 2009
Abstract The focus of this paper is two-fold. First, similarities generated from six semantic models were compared to human ratings of paragraph similarity on two datasets—23 World Entertainment News Network paragraphs and 50 ABC newswire paragraphs. Contrary to findings on smaller textual units such as word associations (Griffiths, Tenenbaum, & Steyvers, 2007), our results suggest that when single paragraphs are compared, simple nonreductive models (word overlap and vector space) can provide better similarity estimates than more complex models (LSA, Topic Model, SpNMF, and CSM). Second, various methods of corpus creation were explored to facilitate the semantic models’ similarity estimates. Removing numeric and single characters, and also truncating document length improved performance. Automated construction of smaller Wikipedia-based corpora proved to be very effective, even improving upon the performance of corpora that had been chosen for the domain. Model performance was further improved by augmenting corpora with dataset paragraphs. Keywords: Semantic models; Paragraph similarity; Corpus preprocessing; Corpus construction; Wikipedia corpora
1. Introduction
The rate at which man [sic] has been storing up useful knowledge about himself and the universe has been spiralling upwards for 10,000 years. —Toffler, 1973, p. 37 Correspondence should be send to Simon Dennis, Department of Psychology, Ohio State University, 225 Psychology Building, 1835 Neil Avenue, Columbus, OH 43210. E-mail:
[email protected]
B. Stone, S. Dennis, P. J. Kwantes ⁄ Topics in Cognitive Science 3 (2011)
93
Nearly four decades later, Toffler’s remark is perhaps even more relevant in today’s internet-driven world. ‘‘Information overload’’ may be regarded as pervasive in many professions, and filtering strategies such as the summarization of text are commonplace. Government leaders and company executives make informed decisions based on briefs or short summaries of complex issues, provided by department managers who have in turn summarized longer reports written by their staff. In academia, the abstract is used to provide an overview of a paper’s contents, so that time-pressed researchers can filter and absorb information related to their fields of study. In many areas it is important to be able to accurately judge the similarity between two or more paragraphs of information. Sorting and extracting useful information from large collections of these types of summaries can prove both overwhelming and time consuming for humans. In an attempt to address this issue, semantic models have been successfully employed at these tasks. For example, latent semantic analysis (LSA; Landauer, McNamara, Dennis, & Kintsch, 2007) has been used to grade student essay scripts (Foltz, Laham, & Landauer, 1999). Similarly, the Topic Model has been used to extract scientific themes from abstracts contained in the Proceedings of the National Academy of Sciences (Griffiths & Steyvers, 2004). In a surveillance application, nonnegative matrix factorization has been applied to the large ENRON e-mail dataset to extract topics or themes (Berry & Browne, 2005). Other models such as the vector space model (henceforth called ‘‘Vectorspace’’) were originally designed to index (or order by relevance to a topic) large sets of documents (Salton, Wong, & Yang, 1975). Semantic models have also been shown to reflect human knowledge in a variety of ways. LSA measures correlate highly with humans’ scores on standard vocabulary and subject matter tests; mimic human word sorting and category judgments; simulate word-word and passage-word lexical priming data; and accurately estimate passage coherence (Landauer et al., 2007). The Topic Model has proven adept at predicting human data on tasks including the following: free association, vocabulary tests, lexical decision, sentence reading, and free recall (Griffiths, Tenenbaum, & Steyvers, 2007). Other models have been developed to reflect specific psychological processes. For example, the constructed semantics model (CSM) was developed as a global-matching model of semantic memory derived from the MINERVA 2 architecture of episodic memory (Kwantes, 2005). 1.1. Different types of textual language unit When making similarity comparisons on textual stimuli with semantic models, several researchers have highlighted the need to delineate textual stimuli into different language units (Foltz, 2007; Kireyev, 2008; Landauer & Dumais, 1997; McNamara, Cai, & Louwerse, 2007). Past research has modeled human comparisons of similarity on four types of textual language units: words, sentences, single paragraphs, and chapters or whole documents (Foltz, 2007). 1.1.1. Word comparisons Griffiths et al. (2007) found that the Topic Model outperformed LSA on several tasks, including word association and synonym identification. Griffiths and colleagues compared
94
B. Stone, S. Dennis, P. J. Kwantes ⁄ Topics in Cognitive Science 3 (2011)
performance by the Topic Model and LSA on a word association task using norms collected by Nelson, McEvoy, and Schreiber (1998). The study used 4,471 of these words that were also found in an abridged1 37,651-document (26,243 word, 4,235,314 token) version of the Touchstone Applied Science Associates (TASA) corpus. Moreover, the TASA corpus was used as a knowledge base for both the Topic Model and LSA. Two measures were employed to assess the models’ estimates of word association. The first measure assessed central tendency, focusing on the models’ ability to rank word targets for each word cue. The other measure assessed the proficiency of each model’s estimate of the most likely target response for each word cue. Griffiths and colleagues found that the Topic Model outperformed LSA on both of these performance measures. Furthermore, they reported that both models performed at levels better than chance and a simple word co-occurrence model. In another study, Griffiths et al. (2007) compared the Topic model and LSA on a subset of the synonym section taken from the Test of English as a Foreign Language (TOEFL). The TOEFL was developed in 1963 by the National Council on the Testing of English as a Foreign Language, and it is currently administered by the Educational Testing Service.2 The synonym portion of TOEFL offers four multiple choice options for each probe word, Griffiths and colleagues only included items in which all five words also appeared in the aforementioned abridged version of the TASA corpus. Similarity evaluations between the probes and possible synonyms revealed that the Topics model (70.5%) answered more of the 44 questions correctly than LSA (63.6%). Furthermore, the Topic Model (0.46) predictions captured more of the variance found in the human responses than LSA (0.3). The Topic Model is a generative model that assesses the probability that words will be assigned to a number of topics. One of the key benefits of this generative process is that it allows words to be assigned to more than one topic, thus accommodating the ambiguity associated with homographs (Griffiths et al., 2007). For example, using the Topic Model, the word ‘‘mint’’ may appear in a topic that contains the words ‘‘money’’ and ‘‘coins,’’ and in another topic containing the words ‘‘herb’’ and ‘‘plants.’’ Griffiths et al. (2007) argue that this attribute gives the Topic Model an advantage over models like LSA which represent meanings of words as individual points in undifferentiated Euclidean space (pp. 219–220). 1.1.2. Sentence comparisons McNamara et al. (2007) used several implementations of LSA to estimate the relatedness of sentences. The human judged similarity of these sentences decreased from paraphrases of target sentences, to sentences that were in the same passage as target sentences, to sentences that were selected from different passages to the target sentences. Likewise, comparing sentences using a standard implementation of LSA and the TASA corpus, these researchers found estimates of similarity were greatest for paraphrases, then same passage sentences, with different passage sentences judged least similar. When human estimates were correlated with the LSA estimates of sentences similarity, it was found that a version of LSA that emphasized frequent words in the LSA vectors best captured the human responses. Subsequently, using data collected in the McNamara et al. (2007) study, Kireyev (2008) found that LSA outperformed the Topic Model at this task.
B. Stone, S. Dennis, P. J. Kwantes ⁄ Topics in Cognitive Science 3 (2011)
95
1.1.3. Single paragraph comparisons Lee et al. (2005) examined similarity judgments made by Adelaide University students on 50 paragraphs that were collected from the Australian Broadcasting Corporations news mail service. These paragraphs ranged from 56 to 126 words in length, with a median length of 78.5 words. Lee and colleagues compared several models’ estimates of similarity to the aforementioned human ratings. These models included word-based, n-gram, and several LSA models. Using a knowledge base of 364 documents also drawn from the ABC news mail service, LSA under a global entropy function was the best performing model,3 producing similarity ratings that correlated about 0.60 with human judgments in this study. LSA’s result in this study was also consistent with the inter-rater correlation (approximately 0.605) calculated by these researchers. More recently, Gabrilovich and Markovitch (2007) produced a substantially higher correlation with the human similarity judgments recorded for the Lee paragraphs (0.72) using the model they developed, explicit semantic analysis (ESA). The ESA model uses Wikipedia as a knowledge base, treating Wikipedia documents as discrete humangenerated concepts that are ranked in relation to their similarity to a target text using a centroid-based classifier. Kireyev (2008) used LSA and the Topic Model to estimate similarity of pairs of paragraphs taken from 3rd and 6th grade science textbooks. It was proposed that paragraphs that were adjacent, should be more similar than nonadjacent paragraphs. Difference scores were calculated between adjacent and nonadjacent paragraphs for both grade levels, with higher scores indi