Gesturecraft: The manu-facture of meaning (Gesture Studies)

Gesturecraft

Gesture Studies (GS) Gesture Studies aims to publish book-length publications on all aspects of gesture. These include, for instance, the relationship between gesture and speech; the role gesture may play in social interaction; gesture and cognition; the development of gesture in children; the processes by which spontaneously created gestures may become transformed into codified forms; the relationship between gesture and sign; biological studies of gesture, including the place of gesture in language evolution; and gesture in human-machine interaction. Volumes in this peer-reviewed series may be collected volumes, monographs, or reference books, in the English language.

Editors Adam Kendon

University of Pennsylvania, Philadelphia

Cornelia Müller

European University Viadrina, Frankfurt/Oder

Volume 2 Gesturecraft. The manu-facture of meaning by Jürgen Streeck

Gesturecraft The manu-facture of meaning

Jürgen Streeck The University of Texas, Austin

Drawings by

Jürgen Grothues and Jacob Villanueva

John Benjamins Publishing Company Amsterdam / Philadelphia

8

TM

The paper used in this publication meets the minimum requirements of American National Standard for Information Sciences – Permanence of Paper for Printed Library Materials, ansi z39.48-1984.

Video clips (.mov) of numerous examples in this volume can be found at http://dx.doi.org/10.1075/gs.2.video

Library of Congress Cataloging-in-Publication Data Streeck, Jürgen. Gesturecraft : the manu-facture of meaning / Jürgen Streeck ; drawings by Jurgen Groethues and Jacob Villanueva. p. cm. (Gesture Studies, issn 1874-6829 ; v. 2) Includes bibliographical references and index. 1. Gesture. 2. Communication. I. Groethues, Jurgen. II. Villanueva, Jacob. III. Title. P117.S84    2009 302.2'22--dc22 2008050994 isbn 978 90 272 2842 0 (Hb; alk. paper)

© 2009 – John Benjamins B.V. No part of this book may be reproduced in any form, by print, photoprint, microfilm, or any other means, without written permission from the publisher. John Benjamins Publishing Co. · P.O. Box 36224 · 1020 me Amsterdam · The Netherlands John Benjamins North America · P.O. Box 27519 · Philadelphia pa 19118-0519 · usa

For Susi Plutôt la vie!

Table of contents

Acknowledgements 1.

Manufactured understanding What this book is about 3 The approach taken 5 Ecologies of gesture 7 Overview 11

ix 1

2. Gestures as interaction: On methodology An interactionist approach to gesture 13 Gregory Bateson and the “natural history approach” 16 Goffman’s micro-studies of the interaction order 19 Context analysis 21 Kendon on gesture 22 Conversation analysis 25 Ethnography 28 Praxeology 30 Visual research on cultural behavior 32 Gesture, gestures, culture, cultures: Some conceptual clarifications 35

13

3. Hands The structure of the human hand 40 Evolution 42 Grasping: Prehension and comprehension 47 Repertoires of manual action 52 Exploratory and practical actions 53 Hands, worlds, and knowledge 54 Conclusion 58

39

4. Gathering meaning The world at hand 61 Gestures of orientation 63 Clearing 66 Tracing: Discovering and drawing lines 69 Exploratory procedures 70

59

 Gesturecraft

Disassembling objects 73 Making action intelligible 75 The world in sight 76 Contiguity 82 Dwelling 83 5. The turn to the hands Gaze, visibility, and talk in interaction 86 The speaker’s look at the hands 87 Directing attention to gesture by utterance design 94 The recipient’s orientation to gestures 104 Gaze and gesture during searches for a word 108 Attentional struggle 114 6. Depiction Articulating gesture space 122 Motion: Real and fictive 132 Drawing 136 Handling 138 Marking and self-marking 142 Mimesis: Depicting action 144 The heterogeneity of representational practices 7.

85

119

148

Thinking by hand Gestures of emotion 152 Gesture as conceptualization 160 Further examples of ceiving 163 Models for theories 167 Gesture and thought revisited 172

8. Speech-handling Kendon’s pragmatic-gesture families 179 Pragmatic gestures, turns at talk, and kinesthetic feedback 182 Open hands and turn-completions 182 Giving and receiving; waiting to receive 184 Mid-turn offerings 187 Shrugs 189 Moving things aside 192 Throwing back (re-jecting) 193 Negations 194 The open hand before tellings 196 Implicit objects 199

151

179

Table of contents

Verbs of speaking 199 Conclusion 201 9. A sustainable art

203

Appendix. Data and transcript conventions

213

Bibliography

215

Person index

229

Subject index

233

Video clips (.mov) of numerous examples in this volume can be found online, at http://dx.doi.org/10.1075/gs.2.video This logo marks the availability of a video clip of an example.



Acknowledgements

The research that is condensed into this book and the cycles of writing, rethinking, and rewriting it took as long to complete as it took to raise a son (mine celebrated his 21st birthday the other day). Conceiving them was easy. Letting them grow at their own pace, recognizing mistakes in relating to them, adjusting to their resistance, giving them what it takes to stand alone, and loving them for their qualities – rather than those one wishes for them – took years of maturation on the parent’s part. In the end, of course, it is all about letting them go. The idea to study gesture in human interaction was first conceived during a morning of socially shared cognition with a small and loyal group of students at the Freie Universität Berlin. We met at Café Montevideo (which we renamed Monte Video), to discuss what we could do together during the coming semester, now that the University had finally given us video equipment, five years after I had requested it. Among others the group included Ulrike Hartge and, notably, Cornelia Müller. Studying gesture in the context of conversation analysis – about which I was teaching a seminar series at the time – seemed to be an exciting project, because so little was known about it at the time: we could count the number of publications on gesture that were based on observational research on the fingers of our hands. And what a journey it has become! Today, gesture studies is a very lively enterprise, carried out by numerous scholars across many disciplines. The journal Gesture, founded by Adam Kendon and Cornelia Müller, is now in its eighth year, and Cornelia and I have also been involved in the founding of a new international association, the International Society for Gesture Studies. Being so closely involved in the growth of a new field of knowledge about the human species has been a very special pleasure indeed. The research reported here was initially supported by a small grant from the Freie Universität Berlin, then by a larger grant from the Deutsche Forschungsgemeinschaft, which enabled me to do field-research and film interactions in the Philippines, and later by a number of small grants from the University of Texas at Austin. I owe thanks to Elnora Held and her family in the Philippines, the Alindayus, as well as to Bonifacio Ramos at St. Mary’s University in Bayombong, Nueva Vizcaya, for their support of my research and for their boundless hospitality. I have also benefitted greatly from the enthusiasm and intelligence of my



Gesturecraft

doctoral students at the University of Texas, especially Curtis LeBaron, Carmen Taleghani-Nikazm, Andrea Golato, Siri Mehus, Kris Markman, Tomoko Ikeda, Juanita Handy-Bosma, Sae Oshima, and Eiko Yasui, and the interest of one or the other of my local colleagues, especially Mark L. Knapp and the late Robert Hopper. My brother Ulrich Streeck inspired me by feeling inspired to also become a conversation analyst (in addition to being a psychoanalyst, a fact that also benefitted me). I could not have written Chapter 7 of this book had I not been a fellow in the research group Embodied Communication in Humans and Machines at the Center for Interdisciplinary Research (ZiF) in Bielefeld in 2005–6, where I learned much of what I know about cognitive neuroscience (while maintaining my conviction that machines will never manufacture meaning). I thank Ipke Wachsmuth for inviting me and Scott Jordan for his collaboration, friendship and spirit. I have generally been reluctant to show my unfinished work to others. However, Susan Duncan read and criticized what I previously had to say about thought and gesture, and Liza Shapiro checked the factual accuracy of my account of the evolution of the human hand (in Chapter 3). Peg Syverson and I kept each other on task for a while by forming a writing group, and my friend Stephan Wolff admirably worked his way through an entire prior draft (and making many suggestions that became obsolete in the next cycle of rethinking gesture). I owe much to an ongoing conversation with Chuck Goodwin, and to Adam Kendon. To be in states of focused interaction with Adam Kendon – and for him to temporarily be focused on my work – has been an immeasurable blessing. If he had not encouraged me, back in the 1980s, in a pub in York, England, to go ahead with my plans to study gesture comparatively, I would never have begun this study. And if he had not spend hours and hours editing my manuscripts, they would amount to so much less. I am equally grateful to all the people who allowed me (or my students) to videotape them and analyze their interactions, foremost among them Hussein Chmeis whom I videotaped for an entire day and who became my friend in the process. My sweetest sense of gratitude, though, is to my wife, Susan Alvarado, whose passion, intelligence, beauty, and sense of drama and comedy sustain and distract me, in equal measure. Austin, Texas October 2008

chapter 

Manufactured understanding

In his report of the Voyage Round the World that he undertook in the company of Captain Cook, the young German writer Georg Foster describes an encounter on the island of Tanna in the South Pacific: We reached the eastern point of the harbour, and there gathered some beautiful red flowers, which had tempted us ever since we came to an anchor; they belonged to a species of yambos or eugenia. As we were going to cross the point and to proceed along the shore beyond it, fifteen or twenty natives crowded about us, and begged us with the greatest earnestness to return. We were not much inclined to comply with their demand, but they repeated their intreaties, and at last made signs that we should be killed and eaten. The same gestures, though much less intelligible, had been made to us two days before, but we had paid very little attention to them, unwilling, upon such slight testimony, to suppose that they had any idea of anthropophagy. (Forster 1777: 299–300)

This encounter between two groups of people who had no common language and no knowledge of one another is an unusually dramatic display of our human ability to understand one another and our world by moving our hands. The people of Tanna may have made themselves look like predatory animals, but the fact that one could communicate with them in a fashion in which man and animal never could must have demonstrated to the Europeans that they were human. Gestures impressed upon visitors and hosts alike that, however different they may have been and however hostile their mutual intent, they were able to share their minds. Their ability to manu-facture mutual understanding was living proof of what anthropologists would later call the “psychic unity” (Tylor 1871) of humankind. However, gesture also entailed the possibility of misunderstanding, even of deliberate misunderstanding, but also of resolving it. Seeking to gain the upper hand in the struggle over the definition of the situation, Cook’s people turned the ambiguities of gesture and intercultural communication into a tactical maneuver; yet, the Tannans saw through the scheme and corrected what was a deliberate misunderstanding. All this was done by gesture: Making them believe that we thought they offered us some provisions, we began to move forward, and expressed that we should be glad to eat something. They were very eager to undeceive us, and showed, by signs, how they killed a man, cut his limbs asunder, and separated the flesh from the bones. Lastly, they bit their



Gesturecraft

own arms, to express more clearly that they eat human flesh. We turned from the point (loc.cit.).

The encounter in Tanna also demonstrates our pan-human ability to improvise understanding by making creative use of our hands and arms, to manu-facture not only meaning, but also the tools with which meaning can be made, to invent symbols in response to unexpectable communication needs that our daily lives as much as our foreign adventures may put before us.1 Forster, embodying the spirit of the Age of Enlightenment, attended to the Tannans’ gestures with the patient gaze of the collector of cultural forms. He appears to have reckoned with the possibility of a local, communal gesture-code, for example when he states that today’s gestures were “the same” as yesterday’s. Some of the signs that he described may indeed have been local conventions, perhaps belonging to a kind of lingua franca. The gesture that eventually convinced Cook’s people that it might be better to retreat – the bite in the upper arm – seems to have been common in parts of Melanesia: Don Diego de Prado, who visited the region before Cook, reported that a well-traveled Big Man, when asked what islands there were further to the east, “indicated more than thirty in different directions, and for those inhabited by cannibals, he bit into his arm” (cited after Hewes 1974). In encounters of this kind, the skilled, mindful bodily practice of gesture unfolds its full potential as a universally available resource from which people can manufacture understandings – of each other and of the world that they share. Moving the limbs, one party can articulate for the other how some part of the world ought to be seen. Gesture is a strange and puzzling practice, richly textured, diverse in the devices that it uses and creates, and in many ways still uncharted and unexplained. Certainly, first encounters in which moving bodies, situated within a shared perceptual world, but not a common culture, are the only communication resource, occur only rarely anymore; still, where spoken language is unavailable, as among the deaf, in situations of intergroup contact or of ritual or perceptual constraints on talk, the evolution of complex, viable gesture-based manual languages . It is not entirely certain that Cook’s people truly understood the Tannans. The anthropologist Obeyesekere (1997, 2005) has suggested that the idea that the Pacific islanders were cannibals was a European projection, fed not only by contemporary European punitive practices such as quartering, but also by deliberate deception on the islanders’ part, who exploited the Europeans’ curiosity and suspicion. Explorers like Cook, he points out, were obsessed with the idea of cannibalism, and they were the ones who generally initiated “cannibal talk”, who asked the islanders whether they ate human flesh, to which these commonly responded: “Why, do you?”. Sensing the fear that the idea of cannibalism evoked in their visitors, they pretended that, yes indeed, they were cannibals, and thereby kept the strangers away from the interior and the women. In New Zealand, the Maori held the Europeans at bay in this fashion for thirty years. Thus is the power of gesture and of the ambiguity that is inherent in it.

Chapter 1. Manufactured understanding

can be observed to this day (Barakat 1972; Kegl, Senghas, & Coppola 1999; Kendon 1980a–c, 1988b; Meissner & Philpott 1975). In the age of global communication via the Internet, even in the technologically most advanced settings of science and work, gesture has lost none of its to importance to human understanding, as can be observed, for example, even among software-engineers who routinely employ gestures to interpret graphic representations (Suchman 1993). Most of us encounter gesture primarily in mundane face-to-face conversations where they do not appear alone, but are interwoven with spoken language and seen and understood within the webs of meanings that spoken language spins. And yet, its co-occurrence with language does not solve the puzzle of gesture; linguistic context may well tell us what to look for when we look at a gesture, but it does not explain how we see what we see. It does not explain away, for example, the pictorial nature of depictive gestures, the fact that we are capable of seeing worlds in moving hands. Gestural understanding requires its own explanation. The reason why we still do not fully understand this perhaps most ancient mode of human communication (Corballis 2002; Donald 1991; Armstrong, Stokoe & Wilcox 1995; Armstrong & Wilcox 2007) is that, like so many other everyday-life practices, it operates quietly in the background, its features, only seemingly obvious, remaining undisclosed because there rarely is a practical need to explore them. Like Captain Cook, we would still trust our bodies’ sense-making abilities if we were visitors in Tanna, and we do trust them in our daily lives and find little reason to question how gestures live up to the trust that we invest in them. Gestures are elusive phenomena; their lifetimes can be measured in fractions of seconds. They leave no traces: by the time we make sense of a gesture, it is usually over. That gesture is meaningful at all is not even beyond reasonable doubt, for while some people habitually visualize with their hands what they are talking about, others hardly move their hands at all. If there are such extreme inter-individual differences in the deployment of gesture, how can understanding possibly rely on it? Conversational gestures are spontaneous and sometimes improvised, rarely subject to regimentation, and impossible to cast in a system of rules. But this makes them all the more interesting: observing gestures, we can witness how communicative action creatively fashions its own tools.

What this book is about This book is an investigation of the practices of gestural meaning-making and understanding, an attempt to explain how, by moving our hands, we gather meaning from and structure our environments, articulate experience, share it with others, and organize our interaction. The book is the product of numerous microscopic studies, conducted over two decades, of brief moments of human action and





Gesturecraft

interaction. The initial motive for this journey into the microcosm of gesture was a naive sense of wonder at its complexity and richness, at the human capacity, to paraphrase the art-historian Gombrich, to “conjure up [by motions of the hands] those mysterious phantoms of visual reality we call ’pictures”’ (Gombrich 1961). When these investigations began in 1986, only a small number of empirical accounts of conversational gestures had been published (Heath 1982; Kendon 1972, 1980d, 1985; Schegloff 1984). These few pioneering studies had already demonstrated that the spontaneous, often despised “chatter of the hands” is in fact often an artful, methodical activity which is skillfully and precisely coordinated with speech. “To marvel is the beginning of knowledge” (Gombrich 1961: 8). If we give it our attention, gesture gives us much to marvel about. Movements of the hands are capable of evoking images of objects, scenery, actions, events. They are capable of making the abstract tangible by expressing it in spatial terms. They can visualize speech acts or responses that are sought, or the structures of spoken utterances. And, in the first place, they can find and mark meaning in the world around us: by movements of the hands that are so slight that one even hesitates to call them gestures – a minimally embellished way in which an object is handled, a finger’s tracing of an invisible line – we can make the features of our lived-in world transparent and intelligible to one another. Gesture is not in the first place a code, a repertoire of conventional signs with fixed meanings and rules of use and combination. Rather, it is a form of human practice – or a family of practices – , and these practices in turn make use of a motley crew of methods and component parts. Without wanting to correct ordinary language, which also, appropriately, calls head-nods and gifts of flowers – and perhaps the design of some buildings – “gestures”, this inquiry is focused on gestures of the hands. For the purposes of this inquiry, gesture is defined as communicative action of the hands. This delimitation is not always defensible, for example because often a movement of a hand is part of a more embracing bodily display and because there certainly are gestures made by the head or parts of the face. Still, concentrating on hand-gestures allows us not only to keep the phenomena to be studied within a manageable range, but also to pay dues to the extraordinary status of the human hand: no part of our body (except the eyes) is as important as the hand in providing us with knowledge of the world, and no organ (except the brain) has played a greater part in creating the world that humans inhabit. The evolution of the human hand was central to preparing the species for the production of culture. How an organ with such skills and insights into the workings of the world operates in the realm of communication is an issue that would be obscured if we lumped it together with other communicative acts that are also reasonably called “gesture”. The gesturing hand is interesting and distinct enough to warrant special treatment.

Chapter 1. Manufactured understanding

The approach taken The aim of this book is to elucidate some of the main ways in which gesture contributes to human understanding. The approach taken can be called a naturalistic one (Sarles 1975; Scheflen 1975; see also Kendon 1979a, 1990): it consists of observational research in the tradition of anthropological linguistics, microsociology, linguistic pragmatics, ethnography, and ethology; and it is grounded in the micro-analysis of human action and interaction in non-experimental, everyday life settings.2 I use the term micro-ethnography to refer to the naturalistic study of human practices in social life (Erickson 1971; Streeck & Mehus 2004). Gesture in this book is conceived as a family of human practices: not as a code or symbolic system or (part of) language, but as a constantly evolving set of largely improvised, heterogeneous, partly conventional, partly idiosyncratic, and partly culture-specific, partly universal practices of using the hands to produce situated understandings. Practice-based (or praxeological) approaches to language, gesture, and social action (Chaiklin & Lave 1993; Erickson 2004; Hanks 1990, 1996; Heath 1986; Lave 1988) locate meaning in the empirical, embodied practice of human actors within socioculturally constituted, social and material settings. Gestural understanding in this view is not the result of a shared grammar or lexicon, but of the coordinated embodied actions of people and their perspectives upon the material, real-world setting within which they interact. As Hanks has stated, in order for two people to . . . communicate . . . it is not sufficient, and perhaps not even necessary, that they ‘share’ the same grammar. What they must share, to a variable degree, is the ability to orient themselves verbally, perceptually, and physically to each other and to their social world. (Hanks 1996: 235)

Gesture is part of this mutual and shared orientation. Gesture is also largely improvised: while undoubtedly members of all human societies utilize gestural forms in their practice that are conventional, routine, and thus “prefabricated”, many gestures that individuals produce are idiosyncratic and opportunistic, making use of locally available material such as prior gestures, practical actions that are currently being carried out, or even props such as objects on the table before them. Gestural practice has these features – hybridity, improvisation, bricolage – in common with other forms of human practice (Erickson 2004).

. While some of the data for this research are “contrived” in that “human subjects” were asked to come to a particular place at a particular time to have their conversation videotaped, the participants were always close friends, lovers, or acquaintances, and they were videotaped having conversations about emerging topics of their own choice. They were never informed that we were interested in their gestures.





Gesturecraft

By “understanding” I mean not only mutual understanding among conversation partners, but also their shared understanding of the world at hand. The primordial setting for human understanding is not of the kind that much contemporary research on human interaction has focused on for a long time, in which two or more people talk about an absent world, but a situation in which they are together involved with the world around them, touching it, modifying it, working, and together trying to make sense of what is going on and what is to be done. This situation of care or Sorge, as Heidegger has called it (Dreyfus 1991), engages not only the actors’ minds and speech, but also their bodies, and understanding or shared sense-making – and specifically understanding by bodily means, including gesture – often arises from their practical, physical actions in the world. Gesture, to use Goodwin’s term (Goodwin 2007), is often environmentally coupled, and it is part of the larger multimodal organization of human action. Thus, a primordial site for the study of gesture consists of a situation in which multiple participants are carrying out courses of action together while attending to 1) each other; 2) the detailed organization of the talk in progress; 3) relevant phenomena in the environment and 4) the larger activities that they are engaged in. Within such a framework gesture does not stand alone as a self-contained system that can be analyzed in isolation from the other semiotic resources and meaning practices that participants are using to build action in concert with each other. Instead gesture occupies an interstitial position within a larger ecology of sign systems that build meaning and action by mutually elaborating each other. (Goodwin 2002: 1)

Communicative processes that are removed from such involvement are, historically and logically, secondary. The point of departure for the research reported in this book, thus, are human beings in their daily activities. The perspective on gesture is informed by the work of phenomenological philosophers (Heidegger 1962 [1926]; Merleau-Ponty 1962; Polanyi 1958) who have argued that we must understand human understanding by finding it, in the first place, in concrete, practical, physical activity in the world, as well as by more recent work in anthropology (Hastrup 1995; Ingold 2000; Keller & Keller 1996; Strathern 1996), philosophy and linguistics (Hanks 1990; Johnson 1987; Lakoff & Johnson 1999), educational psychology (Lave 1988), and sociology (Connerton 1989; Harper 1987; see also Mauss 1973 [1935]), which is defined by the view that the human mind – and the symbols that it relies upon – are embodied. This view is now even more forcefully argued by an increasing number of cognitive scientists and neuroscientists who take issue with the traditional cognitive science view of the human mind as a computer which processes information that is represented by some abstract symbolism (Chomsky 1965; see Gardner 1985 for an early critique of this view). Cognitive activities and concepts, rather, are multimodal, and experience is stored in the sensory modalities in which they are first acquired (Barsalou 1999). Bodily experiences and motoric schemata are the foundations for

Chapter 1. Manufactured understanding

higher, more abstract cognitive structures and functions (Gallese & Lakoff 2005; Lakoff & Nuñez 2000); this is attested by the frequency of body-based concepts for abstract content in all natural languages (Heine & Kuteva 2002). There is abundant experimental evidence that those brain structures that control movement are also involved in perception, especially perception of other humans (Wilson & Knoblich 2005) as well as in the semantic processing of linguistic representations of motion events (Bergen & Wheeler 2005; Tettamanti et al. 2005). The primary function of the brain is to control movement, and therefore thinking is, as Llinàs (2001) has pointed out, internalized movement. However, the approach taken here is also framed by the methodologies of conversation analysis (Sacks, Schegloff, & Jefferson 1974) and context analysis (Kendon 1990; Scheflen 1973, 1974); in fact, it has developed from research that initially was conducted entirely within these frameworks, and exclusively focused on gesture within conversational settings (Streeck 1988; Streeck & Hartge 1992). Only gradually did I realize that we cannot really understand gesture unless we see it in its close connection with practical, bodily acts. However, this recognitions does not relinquish the strong methodological demands that context analysis and conversation analysis impose, namely to examine communicative phenomena such as gestures exactly as they occur in the context of human interaction, coordinated with other modalities, and consequential for – and organized by – the participants’ moment-by-moment relating to one another. In other words, we must study gestures in their natural contexts of occurrence, where “context” not only means the physical and social setting, but also the concurrent and prior acts of co-participants within unfolding turns and sequences of action.

Ecologies of gesture A great deal of research on gesture has focused on a single type of setting, the type in which human subjects discuss or narrate experiences that have taken place outside and prior to the current situation. This a specific type of ecology, a distinct pattern of alignment between human actors, their gestures, and the world. For example, during narratives speakers may report what they have witnessed, and their gestures, although addressed to their current interaction co-participant, refer to these distant events. (This ecology is in play when speakers describe a film-clip that they have just seen, as in McNeill’s laboratory research; McNeill 1992). However, within the same context of narration it is also possible that speakers make gestures in pursuit of a proper term to conceptualize an experience; in this case, gestures do not depict the world, but rather are made in attempts to make sense of it, for example by finding a fitting metaphor for it. These usage modes, although usually lumped together, are clearly distinct, even if they may overlap at times (for





Gesturecraft

example when a patient during psychotherapy seeks to make sense of the emotional turmoil within him by spatializing it with gestures of the hands – a case of congruence between conceptualization and depiction by hands). Speakers usually reveal the difference between the two modes by their overall bodily behavior, for example by the direction of their gaze. Finally, within the same context, the speaker may also perform hand gestures to mark the speech acts that she is performing or to emphasize some part of speech: here the gestures are neither about the world outside the conversation nor are they made in pursuit of a concept, but rather they contribute to the structuring of the communicative process itself (Kendon 1995). In this book, I distinguish six gesture ecologies, that is, six different ways in which gestural activity can be aligned with the world, with concurrent speech, and with the interactants: 1. Making sense of the world at hand Much inconspicuous gesturing occurs while people and their hands are actively engaged with the world at hand. Gestures arise as a by-product of and in the service of practical action, disclosing features of the immediate scene, or otherwise involving the touching, feeling, grasping, and handling of whatever is at hand, and maybe the making of something from it. Gestural practices that are coupled with the world at hand are often excluded from the study of gesture, which is treated as movement in the air by empty hands. However, wherever cooperation involves the handling or making of things (including the making of graphic marks on paper), one finds manifold indexical, iconic, and symbolic actions of the fingers and hands, and often these are entirely indispensable, given the type of activity underway and the communication tasks that it raises for the practitioners (Ochs, Gonzales & Jacoby 1996). Among the ‘jobs’ that gestures do in the world-at-hand are: to structure the participants’ perception of objects; to disclose intrinsic, invisible features and affordances of things; to analyze, abstract, and exhibit action; and to ‘mark up’ the setting. Gestures can arise as minimal modulations of instrumental acts, performed for the benefit of the co-participants, e.g. in the context of demonstrations. 2. Disclosing the world within sight A somewhat similar framework of person-gesture-world alignment is realized when the gesture orients the participants to the visible world beyond the reach of the hands. This is the prototypical realm of pointing: an extended index finger or hand (or another array of body-parts whose constellation can be seen as a vector) enables the participants to coordinate their orientation so that they jointly focus gaze on a distant object, feature or location. Pointing is only one type of orienting behavior, however; posture, gaze-direction and head-orientation can serve similar purposes, as can the placing of an object in the interlocutor’s line of regard. And pointing is not universally executed with an extended index finger; other body-

Chapter 1. Manufactured understanding

parts used to point are the eyes, the lips, the head, and the feet. Gestures made in this mode serve spatial orientation as well as the sharing of sights: once a shared visual focus is established, what is seen together can then be structured by gesture; motions of the fingers and hands project lines, vectors, points, etc. onto seen scenes, indicating how what is visible should be regarded. The visual field – i.e. the culturally structured landscape within sight – is thus augmented and made intelligible by layers of cognitive-manual actions and forms. In this book, the first two ecologies are treated in one chapter (Chapter 4). 3. Depiction The third alignment type of hand gestures is realized when interaction participants turn away from the world and focus attention inside their interactional huddle, and, by their bodily orientation and positioning in space, mark off the space between them as territory of their interaction. The interaction space becomes the stage for hand gestures when attention is, however fleetingly, focused on the speaker’s gesticulating hand(s). This happens whenever gesture is employed as a representation device, to depict aspects of the talked-about world, whether concrete or abstract, real or imagined. When gesture is used to depict or represent, for example in the context of narrative, recipient attention is solicited for the gesturing hands, for example by means of deictic adverbs included in the talk. Descriptive (depicting) gestures represent worlds in collaboration with speech, and they are understood by reference to what is known about the world, not what is seen at the moment. 4. Thinking by hand: Gesture as conceptual action Thinking by hand involves the speaker’s hands producing schemata in terms of which utterance content or narrated experience is construed. This mode is usually lumped together with gestural depiction (under the label “iconic” or “imagistic” gesture); however, although such construal can be a means of gestural depiction (as well as pragmatic gesticulation), it also occurs on its own, within an alignment framework that is different from both gestural depiction and pragmatic gesticulation. I call this mode of gesticulation ceiving or ception3 (from Lat. cap, take; a pattern thus formed is called a cept). Ception is a bodily form of conceiving, i.e. of conceptually structuring content to be articulated in speech: for example, content can be “grasped” – i.e., made sense of – by a schematic prehensile action. One of the more common varieties of ceiving is realized whenever speakers, without attending to the process and without wishing to depict anything, use their hands to give form to – i.e. construe – content. Often we notice a congruence between the form of the gesture and the semantic profile of the word or phrase then chosen; . This term is also used by Talmy (2000) to refer to the unity of conception and perception. My use of the term is motivated by the imagery provided by its root.





Gesturecraft

thus, gesture appears to facilitate “the successive fabrication of thought during speaking” (v. Kleist 1807/8). 5. Displaying communicative action This alignment-type and functional genre involves the use of the hands in the embodiment of communicative action. This includes gesturing by which the communicative act performed (question, proposal, imploring, and so on); aspects of the pragmatic and syntagmatic structure of the unfolding utterance; and coherencerelations between successive or separate utterance parts are shown or foreshadowed. Gestures that display communicative action can be conceived as a variety of cepts, in that they each express an act of communication by a prehensile schema of the hands. A minimal version of this mode of gesturing are “beats” (McNeill 1992), i.e. hand-movements characterized by repetitiveness and synchronization with the speaking rhythm and signifying not so much by form itself than by changes in form. The pragmatic mode of gesticulation also includes pronominal referential gestures: pointing-like motions, often made by the thumb, which individuate and refer figuratively within the universe of discourse, marking acts of reference to non-present people, locations, or points in time. And it includes the display of the stance that the person takes towards an utterance or the content expressed. Pragmatic gesture encompasses all actions of the hands (and a variety of other body-parts, notably the face, head, and shoulders) by which aspects of the interaction are displayed. 6. Ordering and mediating transactions Gesticulation in the pragmatic mode includes an other-centered variety, which, because of its different ecology, should be mentioned as a category of its own. This is gesticulation that is also occupied with the communicative process, but addresses other interaction participants, whose actions it is intended to regulate. This mode of gesturing can involve touching or gesturing toward the other to elicit attention or to allocate a turn, to solicit response or attempt to silence it, or to manage the attention of others to one another. Their non-conversational prototype are the gestures of conductors (Ashley 2005). Pointing gestures (or gestures of orientation) also fall under the definition of this category, but are treated separately because of their coupling with the local environment, which not all interactionmanagement gestures have. Many gestures blend the display of communicative action with the regulation of the behavior of others, and the distinction between these two modes is of only limited analytic value: at one end of the polarity, gestures are aligned with what the speaker is presently doing and convey something about it; at the other end they are performed in attempts to structure the actions of other participants. Often a gesture is aligned in both ways at the same time, or the two functions are realized at different points within the same gestural action.

Chapter 1. Manufactured understanding

These typological categories overlap, and often a single gesture can be assigned to more than one category. Typologies are often of primarily heuristic value, because reality refuses to be neatly divided into mutually exclusive categories. This is certainly true for the unwieldy family of communication practices that we collectively call “gesture”. Still, the typology helps us organize our analysis of these practices, reminds us of the wide range of different uses to which gesture is put, and thus keeps us from drawing overly broad generalizations from a narrow data-set.

Overview Along with this introduction, Chapters 2 and 3 form the stage-setting part of this book: Chapter 2 sketches the methodology of interaction micro-analysis that has shaped this study of gesture; it identifies milestones in the development of this methodology and describes how gesture has been approached within an interaction-centered framework of study. Chapter 3 offers a brief account of the evolution and functioning of the organs with which we gesture, the hands. Its evolved anatomy is explained as the result of its changing grasping tasks, and grasping is described as the paradigmatic function that makes the hands cognitive organs as well: the hands give us the world about which we gesture. Readers who feel sufficiently familiar with the history of interactionist methodology and the functioning of hands can move ahead to the empirical inquiries into gesture, which begin in Chapter 4. Chapters 4 through 8 describe four practice genres of gesture, that is, “kinds of things” that we do with them; and these correspond to four of the ecologies outlined above: Chapter 4 investigates the gathering of meaning from the environment, that is, indexical gestures that interface with the world at hand or the world in sight; the examples mainly come from two settings: an auto-shop in Texas and a rice-farm in the Philippines. In Chapter 5, we turn to gesturing that is about the world, without being in it: the depiction of distant, remembered, or imagined settings. We observe how speakers, in depiction, turn to their hands. Data for this chapter come from diverse sources. Chapter 6 is devoted to the analysis of depiction methods or practices, and at the center of the chapter are three conversations: one about a play, Kleist’s Penthesilea, one on car-accidents, and one in which an architect describes a building-site. Chapter 7 is about ceiving, the conceptualization of content by gestures of the hand. The examples for this chapter were taken from talk about the human psyche, a phenomenon without manifest visual reality that could be depicted; nevertheless, it can be construed or conceptualized. In Chapter 8, finally, we investigate the ceiving of communicative actions, that is, gestures by which speakers display the communicative act or project that they are pursuing. Having studied samples of these heterogeneous gesture practices, we are





Gesturecraft

then in a position to summarize, in the final chapter, what we have learned about some of the competent ways in which people live their social and cognitive lives by gesturecraft and how this craft is sustained.

chapter 

Gestures as interaction On methodology

Gestures are interaction phenomena, and they are analyzed in this book with the methods of interaction micro-analysis or micro-ethnography. How this methodology has developed and how it can underwrite the project of gesture analysis is the theme of the present chapter. I begin by stating the founding idea of interactionism, that meaning and mind are the products of interaction. This idea was first articulated by George Herbert Mead (1934). It found resonance in Erving Goffman’s work, but also in the earliest attempts at analyzing film records of human interaction, which were framed by the natural history method (Bateson 1971), later re-conceived as context analysis (Scheflen 1973), and importantly complemented by conversation analysis (Sacks, Schegloff & Jefferson 1974) with its focus on the sequentiality of human interaction. These perspectives, and the methods of ethnography and visual anthropology, allow us to focus on hand-gestures as components of multimodal actions and contexts produced in ongoing interaction among co-present people, and to describe how gesture “works” in the process of interaction. A more recent research development in a number of disciplines is praxeology, a term which I use to refer to the so-called “communities of practice” school of thought (Lave 1988) in the aftermath of Vygotksy (1978), but which is also used by a group of French visual anthropologists who study gesture in the context of the transmission of cultural skills (Comolli 2003; de France 1983): a proper research perspective that emerges from this work is one we could characterize as visual praxeology, the visual study of human practices.

An interactionist approach to gesture Coincidentally or not, interactionism as a paradigm of thinking and research in the social sciences arose with a disagreement about gesture. A common way of dealing with gestures – that is, bodily communicative acts –, in modern times developed by Wilhelm Wundt, the founder of the first institute in experimental psychology, and espoused by generations of psychologists after him, is to regard gesture and other forms of bodily expression as exteriorizations of individual thought and affect.



Gesturecraft

Mead, who spent a year at Wundt’s institute, developed his own, contrarian view of the human mind and language as the evolutionary products of social interaction by recognizing in Wundt’s theory of gesture and language – formulated in his Völkerpsychologie (“Psychology of Peoples”, 1911) – the shortcomings of individualistic theories of communication, society, and mind. Mead developed these thoughts in a series of articles beginning in 1909 (Mead 1909) and again in Mind, Self, and Society (Mead 1934). Mead posited that, within a theoretical universe which assumes selves and psychic content as given, it is impossible to explain the possibility of communication. A type of theory which assumes individual selves as the presuppositions, logically and biologically, of the social process or order within which they interact. ..., cannot explain the existence of minds and selves. ... [A theory, however, which] assumes a social process or social order as the logical and biological precondition of the appearance of the selves of the individuals involved in that process or belonging to that social order, ... can explain that which it takes as logically prior, namely the existence of the social process of behavior, in terms of such fundamental biological relations and interactions as reproduction. (Mead 1934: 222–3)

Communication, then, to Mead and those among today’s researchers who share his interactionist perspective, is not a product of individual psyches or minds; rather, psyches and minds are epiphenomena of the social process of communication: Mind arises through communication by a conversation of gestures in a social process or context of experience – not communication through mind (50).

Borrowing an idea from Darwin, Mead suggested that a gesture is an “early part” of an act: by its very nature, it projects an imminent action. For example, when a dog bears its teeth, the other organism dog will respond to this gesture (for example, by retreating) as if it were the act itself. Thus, a conversation of gesture arises in which both animals adapt their behavior to each other’s anticipated response. Interaction displays that replace completed actions in social intercourse have survival value for any species: they make it possible to reduce open conflict between its members by ritualizing conflict in interactions. Gestures, Mead insisted, have evolved from physical actions, not from expressions of affect. Affect, though present as a driving force, is irrelevant to the meanings of behavioral displays. Meanings result from objective behavioral interrelations. Meaning is the relation between the gesture of a given organism and the subsequent behavior of this organism as indicated to another organism by that gesture. ... (It is) a relation between certain phases of the social act. (Mead 1934: 76)

The response also constitutes an analysis, an “interpretation” of the gesture to which it reacts:

Chapter 2. Gestures as interaction

Throughout the entire process of an interaction, we analyze the incipient actions of others by our own instinctive reactions to changes in their postures and other signs of developing social actions. (Mead 1909: 219)

This is an important statement, for it posits intersubjectivity as a public phenomenon, as an overt aspect of sequences of action, not as silent mutual understanding between fundamentally separate, individual minds. When the understanding of a gesture by other and self is shared, the gesture becomes a “significant symbol”. Thus, it is by apprehending its own gesture in light of its interpretation by the other, displayed by the other’s overt response, that the individual organism apprehends the meaning of its own gesture, which it can then internalize and constitute as a ”mental” phenomenon. Mind depends upon social symbols and interaction, and thought develops through the internalization of the conversation of gestures. From the beginning, the individual psyche is socialized; its desires and capacities are derived from and geared towards interactive living, and its contents are acquired from the social sphere. Gesture is not in the first place a means of expression; it is a component of social acts. Mead thus taught us to see gestures as components of acts – components capable of relative autonomy and distinction – and the gestural act as being made significant through reciprocal interaction. How other responds to my movement of the hand is what matters. Act and response are constituted in lived time, during irreversibly unfolding moments of shared engagement. Gestures enable the forward-adaptation of actions during interaction: they enable us to act, not on the basis of what is happening now, but what is likely to come next. As will be seen, the conversation of gestures, as Mead conceived it, describes certain modes of gesturing by hand better than others, pragmatic gesturing better than depictive uses of the hands. What one could envision as Mead’s empirical program – the study of the ways in which gestures (quasi-separate action components) and other significant symbols facilitate and mediate social interaction – did not, however, come into being until some fifty to sixty years after he first described the conversation of gestures. And even then, Mead’s own important contribution was occasionally overlooked or forgotten, or, in the case of conversation analysts, curiously denied. In the remainder of this chapter, I will sketch the most important stages in the development of the micro-analysis of interaction within which the present study of gestural practices is situated. I begin with an exposition of the natural history method (McQuown 1971), then make a few remarks on Erving Goffman’s work, move on to context analysis, Adam Kendon’s research on gesture, conversation analysis, and praxeology. I will only have a few things to say about the use of film and video in interaction research. After this overview I sketch the ecological framework that is an original contribution of the present work and used to distinguish a handful of fundamentally different ways of using gesture to produce





Gesturecraft

understanding, among people and of their worlds. This ecological framework for the study of the craft of gesticulation enables us to distinguish the ways in which hand gestures relate and refer to the world. The chapter ends with a few essential conceptual distinctions that are covert in the English language and therefore often overlooked, with confusion and false assumptions being the inevitable result: distinctions between gesture (as praxis) and gestures (i.e. instances of gesticulation); between culture and cultures; and between making and using gestures.

Gregory Bateson and the “natural history approach” The anthropologists Gregory Bateson and Margaret Mead studied an entire people, the Balinese – instantiated by the inhabitants of a single village, Bajoeng Gedé – by examining their bodily comportment as they engage in interaction with one another or in solitary activities. They used photographs and, to a lesser extent, film, to capture the embodied culture of the Balinese, that is, “the way in which they, as living persons, moving, standing, eating, sleeping, dancing, and going into trance, embody that abstraction which (after we have abstracted it) we technically call culture” (Bateson & Mead 1942: xii). Bateson and Mead’s interest was not focused on the details and sequences of interaction as such, but on the broader abstractions that could be studied in them. For example, to shed light on “the importance of levels in Balinese interpersonal relationships” and the “intangible relationships among different types of culturally standardized behavior” (ibid.), they placed photographs from different domains of Balinese life side by side: Pieces of behavior, spatially and contextually separated – a trance dancer carried in a procession, a man looking up at an aeroplane, a servant greeting his master, the painting of a dream – may all be relevant to a single discussion; the same emotional thread may run through them” (ibid.)

In this fashion Bateson and Mead examined notions such as balance, systems of spatial orientation and respect, attitudes towards bodily orifices, modes of bodily (kinesthetic) learning, and uses of the fingertips across different activities. While the couple spent a great deal of time observing parent-child and sibling interactions, few scenes were analyzed in a manner approximating micro-analysis. An example is a sequence involving a young mother and her infant son, in which Bateson recognized what he regarded as the “anti-climactic” values of Balinese culture, a favoring of “steady states” (Bateson 1949). Of greater interest to us are Bateson and Mead’s observations on Balinese modes of using the hands. For example, they noted great “emphasis on the separateness of the fingers and on the sensory function of their tips” (Bateson & Mead 1942: 100). While the greater sep-

Chapter 2. Gestures as interaction

arateness of the fingers is visible in the ease with which the Balinese adopt complex hand postures during diverse activities such as dancing and tying a spur to a fighting cock’s foot, as well as in their rest positions, sensitivity of the finger-tips is visible, for example, during food-preparation or in the careful ways in which an object is stabilized by the left hand as the right hand performs some action on it. The differential allocation of roles to the two hands, partly the result of the cultural definition of the left hand as unclean, results in a pattern in which, roughly, the left hand is dedicated to feeling and holding objects, the right to using tools (Streeck 2003a). This separation of roles is the result of persistent kinesthetic training: The Balinese mother or nurse carries a child, either in or out of a sling, on her left hip, thus leaving her own right hand free. In this position, the baby’s left arm is free, while the right is frequently pinioned in against the breast, or at best extended behind the mother’s back. Naturally, when a baby is offered a flower or a bit of cake, it reaches for it with the free left hand, and the mother or the child nurse invariably pulls the left hand back, extricates the baby’s right hand – usually limp and motiveless under this interference with the free gesture – and extends the right hand to receive the gift. This training is begun long before the child is able to learn the distinction, begun in fact as soon as the child is able to grasp at a proffered object, and discontinued usually when the child is off the hip. (Bateson & Mead 1942: 100)

If the hands – in both their enactive and sensory skills – are to such an extent subject to cultural patterning, one could hypothesize that some of this patterning will also extend to gesture. While Balinese Character is a pioneering work of visual anthropology and of great importance to the study of the enculturation of the body, Bateson’s later work on context and on the natural history method in the study of human interaction has had a more direct influence on the methodology of micro-analysis. When Bateson, who also conducted research on animal communication, observed young sea-otters interacting, he noticed that they engaged in behaviors that were partly identical with those they would show in fight – and yet they refrained from injuring each other. Bateson concluded that there must be a class of behaviors by which otters inform one another that the actions that they engage in are play (Bateson 1956). This observation led him to coin the important concepts meta-communication and context. Certain behavioral displays (in animals and humans) serve to define the present context of the interaction (e.g. as play), which in turn constrains how behavioral messages exchanged within its boundaries are to be taken. Referential communication and meta-communication (communication that defines the context) constitute two distinct logical types (Bateson 1972). However, it is important to consider each action “as part of the ecological subsystem called context and not as the product or effect of what remains of the context once the piece which we want to explain has been cut out from it” (Bateson 1972: 338).





Gesturecraft

In other words, context and act reflexively constitute one another. These ideas later became the cornerstones of the new field, context analysis, that focused microanalytic attention on the behavioral processes by which contextual frames are created and maintained, for example the postures that interaction participants take on over time, in relation to and in concert with one another. The work, however, that is considered the beginning of the micro-analysis of human interaction was a collaboration between Bateson and the anthropologists, linguists, and psychotherapists Ray Birdwhistell, H. W. Brosin, Charles Hockett, N. A. McQuown, H. L. Smith, and G. L. Trager (McQuown 1971; see Leeds-Hurwitz 1987, 2005). This collaboration centered on the detailed analysis of a single interaction (in which Bateson interviewed a psychiatric patient, “Doris”, in the presence of her young son.) With the exception of the kinesic analysis of a single scene (in which Bateson lights Doris’ cigarette; Birdwhistell 1972), the results of this research (which took place at the Center for Advanced Study in the Behavioral Sciences at Stanford University) were never published (other than on microfilm; McQuown 1971). But this did not diminish the influence of the work. Especially Bateson’s conception of the natural history method of interaction research would later inform the deliberate theoretical austerity of approaches such as context analysis and conversation analysis. The method was called the ‘natural history’ method because, in Bateson’s words, “a minimum of theory guided the collection of the data” (Bateson 1971: 6). As Kendon wrote in his historical account of the development of this methodology: The natural history approach proposed the detailed description of whatever could be observed in an interaction. Since what was sought for was an understanding of the natural orderliness of interaction, observations must be made in terms of what is there to be observed, not in terms of pre-established category systems. To decide what will be measured and counted before this is done will prevent the very understanding that is sought. (Kendon 1990: 20)

In what was supposed to have become the introductory chapter to The Natural History of an Interview, Bateson (1971) laid out the principles of what would later be called “context analysis”, by defining meaning as a relation between behavior (or form) and context. He took inspiration from Gestalt psychology’s figure-ground hypothesis, which states that human experience (such as the perception of an object) always appears as a figure before a background; meaning is not inherent in the experience (or the object) as such, but results from its relation to the given ground. For the micro-analyst, this implies the need to simultaneously work top-down, specifying hierarchical levels of context in which a bit of behavior is embedded, beginning with the most inclusive, yet distinctive level, and bottom-up, identifying minimal behavioral units out of which complex activities are fashioned: “our

Chapter 2. Gestures as interaction

data are the individual signals or messages, each in its immediate and extended context” (Bateson 1971: 20). As we climb the hierarchic ladder of Gestalten from the most microscopic particles of vocalization towards the most macroscopic units of speech, each step of the ladder is surmounted by placing the units of the lower level in context (p. 16).

Meaning can only be unequivocally identified “when large units of the communicational stream are admitted to examination” (p. 17). Bateson believed that linguistic methodology had already demonstrated that “there will always be one or more hierarchies of Gestalten which will be correct in the sense of describing how the message stream is created and/or how it is received and interpreted by the hearer” (Bateson 1971: 14). He regarded it as a historic accident that the disciplines of linguistics and of kinesics, the term given to the method by Ray L. Birdwhistell (Kendon & Sigman 1996), developed separately: That the scientists have become specialized in this particular way does not indicate a fundamental separateness between these modalities in the stream of communication. . . .It is for this reason that [our] work . . . starts from concrete natural history – from the recorded interaction between Doris’ speech and movement and the speech and movement of Gregory (p. 19).

Bateson hoped that kinesics could be developed in analogy to structural linguistics – a position that became the basis for Birdwhistell’s program of kinesics (Birdwhistell 1979). But, more importantly for us, Bateson also laid the foundation for an interactional view of context: the context of a signal emitted by [the speaker] . . . is not merely those other signals which she has recently emitted plus those which she emits soon after; it is also the room in which she is speaking, the sofa on which she is sitting, the signals emitted by [the person] with whom she is talking . . . and the inter-relationships among all of these. . . . At this point our concept of communication becomes interactional and our intellectual debt is to G. H. Mead. (Bateson 1971: 20)

Goffman’s micro-studies of the interaction order Someone who was deeply influenced by Bateson’s ideas about social interaction and its context, but whose own work was perhaps even more influential in the development of micro-analysis, was the sociologist Erving Goffman1 (see especially Goffman 1961). Goffman’s interest in social interaction was focused on the . For comprehensive overviews and accounts of Goffman’s work, see Ditton 1980; Drew & Wootton 1988; Fine 2000; Manning 1992; Riggins 1990. For an account of Goffman’s impact on the micro-analysis of interaction see Kendon 1988.





Gesturecraft

emergence and workings of the public order, that is, social order, both enabling and restrictive, which emerges from the fundamental fact that during moments of co-presence, individuals organize their bodily actions in terms of their visibility to others: whatever move someone makes, it is responsive to whether or not it can be seen by others, and if it can, how they will see it. From visibility and reciprocity come reflexivity, i.e. the ability to take a stance on the acts of self, and perspectivetaking, the ability to relate to self ’s acts from the assumed vantage-point of others. Goffman has described a host of gestural acts that display such self-awareness, for example body glosses, acts by which people who have committed a faux pas display their awareness of having done so, perhaps even what made them blunder. Goffman helped to establish the face-to-face domain, the primordial domain of human social organization, as an analytically viable domain of study, amenable to micro-analysis. He regarded the interaction order as a “relatively autonomous form of life” (Goffman 1983: 9), enacted not so much by individuals as by participation units such as withs (units comprising more than one individual, acting in concert) and solos. In public life, solos often exhibit a self-reflexive attitude, attesting to the primordially dialogical nature of human existence. Where people are mobile, they orchestrate themselves into momentarily sustained arrangements within which differential positionings and postures display and facilitate differential participation roles (eaves-dropper, bystander, focal participant, and so on). Types of gatherings are distinguished by the ways in which the participants allocate their attention (unfocused and focused, single-focused and multi-focused gatherings). Participant frameworks can change over time during encounters; such changes are brought about by the participants’ changing ways of looking at and away from one another (Goodwin 1981). Goffman made sociologists aware that it is possible to study sense-making and social organization without conjecturing the content of subjects’ minds. It is possible because these are public phenomena and processes, enacted by the participants’ visible (or, more generally, overt) doings. Nothing is available to the parties other than what they make available to each other; communication results from and in inter-subjectivity.

Context analysis According to Bateson, communication takes place at two different logical levels: at one level, the participants communicate “messages”; at the other, they “metacommunicate” by informing one another about the context or frame within which the given message operates. Context analysts have been interested in the behaviors through which these contextual frames are constructed, displayed, and calibrated, in the methods and devices by which the parties tacitly let each other know what context they are in. The initiator of context analysis as a system-

Chapter 2. Gestures as interaction

atic and sustained empirical research effort was the psychiatrist Albert Scheflen (Scheflen 1964, 1973). Scheflen observed during his studies of therapeutic encounters that patients and therapists collaboratively assemble posture-configurations and sustain them over extended periods of time. When they change postures, they typically change them together and produce a new configuration which is then sustained until another collaborative rearrangement occurs. Scheflen concluded that posture-configurations serve as embodied frames or contexts within encounters and that posture-shifts cue topic shifts or other changes in the activity carried out by the parties. A posture-configuration, collectively sustained, can thus be regarded as an “embodied answer” to the parties’ question “what is going on?” (McDermott & Aron & Gospodinoff 1978: 247; McDermott & Roth 1978). By assembling postural configurations, interaction participants display their working consensus (Goffman 1959: 9–10). Scheflen’s focus was on the overall “program” of interaction events, and his empirical research was mainly devoted to events that have a relatively repetitive structure, namely family-therapy sessions. He made anecdotal observations on some gestures, for example those he labeled “monitors”, which serve “to regulate or extinguish a deviancy and thus maintain the usual course of events” (Scheflen 1972: 105). But otherwise, Scheflen offered only one detailed contextual analysis of a gesture. The gesture is a bilateral positioning of the cupped supine hands, which the researchers labeled “the bowl” (Scheflen 1974: 137–154). This gesture, according to Scheflen, occurred again and again exactly nineteen minutes into the family-therapy sessions with a certain mother-daughter pair. The recurrence of the gesture marked it as customary. Scheflen’s objective was to establish the function that the gesture served with respect to the program of the occasion: “the bowl gesture occurred each time in the same configuration: [the patient’s] seductive-type behavior, relating a daydream, postural shifting . . . this is how patterned human behavior is” (Scheflen 1974: 143). Invariably, according to Scheflen, “the bowl unit marked the transition from the covert to the overt form” of courtship behavior (p. 143), which then allowed the therapists to openly discuss issues of the patient’s sexuality. Marking this transition was the contextual function of the bowl gesture. The stability of embodied contexts such as posture-configurations is itself the product of continuous spatial and kinesic maneuvering by the parties. Changes in the configuration which, at first glance, simply appear to “happen”, reveal themselves upon analysis to be accomplished through sequences of interactional moves, involving “pre-enactments”. The “systems of behavioral interrelationship” (Kendon), in other words, that are the central concern for context analysis are assembled through sequentially organized interaction. This was demonstrated by Adam Kendon’s studies of the spatial organization of encounters (Kendon 1976a, 1990). Kendon thus presented a more dynamic version of context analysis (Kendon 1992b), compared to Scheflen’s rather static, hierarchical view of





Gesturecraft

contextual organization (Scheflen 1973). Of even greater importance to our project are Kendon’s contributions to the methodology of studying gesture in interaction.

Kendon on gesture Kendon’s initial focus was on the coordination of gesture and speech and the roles that gesture plays in the constitution of meaning in interaction (Kendon 1972). He regarded gesticulation and speech as distinct, yet interrelated aspects of the “process of utterance” (Kendon 1980) – a view that differs, for example, from McNeill’s, who regards speech and gesture as parts of the same psychological system “language” (McNeill 1992). While gestures emerge along with vocal components, those that become parts of a shared communal repertoire also undergo processes of evolution in which they can become like words (Kendon 1988a). In his studies of a signed language used by deaf people in New Guinea (Kendon 1980a–c) and alternate (taboo-based) sign-languages used among bereaved women in Aboriginal Australia (Kendon 1988b), Kendon showed that the morphological properties of these systems, that is, the formational properties of signs, cannot be explained without thorough consideration of the features of the human body by which they are articulated: Sign languages are organized in the kinesic medium, the medium of visible bodily movement. . . This has properties very different from the medium of speech. Whereas, in speech, linguistically significant units can be organized in temporal sequence only, in the kinesic medium forms may be constructed which contrast in their spatial organization, as well. Furthermore, the instrument of sign language expression, the body, has a number of spatially separated parts which can be used as articulators, simultaneously. (Kendon 1988b: 6)

Kendon thus emphasized the materiality of the body that produces signs, and this emphasis also characterizes some of his research on hand-gestures during spoken conversation, for example when he explores the practices by which speakergesturers seek to bring speech and gesture into alignment, which requires constant attention to the physical (temporal and spatial) properties of both modalities (Kendon 2004: Chs. 7, 8). In his studies of gesture from 1980 on, drawing on video-recordings of everyday interaction in the U.S., England, and Southern Italy, Kendon has adopted a functional perspective on gesture, closely examining their formational properties, their integration with speech, their semiotic properties, and their roles in interaction. A natural unit of gesture action is initiated when the hand begins its excursion from a position of rest – or wherever it is presently located in the pursuit of

Chapter 2. Gestures as interaction

practical action –, and ends when the hand returns to its place of rest or resumes the practical activity that the gesture temporarily suspends. An act of gesticulation typically comprises several stages: a preparation phase during which the hand travels to the position from which it executes the stroke, and during which the hand is configured to a particular shape; the stroke itself, that is, the main expressive movement that viewers recognize as “the gesture”; and finally the return. Often, there is a post-stroke hold (Kita 1993; Kita, van Gijn & van der Hulst 1998), a freezing of the hand in the location that it reaches at the end of the stroke and from which it may then perform another stroke, either of the same or of a different kind. Kendon calls a unit of gesticulatory action – from onset to return – the gesture unit. Gesture units can comprise one or several gesture phrases, for example a series of repetitive strokes, then a series of different strokes, perhaps with a differently configured hand, then perhaps a resumption of the first stroke pattern, and so on, until the hand finally returns to its rest position. Such extended gesture-units are common when interaction participants’ produce extended turns-at-talk. We may note that, in the context of conversation, it is usually not difficult at all to identify gesture units and their boundaries. In contexts of work, however, gestural communication may consist in nothing more than a repetition or a slight embellishment or exaggeration of an instrumental act. The boundaries of these communicative micro-acts may not be obvious, because the very distinction between instrumental behavior and communicative behavior may not be: a communicative dimension may simply come into play by the fashion in which an instrumental act is carried out. Many gesture researchers have noted the frequent simultaneity of the gesture stroke and the “information center” of a unit of speech. As Kendon emphasizes, this is an achievement (Kendon 2004: 127), made possible by the mutual adjustment of delivery in either medium: speech can be slowed down to accommodate the production of a complex gesture phrase (p. 143), or interrupted to “allow gesture to stand on its own” (p. 147). The simultaneity of the gesture and the semantically affiliated unit of speech is not, in other words, an automatic process. An implication of this complexity of the speech-gesture coordination is that it is never sufficient during research to identify tokens of a certain gesture, count their frequency, and correlate these with some other variables: what a token of a gesture does and how it is taken is contingent on its precise temporal relationship to the utterance or utterance sequence within which it occurs, the unit that it precedes, the ones that it follows, and on how vocal and kinesic units are delivered moment by moment in relation to one another. Kendon examines the ways in which hand gestures contribute meaning to utterances under the term “semantic interaction” (Kendon 2004: 158). He distinguishes referential, pragmatic, and interactive (interpersonal) functions. Referential gestures contribute to the propositional content of utterances: they are part of what





Gesturecraft

is being predicated of something. Among pragmatic gestures (which contribute non-referential meaning), Kendon distinguishes modal gestures, which mark the “frame in terms of which what is being said . . . is to be interpreted” (p. 159), for example as in some way being related to something that has been said before, or as stated without conviction; performative gestures, which display the speech act that is being performed; and gestures which parse the utterance, that is, display its syntagmatic structure. Interactive gestures include those that indicate the addressee of an utterance, a participant’s claim to the floor, and other regulatory aspects of the interaction process. Kendon has identified several “gesture families”, that is, variants of types that are related to one another by family resemblances. One such family is the “G-family”, gestures using a “finger bunch” (or grappolo). Kendon’s analysis of this type demonstrates the importance of micro-analysis if one wants to establish the functioning of a gesture in its context: the closing of the hand to a finger-bunch occurs when – and displays that – the speaker “establishes a topic which is to become the focus of attention” (Kendon 2004: 231); the subsequent opening of the hand “is associated with the comment”, i.e. the part of the spoken utterance that predicates something of the topic. If the coordination of the gesture with the moment’s talk were not rigorously and precisely taken account of, it would be impossible to establish this dual function of the gesture. Micro-analysis always combines the analysis of form with attention to the exact moment of production, in relation to the ongoing utterance and the actions of the co-participants. Gesture, like speech, is situated in spaces, and its production is tied to the interactional structuring of the spatial setting. Gesture itself symbolizes by constructing structures in space. The shape, orientation, trajectory, and motion pattern of a gesture can therefore not be explained solely by reference to the content that is conveyed through talk and gesture; rather, the features of gestures and the effects that they achieve are also contingent upon the locale and spatial organization of the encounter. Kendon has suggested that “kinesic components” that accompany speech simultaneously participate in, and contribute to, several contextual systems, namely a locational system, by which the individual maintains his location to both the gathering of co-present individuals and the physical environment; ... systems concerned with the production of referentially encoded meaning, ... and systems that have to do with the ways in which the individual manages his interaction with the physical environment. Any given piece of behavior that we might look at ..., say, a phrase of movement, ... is not to be classified as belonging exclusively to one or other of these functional systems. Rather, it must be regarded as participating in these systems. Further, it may, and, indeed, usually does, participate in several of these systems simultaneously. (Kendon 1989: 2)

Chapter 2. Gestures as interaction

Conversation analysis Like other theory-minded sociologists, ethnomethodologists, including those who study talk in interaction, are primarily interested in sociology’s foundational problem: how is social order possible? Ethnomethodologists believe that social order not only constrains and patterns social life, but is also the product – an “ongoing accomplishment” – of situated social action (Garfinkel 1967; Heritage 1984). Therefore, it is necessary to study the practices by which social order is achieved, the “ethno-methods” by which a society’s members bring about the familiar scenes and events that make up their routine everyday life. Among these methods are those that enable us to carry out the most ubiquitous activity which, more than any other, makes us human: talk. To study talk (i.e., the methods by which talk in interaction is organized) is one possible and important way to study the production of social order. For conversation analysts, the most important unit of analysis is action. Action is “omnirelevant” in social life and interaction (Schegloff 1995). For each unit of talk and behavior, there is “virtually always an issue ... for the participants ... of what is getting done by its production in some particular here-and-now” (187, my emphasis). The first order of business for the analyst, therefore, is to establish what action a bit of behavior or the uttering of a linguistic form constitutes for the conversationalists. The criteria for this analysis are inherent in the interaction itself, because the parties demonstrate by their responses what kind of action they have seen each other perform. In contrast to other approaches (e.g., speech act theory, discourse analysis, and the ethnography of speaking) conversation analysts have not relied on members’ categories of communicative actions, but rather sought to find and describe methods and devices which we routinely deploy as interactants, but often cannot name or explain. We have no words for many of the speech acts that we routinely perform, nor are we aware that we perform them. As analysts, we therefore cannot consult our own competence or members’ judgements to identify the functions of many conversational forms. The research process centers on the discovery of overt phenomena – phenomena that are “external” and thus public and brought about and attended to by the interactants themselves, and yet often unknown to them. Conversation analysts seek to account for the interactional work done with such “conversational objects” and “formats” in sequences of action. This orientation to “undisclosed” background phenomena and practices reflects the phenomenological background of conversation analysis. Conversation analysts work from the assumption that human societies have “as a central resource for their integration . . . an organization of interaction informed by the use of language” (Schegloff 2007: xiii). This organization itself comprises several sub-organizations that deal with genres of contingencies with which





Gesturecraft

interaction participants continuously have to reckon if there are to be intersubjectivity and social order (see Schegloff 2007: xiv). These contingencies include, among others, the turn-taking problem, i.e. the need to take or allocate turns at talk one at a time, in succession – a necessity given the linear nature of speech and the systemic inability of humans to produce speech and understand the speech of others at the same time –, and the attendant constraints upon the organization of utterances as turns; the action-formation problem – “how are the resources of the language, the body, the environment, and position in the interaction fashioned into conformations designed to be, and to be recognizable by recipients as, particular actions” (ibid.); the sequence-organizational problem, that is, the need to design turns at talk so that they are recognizably responsive to prior actions and/or show what response actions they seek to engender; and the trouble problem, that is, the need to deal with contingencies resulting from actual or possible problems of hearing and understanding. Gesture is relevant to and impacted by all four organizations. Thus, gestures can display a participant’s request for the next turn (Mondada 2007) or the speaker’s solicitation of a response (Streeck 2007). Gestures can show, often at or even before turn-beginning, the type of action a turn is going to be (Streeck & Hartge 1992; Streeck 2009a) or combine with talk and other resources to form multimodal actions (Bolden 2003). The continued production or repetition of a gesture can demonstrate that a sequence is not complete (Streeck 2008a) which exemplifies how gesturing is constrained by and contributes to sequenceorganization. And gestures have been found to be operative in word-searches (Goodwin 1986; Streeck 1993) and self-repair (Seyfeddinipur 2006). Nonetheless, there has actually been relatively little research on gesture within a conversationanalytic framework (but see Lerner 2002; Schegloff 1984) – most certainly due to the fact that many of its practitioners regard talk in interaction – and not interaction per se or action in interaction – as the crucible of human sociality; gesture, accordingly, is relegated to its customary role as a hand-maiden to speech. Some conversation analysts, however, notably Christian Heath and Charles and Marjorie Goodwin, have always considered bodily action to be as important as talk. Heath (1982, 1984, 1986, 1992) has investigated the organization of talk, bodily conduct, and interactional participation in medical consultations, focusing in particular on how coordinations of body-motion and talk are implicated in achieving the various stages of the event (opening, medical history-taking, physical examination, delivery of diagnosis, and leave-taking). He described how patients conduct themselves so as to attract the physician’s attention to their gestures (Heath 1986: 87–92) and showed that hand gestures as well as movements of other body-parts in and out of medical consultations are often used to solicit the gaze of a co-participant occupied with other activities (e.g. note-taking) – that they have the potential of attracting attention to themselves. Gestures can solicit increased

Chapter 2. Gestures as interaction

co-participation and encourage visual displays of resiliency and agreement (Heath 1992). In general, they “transform the environment within which [talk] . . . is received” (Heath 1992: 109). Heath’ work discloses the sequential nature not only of talk, but also of embodied conduct. Charles and Marjorie Goodwin have investigated the work done by gestures within specific (conversational) activities, writing that by investigating gesture within particular events, it is possible to begin to study in some detail not only how participants find it to be meaningful, but also how they use that meaningfulness as a constitutive feature of the social organization of the activities they are engaged in . . . Gesture obtains its meaning by virtue of its placement within an activity which is clearly recognizable to the participants. (Goodwin & Goodwin 1986: 51)

The activity within which they initially studied gestures were word-searches. These are made recognizable by characteristic features of speech-delivery (pauses, hesitations, perturbations) and embodied conduct (gaze withdrawal, a “thinking face”, p. 57). While these features cue recipients as to the activity underway, the talk prior to the word-search usually indicates what category of word is being sought and what could constitute a solution to the search. Within this highly specified context, recipients can then make sense of the gesture that the speaker performs in lieu of the word, and, when they succeed in finding a word or phrase that expresses its meaning and put it in the proper grammatical frame, they effectively solve the word-search and complete the talk for the primary speaker. Building on his seminal work on the sequencing of gaze during speakerlistener interaction (Goodwin 1981), Charles Goodwin also investigated differences in recipients’ gaze to gesture as opposed to other types of bodily action. He found that, while (depictive) gestures attract recipient gaze, self-touch – and especially facial self-touch – tend to drive their gaze away (Goodwin 1986). Therefore, these different types of manual action can be used to differentially organize co-participation: while gesture can serve to increase the recipient’s active participation, self-touch can decrease it and thus help to bring about a state of mutual disengagement, potentially ending the encounter. In his more recent work (Goodwin 2000a, 2003, 2007), Goodwin has examined gesture not only in the context of activities, but also the material environment, which he conceives as an ensemble of semiotic structures. He writes that a theory of action must come to terms with both the details of language use and the way in which the social, cultural, material, and sequential structure of the environment where action occurs figure into its organization. . . . Human action is built through the simultaneous deployment of a range of quite different kinds of semiotic resources. . . . Both talk and gesture can index, construe or treat as irrelevant entities in the participants’ surround. (Goodwin 2000a: 1489)





Gesturecraft

Such “contextual configurations” (p. 1490), which also include the participants’ bodily and cognitive orientations, can change at any time (with changes themselves being brought about by participants’ actions), and particular kinds of spatial and linguistic work may have to be undertaken by the actors to have their gestures recognized and make them effective as social actions. How much interactional and interpretive work may be required to make sense of gestures is shown in Goodwin’s studies of conversations between an aphasic man, whom a severe stroke left with a repertoire of only three words, but an undiminished ability to gesture, and his family (Goodwin 1995, 2000b). He shows that, due to the family members’ methodical and sustained efforts, this man was able to maintain his status as a competent speaker (and story-teller), even though he no longer could speak (Goodwin 2004). Goodwin’s work, which locates gestures squarely within the multimodal fabrics of social encounters and collaborative action, demonstrates that, if we want to identify and explicate the significance of gestures, it is not only impossible to abstract them from the utterances and communicative action-sequences in which they occur, but also from the spatial (bodily and built) environment. More often than not, significant gestures are “environmentally coupled” (Goodwin 2007).

Ethnography The work of the Goodwins, Heath’ more recent work (Heath & Luff 1992, 1996), this author’s research and the work of a number of other researchers (especially Koschmann & LeBaron 2002; Koschmann et al. 2007; Mondada 2007), blur the distinction between context analysis, conversation analysis, and (micro-) ethnography. Attentive to the turn-by-turn, sequential organization of interaction, all of these authors investigate gesture within the context of activities in complex social and material environments. Prior to the contemporary efforts in micro-analytic ethnography, there have been a few isolated attempts to study the uses of hand-gestures (or of specific types or families of hand-gestures) in their ethnographic contexts, beginning with Andrea de Jorio (2000 (1832)), whom Kendon (1995) called “the first ethnographer of gesture”, because of his careful and detailed observations and descriptions of the uses of conventional gestures in the everyday life of Naples. De Jorio not only described the gestures’ forms and explicated what they signified, in the process exhibiting the semantic mechanisms involved, but also delineated the typical circumstances under which these gestures were being used. What was genuinely new about de Jorio’s approach was that he treated gesture “as a component of Neapolitan culture, showing its interrelationships with Neapolitan life and language as well as . . . its continuity with antiquity” (Kendon 1995a: 374). De Jorio explained

Chapter 2. Gestures as interaction

in explicit detail his painstaking methods for interpreting gestures, for example the need to pay attention to “1) the general position of the entire body. 2) The expression and movement of the face as well as that of the eyes. 3) The positions of the palms, of the fingers, and their movement”, as well as the subject of the conversation (DeJorio 2000 [1832]: 34). Equally important to him was the context or what he called the “subject of the situation” (36), which provided the basis for systematic variations in meanings of recurrent gestural forms. In this fashion, de Jorio compiled an extensive, ethnographically based “ABC” of the gestures used on the streets of Naples in the early part of the 19th century. Kendon has, in a sense, resumed de Jorio’s work in a modern, micro-analytic vein, for example by studying differential hand-shapes used in pointing in Naples (Kendon 2003) and by reflecting on the ecological conditions (or communication economy) that favored the rise and durability of the unusually rich gesture tradition of Naples (Kendon 2004: Ch. 16). Another important contribution to the study of gesture in everyday life, published more than 100 years later on the basis of research in New York City, and billed, surprisingly, as an “experimental study”, was David Efron’s Gesture, Race and Culture (Efron 1972 [1941]). Efron, a student of the cultural and linguistic anthropologist Franz Boas, treated gesture as a component of culture, aiming to falsify the racist ideology of Nazi “anthropologists” who, following a tradition begun during the Spanish inquisition (Schmitt 1990), treated expression movements of Jews as indicative of “racial”, i.e. biological, traits. To counter this claim, Efron, who drew on de Jorio’s pioneering work, compared conversational gestures of first- and second-generation Eastern European Jewish and Southern Italian immigrants in New York City, demonstrating that the gestures of newly arrived people from the two groups differed massively both in kinetic style (“spatio-temporal aspects”, p. 67) and function (“referential aspects”, ibid.). However, these styles showed abundant signs of hybridity and convergence with the American mainstream among the second generation of immigrants. Gesture, in other words, revealed the properties of diffusion, hybridization, externally induced change, and transmission by learning that was known from other modes of cultural behavior, notably language. Efron drew these conclusions from a massive, empirical and, in part, “micro-analytic” research effort that comprised (1) direct observation of gestural behavior in natural situations, (2) [2000] sketches drawn from life . . . under the same conditions, (3) rough counting, [and] (4) motion pictures studied by (a) observations and judgements of naïve observers, and (b) graphs and charts, together with measurements and tabulations of the same. (Efron 1972 [1941]: 66)

Since then, only smaller, far more circumscribed ethnographies of gesture use have been published. Among these are Sherzer’s studies of lip pointing among the Kuna





Gesturecraft

of Panama (Sherzer 1972; see also Enfield 2001), and of the thumbs-up gesture used in Brazil (Sherzer 1991), which both explore a single, highly standardized form in its manifold context of use in everyday life. Haviland has studied pointing gestures in societies with different systems of spatial orientation and representation (Haviland 1993, 2002); and Brookes has produced detailed ethnographic and video-based studies of the multifunctionality of a number of “quotable gestures” or “emblems” among young male South-Africans (Brookes 2001, 2004, 2005). With the exception of Haviland’s studies of pointing, these ethnographic studies examine single, conventional forms and their uses across contexts; the study presented here, in contrast, focuses on gestural practices, which may or may not include conventional forms.

Praxeology A loosely connected field of studies that has some bearing on the research presented here has evolved in the social sciences (mainly in the U.S.) in recent years, owing in large part to the belated discovery of the work of the Soviet psychologist Vygokstky (1978) in the U.S. Vygotsky, who lived from 1986 to 1934, founded the “cultural-historical” school of psychology (Wertsch 1981a; Wertsch, Râio, & Alvarez 1995), which also became known as activity theory (Cole, Engeström & Vazquez 1997; Engeström 1987; Wertsch 1981b). Central for Vygotsky was the realization that human minds cannot function without cognitive tools (such as concepts, methods, language, media), which are the historical products of communities and acquired through social practice. “External” – or inter-individual – practice always precedes “internal”, i.e. intra-individual, functioning, and it produces cognitive and communicative tools that actively participate in individual cognition. Paralleling a view expressed by G.H. Mead (Mead 1934), Vygotsky considered thought to be an internalized form of dialogical praxis. The discovery of Vygotsky’s work has given center stage to Marx’ concept of praxis in a growing field of research on cognition, learning, and communication, and led to the conception of communities of practice as the primordial sites for their study (Lave 1988, 1991). Instead of studying cognitive labor in laboratory settings that isolate the individual not only from other actors, but also the material and symbolic environments within which most human cognition naturally takes place, researchers in the Vygotskyan tradition have turned to studying cognitive apprenticeship within real-world practice communities, from grocery stores to butcher shops and the workshops of tailors and blacksmiths. Most important in our context is the fact that these researchers have recognized the importance of multimodal communication for the cognitive collaboration of teams; gesture has a natural place in this framework (Hutchins 2006; Hutchins & Palen 1997). In

Chapter 2. Gestures as interaction

this fashion, traditional separations between internal and external or mental and material resources, and between cognition and communication, embodied communication and practical work, between language use and practical action, are removed. Each element – and each communicative form – is now recognized as being situated within a complex, changing, multimodal fabric of meaning, action, and interaction (Goodwin 2007). Gestures in such contexts are often intricately interwoven with practical actions, turning these, for example, into demonstrations or practical lessons. Many scholars who define their work as a study of practices cite the Outline of a Theory of Practice by French sociologist Pierre Bourdieu (Bourdieu 1977) when they introduce the term, but it is doubtful that the specific content of this difficult work had much of an effect on empirical praxelogical research. We must mention Bourdieu’s concept of habitus, however, by which he refers to “systems of durable, transposable dispositions” (Bourdieu 1977: 72) that human bodies acquire in their social transactions and class-specific everyday lives. The concept of habitus – “the durably installed generative principle of regulated improvisations” (p. 78) – offers an alternative to traditional conceptions of competence which regard competence as a matter of (grammatical) rules – a concept whose shortcomings in explaining intelligent performance have been known since Wittgenstein (Wittgenstein 1953; see also Polanyi 1958, 1966). In the conception of habitus, it is in the first place the body that acquires and exercises knowledge of the world, by unthinkingly, if mindfully, coping with its situated, practical demands. In this regard, Bourdieu’s conception is similar to, and informed by, the views of phenomenological philosophers, notably Heidegger (1962 [1926]; see Dreyfus 1991) and Merleau-Ponty (1962).

Visual research on cultural behavior Before the 20th century, the exact, observation-based analysis of the dynamic processes of human interaction and gesture was a near impossibility, due to the lack of technologies that would have made it possible to record specimens of interaction. Even though body-motion is visible and can be observed by anyone, its precise analysis requires that it is documented so that it can be repeatedly scrutinized by a community of researchers. Until cinematography was invented, no such technology existed. The most detailed representations of interaction were produced by painters, and despite the exquisite insights that were embodied in many of their works (Streeck 2009b), the scenes that they showed were usually contrived, synthesized from multiple sketches of ordinary events and staged arrangements of models in the painter’s studio. These works of art inspired philosophical debates about the merits and possibilities of different art forms – which is more apt at





Gesturecraft

capturing motion: poetry or sculpture? how do both render time? (Goethe 1953 (1798); Lessing 1962 (1766)) –, but they did not – and could not – stimulate empirical research into the moment-by-moment unfolding of human interaction. The increase in attention during the late 19th and early 20th centuries to the empirical facts and details of expression behavior made the need for accurate representations more acute. Various notation systems had already been developed for normative purposes, some designed as aids for rhetorical instruction, others for choreographers, but because so many phenomena of expression and communication do not reach the threshold of observers’ awareness, these could not be applied to the study of ordinary face-to-face interaction. The human mind cannot consciously register every detail that lends structure and meaning to a moment of interaction: analysis of the microscopic level of human communication had to await the invention of film. Only film could enable researchers to disclose the many tacit background processes that lend meaning and structure to the foreground processes that participants and observers attend to. An immediate forerunner of cinematography were the time-lapse photographs of human figures and animals in motion by E. Muybridge (Muybridge 1955 [1887] a, b). Muybridge’s voluminous work – and the popularity that it enjoyed – demonstrate the fascination that people felt when they were exposed to the microcosm of mundane behavior. For the first time, they could witness the subtle and precise coordinations that underlie their own quotidian activities, actions such as walking down a stair, getting up, or jumping, and interactions such as boxing and fencing. Muybridge produced entire encyclopedias of such uneventful events, and some of the early cinematographers that followed him also applied themselves to making this newly discovered microcosm visible. But these early photographic records of people in motion were difficult to read. Photography cuts movement into arbitrary segments, often capturing an act at an insignificant, figure-less stage: the single photograph may be meaningless because it does not capture the Gestalt that participants apprehend. Rather than aiding and enhancing perception, the new representations at first presented new perceptual and interpretive tasks. Cinematography – the discovery that these time-slices, if projected in rapid sequence, are re-synthesized by the human eye into motion – took care of this perceptual problem. Now people became able to immerse themselves in what appeared to be simple reproductions of the familiar world itself. Decomposing these synthetic images into their constituent parts and analyzing the component behaviors out of which familiar actions are fashioned became the province of specialists (including those who sought to maximize the speed of workflow in factories). All of the relevant micro-analytic studies of embodied interaction until approximately 1975 used film (usually 16mm film) to produce researchable specimens of behavior (Kendon 1979: 67). From the mid-seventies on, video-technology began to replace film.

Chapter 2. Gestures as interaction

There is no need here to retell the history of the use of film and video in the cultural analysis of human behavior. Numerous books and articles have been written about the development of ethnographic film and visual anthropology (Banks & Morphy 1997; Boonzajer Flaes 1993; Collier & Collier 1986; Hockings 1995; Pink 2006; Kuklick 2008) and a few historical, critical, and practical accounts of the use of film and video in the micro-analysis of human interaction are available as well (Goodwin 1993, 2000c; Kendon 1979b). Many practitioners of micro-analysis regard Bateson and Mead’s Balinese Character (Bateson & Mead 1942) as an important pioneering effort, even though film played a relatively small role in their research. Micro-analysts today agree on the importance of Charles Goodwin’s efforts to educate researchers about effective and professional methods for producing researchable videotapes. One methodological issue that needs to be addressed here is the widely shared convention that social interaction should be filmed with a static camera, mounted on a tripod, and without the use of zoom. There appears to be strong agreement that this strategy is the only one that precludes distortion of the events by the videographer’s subjective gaze, and the strategy is further motivated by the need, emphasized both by context and conversation analysts, to present and analyze the behavior of individuals within the context of the entire interacting ensemble. Thus, the camera should always include all participants of an interaction. While it will never be possible to produce “a truly comprehensive view of any event”, Kendon emphatically asserts that the limitations of a film recording must be fully recognized but attempts to overcome them, for example by moving around with the camera while one is filming, should be avoided at all costs. (Kendon 1979b: 75)

I disagree with this view. While it is surely good advice against the tendency of amateurs to pan the camera back and forth between different participants, trying to capture different speakers as they take turns, the static camera is itself a biased one and, depending on the subject matter or activity that the researcher is trying to record, it may well be advisable to choose a different strategy. In her study of visual research on culture, Filming Culture (Mohn 2002), the German visual sociologist and camera ethnographer Pina Elisabeth Mohn has called this research strategy gaze-less camera (“blicklose Kamera”). She emphasizes how different the perspective it embodies is from that of interaction participants: it does not look around nor linger, search, focus, look away or go out of focus. But capturing the participants’ perspective is the avowed objective of many micro-analytic researchers, notably conversation analysts. There is thus a paradoxical relation between the goals and the data-gathering strategies of film- or video-based micro-analysis. To capture the perspective of (individual) participants, it would often be necessary to zoom in and out, to pan the camera, and surely to have it move about the scene.





Gesturecraft

To secure that all participants are always in the frame, the camera should not be moved or zoomed. The video-recordings for this book were partly made with a static, tripod-mounted camera, partly with a moving, hand-held camera, which was usually focused so that all interaction participants were included in the frame, but occasionally zoomed in on the area near the participants’ hands, where their gaze was focused. A field of studies that is virtually unknown among interaction researchers and ethnographers outside of France is the cinematographic anthropology (“anthropologie cinéaste”), as practiced by Claudine de France (1978, 1983, 1989) and Annie Comolli (1991, 1993, 2003; Comolli & de France 2005), two disciples of the ethnographic filmmaker Jean Rouch (2003). De France’s idea to use film as the research method for the praxeological analysis of human activities and the emphasis on gesture in Comolli’s film-studies of cultural transmission (Comolli 1991 – originally published in 1983!) are of great interest to researchers of gesture, but also of those who are primarily interested in the incarnation of culture or cultural transmission. Comolli and France trace their work to Marcel Mauss’ essay on the techniques corporelles (Mauss 1973 (1935)). Mauss defined the human body as the main locus of culture, imprinted by society. He stated that “dans tous ces éléments de l’art d’utiliser le corps human, les faits d’éducation dominaient” (Mauss 1968: 369). Accordingly, in their film-based studies of culture and cultural transmission, de France and Comolli concentrate on the bodily practices by which culture is enacted and passed on – for example domestic practices such as setting a table, folding napkins, or washing clothes; musical practices such as playing the violin; and particular forms of apprenticeship through which these practices – and the skills enabling them – are apprehended and passed on. Central to the praxeological approach is to conceive of cinematic decisions – how to focus, what camera angle, when to cut – as methodological choices that affect the analysis of an activity. Thus, for the praxeological analysis it is important to get the temporal articulations right, so as not to obscure, but to reveal the underlying enchâinement materiel, that is, the sequence of interconnected postures and gestures from which the activity arises, and of pauses and acts. Praxeological analysis aims to exhibit the junctions between bodies, tools, and objects, the sequences of optional and necessary steps, and the articulation of bodily actions and gestures by the different agents that cooperate in the activity. The body is both agent and object in many activities, and in order to reveal its roles and acts, it may be necessary to change the camera angle repeatedly and to record sequences of close-up and wide-angle shots. “Anthropologues cinéastes” aim to let the film reveal the structure and logic of the activity, rather than relying on spoken or written commentary to carry and communicate the analysis and using the moving image only as illustration.

Chapter 2. Gestures as interaction

Comolli’s book, Gestes de Savoir (Comolli 1991), and her films, for example Gestes Guides, Gestes Guidés (1995), give center-stage to gestures, which she regards as the main vehicle for the transmission of embodied culture. “Gestes”, however, does not exclusively, maybe not even primarily, refer to manual symbols, but all motor unit-acts that are part of some mastered practice, including those that repair faulty performance. The methodological viewpoint underlying the studies in this book is consonant with the visual praxeology of Mohn, de France, and Comolli. The focus adopted – and the way the camera was used – varies from chapter to chapter, in line with the ecological frameworks within which gestures function: whether they depict a world that is not here or reveal the features of an object at hand, and so on. A more important difference is that the studies presented here are pointillist rather than ethnography in the classic sense: I give little ethnographic context for the data, even when the data are analyzed for their contextual ties. But this is appropriate given the aim of this book, to give a broad overview of the functional diversity and heterogeneous properties of some of the main genres of gestural praxis and at the same time show in microscopic detail how these function within their immediate contexts, the sequence of interaction or the currently pursued activity.

Gesture, gestures, culture, cultures: Some conceptual clarifications Sometimes one can detect conceptual problems in the way in which lay people and professionals refer to gesture events. For example, an issue that is currently of great interest to many people is whether gesture is “culture-dependent”, whether “members of different cultures” “use different gestures”. Even posing the question in this way makes a number of stipulations that are neither obvious nor realistic: confusion often results from failure to recognize covert conceptual distinctions that are obscured by the surface grammar (Wittgenstein 1953) of ordinary English. When we talk about gesture(s), the first conceptual distinction that we must observe is that between “gesture” as mass-noun and as count-noun, between “gesture” and “a/the gesture”, or, somewhat inaccurately, between “gesture in the singular” and “gestures in the plural”. English count-nouns are distinguished from mass-nouns in that the former can take articles – and thus be “individualized” – whereas the latter cannot. Count-nouns designate entities that come in bounded, discrete, countable units. Things are the prototypical countables, but events – ”bounded” events – are also countable and are denoted by count-nouns. Massnouns, in contrast, denote “unbounded” entities, prototypically masses such as water and sand, and by extension ongoing (i.e., “unbounded”) processes, activities, and states (e.g., rain, skiing, sleep), as well as dispositions (musicality, knowing Spanish). This grammatical distinction in English is a good example of





Gesturecraft

what linguists call “covert categories”: conceptual distinctions that are not explicitly marked in the morphology of words, but visible in differences of grammatical behavior. “Gesture as mass-noun” (without article) denotes a praxis, a human activity. “Gesture as count-noun” (with article) denotes gesture-events, single instances of the praxis. Other languages make this distinction overt. In German, for example, the praxis is denoted by the mass-noun Gestik, the event by the count-noun Geste. However, there is not one, but there are several count-noun varieties of the term “gesture”. Gestures are processes, but we also treat them as products: as things made; English treats gestures as things. And we usually do not distinguish the gesturing from the gesture; or rather: we assume that the gesturing is the making of a gesture, which is like the making of a thing. The English word “gesture” (the countnoun) is indifferent to this distinction: “gesture” can designate the event, the process of making, but also the product of this process. Other languages would mark the difference, for example, by aspectual inflections, or by making one a verb, the other a noun. But this is not a failure that has grave consequences. Another distinction that exists, albeit covertly, is of far greater consequence, and failure to observe it invariably causes confusion. This distinction is that between gesture-as-product and gesture-as-means. It becomes overt in the following pair of sentences: (1) a. X makes a gesture. b. X uses a gesture.

In (a) we characterize the gesture as a product, an achievement of a singular act. In (b), however, we construe the gesture as something durable, as a “thing” whose lifespan goes beyond the single event: only something that has prior existence can be used. Sentence (1)(a) is open to the possibility that the gesture is a moment’s invention; sentence (1)(b) excludes this possibility. Built into (1)(b), therefore, is a construal of the gesture as an entity that has an existence similar to that of a word, as an enduring “type” or “template” of which the event is a “token” or “instantiation”. It is very common to unwittingly move from sentence (1)(a) to (1)(b) without recognizing the difference and to thereby accord gesture (the praxis) a status similar to language: as a repertoire of ready-made, conventionalized signs, a code. But notice how different the sentence-frames are when we apply them to “word”: (2) a. X makes a word. b. X uses a word.

Whether the praxis “gesture” is better described by sentence-type (a) or (b), or if it is a combination of the two kinds of events, is an empirical question. My data lead me to believe that it is a combination of the two, that it involves both “uses” and “inventions”, a combination that is also typical of other types of practices that we consider to be improvised. An improvised performance is always unique, but

Chapter 2. Gestures as interaction

it combines parts that are pre-fabricated (traditional, rehearsed, or routine) with situated inventions. The same, I believe, can be said about how we commonly gesture. Finally, “culture”, that is, the issue whether gesture is “cultural” or “varies from culture to culture”, which holds great fascination for many people today. There are different ways in which such questions can be stated, and it is important that they be stated clearly. “Culture”, too, exists in a mass-noun and a count-noun version, and the empirical relation between the phenomena denoted by the two grammatical versions of the term is anything but obvious. The word “culture” in the anthropological sense in which we use it today is an invention of the second half of the 19th century and is roughly equivalent with “civilization”: it denotes all the “stuff ” that distinguishes the human species from other animals, the “set of prosthetic devices by which human beings ... exceed ... the ‘natural limits’ of human functioning” (Bruner 1990: 21), and which are the product of invention and tradition, not biology. This is “culture as a mass-noun”, “in the singular”. All human societies “have” culture. Undoubtedly, gesture, the praxis, is such a prosthetic, human-made device and therefore “cultural”. “Culture-in-the-plural” (that is, as a count-noun) is an even more recent terminological innovation. We seem to understand the relationship between “culture” and “cultures” (and construe the meaning of the latter) by analogy to that between “language” and “languages” (another word that comes both in a massnoun and a count-noun variety): “language” is something that the entire species has, a faculty; “languages” are more or less bounded, individuated entities, and there are thousands of them. However, the faculty “language” is always realized as mastery of “a language” (at least one). By analogy, we construe the possession of “culture” as mastery of (or membership in) “a culture”. But the empirical basis of this analogy is highly questionable, and the relationship between “languages” and “cultures” is a matter of very serious contention in many parts of the world. The cultural stuff that is indexed by the mass-noun “culture” simply does not so easily cluster into segregated, bounded units as language, to a certain extent, does. While it is generally possible to name the language to which a word belongs, it is often difficult to identify the communities (or “cultures”) to which other kinds of cultural stuff belong (say, chairs, winks, or the gesture of giving flowers). The new field of “cultural studies” has for very good reasons moved from assuming the existence of cultures as bounded entities to an analysis of the discourse and social and political movements within which cultural membership and boundaries are invoked, symbolized, and contested. In light of this, the unproblematized use of terms such as “French gesture” or “Japanese nonverbal communication” appears naive. If we take these covert distinctions into account, I believe we can make the following observations: first of all, “gesture”, the “praxis” (the mass-noun), is an example of “culture” (the mass-noun), one of the “prosthetic” devices (Bruner





Gesturecraft

1990: 11) which human beings have fashioned and continue to fashion. It is not only possible but empirically the case that there are “stylistic variations” in the ways in which this praxis is carried out, for example in the patterns of motions of the arms (whether these motions mainly pivot around the wrists, elbows or shoulders), but some clear-cut cases notwithstanding, it is not at all clear how pronounced these differences are nor along which boundaries they cluster. Nor is it clear whether such stylistic differences affect the logic of gestural communication, that is, the methods by which meaning is made. The differences may turn out to be shallow. It is also true that conventional gestures (“emblems”) exist and that these are anything but universal; but again it is (some clear-cut cases notwithstanding) not clear where the boundaries of the areas are within which different repertoires of emblems are used, nor how these boundaries are maintained. The number of conventional gestures is probably much greater than those that have been described, and whether those that will be discovered in the future are the property of bounded communities or in use across large, fuzzy geographical and cultural areas, is an open question. This book is a study of gesture understood as cultural praxis – or, more precisely, as a family of cultural practices –, and usually I will refrain from attributing them to any one culture. The primary focus is on the situated making and functions of gestures in human activities and interaction, that is on situated gestural practice, on gesture “as a mass noun”, which nevertheless can only be studied by investigating individual “gesture events”, which may in turn comprise several gestural acts, some of which may be conventional, that is, “tokens” of gesture “types”, while others may be improvised on the spot.

chapter 

Hands

Most books on human language and linguistics contain some chapter on the vocal apparatus, and there is surely wide agreement that the structure of (spoken) human languages is to some extent shaped – or constrained – by the properties of our vocal organs: the importance of syllables, the voiced/voiceless distinction, the role of features such as tongue position in phonological systems – these and many other “distinctive features” are by-products of contingent features of the vocal apparatus. The ability to process language sounds, to produce and recognize very small articulation differences, is a contingent and convergent by-product of other developments, but it is nevertheless essential to the structures of spoken human languages which, in all their diversity, make systematic and economic use of these possibilities that the human body provides. In the case of gesture, the correspondences between the features of the medium and those of the organs by which its units are produced are rarely examined or established. It is common to treat gesture as a medium of expression, which meets both informational and pragmatic or social-interactional needs, but whose “manuality” is accidental and irrelevant. Here, a different view is proposed, which takes inspiration from the only seemingly trivial fact that (hand-) gestures are made by the hands. For one thing, the forms that we can give our gestures are constrained by the degrees of freedom of our hands, by the sum total of motions that the ensemble of our fingers is capable of making. Within anatomical limits, cultural training can increase the flexibility of the hands and expand the range of possible motions and postures somewhat, but the number of possible configurations is nevertheless limited. But there is another dimension to “manuality” that is of greater importance. This is the fact that gestures are made by the organs that give us our world: hands are not only organs of action and expression, but also of cognition and knowledge acquisition; much of our acquaintance with the world comes from the actions and perceptions of our hands. Thus, to come to terms with the manuality of gesture, we should try to understand what kind of an organ the hands are, how they come prepared for the tasks of gesture. Understanding the concrete properties of the hands will help us understand some of the concrete features of gesture. Sennett writes, “all skills, even the most abstract, begin as bodily practices” (2008: 10) and



Gesturecraft

“the varied movements [of] . . . the hands . . . affect how we think” (150). Thinking is “internalized movement” (Llinàs 2001). In this chapter, I therefore briefly tell the story of human hands, how they came into being, what their primary modes of engagement with the world are, how specifically they are adapted to these modes – and how some philosophers and anthropologists have conceived of the hand as a “vehicle for being in the world” (Merleau-Ponty 1962).1 After taking stock of bone structure and degrees of freedom, I sketch in the shortest possible terms the evolution of the hand from an organ of transportation to one of grasping and, eventually, tool-making and gesture. The execution of movements of course is a matter of muscles and tendons, which in turn are controlled by the central nervous system. We cannot include these components here, for lack of space and expertise. I will concentrate on the action-perception loop in grasping, the way in which the mechanics and dynamics of prehensile postures and the hand’s perceptions work together to produce “intelligent” handlings of objects. Grasping is without doubt the raison d’être of the human hand, and we must examine this activity a little more closely: grasping, more than any other primordial action of the human body, combines perception and action. It is, at once, at the very core of our sensory experience and our active making of the world. This dialectic is most apparent in the phase of grasping known as “enclosing”, during which the action of the hand is at every moment shaped by the feedback that it receives from the object. Grasping is, as the metaphorical use of the word reminds us, a cognitive act as much as a physical one.

The structure of the human hand Humans share the basic structure of their hands with all amphibians, reptiles, birds, and mammals: all tetrapodes have hands with five fingers. Four fingers always have three segments (phalanges), while the fifth, the thumb, only has two. (There are of course animals that only have four fingers – or hoofs – . but these hands are later variations on the underlying, five-finger theme.) Other hand-bones make up the palm (metacarpals) and the wrist (carpals). The human wrist has a feature that distinguishes it from other primate hands and is a relatively modern adaptation: this is the fusion of the two wrist-bones to which the thumb and index finger attach. It stabilizes the opposition of index-finger and thumb, which is the most important structural feature of the human hand. Many of the actions that . For appraisals of the hands in the context of craft see Sennett’s The Craftsman (2008: Ch. 5) and the beautiful account in McCullough’s Abstracting Craft (1996: Ch. 1). See also Foçillon’s classic “In praise of hands” (1992 (1934)).

Chapter 3. Hands

Figure 1. Structure of the human hand. From John Napier, Hands (1980). Reprinted by permission of Princeton University Press.

the human hand can take depend on it. What a hand can do is also dependent on the bone-structure of the arm. Particularly important for humans is the forearm’s ability to rotate; it greatly expands the range of effective action that the hands can take. The movement range of the hand is also conditioned by the structure of the joints between the phalanges and their resulting degrees of freedom. Compared to the digits, the thumb has the additional capacity to rotate and thus an additional degree of freedom. Two additional features essential to our grasping skills must be mentioned (MacKenzie & Iberall 1994). These are the existence of fatty pads as well as the texture of the skin on the inside of our hands. Pads are the cushions on the inside of the hand, of which there are nine, five on the finger tips, three on the palm’s distal side, between the fingers, one on the palm’s outer ridge, on the far side of the thumb. These pads do not only cushion and protect the bones where they are most





Gesturecraft

vulnerable, but also enable the precise adaptation of the hand to objects of different sizes and shapes as it is grasping them, and their distortion and displacement is an important sensory mechanism through which our brain and central nervous system gain information about the material qualities of the world. As we will see, most of this information is gathered during acts of grasping, and many if not most of the features of the human hand are the result of adaptations to grasp-tasks. This is true for the skin on the inside of the hands. Despite appearances to the contrary, it is not at all smooth. Rather, the skin of the palm and the inside of the fingers are marked by flexure lines, from which palmists read the future but which really work like hinges that open and close during grasping, affording the skin the slack it needs for its many prehensile postures. Covering the fatty pads is skin marked by “papillary ridges”. They are located in the areas most likely to have contact with object surfaces. These ridges not only allow the authorities to keep track of our actions (through finger-prints), but work like the treads on tires: they provide traction. Their concentric arrangement guarantees that, whatever the position of the object in hand and the vectors of gravity, some ridges will always be perpendicular to the shearing forces, keeping the object from sliding down while minimizing the effort needed for this. The papillary ridges also house nerve endings that transmit sensations to the brain as soon as fingers come into contact with things; and they contain sweat glands, so that the skin gets lubricated exactly in those places where contact with objects occurs. Lubrication increases skin-adhesion and reduces the amount of force required for a grip; it also enhances our sense of touch. The sweating of our human hands is not exclusively, not even primarily induced by heat, but rather by psychic stimuli. It has been called “cerebral sweating”: controlled by the same brain region as grasping, it anticipates the right amount of traction that will be needed for a grasping-event and produces the right amount of lubrication.

Evolution Although these features of the hands have enabled humans to make the world as we know it, in its basic, underlying design the hand is in fact extremely old and not even specific to mammals, let along primates or humans. Napier noted its “extraordinary degree of primitiveness” (Napier 1980: 24). The approximate age of the five-fingered hand is some 400 million years. It was in place when the first tetrapodes left the oceans and sought to move on land. In other words, the hand developed under water, formed to its pre-terrestrial shape by the need to get around in swampland, and, as a result, fins transformed into locomotion-devices for a new coastal habitat. These devices would then eventually enable us to walk out of water and gain ground (Clack 2002). Our hands are homologous with those

Chapter 3. Hands

of lizards and frogs and with the wings of birds and bats, all co-descendants of those ancient fin-feet of the tetrapodes, each form an adaptation to the medium in which the movement takes place (in water, air, or on land; Strickberger 2000.) When we take a synoptic view of the hand’s history, we easily recognize that the most important innovation was its stepwise transformation from an organ of locomotion to one of prehension (grasping) – or of feet into hands. This began approximately 60 million years ago when primates branched off from their rodent ancestors. These were tree-dwellers that must have looked somewhat like opossums, and they were insect-eaters, feeding by night and navigating by olfaction – “noses on legs” (Napier 1980: 86). Primates then became day-time dwellers and established the reign of vision. This enabled them to move about and navigate in ways that only vision enables, and eventually, with eyes in the right place (in front of the face, enabling three-dimensional vision), they became capable of precise eye-hand coordination, enabling them to swinging from branch to branch (bracchiation). Hand-eye coordination enabled the grasping of branches, but also the grasping of food. The repositioning of the eyes was complemented by reconfiguration of the hand so that prehensility – grasping a food-item with a single hand while holding onto a branch with the other – became possible: this is the characteristic feeding mode among primates. Only primates have prehensility and initial lateralization (division of labor) in activities involving both hands. This single adaptation, prehensility, achieved by the opposition of the thumb, is considered the most important change on the road to humanity prior to the evolution of speech; after all, it afforded not only new forms of feeding, but also making, the production of new things from old or from raw matter. Prehensility emerged in a succession of stages, and only the most recent adaptations – mainly the ability to oppose the thumb to all other fingers and the ability to perform forceful and shape-adaptive precision grips – are unique to humans. Other novelties of the primate hand were the replacement of claws by nails and the development of sensitive, ridged pulps on the inside of the hands. With these adaptations in place, an arboreal life became possible: climbing and movement on branches, grasping and holding of fruits and insects, and navigation in three-dimensional space. The “geometrically more complex” living environment of the early primates (Wilson 1998: 20), compared to that of their nocturnal predecessors, contributed greatly to the increase in brain-size in monkeys; the further evolution of the brain followed that of the hand. These changes were complete some 35 million years ago. By then the hand was established as a compromise organ, serving both locomotion and manipulation – the “foot-hand” (Napier 1980) of monkeys. The Great Apes, however, underwent yet another change in life-style, yielding new adaptation pressures on the hand. The hands morphed from tools for transportation into tools for manipulation via their role in bracchiation, the odd method of moving by swinging from branch





Gesturecraft

to branch practiced by the Great Apes. The Great Apes had become too large to support themselves safely on the upside of branches, their balance had became all too precarious. Moreover, some species not only grew in size, but also changed their dietary habits from insect- to fruit- and leaf-consumption, which required them to “go out on a limb”, i.e. to climb out towards the periphery of canopies where most fruit and edible leaves were found. Moving along the upper side of slim branches became a precarious affair, the more so the more the animals gained in size. Eventually, a critical point ... [was] reached where a potentially unstable system is produced and lateral displacement of the axis of gravity will lead to overbalancing. Without some compensatory trick such as prehensility, large animals would find it extremely difficult to move about slender, leaf- and fruit-bearing branches and keep their balance; thus, a rich food source would be closed to them.2 (Napier 1980: 93)

Bracchiation led to a significant refinement of the hand and its neuronal control mechanisms. We know that the arboreal life of our primate ancestors did not go on forever. Their habitat changed slowly but dramatically approximately 5 million years ago, when a drop in global temperatures led to drier climates in the tropics and the rain forests of East Africa gradually turned into savannah. As a result, our ancestors lost their arboreal habitat and became ground-dwellers, a circumstance to which they adapted by evolving bipedalism, the ability to walk on their hind legs alone. Once only the feet were needed for locomotion, the hands could become entirely specialized for manipulation, for “grasping, manipulating, and carrying tools, food, and offspring” (Strickberger 2000: 474). Paleontologists now believe that the first hominid-like, bipedal species, Ardipithecus ramidus, appeared approximately 4.5 million years ago in Ethiopia; it is considered the missing link between apes and humans (Strickberger 2000: 473) and the ancestor of the several species of Australopithecus that lived from about 4 to 1 million years ago in Africa and that developed the manufacture of stone tools, which some researchers credit as the driving force behind the most recent and final adaptations of the hand. To understand these adaptations we must make . Bracchiation occasioned further adaptations in different species of primates. One was an increase in length of the fingers (except the thumb) and, in some cases, the palm, which made it possible to use the hands as hooks. Human fingers are half the length of the hand, a ratio that they share with those of primates who live partly on the ground, partly in trees. Yet, the human hand is very short compared to that of chimpanzees. In chimapanzees and orang-utans, the great length of the palms enables a mechanism called “double-locking”, in which the fingers can be tucked into the skin-fold at the base of the fingers, which can then be rolled further into the palm. This enables a very tight grip on small branches, enabling the ape to hang suspended from them while feeding with the other hand.

Chapter 3. Hands

use of a distinction, initially made by Napier (1956), between two ways in which our hands grasp objects, namely the power and the precision grip. In power grips objects are squeezed between fingers and palm, while precision grips involve only the thumb and usually one or two fingers, the palm is being used only for support. In contrast to the grips performed by other primates, human precision grips allow greater force in securing objects between thumb and finger (precision pinching), and they also allow the adjusting of the orientation of gripped objects while they are being held in the hand (precision handling; see Marzke 1997). These two abilities, precision pinching and precision handling, were needed to conduct stone-tool manufacture on an industrial scale, as was characteristic of the hominid period in East Africa, involving workers for hours at a time. Precision “pinch” grips enabled homo erectus to hold a small object firmly in place or to constantly reposition it so that it is always properly placed and held in place for the impact of the stone-hammer wielded by the dominant hand. Primates cannot exert such forces in a precision grip. As re-enactment studies have shown, three kinds of grip that only humans can make are involved when stone-tools are made in repetitive, industrial fashion. Marzke describes them as pad-to-side grip between thumb and index finger; cradle precision pinch grip by thumb and finger pads, and 3-jaw chuck ‘baseball’ grip (93). Common to all of them is that they provide “extensive contact surfaces on the hand” and thus enable the handling of hammer stone and preform each by one hand alone (94). Only the human hand is capable of holding tool and preform in the firm, but constantly changing positions that allow for the rapid and habitual shaping of stones (Figure 2). According to Marzke (1997), eight morphological changes are responsible for the specifically human abilities of precision pinch and precision handling; most important are those that allow humans to accommodate their hands to irregularly shaped objects while exerting considerable force. Thus, the development of the morphology of the human hand can be explained as a series of functional adaptations that have enabled specific kinds of grips and which together have facilitated the habitual, repetitive manufacture of stone-tools.3 Reynolds (1994) has emphasized that “human technology involves the cooperative construction of artifacts, and even stone tools are normally made by groups, not individuals” (Reynolds 1994: 410, my emphasis). Moreover, while simple stone-tools were the main technology of homo erectus, the more characteristic form of artifacts made by homo sapiens is the polylith, which is . Different species possess different degrees of manipulation skills, and these variations have to do with differences in hand-configuration, especially relative length and position of thumb and index-finger. This is expressed in the opposability index. The index is highest in humans and gorillas, low in chimpanzees. High opposability means great skills at manipulating small objects.





Gesturecraft

Figure 2. Three types of human prip. From M. Marzke, Precision grips, hand morphology, and tools, American Journal of Physical Antropology. Reprinted with permission of WileyLiss, Inc., a subsidiary of John Wiley & Sons, Inc.

made of distinct parts, often useless in themselves, which fit together to make a functional whole (407).

Polyliths are characteristically produced by teams whose members work together in heterotechnic cooperation (i.e. by taking different roles in the activity). In this context, the essence . . . is anticipation of the action of the other person and performance of an action complementary to it, such that the two people together produce physical results that could not be produced by the two actions done in series by one person (412).

In other words, it requires sequential interaction and the possibilities for projection and anticipation that a sequential interaction system affords, and it is this complex context in which, in Reynolds’ view, gesture has evolved.

Chapter 3. Hands

Grasping: Prehension and comprehension We have seen that the hand has evolved through successive adaptations to changing demands on its grasping skills. Although various other types of manual action are also indispensable (e.g. active touch, see below), grasping is by far the most important activity of the human hand. Much of what we do during our waking hours requires us to hold something, if only to put it away. We gain most of our tactile information about the world through taking, holding, using, and handling things, all of which involve grip as a stable form of body-to-object contact. Because grasping is so central, we need to understand somewhat more closely what happens when humans grasp. Thus, I will briefly review the formation and types of prehensile postures; the micromanagement of grasp; and the sensory and cognitive feedback-loops involved in such management. I draw on the study The Grasping Hand by C. L. MacKenzie and T. Iberall (1994) here, a book which combines neuro-biological, cognitive, and computational perspectives. MacKenzie and Iberall give the following definition of prehension (grasping): Prehension involves the taking hold of an object for a purpose such as manipulating it, transporting it, or feeling it. ... Prehension entails applying forces during interaction with an object. ... [and] the orienting and posturing of the hand and fingers, with the appropriate transportation of the limb to the correct location in space. ... Prehension is ... the application of functionally effective forces by the hand to an object for a task (15).

A grasping episode comprises a sequence of stages: in the preparatory stage of reaching out the hand is oriented, brought to the object, and configured according to the properties of object and task; during the contact period the grip is fine-tuned and the object perhaps repositioned in the hand; the event ends with release of the object and recovery of the hand. In more detail: during aiming the arm is moved so that the hand comes in the vicinity of the targeted object. This stage is initially controlled by peripheral vision: we begin to reach for an object without directly looking at it. The motion during this stage is fast or ballistic. Ballistic movements are brief, starting and ending abruptly (Phillips 1986),4 and cannot be modified after they have been begun. During the transport phase the arm and hand are responsive to the object’s extrinsic properties: its location, orientation, and, if it is moving, velocity and direction of movement. About two thirds into the total movement time, corrective movements are made to secure that the hand will arrive at the target location (Jeannerod 1984). . “A ballistic movement is a literal throwing of a limb or part of a limb by a brief muscular contraction. ... A voluntary ballistic movement, once launched, completes its brief preprogrammed trajectory before it can be influenced by any feedback from the moving parts, or from vision” (Phillips 1986: 86).





Gesturecraft

Figure 3. Hand reaching for target From M. Jeannerod, La Recherche, 12, 376–78, 1981. Reprinted by permission of Marcel Jeannerod.

The reaching movement of the arm, thus, begins with a “ballpark” assessment of the target location and is fine-tuned during the final approach. During the final approach, several parts of the arm and wrist collaborate to appropriately orient the palm so that the configured hand makes contact with the target in the right places. At this stage visual feedback influences the choice of hand configuration: without it, the fingers could not be shaped accurately (Jeannerod 1997). Backing up to the beginning, we note that during the ballistic transport phase of grasping, the hand is also being opened (Figure 3). When it reaches its maximum aperture, its movement towards the object begins to slow down. At the same time, the hand’s prehensile posture is configured which is the result of a ballpark estimate of the object’s intrinsic properties: its size, shape, density, and weight, computed in relation to the mechanical properties of the intended task. (A different prehensile posture is chosen for a pen-to-be-put-away than for a pen-tobe-written-with.) More time is required to prepare the prehensile posture when the task is complex. Prehensile postures – hand configurations for grasping – can be defined as “task-specific postures for applying forces” (MacKenzie & Iberall 1994: 11). They can be described in terms of opposition spaces, i.e. the spaces between the (virtual) fingers that will exert forces on the object. Grasping always involves the application of two (more rarely three) opposing forces on the object-in-hand. Three types of opposition have been distinguished: pad opposition between the surfaces of the thumb and one or more digits (Fig. 4); palm opposition “between hand surfaces along a direction ... perpendicular to the palm” and characteristic of power grips (Fig. 5); and side opposition, for example between the pad of the thumb and the inside of the index finger or other digits (Fig. 6). This is a posture that we adopt,

Chapter 3. Hands

Figure 4. Pad opposition. From John Napier, Hands. Reprinted by permission of Princeton University Press.

Figure 5. Palm opposition. From John Napier, Hands. Reprinted by permission of Princeton University Press.

Figure 6. Side opposition.

for example, when we turn a key, and only we humans are capable of enacting it (Marzke 1997). Prehensile postures can be described as constellations of two or three virtual fingers: every posture is a configuration of two or three force-vectors, provided by the thumb, perhaps the palm, and a single digit or several digits acting as a unit. The term “virtual finger” refers to discrete functions of individual fingers or groups of fingers in forming an opposition space and applying forces to the object. In grasping the handle of a mug, for example, one virtual finger (the thumb) exerts downward force from above the handle, one provides upward force from within





Gesturecraft

Figure 7. Preparation of a prehensile posture adequate to the target object and task. After Arbib, Iberall and Lyons, 1985.

the handle (when larger mugs are handled typically the pair of index and middle finger), and one may stabilize the handle from below (see Fig. 7). Studies have revealed that the forces applied by these virtual fingers or “jaws” are almost always perfectly calibrated: the combined force is only marginally greater than what is needed to prevent slip, given the weight of the object and the friction between object-surface and skin, which depends on the object’s surface texture and the skin’s lubrication. Prehensile postures are formed, then, while the hand approaches, before it reaches the object. Embodied knowledge participates in the formation of prehensile postures, knowledge, gained over time, of the fit between hand-configuration and force application on one hand and type of object and task on the other. It happens rarely enough that, as we begin to lift an object such as a mug, we suddenly realize that it is “too heavy” or “too light” for the quantities and vectorization of the forces that we apply through our grip. But remember that “it is the intended activity [not the object per se, J. S.] that is the main determinant of the type of grip for each

Chapter 3. Hands

given action” (Jeannerod 1997: 32).5 Embodied knowledge in this context concerns the structural “tripling” (cf. Maturana & Varela 1992) between hand-shape, object-characteristics, and task. This type of knowledge usually remains “tacit” (Polanyi 1966). The last phase of grasping, when the fingers begin to enclose the object, has been described as “guarded movement” (MacKenzie & Iberall 1994): the hand now searches for tactile contact and becomes responsive to tactile feedback. It is in pursuit of tactile information to fine-tune itself. The result can be stable grasp, which involves fixed contacts. Citing Polanyi (1958), MacKenzie and Iberall write about this final phase of grasping that “in acquiring an object into stable grasp, the system being controlled changes from the hand to the hand plus object. The object becomes an extension of the hand” (267). Equally important for the daily work of the hands, however, is dynamic grasp which involves shifting contacts, including rolling, sliding, and repositioning of the object in the hand. The formation of a prehensile posture in everyday life relies on prehensile schemata that are stored in the nervous system as residues of prior prehensile experiences. In this sense, grasping exemplifies what is meant by “embodied knowledge”: underlying our routine abilities to grasp, hold, reorient, and transport objects is a wealth of knowledge not only about objects and the prehensile postures that suit them, but also of the mechanics of tasks in which objects are handled (MacKenzie & Iberall 1994: 80). Grasping is a sophisticated, knowledgebased activity, even though most of us rarely, if ever, become aware of our ceptive know-how.

Repertoires of manual action The importance of grasping to the entire range of actions that our hands routinely perform can be gleaned from a typology that A. Mulder has proposed and that is grounded in the set of verbs for manual action of ordinary English (Mulder 1996: Appendix 1). I have marked actions that always involve prehension with an asterisk; the categories are Mulder’s: Changing position: *lift, move, *heave, raise, *translate, *pull, *draw, *tug, *haul, *jerk, *toss, *throw, *cast, *fling, *hurl, *pitch, depress, jam, thrust, . It has been shown that different motor-neurons are involved in grasping identical objects with the same muscles when different tasks are involved. “The result indicates that a given motoneuron can be connected to several sets of cortical cells, and that each set becomes activated in relation to a certain type of movement, not to the contraction of a certain muscle” (Jeannerod 1997: 41).





Gesturecraft

*shake, shove, *shift, *shuffle, *jumble, *crank, *drag, *drop, *pick up, slip Changing orientation: *turn, spin, rotate, revolve, *twist Changing shape: *mold, *squeeze, *pinch, *wrench, *stretch, *extend, *twitch, smash, thrash, break, crack, *bend, *bow, *curve, deflect, *tweak, *cut, *spread, *stab, *crumble, rumple, crumple up, smooth, *fold, wrinkle, wave, fracture, rupture Contact with the object: *grasp, *seize, *grab, *atch, ?embrace, *grip, *lay hold of, *hold, *snatch, *clutch, *take, ?hug, ?cuddle, *hold, *cling, support, *uphold Joining objects: *tie, *pinion, *nail, *sew, *button up, *shackle, *buckle, *hook, *rivet, *fasten, *chain up, *bind, *attach, *stick, *fit, *tighten, *wriggle, *pin, *wrap, ?envelop Indirect manipulation: *whet, *set, *strop

The list shows how pervasive grasping – or grip – is as an embedded component of physical acts. Grasping is our most pervasive immediate contact with the physical world.

Exploratory and practical actions A distinguishing feature of the hand in comparison to other body parts is that it is both an organ of action and of perception. The hands are the organs with which we shape the world while at the same time receiving information about it. Tactile perceptions occur in what neurophysiologists call the “afferent-efferent” or “action-perception cycle” (Neisser 1976), i.e., the cycle of sensation and neuronal control of action. As McCullough has put it, hands act as conduits through which we extend our will to the world. They serve also as conduits in the other direction: hands bring us knowledge of the world. (McCullough 1996: 1)

In contemporary cognitive neuro-science, these abilities are explained in ways that, as Sennett has observed, allow “touching, gripping, and seeing to work in concert” (Sennett 2008: 153). Hands that are not engaged in deliberate activities can often be observed to be fidgeting, engaged in mutual play or play among the fingers of one hand. They thus sustain a state of perpetual perceptual and cognitive stimulation for themselves. “They have a life of their own that leads them into explorations” (McCullough 1996: 8.) The hands also possess the most practical sense of all the sense organs: they acquire knowledge in the process of changing the world. While most of the embodied knowledge that the hands gather is gathered unthinkingly in the course of their ordinary lives, without attention being focused upon them, there is also

Chapter 3. Hands

a specific class of exploratory procedures that we use when we methodically gather information. Gibson (Gibson 1962), following Katz (Katz 1925), has called these active or dynamic touch. Active touch includes movements of tactile exploration or scanning that we perform in order to identify intrinsic object-features such as shape, texture, and substance, or the generic category to which the object belongs. To extract these features we must move our fingers: “the stimulus has to be a change in time” (Gibson 1962: 479). Mulder lists these varieties of “haptic exploration”: touch, stroke, strum, thrum, twang, knock, throb, tickle, strike, beat, hit, slam, tap, nudge, jog, clink, bump, brush, kick, prick, poke, pat, flick, rap, whip, hit, slap, struck, caress, pluck, drub, wallop, whop, thwack, rub, swathe. (Mulder 1996: Appendix 1)

Note that the perception and recognition of objects is in no way possible by our sense of skin-touch alone, but requires the simultaneous processing of kinesthetic, body-internal stimuli. Our so-called sense of touch “involves the play of inputs from the whole skeleto-muscular system” (Gibson 1962: 479). Gibson studied active touch in a variety of experiments in which subjects explored objects with their hands but were unable to see them. Generally, when an object is placed in the hand of the subject, the subject tends to bring the other hand to the object if allowed to do so; ... tends ... to curve his fingers around [it]...; [and] to move its fingers over [it] ... in complex ways (481).

These ways include tracing movements; thumb-opposition; and rubbing, grasping, and pressing movements of the fingers. Interestingly, subjects also tended to “name the object if ... [they could] do so, or to compare it with a familiar object if ... [they could] not” (481); they always cease exploration once the object-category has been identified. Lederman and Klatzky (1987) showed that the “motion patterns [used] for obtaining information about objects [are] highly stereotypic ” (cited after MacKenzie & Iberall 1994: 231): rapid, lateral motions between skin and object are used to extract texture; pressure is used to determine hardness; “unsupported holding” and “jiggling” serve to identify weight; enclosing determines volume and overall shape; “static contact” serves as “thermal sensing”; and “contour following” – ”smooth, non-repetitive movements within a segment of the object’s contour” – is used to ascertain the exact shape of the object at hand (Fig. 8). Generally, haptic perception is superior to vision when it comes to identifying texture, hardness, temperature, and weight. Vision is superior in determining size and shape. J. J. Gibson has called the perceptual and neurological system that underlies our manual skills, haptic system (Gibson 1966), and distinguished it from the





Gesturecraft

Figure 8. Exploratory procedures. From S. J. Lederman & R. L. Klatzky, Hand movements: A window into haptic object recognition. Cognitive Psychology, 19 (3), 1987. Reprinted by permission of Elsevier Science.

auditory and visual systems. The haptic system not only includes our sense of touch, but also our body-internal kinesthetic perception; it integrates manually acquired information of the world with our body’s self-perception and supplies us with knowledge about our environment only by letting us feel the motions and relative positions of our joints. The haptic system ... [is] ... the sensibility of the individual to the world adjacent to his body by use of his body... (It) is an apparatus by which the individual gets information about both the environment and his body. He feels an object relative to a body and the body relative to an object. (Gibson 1966: 97)

Chapter 3. Hands

Hands, worlds, and knowledge Our practical manual actions in the world thus always have an “internal”, proprioceptive component, even though our proprioceptions usually remain undisclosed; our consciousness is focused on the objects of our actions, not on our body-internal perceptions while handling them. When we act “in the natural attitude” (Schütz 1982), both the tactile experience of the world and the kinesthetic experience of our own body remain in the background, undisclosed as a “tacit dimension” (Polanyi 1966) of our practical pursuits and understanding. When we use a hand as an organ of knowledge acquisition (active touch), we cannot feel it, but only some other object through it.6 This makes the body an altogether different kind of entity than other physical objects, as Merleau-Ponty did not cease to emphasize. He wrote that to move one’s body is to aim at things through it; it is to allow oneself to respond to their call. (Merleau-Ponty 1962: 138)

The hands are “vehicle[s] of being in the world” (82), and “consciousness is beingtowards-the-thing through the intermediary of the body (139)”. Similarly Polanyi: Our own body is the only thing in the world which we normally never experience as an object, but experience always in terms of the world to which we are attending from our body. It is by making this intelligent use of our body that we feel it to be our body, and not a thing outside. (Polanyi 1966: 15)

And yet, the knowing of the world through our bodies, attending away from our bodies, is only possible because our internally felt dynamic constellations of bones and muscular forces make the world internally available to us. This is how Gibson describes it: The sailor can feel the rope and tie the knot even in darkness. The violinist can feel the shapes and spaces of his instrument with extraordinary precision while keeping his eyes closed. The man with a walking stick can even feel stones, mud, or grass at the end of his stick. Yet all these perceptions come from the contact between the adjacent surfaces and the contacts of bones upon another. (Gibson 1966: 112) . The philosophical anthropologist Gehlen has highlighted the tacit, undisclosed nature of tactile sensations by describing what happens when we touch ourselves: “Whenever I touch my own body with my hand, the movement’s tactile sensations are produced in the touched limb (B) as well as in the touching limb (A). At first, sensation in the active limb is drowned out, in a manner of speaking. Because the moved limb is also moveable, however, this relationship may immediately be reversed: the sensation in A can be expressed, if this limb becomes passive and is touched in an active way by limb B. In that A and B are active alternately, the ...“passive-active double sensation” (Palagyi 1924) separates itself out; in each case, the sensation aroused in the passive limb drowns out the other. (Gehlen 1988 (1958): 154)





Gesturecraft

In this fashion, bodies become structurally coupled with the environments that they inhabit (Maturana & Varela 1992), and, at the same time, knowledgeable. Our fixed motor patterns correspond to affordances of objects which these reveal during our daily, repetitive tasks. The haptic system encompasses a diverse set of bodily skills, and it can reach a high degree of precision. Gibson has pointed out that we have very accurate and distinctive propioceptions of hand-shapes, e.g. of the opposition spaces of our hands, which allows us to perceive relatively minor differences in size between two objects when we grasp each in one hand. We are also able to compare height and width of an object by “successive spanning with two fingers” (Gibson 1966: 119): in fact, the relative dimensions of an object can be simultaneously perceived by the relative spans between all five fingers... When all five fingers touch an object, there are five distinct sensations of touch but there is a perception of only one object. ... Multiple touching of this sort yields haptic perception in the literal meaning of ‘laying hold of ’ (119).

Of course, we also perceive and know our own gestures through kinesthetic perception – independently of the visual access to them that we may also have. The knowledge that our hands possess is not a set of abstract technical skills, but rather is constituted by our practical familiarity with specific life-worlds within which our bodies have come of age. Bodies acquire skills and distinction over a life-time; a human body comprises more than phylogenetic skills. Mature hands incorporate worlds and possess “habit memory”, “keeping the past [alive]... in their continuing ability to perform certain skilled actions (Connerton 1989: 72). The coupling of hands with particular life-worlds has been described in a handful of ethnographic studies. For example, Harper (Harper 1987) portrayed a carmechanic who possesses a detailed working knowledge of materials and develops precisely the kind of tactile, empirical connection that leads to smoothly working rhythms, appropriate power and torque, and the interpretation of sounds and subtle physical sensations. (He) reads his body’s messages and measures out the appropriate force of a blow or a twist. (Harper 1987: 117–8)

Keller and Keller (1996) describe the incarnate intelligence of blacksmiths, who develop a feel for the right temperature at which a particular piece of metal will comfortably bend, given the right amount of pressure. This kind of knowledge has no other place than the hands in action; it cannot be abstracted, objectified, formulated in so many words or cast in rules. It is inalienably tied to the single body that has obtained it in endless, both repetitive and variable practice. It is what Polanyi calls personal knowledge (Polanyi 1958). A specific form of embeddedness of manual skills within spaces or settings or physical materials is the familiarity that a musician develops with his instrument of

Chapter 3. Hands

choice. David Sudnow, an ethnomethodologist and classically trained pianist, has described how his hands’ knowledge of the piano keyboard became reconfigured when he learned to play jazz. Now he had to learn how, during improvisation, he could methodically get from one structured ensemble of keys to another, where no glance of the eyes and no thought or mental calculation could ever be fast enough to manage such chord progressions: the required knowledge can only reside in the practiced, unthinking relationships (or structural couplings) between a single pair of hands and the arrangement of keys on a keyboard (Sudnow 1978). Moreover, one of the hands of the jazz pianist must also carry the rhythm, a task that is often passed, via the interplay of the thumbs, from one hand to the other. Our hands incorporate worlds. This means that their habits – the motion patterns which they are apt and prone to execute – are coupled with instruments and objects in whose company they spend their lives; these instruments become part of the hands’ identity: actions in which I habitually engage incorporate their instruments into themselves and make them play a part in the original structure of my own body. (Merleau-Ponty 1962: 91)

Thus, “hands get shaped. They may get callused or stained. They pick up experience” (McCullough 1996: 2). When my hand reaches out to take hold of a mug, it remembers how it took hold of it on prior occasions, and it configures itself accordingly, without my being aware of it. This is a paradigmatic cognitive act: I enact something that I know about the world. As it first reaches out, my hand relies on habit-memory alone, configuring itself as a token of a prehensile type, chosen in light of the kind of object to be taken and the kind of task to be solved. To some extent I share my manipulation habits with other bodies. But at the same time the particular configuration of techniques that make up my body is unique to me; it belongs in its specificity to my concrete, singular, incidental life: my body is “my point of view” (Merleau-Ponty 1962: 70). “I ... take my place through the medium of my body”(106). To Heidegger (1962 [1926]), manipulating things and unthinkingly using equipment are our most basic modes of understanding the world. By using tools and handling things we also incorporate and make sense of the world that others have made and left for us in form of the artifcacts that we are handling and the techniques that we apply: we incorporate their motives and make them our own.





Gesturecraft

Conclusion The actions and the play of our hands are a form of thinking, because thinking is an activity that involves the entire person. Not only are most of our experiences mediated by our own actions – a condition that is nowhere more visible than in the action-perception loop that is grasping – , but “internal” thinking – thinking without use of the hands – is also, as neuroscientists increasingly believe, internalized movement: ”the ability to think arises from the internalization of movement” (Llinàs 2002: 62). The brain, according to Llinàs, has developed from the need to make predictions about the world so that the organism can move forward; features of the world become embedded in the neural patterns by which motion is controlled, and abilities to simulate space are developed. This is visible in an immediate way in the picture of the hand configuring itself as it reaches: this is the basic form of thinking in Llinàs’ sense. I will take it up in Chapter 7. This study of gesturecraft, of everyday practices of gesticulation, therefore begins in the domain that the hands know best, in the “world at hand”, also known as “peri-personal space”: we look at gestures that illuminate actions and their local settings, and then step by step investigate gestures at greater distance or remove from the world: in relation to visual fields or scapes; the talked-about world (universe of discourse); the conceptual world; and the abstract, yet immediate world of communicative process and encounter.

chapter 

Gathering meaning

The stage-like, fictionalizing “gesture space” (McNeill 1992) in which we place many of our gestures during conversation is not the primordial site where hand gestures first emerge. A more basic positioning of the gesturing hand is within the physical and phenomenal world within which people, as they communicate, operate. Gestures emanate from people as they touch and handle things and go from one place to another, and many of their hand gestures serve to make the settings that they navigate and the objects they become occupied with intelligible and meaningful in relation to a project. For convenience, we can distinguish two zones of proximity to the actor: the world within reach of the hands, that is, the zone within which we are able to take and touch (even if we may have to shift our position a bit), and the far larger zone that we can see (bounded by walls or the horizon). Of course there is no clear border between the two zones, they fade into one another, and it is possible to communicate about the things right next to oneself as targets of vision, not touch – to point to them, rather than handle them. But whether referential objects are within sight or within reach can make a difference in the forms with which we make gestural reference to them. Both varieties have in common that hand gestures are perceived and understood along with the present environment, that they are “environmentally coupled” (Goodwin 2007). An endless variety of indexical practices of the hands can be observed, methods by which people make discriminations, and highlight, emphasize, elaborate, and interpret the present world and orient each other to it, in lived-in settings and scapes. Pointing gestures are the single variety of indexical practices that have received sustained attention, but there are many other methodical ways in which people move their hands in pursuit of common understandings of the setting at hand. The practices collected in this chapter can be subsumed under a term that Heidegger (1962 [1926]) has introduced and that Dreyfus (1991) has given a concrete interpretation: clearing. “Clearing” means that an objective, merely existing, uncomprehended setting is transformed into a field that is jointly known and understood by the parties. Where there was opacity, there is now transparency: we can see what is the case, what is going on, what is wrong, and what needs to be done. We show each other what has happened here and what we will do. Such clearings of the field are routine components of many forms of cooperation among people.



Gesturecraft

We will enter this domain by following a speaker-actor as he walks onto a scene: Mr. Hussein C., the owner of the car-repair shop Hi-Tech Automotive, as he checks up on an employee and a shared sense of “what is needed to do what needs to be done” is being worked out. This and a few comparable incidents give us a category of gestures of orientation, gestures that aid the parties in working out a shared perspective upon the situation, a shared focus of attention or aspect under which objects are to be attended or featured. Before I describe further gesture practices by which meaning in the environment is mobilized, I anchor these in a concrete conception of clearing, which is illustrated by another episode from the auto-shop. The practices which I then describe and which, emphatically, only constitute a small, if diverse, subset of what we must presume to be an open-ended, growing family, include: tracing (the drawing – or discovery – of lines); exploratory procedures (gestural versions of feature-extracting practices as described in Chapter 3); dis-assembling (when things are virtually taken apart); and augmented action (practical action made intelligible by gestural enhancement). These practices all pertain to the world at hand: they are predicated on the contiguity of reality and representation. In the case of practices that belong to the more distant world in sight, this contiguity is no longer a tactile, just a visual one. We will observe only a single episode in which the landscape in which the interaction takes place is gesturally structured, especially varieties of pointing and tracing. That episode ends when the speaker turns away from the land around her to face her own gestures, with which she depicts, from up close as it were, what one would see if one visited a certain place somewhere out there in the distance. This turn will take us in Chapter 5 into the next realm of gesture, where the hands act as makers of representations of distant places or virtual scenes, i.e., depiction. The examples for this chapter come from two settings: the auto-shop gives us examples of gesturing in the world “at hand”, while the examples of gesturing in the world “within sight” come from a rice-farm. The activities – and what needs to be achieved, cognitively, in the two settings – are not all that different, however. In all cases, the objective is to “clear the field”, to figure out what is the case, what is wrong, what has been done, what can be done, what is to be done. Borrowing a term from computer engineering, we could also call these methods augmenting reality: visible reality is visibly enhanced by actions of the hands.1 This term emphasizes the layers of meaning that are superimposed upon or projected onto the world that we can see. The term gathering meaning, in contrast, that I take from Ingold (2000) and prefer, emphasizes that meanings are not usually brought into existence by indexical practices, but that these methods aid in select. An example of technologically augmented reality are goggles that allow someone to view a prior historical incarnation of a building while looking at the present, tangible one.

Chapter 4. Gathering meaning

ing, disclosing, emphasizing, and elaborating meanings that are already inscribed in the world, residues of prior human action. Either way, the following are practices which require that the gesture of the hand be seen and understood within its environment.

The world at hand At about 10 o’clock on a Thursday morning, the owner of Hi-Tech Automotive is making his round of the shop as he does every day, making sure that his employees know their assignments and have the resources that they need to do their work. The Chinese welder whom everyone calls Uncle Ahm is working on the front end of an old automobile which, as most cars that he works on, is being remodeled to be sold. Ahm speaks little English. When Hussein arrives, the following dialogue unfolds: 4:1 Hi-Tech Automotive 1 2 3 4 5

Hussein Ahm Hussein Hussein Ahm

Hi Uncle Ahm. Hello How are you. We got all the part you need? Mmm?

6

Hussein Every• thing you need we got, right?

Fig. 4.1.2





Gesturecraft

7

Ahm

8 9 10 11 12 13

Hussein Ahm Hussein Ahm Hussein Hussein

Uhhh• hhh hhh ( - - - - - - - - - - - - - - - - - -) This ‘cross the street. Uhh. This ( – ) ‘cross the street. (nods) I have another car to cut. Everything over there.

This exchange is accompanied by several hand-gestures by both men; given Ahm’s very limited ability to participate in English conversations, they may well carry the main load of the exchange. Helped by their gestures, the two men quickly come to an understanding. As he formulates part you need (line 4), Hussein points to the car that Ahm is working on. The car is deictically introduced as an argument, as if the speaker were saying “all the part(s) that you need for that car”. Ahm turns around as Hussein rephrases the question, changing the word-order: “Everything you need we got, right?”. This time the talk is accompanied by a rapid series of waving gestures with the prone open hand (Fig. 4.1.1), back and forth between a hood that he is leaning on and the body of the car, thereby gathering together the various items into a set. Instead of giving a spoken answer, Uncle Ahm now moves to the front-end of the car, looks at the area where the head-light will go, and draws a line across it with his extended index-finger (Fig. 4.1.2, line 6). Hussein takes this as an answer: as his next action reveals, he understands Ahm to be telling him that he is still in need of a certain part (this, in line 8). Thus, like Mead’s prehistoric protagonists, Ahm and Hussein conduct a conversation of gestures, demonstrating to each other what they see each other to mean. In an instant, a scene that Hussein does not understand is transformed into one that the two understand in common. They agree on a need. With his next utterance, the shop-owner offers a solution. He points out that the unnamed missing part is “across the street” (in the small junk-yard for salvageable cars and car-parts that is maintained there.) Interwoven with the utterance are again two pointing gestures. Hussein first aims his hand towards the headlightsocket, as he utters this. Ahm is still looking at that part at the time. Then he points to the junk-yard, as he utters ‘cross the street. Hussein’s first gesture succeeds in attracting Ahm’s attention, but too late: by the time Ahm looks to Hussein’s hand, this hand has already moved on to a new position, ready for a new gesture. Ahm seems to show that he does not follow (line 9). Hussein repeats the utterance and the two-part pointing: with his left he points to the head-light socket, then, with the right, to the junk-yard (Fig. 4.1.3, line 10). Now Ahm nods and thus shows “receipt” (line 11).

Chapter 4. Gathering meaning 4.1.2

10 11

Fig. 4.1.3 Hussein This ( – ) ‘cross • the street. Ahm (nods)

Notice that the two pointing gestures at line 10 (left hand to the car, right hand across the street) in effect constitute a truncated, but logically complete proposition: an object is selected and connected to a location. Gestures are embedded in a landscape that the participants inhabit together and know in common. Pointing gestures and other physical acts of orientation do not simply select physically present entities, but often also evoke their implicit, known-in-common meanings: an area of the car is the place where the headlight will go, across the street refers to a junk-yard where headlights can be found. It is their shared knowledge of the cognitive landscape that enables the parties to communicate via a minimal set of gestures of orientation.

Gestures of orientation Episodes of clearing often begin with the establishing of a shared focus of attention: some object or place, a figure, is selected from the surrounding world, the ground (Goodwin 1994); a single entity, of whatever kind, is individuated, elevated to central position within its field. A common way in which this gets accomplished is by acts of pointing: the very point of pointing is to direct someone’s attention to an object or location within the currently visible part of the world. Not always are index fingers or thumbs used to this end; pointing can also be done, in some societies preferably, by head-gestures, pursed lips, or other body-parts (Enfield 2001; Sherzer 1972; Wilkins 2003); the eyes, too, can serve as pointers (Goodwin 1981; Streeck 1993). Looking more closely, however, we realize that the term “pointing” is itself too narrow to denote the whole family of focusing and topicalizing actions of the hand that we can observe in settings that are populated by material objects and inhabited by mobile human actors. What we find there is an array of functionally related practices of which pointing gestures – which individuate their targets – are but one component. By way of choosing among different possible hand-shapes, actors can construe targets as individuals, groups, collectives, relations, or distributed sets, corresponding to linguistic classifier-like words such





Gesturecraft

as “bunch of ”, “those there (but not those others)”, “this as it relates to that”, “the whole”, or “all”. These gestures of orientation – or the hand-shapes or motion patterns chosen – behave somewhat like classifiers in spoken and signed languages in so far as they categorize matter by the ways in which it can be grouped, divided or combined. But the function of these gestures is not primarily a descriptive one. Rather, they are made to figure intentional relations (Barresi & Moore 1996), that is, patterns in which self and other will relate to one another and to the object of their attention in the interaction to come: Mr. C., as he approaches his welder and figures a vaguely inclusive gesture of “gathering”, while uttering everything you need, opens the entire setting of the activity as object for shared scrutiny; Ahm, in his answer, selects a single (absent) object as target of their joint attention. In both cases the gesture indicates “what is being attended” and “how the object attended features in the project at hand” (as a specific object of attention, one of a bunch, in relation to another, etc.). This is similar to the distinctions made by some gestures that Kendon describes, which refer “to an object in virtue of its aspect of spatial extent, or several objects considered as an example” (Kendon 2004: 167). In example 4.2, as he approaches Cedric, a mechanic, Mr. C. extends his hand with an extended index-finger towards a pile of tires on the shop-floor, but moves his hand back and forth, thus selecting not a single object, but the entire set (Fig. 4.2, at line 1). 4.2 Hi-Tech Automotive

1 2 3 4 5 6 7 8 9 10 11

Fig. 4.2 Hussein Let’s take • all the used tire across the street, those Cedric Okay. Okay they belong (in the hallway) These- ( - - -) I mean (1.5) belong (in the hall) anyway Hussein Okay If it’s cut and ba:d Who brought these here? Cedric A:rt He he was trying the rims out,

Chapter 4. Gathering meaning

The gesture in this segment involves a similar motion as that produced along with “everything you need” in segment 4.1; the hand moves back and forth about the floor and “gathers” things into a set. In this example, not much work is required to single out the object in focus: the tires are right there, in front of the protagonists, and rather than singling out specific ones from the set, reference is to all of them. In contrast, pointing gestures that individuate targets by selecting them from their surroundings or from a set often require focused visual attention, and it can take interactional effort to bring it about, as in the following example, which involves the same protagonists and a different set of tires. 4.3 Hi-Tech Automotive

1

2

Fig. 4.3.1 Hussein (summons Cedric with an ‘index up’ gesture) (5.5)

Fig. 4.3.2 Hussein The • front right side tire

Fig. 4.3.3

3

the fro:nt •





Gesturecraft

Fig. 4.3.4

4

Cedric

(

)

4 5 6 7

[ Hussein • this side Cedric Okay Hussein you need to switch with that tire Cedric Okay.

Here, it is essential that the recipient can single out a single object from among several of a kind. It is not enough that the recipient looks in the general direction that the hand is aimed at, but he must focus correctly. Mr.C. secures Cedric’s proper orientation by holding his arm at the ready (Fig. 4.3.2) until Cedric has turned around and looks in the right direction. Cedric signals success by briefly pointing in the same direction (Fig. 4.3.3), but Mr.C fine-tunes his recipient’s gaze by small, lateral movements with the sagittal palm (Fig. 4.3.4). In their gestures of orientation, the members of the auto-shop use handshapes to make minimal practical distinctions: objects of attention are construed by the hand as individuals, groups, wholes, relations, distributed sets, and so on. Gestures of orientation are made to orient co-participants to thematically relevant sectors of the perceptual field – or the field of action – , and distinctions between hand-shapes and motion patterns can be exploited to project elementary practical classifications onto the field of action. By means of gestures of orientation the parties reveal to one another what within the world at hand matters to their project at hand, and whether it is something specific, a part, the whole, a collection or a set. Gestures orienting to a field are often made with a flat hand, gestures referring to a collective with a cupped hand, and individuating gestures with an extended finger (generally the index finger, sometimes the thumb). In the pages and chapters that follow, we will come upon other varieties.

Clearing The knowledge that the inhabitants of a world share is not only presupposed by their gestural communication, it is also in part the result of it. When we observe how people in work-places such as construction sites, farms, design offices,

Chapter 4. Gathering meaning

or auto-shops plan their work and organize their tasks, again and again we find gesture-type actions by which they annotate and highlight the setting at hand. These acts of clearing transform settings into “spaces of possibilities” (Dreyfus 1991: 189) and enable the shared understanding of the “involvement-whole” that is structured by such meaning-relations as “in-order-to”, “towards-which” and “for-the-sake-of-which”. The interconnection of these relationships ... [is] ‘significance’ . . . [whose] unity makes up what we call the ‘world’. (Heidegger 1962 [1926]: 364)

Gestures can contribute to clearing by selecting from complex objects those parts that are relevant now; by bringing to light features that are hidden to the uninformed eye; by suggesting actions that can be performed in the setting; or by indicating a temporal order in which objects should be handled, to name only a few functions. In the following scene, Hussein and Mike, an apprentice, inspect the trunk of a car that Mike is preparing for sale. He wants to show his boss what he has done and what he thinks he still needs to do. At the beginning Mike opens the gesture track of the interaction with another example of a “classifier”; this one corresponds to a category like “distributive”. It indicates objects or a substance that are dispersed. Mike moves his left hand, whose fingers are abducted, back and forth inside the trunk (Fig. 4.4.1). This depiction accompanies the statement The carpet is dirty. 4.4.1 Hi-Tech Automotive 1 2 3 4 5 6 7

Mike

The interior was really clean and this rear light all this is fine,= Hussein =Yeah. [ Mike Maybe the • carpet is dirty, I don’t wanna change the seats because it takes too long ‘n I have to wait (in) line. You know . At the car-wash it’s real quick and it picks up all the dirt and [ Hussein Yeah.

Hussein’s gaze follows Mike’s hand. The gesture, which directs Hussein’s attention, is another example of a gesture of orientation that establishes the thematic focus of the scene by selecting an object for joint attention, but it also classifies it in a certain way, as something that is dispersed – as, indeed, dirt might be on a carpet. This is achieved by a hand shape and motion trajectory which structure the recipient’s vision and propose how what is visible should be regarded. Mike then models for Hussein how he might vacuum the trunk at the carwash: he configures his hand to a prehensile posture which could hold the nozzle of a vacuum cleaner, and moves it about the carpet, in a highly schematic simulation of vacuuming (Fig. 4.4.1, at line 6). The boss is hereby given an enactive demonstration of further work to be done. This is a good example of “augmenting





Gesturecraft

reality”, for the scene at hand is elaborated – augmented – by motions of the hands that add a layer of significance to it, in this case a layer of possible future action. The “occurrent” (Heidegger) is enhanced by the possible.

Fig. 4.4.1

In the following sequence, occurring just a few turns later, an object is assigned its place within an involvement-whole. Hussein discovers a bolt, grasps it, takes hold of it, and asks about its function (line 1). 4.4.2 Hi-Tech Automotive 1 Hussein What this bolt here for, need to go inside?

2 3 4 5

Mike Hussein Mike Hussein

Fig. 4.4.2 Oh no, there’s- an ( - - - - • - - - - - - - ) Something to hang from this? For the cover right here. Okay. ( - - ) You just inspect everything.

Mike gives partial and inconclusive answers. The dialogue appears truncated: note Mike’s abandoned turn at line 2 and the many deictic terms: this, inside, right here. The videotape, of course, shows a perfectly coordinated and successful clearing sequence. The main component of Mike’s answer is a practical demonstration: he extends both arms, bringing his hands to tracks inside the hood of the car, then, with his index-fingers pointing, moving them along these tracks, tracing each twice (Fig. 4.4.2, at line 2). In this fashion, Mike visualizes the role of the bolt in the operation of the hatchback: when the hood is opened or closed, the bolt moves back and forth along the tracks. It is impossible to imagine this clearing without the support of gestural actions.

Chapter 4. Gathering meaning

Mike, the apprentice mechanic, and the other personnel at Hi-Tech Automotive utilize a repertoire of differential hand-shapes and motion patterns to indicate referential objects to one another when they illuminate scenes of action. The philosopher W. V. O. Quine (Quine 1960) has suggested that the ultimate reference of any linguistic term is opaque and inscrutable. For example, when we see a rabbit running by and our language teacher says gavagai, we cannot ever decide (without the help of other terms) whether the word refers to the rabbit, or its running, or to running rabbits or running rabbit-parts. No ostensive definition (or pointing gesture) could ever resolve the matter. However, when we examine how people actually proceed when they clear a field, we find them making subtle distinctions in the ways in which they use their hands and point to or otherwise select objects, which allow their interlocutors to infer what aspects of these they wish to highlight and focus attention on. If Mike or Hussein were the language instructors in Word and Object (Quine 1960), they would know how to shape their hands or alter their motion to ostensively distinguish between rabbit-as-individual, running-ofthe-rabbit, and rapidly-moving rabbit-part; reference may not be as inscrutable as Quine has suggested.

Tracing: Discovering and drawing lines One may disagree as to whether the people at Hi-Tech and other gesturing collaborators impose meaning with their gestures onto the scene at hand, or rather gather and disclose meanings that are “already there”. The former view would correspond to a constructivist view of intersubjectivity and human action – meaning is the result of sign-production and usage – ; the latter to the phenomenology of embodiment, which posits that meaning is, in the first place, the product of intentional action in the world, incorporated in acts and their instruments, objects, and settings, from which it can be recovered and made salient and public by indexical practices of use. But it would be imprudent to make too much of this quasi-epistemological dichotomy, because the dialectic is inherent in the actions of the hands: human hands are foveation organs – data-gathering devices – , in as much and at the same time as they are organs of making. What is interesting to observe is how the hands’ data-gathering methods are used as the basis for gestural communication: features of reality discovered by public manual foveation can be broadcast to others by giving distinct, visible profiles to the motions of the hands. In the following scene from the auto-shop, Hussein examines a car in the presence of its owner. He configures his right as “antenna hand” and moves it laterally back and forth along the head-gasket of the car. He thus displays the discovery of a line, which in turn betrays a leak. The customer then repeats the action, and this has a double effect: it enables him to gather the same sensory information – to





Gesturecraft

repeat Hussein’s experience – and to show that he understands the significance of Hussein’s embodied, communicative act. 4:5 Hi-Tech Automotive

H. traces line _______|___

1 2 3 4 5

| We have bad leak here ____________ | you see here leak? Customer ( ) Hussein The valve cover gasket we should take care of this I think. Hussein

Hussein and his customer thus communicate to one another their joint understanding of the problem at hand. Hussein’s tracing of the leak visualizes it as a line but also provides a model for the customer to re-discover the leak for himself. What is interesting are the multisensory nature and transformative potential of such tracings: while the actor’s foveating finger may follow the lead of tactile discoveries, to the interlocutor this action provides visually mediated information. Tactile experience is visualized; the finger, while moving along in microscopic increments, following locally available input, ends up drawing what others can perceive as a straight line. Tracings have been described in the context of various work-place activities (Goodwin 1994), including ones that involve graphic representations (Henderson 1999; Ochs, Gonzales & Jacoby 1996). In some contexts, as we have seen, what is going on is rather a tactile gathering of texture, which produces the appearance of a virtual line as a communicative by-product. The hands often participate in tactile (and haptic) and visual contexts at once, which provides for easily intelligible connections between the two sensory realms. Hand-gestures enable translations between the senses. Thus, in a powerful way, the dual nature of the hand is recruited for communicative purposes; tactile features of the world, presently available only to a single party, are visually broadcast to everyone present.

Exploratory procedures Tracing is an exploratory procedure. Exploratory procedures are epistemic actions (Kirsh 1995) of the human hands, which, rather than making or changing the world (as practical or instrumental actions do), are performed for the purpose of gathering information. Exploratory procedures characteristically differ from practical actions by their motion patterns, typically showing features such as repetition and rhythmicity or prolonged tactile contact. Their purpose is to discover and

Chapter 4. Gathering meaning

identify those features that objects reveal to active touch (Gibson 1962; MacKenzie & Iberall 1994), characteristics such as texture, consistency, and temperature. We cannot identify these “intrinsic “ features unless we move our fingers: “the stimulus has to be a change in time” (Gibson 1962: 479). Only temperature can be extracted without finger-motion. Extractable features correspond to performance characteristics of exploratory actions: a rough surface implies different movementcharacteristics of the hand than a smooth or slippery one. The audience can infer invisible features of the object from visible properties of the act. This multimodality of manual action forms the basis for its communicative potential. Exploratory actions can become communicative in a number of ways. For example, if someone performs an exploratory procedure on an object of shared concern, this can be taken by a collaborator as an invitation to engage in the same activity, with the result that a bit of sensory knowledge is being shared – and known to be shared – by the parties; in this fashion “common ground” is being built (Clark 1991). Or, more commonly, a gesturalized version of the procedure serves to visually transmit tactile properties. In the following segment, Mr.C. interacts with a customer, Ms. Nancy, who has come to the shop because she has a difficult time getting her car to start. One of the diagnostic actions that Hussein takes is to test the electric system, including the charge of the battery, by means of an instrument that he calls “battery tester”. As he uses the device, he gives a running commentary on what he is doing, in an apparent effort to make his actions intelligible to the customer. Presently, Mr.C. is draining the battery to control its ability to accept and hold a charge. He makes several pointing gestures and puts his flat, adduced hand on the battery-charger several times, each time leaving it there for a second or two (Fig. 4.6, at line 2). This action is a transparent part of the diagnostic activity: the “heating” of the battery charger is visualized by the exploratory procedure of extended, intense touch by which temperature is routinely examined (Gibson 1962). 4:6 Hi-Tech Automotive 1

Hussein Now- I wanna see ( - ) if the battery good. Okay? (2.0)

Fig. 4.6





Gesturecraft

2 3

Hussein I’m loading the battery now, we’ll go ( - • - ) to heat this. It’s mean we drain the battery down.

Ms. Nancy can easily identify temperature as a relevant contextual feature of the object in hand. An intrinsic property, available only to touch, is visually displayed, while the overall activity is made intelligible to a novice. What makes Hussein’s action gestural is that it is designed with a view to being seen: it is a communicative version of an exploratory act. The relevant feature in the following example is texture. This time a rubbing procedure is used to evoke it. This interaction involves the researcher, Mr. C., and the mechanic Chito, who specializes in refurbishing automatic transmissions. Hussein and Chito hold clutches up to the camera and rub them between indexfinger and thumb, as if exploring the surface texture between their finger tips (Fig. 4.7). 4:7 Hi-Tech Automotive 1 2

3 4 5 6

Chito

These right here you can see the difference between (

)

Fig. 4.7 Hussein This is the friction- ( - • - ) material Chito a bad one and a good one Hussein And this already burnt. The friction? It’s already gone.

The two experts thus enable the researcher, who at this time is looking through the view-finder of his camera, to infer the surface-texture of the two clutches from the fashion in which they rub the pads of index-finger and thumb along them; at the same time they provide a tactile-visual explanation for the need to replace one clutch, but not the other. The exact way in which the exploratory procedure is enacted and the fingers are moved demonstrates that there is too much friction in one, and adequate friction in the other. Are these gestures? These exploratory acts are gestures in the sense that they are here strictly performed for communicative purposes, not to provide the actor with new information. And yet, there is very little difference “in substance” between the two acts: they are nearly identical, the main difference perhaps being

Chapter 4. Gathering meaning

that in the case of exploration, the actor is more likely to look at the object explored, while in the communicative version of the act the actor is looking at the interlocutor. Thus, this is a practical act performed for communicative purposes. The transformation, one can contend, is one of the very foundations of gesture: the “empty”, inconsequential performance of an act for communicative reasons – in the present case to enable the other to vicariously experience texture.

Disassembling objects So far, we have looked at gestural versions of exploratory procedures that seek information about the intrinsic features of simple, homogeneous objects, features such as temperature and texture. A more complex genre of exploratory procedures concerns the composition of complex objects, that is, how “polyliths” are put together. These actions seek to establish (or demonstrate) the structural and relational features of objects, and they comprise characteristic acts of taking, holding, and taking apart. My single example again involves Hussein and Uncle Ahm, the welder. One line of business at Hi-Tech Automotive is the remodeling of cars that have been damaged in accidents and that Hussein has purchased at auctions. A recurring decision that needs to be made in the course of remodeling work is whether missing or broken parts can be repaired – for example by welding – or whether new or used replacement parts must be purchased. These decisions bear upon the profitability of the remodeling business. In the following episode, which is centered around a replacement part that has been bought from a junk-yard but which is broken, Hussein presents Uncle Ahm with a headlight that must be fitted into the socket on the front of a car,2 but whose “frame” or “door” – the part that connects it to the body of the car – is broken. Hussein wants to know from Ahm whether this part of the headlight can be purchased separately. Buying an entire new headlight would constitute a considerable investment, given the low sales prices of these remodeled cars. Note that Hussein, in an interference from his native Arabic syntax, consistently refers to what is properly called “head-light door” as “door-light”, an error, however, that is inconsequential in the current interaction. The episode begins when Hussein approaches a Chevrolet Corsica, opens the door, and picks up a head-light from one of the seats. He calls on Uncle Ahm who is nearby and, walking towards him, holds out the light to him. As he says “they sell this separate?”, he first points with his index finger to a corner of the frame and then takes it between thumb and forefinger and pretends to bend it a little away . This is a different car from the one in Segment 4.1.





Gesturecraft

from the headlight (Fig. 4.8.1, at line 1). Hussein selects an object-part as a figure from a ground; then, by taking it between his fingers and seemingly bending it, he figures a hypothetical act of separating the part from the rest of the headlight. “Separability” is thus extracted and exhibited as a salient feature. 4:8 Hi-Tech Automotive

1

Fig. 4.8.1 Hussein Uncle Ahm. ( - ) Uncle Ahm? They sell this separate you • think?

2

Fig. 4.8.2 Hussein The door-light? ( - - - - - - ) This junk-yard but a-broken.

3 4

Fig. 4.8.3 Hussein They sell door-light sepa• rate you think? (----------) Ahm (I think.)

Then, as Ahm turns around and faces him, Hussein taps one hand on the glass of the headlight (Fig. 4.8.2, at line 2), as he says this junk-yard. The (potentially) separable part of the head-light is contrasted with the object-whole, which is selected by a tapping gesture made with a flat hand, which nearly covers its side. Hussein then goes on to point out that this head-light is broken. He instructs Ahm to see

Chapter 4. Gathering meaning

it this way by twice tracing the fault-line (Fig. 4.8.3, at line 3). Each of these four methods – pointing, taking hold, flat-hand tapping, and tracing – are instantiated several times during this episode of 20 or so turns. While these few kinds surely make up only a small part of the repertoire of indicative acts used in the collaborative exploration of complex objects, their recurrent use suggests that they are not random, but methodical procedures. In both sequences, gestural actions annotate an object while language is used to label parts and states (“broken”). Indexical gestures serve to divide the object into separate components, to select the figure “headlight door” from the ground “headlight”. Tapping with the flat hand makes the object in its entirety salient, whereas pointing gestures made with an extended index-finger select individual parts. Tracing, finally, marks the line where parts of the polylith come together, while holding a corner and pretending to bend it raise separation as a possibility. In this fashion, the complex object is virtually decomposed into its component parts, in a fashion that makes its complexity transparent and that is relevant to the project at hand. The headlight is thus embedded in a stream of practical concerns, established as an object with certain intelligible, known-in-common, and currently relevant properties.

Making action intelligible We have moved from the indication and featuring of settings to the gestured exploration of simple and complex things. Another kind of phenomenon that gestural actions can indicate, annotate, elaborate, and interpret are practical actions, and their manner, objects, instruments, and accessories. Actors can also adopt a variety of reflexive (and oblique) stances toward their own actions, stances which can find gestural expression, and actions are often elaborated for informative or instructive purposes, so that their logic becomes intelligible and transparent to others. This occurs abundantly at Hi-Tech Automotive, for example whenever Mr. C. explains to customers what he is doing while repairing their car. Here, I will only describe a single, minimal example; a more detailed description of gestures by which Mr. C. makes his work intelligible has appeared elsewhere (Streeck 2002b). Communicative enhancements of practical action can be built via operations such as repetition and exaggeration, or small gestural components can be inserted into the practical act. In Segment 4.9, the transportation phase when an object is moved by hand from one location to another is elaborated by a small gestural act. A small segment of the action is selected and formally elaborated: the object is thereby marked as crucial and in need of further attention. (The talk links the modulation to the distributor cap, not the action.) In the example, Hussein and Alex, a young mechanic, are trying to find out why there is no spark when the engine of an Audi is started.





Gesturecraft

4:9 Hi-Tech Automotive 1 2

Hussein Okay now. I wanna make sure we got enough spark now.

3

_____|________ | | Hussein From where you got this cap?

taps cable

Hussein, who is holding the ignition cable, transfers it from one hand to the other, and uses this transfer as an opportunity to quickly tap, first one part, then another section of the cable, thus elevating it to thematic status and recommending it to his co-interactant’s attention. The format is used again when he then asks “From where you got this cap?”: Hussein taps on the distributor cap while it is still hidden. Tapping can thematize an accessory during the course of an action. Tapping is visibly different from the trajectory of actions within which it is embedded in that it involves strategies such as repetition, rhythmicity, and exaggeration in size, which are otherwise not characteristic of the practical action underway, but rather features that are also characteristic of ritualized behavior (Huxley 1966). In this example, repetition marks the communicative segment of the action off from the merely practical phases. One genre of indexical gesticulation, then, consists of explicative performances of practical, non-symbolic acts. In Frame Analysis, Goffman (1974) spelled out a number of possible modulations or “re-keyings” of actions, including makebelieve, play, contest, ceremony, and “technical redoings”, about which he writes: Strips of what would have been ordinary activity can be performed, out of their usual context, for utilitarian purposes openly different from those of the original performance, the understanding being that the original outcome of the activity will not occur. (Goffman 1974: 59)

“Make-believe” is the most common version of this modulation, and by this term Goffman refers to practice sessions, where events are “decoupled from their usual embedment in consequentiality” (59). Mimesis, the performance of practical actions for representational, informative, entertainment, and meta-discursive purposes, is part of the ongoing reproduction of culture in everyday life (Comolli 1991, 1993). Everywhere, the world needs making intelligible, and indexical gestures are readily at hand to help with that.

The world in sight In the scenes that I have so far examined in this chapter, the world is literally at hand and things within it can be handled by the interactants. In the following

Chapter 4. Gathering meaning

episode, the world that is illuminated by gestures lies at a distance, but it can be seen. It appears with distinct features, available not only to the eye, but also to the hands which can point to them, trace them, and annotate what the eyes can see. Gesture operates in the field of visual perception, which is being imbued with knowledge. The landscape in this part of the world is defined by lines and fields, and lines within fields. This is the landscape of rice-paddies, and the scene is a rice-farm in Northern Luzon in the Philippines. The participants are standing near the farm-house on a hill overlooking the farm. Cita and her husband (who is not present during this encounter) are tenant farmers on the farm, which is owned by a relatively wealthy man; his nephew Cito has come to the farm with a few other young men to inspect the irrigation system and discuss farming-methods. Here, Cita and Cito discuss fertilizing and water flow. The language spoken is Ilokano, the local lingua franca and one of the most widely spoken languages of the Philippines (see Streeck 1996). Before, we saw how someone, using an extended index-finger to explore a surface and in the process drawing a line can reveal features of an object at hand that are not available to the unaided or uneducated eye. Something analogous happens when a knowing interactant moves her index-finger along the canvas of the land and thereby discloses to a novice how this landscape “works” and how it ought to be seen. Cita’s first gesture in our extract appears during talk about fertilizing, at line 8 in 4.10.1. As she utters the verb kabil, ‘to put’, which is inflected for iterative aspect: kabkabil, “put again and again” or “put here and there”, Cita moves her extended arm in a wide horizontal arc and incorporates five quick pointing gestures in this motion (each marked by ‘o’ in the transcript, Figs. 4.10.1–3). She thus marks a set of locations – the locations, as her talk indicates, where ‘chicken manure’ is disposed. This happens in the context of an explanation why Cita prefers chemical over natural fertilizer; the gesture shows why: because of the uneven distribution of the latter. 4:10.1 Cita and Cito 1

2

Cito Napig- napigpigsa no aya kwa. Dadiay takki ti manok iti::Which one is strong- stronger. Chicken manure or[ Cita Haan, No,

3

maymayat diay abuno nga:: ( ) nga abuno ngem didiay may, ti we feel that the commercial fertilizer is better

4

panagriknami ta no daydiay takki ti manok, mayat met, ngem the chicken manure is also good, but





Gesturecraft

5

intokuma no (topdresser) nan, adda kuma met diay abuonon, mayat. But if there is (topdresser), if there is chemical fertilizer already, it’s better.

6

Ngamin no takki to manok ke, dadiay latta takki ti manok ke:: agaradu If there is chicken manure only, for instance, they will plow under the chicken

7

kuma tattan, manure

Fig. 4.10.1

8 9

Fig. 4.10.2

Fig. 4.10.3

_____________________________ ◦ ◦ ◦| |◦ ◦ ika• bil-kabilda diay takki ti • manok • idta. they just randomly put out the chicken manure there Cito Ah.

In Luzon, as in most rice-farming areas, irrigation is communally organized. Collectively owned, water and irrigation canals form particularly strong connections between those farmers who share a “line” of water, and conflict, for example by one party diverting water, is always a possibility. This is the cultural backdrop of the following segments. When Cita continues her gestural annotation of the land, she extends her left arm, but keeps her fingers abducted (Fig. 4.10.4). As we have seen, this hand-shape, which contrasts with an extended index-finger, marks a region rather than a specific location. Then, with three long and straight motions of her extended arm and forefinger, Cita marks a large triangle, which is anchored by the deictic ditoy, ‘here’, in the talk (Fig. 4.10.5). By using this deictic, rather than dita, ‘there’, Cita marks a new origo (Bühler 1978), that is, a new reference point which Cito is thereby asked to assume as his ‘here and now’ as he processes the following description, namely the triangle that Cita draws in the air.

Chapter 4. Gathering meaning 4:10.2 Cita and Cito

Fig. 4.10.4

Cita

Fig. 4.10.5

• Derecho ditoy, kasta. Adiay ku• kwami met (It comes) directly here, like that. That’s because there are our what do you call them

Cita enacts a relative system of spatial cognition and representation. Cognitive anthropologists and linguists (Gumperz & Levinson 1996; Levinson 1996, 1997, 2003) have recently conducted a great deal of research on spatial representation, including how gesture is coupled with linguistic systems of spatial representation (Haviland 1993, 1996, 2000). One major division in systems of spatial cognition is that between relative systems – in which location and direction are encoded relative to speaker, addressee, character, or landmark – and absolute systems, in which these are always expressed in terms of an absolute system of coordinates, either the cardinal directions as in the Australian language Guugu Yimiddhir (Haviland 1993, 1996) or some objective feature of the landscape in which the community settles, such as the “up-hill/down-hill” dimension in the Mayan languages Tzeltal (Levinson 1997) and Tzotzil (Haviland 2000, 2002). Research shows that gestural pointing or spatial reference (for example, during narratives) varies in accordance with this difference. Within a relative system, Cita imports the cardinal directions of the reference point she has just defined into the “gesture space” (McNeill 1992), and North and South as oriented to from that vantage point can lie in any direction relative to the current location of the parties. In other words, imagined or narrative space can be rotated for the convenience of the interaction participants. Within an abolute system, this is not possible: both the reference point and any points or directions defined relative to it are located where they actually are. Like Hussein’s orienting gestures at the auto-shop, Cita’s gestures serve to annotate the environment for the benefit of her interlocutor. Thus, when she explains that there is a canal “that comes over here”, she drags her extended index-finger from the left to the right of the shared perceptual field, thus instructing Cito to augment what he sees by an invisible element.





Gesturecraft 4:10.3 Cita and Cito

Cita

Fig. 4.10.6 Ta ‘dda daytoy ka- daytoy::• ::: kali nga aggapo diayBecause there is this ka- this canal that comes over here, that-

Here, a line is not produced by a finger that is gathering tactile information, as in previous segments; rather, Cita’s index-finger moves along visible features of the landscape. Again, we could describe this action as an imposition of meaning or a gathering of structure from an already structured field: an illumination of a meaningful landscape that Cita inhabits and knows (Ingold 2000). Like the episode of object-disassembly at the auto-shop, this interaction in a rice-farm in Asia shows a number of characteristic annotation methods that are instantiated several times: pointing by index-finger to individuate a place or object; pointing with a flat or bunched hand to mark a region; tracing (and drawing) of lines; and dragging with a grasping hand where the width of the hand’s opposition space is used to represent width: a gesture to indicate a canal. Another motion that is responsive to the lay of the land is an elaboration of pointing gestures: as the hand is pointing atop an extended arm, a quick upward-extension of the hand is added, and it is coupled with references to locations beyond the boundaries of the visual field (e.g., on the other side of a line of trees). Cita also modifies her pointing gestures to solicit response: once, she adds a “beat” of emphasis when Cito fails to turn his gaze in the proper direction; in response to her “tag-gesture”, he does. Later during their conversation, Cita and Cito discuss the irrigation system of the farm in greater detail. Because several farms share the same canal, it is possible that several farmers compete for the same water. This possibility is addressed in Cito’s question with which the following sequence is begun: speaking about a particular section of the farm, Cito wants to know whether any farmers “come after” Cita and her husband, so that the water directed there must be shared. At the beginning, Cito turns to the right, raises his arm and extends it, and makes a pointing gesture (line 1). As he searches for a word (see nga::, in the transcript, roughly translatable as “namely”, the stretch indicating a search for the next word), he turns to Cita, then makes another pointing gesture in the same direction, this time with a flat, prone hand, in the horizontal plane, marking an area, not a point, where the other farms are. In response, Cita begins to gesture a depiction of the area, but Cito has simultaneously begun a follow-up question (at line 5). For a brief moment they both point in the same direction, producing conflicting anno-

Chapter 4. Gathering meaning

tations of the landscape, until they finally resolve the problem at hand (at lines 6 and 7). 4:10.3

Fig. 4.10.5

1

2

Cito Awan met ti kwan sa’dtan sa• bilan nga::: Aren’t there any others who are[ Cita agtaltalon? farming?

Fig. 4.10.6

3 4

5

6

Cito agtaltalonto yen su• marunonto kanyayo? farming there who come after you? Cita Awan. DaytaNone. That[ Cito Di mabalenkay’ latta nga agpadanom ditoyen. So can you just direct the water over here? [ Cita Wen, Yes,

Fig. 4.10.7

7

dir• etso adiay danum ‘tta. the water (goes) directly there.





Gesturecraft

As she confirms Cito’s hypothesis about the water-flow (at line 7) – ”the water goes directly there” – Cita produces a truncated repeat of Cito’s gesture: her hand moves in a parallel, if shorter arc, but faster. An exchange of gestures, both coupled with the environment, thus establishes common ground. Now Cita begins to describe the location that she has just pointed out, but to do so, she turns away from it. 11 Cito Paggapgapan na ngay diay danom yo ngay? They don’t collect your water then? 12 Cita

Maymaysa met amin ti irrigasiyon mi, Ading. We all have one irrigation, Little Brother.

13 Cita

dta ‘ni banda na’d There is this channel.

14 Cito

diay. Apan yo’nto kitaen itta. You should go an look at it. [ Hm, adda met ditan. Hm, there is also one over there.

Cita begins to construct a representation in gesture space of a place in the world “out there” and turns her attention – and that of her interlocutor – to her hands. We will explore this moment in the next chapter. The scenes visited here show gestural annotation practices of the visible world that enable the parties to recognize individual targets and locations, regions, entities that are dispersed, and those that exist just beyond the boundaries of the visible field. The practices only work, as most gestures do, by virtue of their coupling with verbal behavior; in fact their very raison d’être is to facilitate the languaging of the world: to enable others to see a scene so that it corresponds to a concurrent linguistic description.

Contiguity Contiguity – or adjacency – between sign and referent is the foundation of indexical signs. In the case of gesture, the meaning of the motion of a hand can derive from how it is positioned and oriented or how it incorporates objects that are nearby. From an ecological perspective, gestures such as those described in this chapter acquire meaning by the way they physically relate to the scene and how they are perceived within the setting. What we can not investigate here (but see Streeck & Kallmeyer 2001) are the varied practices in which hand-gestures cooccur with other sign systems, for example graphic signs, which are themselves the result of micro-gestures. Acts of writing, drawing, and doodling can be interwoven with other gestures or modulated and stylized for social-symbolic purposes. Writing can be embellished, performed self-reflexively or obliquely, or staged. And, as a number of studies of architects (Murphy, to appear), agronomists (Mondada 2007), software engineers (Suchman 1993), and physicists (Ochs, Gonzales,

Chapter 4. Gathering meaning

& Jacoby 1996) have shown, where people draw upon graphic representations such as diagrams or blueprints, they often enhance and augment them by indexical gestures. Gestures can add the third dimension to a blueprint and the dimension of time, change, and motion to static representations; they can even make visible what can be seen on a video-screen (Goodwin 1994). The same is true for gestures that direct us beyond the immediate scene towards the larger scape. While this Zeigfeld (Bühler 1934) – the deictic field – has traditionally been studied as the domain of pointing, we have seen that much more is involved when people “clear a scene” and show one another what there is, what can be seen, where things are, or where something is in relation to other things, and how things work, what we are presently doing, and so on. Indexical gestures provide an ongoing report of the speaker’s positioning in the world and in relation to the other.

Dwelling The practices that we have found and described in this chapter suggest that we approach gesture not as the work of interactants who stand apart from the material world, inhabiting only the “o-space” of their conversation (Kendon 1990), but as the work of those who inhabit or dwell in the world (Ingold 2000). I have begun this exploration of hand-made understanding within the hands’ own, that is, the haptic, manipulative, and tactile, realm, where hands meet things and hold and handle, fondle and feel. This is a shared, a public world: although people can act and explore alone, most of the skills that mature hands possess have grown in part through the observation of, and in interaction with, others. Instrumental actions are routinely interwoven with communicative acts, and the two types do not so much appear as separate sets, but rather as modulations of one another (Goffman 1974). The world at hand is a sphere of “adjacency” (Gibson), where there is direct contact between the body and the surrounding world, where the world enters the body in the form of kinesthetic memory, patterns of adaptation, and procedural knowledge, and the body makes the world by transforming what it finds. Traditionally, human communication and the use of signs and symbols in it are conceived in terms of a triadic relationship between a human subject, a sign, and the world to which the sign refers. Moreover, person and sign (and perhaps recipient) are placed on one side of a divide, the world about which we communicate on the other. Communication is thus separated from the world: it is portrayed as being about, but not of this world. But, as far as gesture is concerned, this disembodied and worldless approach to communication obscures the fact that





Gesturecraft

gestural forms often emerge through the confluence of practical, environmental, representational, and conceptual factors. What we have observed in the examples in this chapter instead recommends an account that situates the communicating person within his or her lived-in world, not apart from it. This is the perspective of a being who, quite unlike the dislocated, closed-in subject confronting an external reality, is wholly immersed, from the start, in the relational context of dwelling in a world. For such a being, this world is already laden with significance; meaning inheres in the relations between the dweller and the constituents of the dwelt-in world. And to the extent that people dwell in the same world, and are caught up together in the same currents of activity, they can share in the same meanings. Such communion of experience, the awareness of living in a common world of meaningful relations, establishes a foundational level of sociality which exists – in Bourdieu’s (1977: 2) phrase, ‘on the hither side of words and concepts’, and that constitute the baseline on which all attempts at communication must subsequently build. (Ingold 2000: 453)

Dreyfus, interpreting Heidegger, writes that “dwelling is Dasein’s basic way of being-in-the-world». (Dreyfus 1991 : 45) and that «theoretical knowledge depends on practical skills” (147). “Neither practical activity nor theoretical knowing can be understood as a relation between a self-sufficient mind and an independent world” (49). Rather, “all relations of mental states to their objects presuppose a more basic form of being-with-things which does not involve mental activity” (52). We have seen the people at Hi-Tech Automotive explicate scenes – objects, properties, behaviors, actions – by highlighting and handling what is at hand. Gesture links objects and object-properties to concepts and selects parts and properties that afford or require further action; sometimes it models action, at other times it elaborates it. The rice-farmer, instead, annotates a world that is presently beyond the reach of her hands, but is available to vision. She draws upon the landscape as if it where a canvas, marking it up with dots and lines. The embodied skills that Cita enacts in this interaction are not derived from more basic practical skills belonging to the handling of equipment, as some of those made by the mechanics. Cita’s are strictly gestural skills, enacted and acquired within conversation. And yet they belong to the world that her body inhabits because they embed its specific visual features. It is by virtue of these features that she is able to draw her interlocutor’s gaze across the scene, to point to locations “beyond” what is visible, to create templates for structured and intersubjective perception. The movements of her hands mark up a kind of landscape – the landscape of rice. Cita’s gestures and her livelihood thus belong together.

chapter 

The turn to the hands

At the end of the last chapter, we saw Cita, the farmer, turn attention to her hands. Having used her hands to enliven and elaborate the world around them, she then turned away from this world and, instead, focused her gaze on the structures emerging from the motions of her hands – structures that depict for the visitor one of the places she had previously pointed to. With Cita, we move from a communicative ecology in which her hands interface with the visible world to one where the hands represent worlds that can not presently (or ever) be seen. In Chapter 4, we studied gestures that are environmentally coupled. We now move on to study those that are uncoupled from the environment: they are about the world beyond the present encounter. This is the ecology of depiction or representation, of gesture as a substitute for the world. In this chapter, I analyze how this turn – which is a behavioral, semiotic, and epistemic turn all at once, is accomplished, that is, how attention is drawn to gestures, both by bodily action and speech. In the next chapter, I will analyze some of the major methods of depiction that are being utilized when attention is on the hands. Different constraints upon gaze obtain when the parties are engaged in the mutuality of face-to-face interaction than when they are oriented together to a “third” object (“joint attention”). I will begin by summarizing very briefly some of what is known about the sequencing and constraints of gaze in conversational interaction, drawing on Kendon’s (1967) and Goodwin’s (1981) work. In the remainder of the chapter I delineate the main methods or organization by which hand-gestures that describe the world are made relevant to others and the turnat-talk. This organization draws upon gaze as well as on deictic expressions; both are utilized to mark the salience of depictions by hand to the turn at talk. Having taken stock of some of the component devices and methods I will then demonstrate how they are utilized together when speakers search for a word. Finally, in an analysis of an interaction between an artist and her audience at an art-show, I show that the shape and explicitness of a gesture is contingent on the audienceattention that it gets. Five languages are spoken in the examples for this chapter: Japanese, German, English, French, and Ilokano, an Austronesian language from the Philippines. Language differences come into play when we look at the resources that languages provide to make hand-gestures salient to the talk.



Gesturecraft

Gaze, visibility, and talk in interaction Face-to-face interaction is structured by an asymmetry in the access that self and other have to the displays on a person’s body: one’s body is visually available in its entirety only to other, not self. Only you – not I – can see my facial expression and my posture. This is true for gesture as well: only others can see all of my gestures. Moreover, recipients can normally see gestures peripherally even when their gaze is directed towards the speaker’s face. Speakers (or gesturers), on the other hand, can see their hands only under two conditions: either when the hands move within or near the line of regard between them and their recipient; otherwise (e.g., when the hands gesture near the lap), when they turn their gaze to their gestures. We must also take into consideration that gaze is not only a method for taking in, but also of giving information, notably about the direction of one’s attention. Thus, looking at or not looking at one’s gesture can have repercussions on what others do with one’s gestures. In any case, other things being equal recipients have privileged visual access to the speaker’s gestures. In conversational interaction, gaze is implicated in several ways in the management of talk, in particular in the organization of attention to the talk and turn-taking (Goodwin 1981; Kendon 1967). Eye contact is the primary method by which the interactional axis between speaker and listener is sustained; speakers judge from the interlocutor’s gaze-direction whether their talk is being received by them. But speakers and listeners do not constantly gaze at one another; rather, their gaze is sequenced in methodical ways. Speakers characteristically seek to achieve mutual gaze at the beginning of their turns (Goodwin 1980): when they turn their gaze to the hearer as they begin to speak, they expect the hearer to gaze at them. Listener-gaze to speaker, in other words, should be there before speakergaze at listener. When speakers do not find a gazing listener, they can be observed either to utter repair-initiators such as prosodic and syntactic breaks and hesitation markers, or to cease talk altogether until gaze has arrived (Goodwin 1981). The two methods are used under different circumstances: breaks are produced when the addressee is looking elsewhere; they solicit recipient gaze. Pauses are used when the recipient’s gaze is already turning towards the speaker. In both cases, when the listener’s gaze reaches the speaker, the utterance is either continued or started from the beginning. The same devices are used when the speaker, who is looking at an object in the environment, seeks to direct the listener’s gaze in the same direction. Thus, by shifting focus and soliciting a parallel shift by other, speakers can bring the environment to bear on the moment’s talk. Speakers, furthermore, also look at listeners – and listeners at speakers – at junctures between turn-components (for example, between successive clauses) and during the completion of turns. During mid-turn, in contrast, speakers commonly withdraw their gaze from the recipient, often adopting a middle-distance look (not ostensibly looking at any particular

Chapter 5. The turn to the hands

target); listeners also tend to look away from speakers for brief periods of time when an extended turn is underway, and they rely on their understanding of the turn’s unfolding grammatical structure to determine when to return their gaze to the speaker. These, then, are the basic facts that need to be taken into account in any analysis of gaze as it relates to the production and perception of gestures: recipient has privileged visual access to speaker’s gestures; speaker- and listener-gaze are sequenced: mutual gaze during turn-beginning, speaker withdraws gaze from recipient during mid-turn without focusing, recipient withdraws gaze from speaker, and both re-establish mutual gaze during turn- completion and transition.

The speaker’s look at the hands We now examine where speakers look when they make depictive gestures. Because here we are interested at first only in the organization of attention to gesture, we assume that we know what the gestures refer to (and indeed, in most cases there is little doubt about this). We will deal with methods of meaning-making in the following chapter. The gestures will therefore be represented in the transcripts by a label that indicates their reference; their forms, however, will be described “etically” in the text. The first example comes from “The Tonight Show” with Johnny Carson. The guest, comedian Richard Jenni, is talking about his adolescence in Brooklyn. Playing upon the stereotype that Brooklyn is a borough of thugs, he explains why he always felt like an outcast: unlike “Joey the Frog” and “Jacky the Salamander”, he did not have a middle name. Before Jenni utters middle name, he makes a twohanded gesture: with his extended hands, palms facing each other, he performs a rapid sagittal, cross-wise “cutting” motion which figuratively defines a slot for the middle name. During the onset of the gesture, Jenni withdraws his gaze from Johnny Carson (Fig. 5.1). 5.1 The Tonight Show

Fig. 5.1





Gesturecraft

1 2 3 4

____________________ | ∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼ | Jenni I didn’t ha:ve- I didn’t • have a- uh (- - - - - - - - ) middle-name. You need that you know. Joey the Frog, Jacky the Salamander

We know that speakers in mid-turn routinely remove their gaze from their interlocutor. Jenni, however, instead of adopting a middle-distance look, redirects his gaze to his hands, as these are making the gesture. (This is marked by the wave-line in the transcript.) Jenni returns his gaze to Carson once the gesture is done, just before he utters the phrase that the gesture has visually prepared: middle name. In this fashion, by looking at his gesturing hands, Jenni proposes that this gesture is something to look at. In effect, with comedic timing, he first presents a visual riddle – “I didn’t have + gesture” – and then solves the riddle by providing the item to go with the action of the hands: “middle name”. Notice also the repair components in Jenni’s utterance: he self-interrupts, recycles the beginning (I didn’t have), self-interrupts again, utters a hesitation marker (uh), leaves a gap of eight tenths of a second, and only then completes the sentence with the phrase middle name. We know from Goodwin’s work (Goodwin 1981) that self-repair is routinely used to solicit the listener’s visual attention or to direct it to a target that is itself identified by the direction of the speaker’s gaze. The object in this case, of course, is the gesture. Thus, when the phrase middle name arrives – which is the center of this brief comedy routine – , it has been prepared by a gesture which itself has been prepared by interactional work that seeks to secure that it is seen. Gaze is returned when the gesture is done and the affiliated word or phrase is being uttered. The following is from a Japanese conversation. The parties, two women who are good friends and live in exile in Germany, talk about car-accidents they were involved in when either was still living in Japan. Tomoio describes how the car she was in in hit another and then spun around. Using the loanword supin (‘spin’), she depicts the spinning several times, each time drawing a circle with a downward pointing index-finger. Both times, Tomoio, who is looking Satomi in the eye before the gesture begins, withdraws her gaze from her (Fig. 5.2.1, 2) and brings it to her hand; she returns it to Satomi when she completes the gesture (Fig. 5.2.3). In this methodical and routine fashion, she alerts the listener to the salience of her gesture.

Chapter 5. The turn to the hands

5.2 Car Wrecks

1

Fig. 5.2.1 ____________ | ∼∼∼∼∼∼∼ | Tomoio ‘hh handoru kirenakute sa, • supin • shichatta no

Fig 5.2.2

steering wheel couldn’t turn spin couldn’t help The car was spinning and he lost control of the steering wheel 2

Satomi

nn:::? ja Tomo chan no suwatteru ho ni butsukatte kita wake? so Tomo chan s` was sitting side to crushed came So it rammed into your side? ((two lines deleted))

5

6

Satomi

A so: sorede do natta no? then how turned out Then what happened? [ Tomoio (sorede) then

Fig. 5.2.3

7

___________ | ∼∼∼∼∼ | Tomoio spin shite sa •, spin did It was spinning

To redirect one’s gaze and focus it upon an object is one method by which a co-participant’s attention and gaze can be informed, if not directed. Gaze then





Gesturecraft

functions like a pointing-gesture: the speaker uses her eyes to point to her own hands as a locus of relevant information. But gaze, of course, is also a way of acquiring information. We have noted above that speakers may have to shift their gaze in order to see their own gestures. By turning gaze to their hands, gesturing speakers thus also secure their own visual access to them. The content that is gleaned by the speaker from her own hands in this fashion can also have an impact on how she continues the talk, as the next segment exemplifies. In this segment (5.3), from a German conversation, the speaker, taking a look at her gesture, then changes the trajectory of her utterance; she brings it in line with the figure that the gesture provides. Describing a stage-play, the speaker reports how the lead actress was standing before the audience with two microphones in her hands. Christine says “she had these two microphones” (hat so zwei Mikrophone); then, having looked at her gesture, she adds “in her hand like this” (so inner Hand). A prosodic break and the otherwise ungrammatical repeated use of the deictic and focus particle so show that the second phrase was appended to the utterance and not initially planned for: the syntactic construction implies a focus-shift. 5.3

Penthesilea

Christine 1 Und danach dann hat se auf Italie:nisch? And after that, in Italian, she 2 Aber total deklamiert, ne? really got into reciting, right?

Fig. 5.3

__________________________________ | ∼∼∼∼∼∼∼∼∼∼ | 3 Und hat so • zwei Mi:krophone, so inner Hand, nich? And she has like these two microphones, in her hand like this, right?

The gesture in this example is made with two hands, which are configured in grasping (or “prehensile”) postures and held in front of the torso. The image of someone holding microphones is thus evoked. After she has turned her gaze (Fig. 5.3) and seen the gesture, the speaker adds the phrase “like this in her hands”; this phrase is not about the microphones, but about the action of holding them,

Chapter 5. The turn to the hands

which the gesture has more literally shown. The utterance becomes more fully aligned with the gesture. Thus, the visual profile that a gesture provides and which may be fully available to the gesture-maker only as a result of a readjustment of gaze, can in some measure provide “new information” even for the speaker which, in turn, may lead the speaker to an adjustment of ongoing talk. The following example, from the same conversation, demonstrates the importance of visual access for the speaker ex negativo. Here, Christine does not see her gesture and produces an utterance which is for a moment entirely incongruent with it. She only discovers this after the utterance is already syntactically complete. By positioning one hand near her chin and the other near her thigh, the speaker unmistakeably enacts the playing of an upright string-instrument; during this enactment, she maintains eye-contact with her listener and can not see what her hands are doing (Fig. 5.4). 5.4 Penthesilea 1

Christine .hh Dann hat- (- -) links und rechts saßen so Kna:bn. And then on the left and right sat these boys

Fig. 5.4

_____________ | Irgendwie blonde Kna:bn (.) ham (.)• Geige oder Somehow blond boys, were playing violin or ________ | Klarinette gespielt. clarinet.

2

3 4

Nee, Cello wa:rs. No, cello is what it was.

5

◦

(0.3) B

Cello. Cello.

In this example, the speaker’s hands are not under the control of her visual system, and the visual information that they provide to the interlocutor is not available to herself. As she begins to utter the the verb-phrase “playing violin” (ham Geige gespielt), Christine produces an enactment, which presupposes a standing instrument. This positioning does not accord with the playing of a violin. Christine





Gesturecraft

corrects herself but incongruously selects clarinet as an alternative; this choice is even less consistent with her body-positioning. Only after she completes the sentence does she discover that this, too, is an error, and she now corrects herself, saying that the instrument was in fact a cello. We know from Gibson’s work about the haptic system that gesturers feel their gestures propioceptively, with their “muscle sense”. Yet the segments in this chapter suggest that proprioception may not be as effective as vision in bringing a gesture to its maker’s attention. Note that the speakers in these segments orient to their own gestures as phenomena in the external world that are simultaneously and publicly available to participants: as visual objects to look at. This public “exhibition” of description gestures is particularly visible when Cita, the rice-farmer, makes her “turn”. Prior to the turn, Cita explains the direction in which water flows through the farm’s irrigation system to her visitor, Cito. After the turn, she describes a water-basin in a remote part of the farm with her hands. At the beginning of the segment below, Cita’s gaze is on Cito. Then, as her arm moves up and unfolds into an extended tracing gesture with which she appears to be drawing lines on the canvas of the landscape, her gaze parallels the vector of her arm (Fig. 5.5.1). As her hand travels to other segments of the horizon, her eyes do the same. She changes hands (Fig. 5.5.2). Preparing for her gestural representation of the water-basin (at the beginning of line 11), Cita then turns halfway to Cito and thus brings the center of attention back into the interaction space. While she then produces a series of depictive gestures, she looks down at her hands (Figs. 5.5.3–6). The function of Cita’s gaze remains the same in the two parts of her discourse: it selects the area or object which the interlocutor should attend. What differs is the object selected by the gaze: first the land, then the hands. 5.5

Cita and Cito

Cito 1

Di mabalin kay’ latta nga agpadanum ditoyen. So you can water here just like that.

Fig. 5.5.1

Cita

2

Wen, diretso adiay • danum nukua. Adda met ittan bandana’ Yes, the water comes directly here. There are also channels

Chapter 5. The turn to the hands

Fig. 5.5.2 3

diayen, haan nga maka• ballasiw diay danum da nga here, the water can not cross over to

4

umay idtoyen. Uray agbagyo haan nga makaballasiw ditoy, come here. There is too much ((transl.unclear)) here,

5

ta adda daytoy ka- daytoy kali nga aggapo diaybecause there is this ka- this canal that comes over here, that-

Cito 6

Paggapgapwan na ngay diay danom yo ngay? (( )) they your water yet?

Cita

7

Maymaysa met amin ti irrigasiyon mi, Ading. Idta ‘ni bandanad We all have one irrigation, Little Brother. There is this channel

Cita

8

diay. Apan yo’nto kitaen. You should go and look at it. [ Hm, adda met ditan. Hm, there is also one over there.

Cito 9

Fig. 5.5.3

Cita

10 Adda maymaysa nga • kayo’t diay nga • nagtubo. There’s only one wooden basin for rainwater.

Fig. 5.5.5

Cita

Fig. 5.5.4

Fig. 5.5.6

11 Adda ti simentona • , asawa, nga kasta • . He has some cement, the husband, like that.





Gesturecraft

I have argued that the speaker’s gaze at the gesturing hands proposes to the recipient that a currently produced gesture is significantly related to the moment’s talk. It is not surprising, therefore, that the shift in gaze-direction is frequently combined with the use of linguistic devices which also highlight the saliency of the simultaneous gesture. We will now describe some these devices as they are used by speakers of a few different languages. Prominent among these devices are indexicals and demonstratives of various types, which alert the recipient to the fact that, in order to achieve full understanding of the sentence in progress, meaning available in the environment must be apprehended.

Directing attention to gesture by utterance design During the initiation and production of descriptive gestures, one can often find deictic tokens in the concurrent spoken utterance, words such as this or various equivalents of such or like this, that suggest that some meaning-component is demonstrated outside the spoken utterance. In German, the deictic adverb or particle so (“such, like this”) is used to this effect; in contrast to its English counterparts, it can be used more freely in a sentence, in combination with different word-classes, including verbs, adverbs, and – by way of its fusion with an indefinite article – adjectives and nouns (see Streeck 2002a). A good example for the gesturerelated usage of so is the previously discussed segment 5.3, where so appears in both prenominal and adverbial positions.

5.3 Penthesilea ________________________________ | ∼∼∼∼∼∼∼∼∼∼ | Christine Und hat so zwei Mi:krophone, so inner Hand, nich? And she has these two microphones, in her hand like this, right?

The first occurrence of so indicates that the subsequent noun-phrase (“two microphones”) is further specified by contextual information (in the sense of “this kind of microphone”, where the kind is somehow shown); the second occurrence asserts that the otherwise unspecified manner in which the actress held the microphones in her hands can be seen somewhere. The suggestion that so makes – that the context elaborates the description – is fulfilled by the gesture: the gesture is the contextual element to which so alerts. The word links the utterance to the

Chapter 5. The turn to the hands

gesture and, by the same token, gives the gesture a status as a fully incorporated, grammatical sentence-part: it marks its syntactic and, thereby, its semantic role.1 So is abundant in spoken German. Given what we know about the direction of semantic and pragmatic change, it is safe to say that the prototypical and, historically primary, use of of so is as a demonstrative adverb: it indicates that the manner in which something is (or was) being done is shown somewhere. The most natural locus for that is the speaker’s body. Thus, so is linked to mimesis and enactment: a word that alerts the recipient that something is being demonstrated through action of the body. In segment 5.6, which is from a conversation between two young German physicians who discuss the hardships of seeking employment, the speaker has trouble finding the word Merkblatt (“instruction sheet”) and repeats the form son several times, as he turns his gaze to his gesturing hands (Figs. 5.6.1, 2). So-n fuses so with the indefinite article ein; it means “such a” or, in colloquial English, “this”. 5.6 BWB 1

2

Dirk Also ich hatte- ich hatte letztens w- irgendwo a äh äh ähm aufm Well, I had- I had- the other day wh- somewhere- uh uh uhm at the

Fig. 5.6.1 _____________ | Dirk Arbeitsamt?oder (.) was das war. Da hatte ich ge- äh äh • Labor Office or whatever that was. I had (ge-) uh uh

. Historically, so originated as an adverb meaning in this manner. The plurality of its contemporary uses makes it more appropriate to refer to it as a particle. A more extensive account can be found in Streeck (2002a).





Gesturecraft

Fig. 5.6.2

3

______________________________________ ∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼ | mal son son son son • son son Merkblatt äh inner Hand? one time this this this this this instruction sheet uh in my hand?

The literal meaning of so is that the next item in the sentence – noun, adjective, verb, or adverb – is further specified by extra-linguistic information. Often, this implicit claim remains unsubstantiated, and so is but a “filler”; the utterance within which it occurs appears vague, its producer indecisive. Repetitive use of so, as in this example, is a common mannerism among speakers of German. And yet, in the uses described here, so is motivated by the gesture that is concurrently produced, and it is anything but a filler: so is the element needed to connect gesture and talk. But so only proposes that contextual details specify meaning, not where these details are found. That place is selected by the speaker’s gaze-direction. Thus, when the speaker shifts gaze to the hands when so is produced – and the coordination is usually very precise – , the interlocutor is not only alerted to the fact that contextual information is available, but also that it is to be found in the hands. Speech, gaze, and gesture are combined in a tight and robust tri-modal construction, the basic shape of which appears to be the same across cultures and languages; what differs are grammatical features of the linguistic resources, as well as cultural constraints on gaze. Whether the gesturer can expect to ever get the full gaze of the recipient may be constrained by cultural taboos on eye-contact. So, then, is a word specifically suited to integrate talk and gesture. Extract 5.7 illustrates this point by showing repetitive coordinations of so and shifts of gaze (at lines 3, 5, and 10). In this example, Christine describes the third act of the play, “Penthesilea”.

Chapter 5. The turn to the hands

5.7 Penthesilea 1

Christine Das war dann irgendwie so:- m .hh sehr metaphThat was somehowlike very metaph-

Fig. 5.7.1

2

__________ | uhm najam. Metaph(h)ysisch, w(h)eiß • auch nich. uhm well. Metaphysical, I d(h)on’t know either.

Fig. 5.7.2

3

_____________.....___ ∼∼∼∼∼∼ |∼∼∼∼∼ .hHHH Hatten se so::- • Fetzen? Von:::- von Fo:tos? They had like these- scraps? From photos?

Fig. 5.7.3

4

_________________________..... ∼∼∼∼∼ | Also- vergrößerte, zer• schnippelte Fo:tos? Like- enlarged, torn-up photos?





Gesturecraft

Fig. 5.7.4

_______________................................. ∼∼∼∼∼∼∼

5

So in diesem For• mat .hh- ( - - - - - - - - - ) Like about this size wiggles

_____________|_______________ Köpfe und alles mögliche, wie son Himmel, heads and everything, like a sky? ___________________________ | über- drübergehängt, wie son Mobile praktisch? .hh hanging up- above, practically like a mobile?

6

7 8

Susi

Mhm. Mhm.

9

Christine Sie stand unten? She stood down below?

Fig. 5.7.5 _____________ | |

Fig. 5.7.6

∼∼∼∼∼∼∼

10

Mit som • weißn Rock • nur? Had this white skirt on, only?

Initially, the speaker tries to give an overall characterization of the scene (lines 1– 2). The particle so (at line 1) projects a description by the hands, but the characterization that Christine then begins to deliver turns out to become abstract (“metaphysical”); she cuts it off mid-word, but then settles for it anyway, marking the descriptor metaphysisch as a tentative, second-choice description (weiss auch nich, “how can I say”). Then, as she moves on to describe the stage-design as well as the location and costume of the main character, she withdraws gaze (Fig. 5.7.1)

Chapter 5. The turn to the hands

and begins to gesture. So is used six times during this extended description (lines 1, 3, 5, 6, 7, and 10, Fig. 5.7.2–6). The examples show that speakers not only alert their interlocutors to the salience of a gesture by “pointing” their gaze at it with their eyes, but also include indexical references to the gesture in their talk. The task of making descriptive gestures relevant to the moment’s talk and soliciting attention for them is a universal one; the linguistic resources which diverse languages provide for solving it, however, may vary. One dimension in which they vary is the range of syntactic contexts in which given indexicals or demonstratives may be used. Japanese has a particle – ko or kou – that is used in very similar fashion as German so, and it corresponds in meaning with so’s original meaning as well: it is a deictic adverb of manner. The dictionary lists its meaning as “thus, so, in this manner, like this”. Not unlike German so, ko is popularly thought of (and used) as a “filler”, used when someone does not know how to continue a turn. Analysis of video-data, however, suggests that a more methodical use made of it is to connect words and gestures. In segment (5.8) ko occurs with a gesture which depicts how two cars moved along a road next to one another immediately before they collided. 5.8

Tomotio

Car Wrecks ‘parallel motion’ _______|___________ ∼∼∼∼∼∼ | .hh Nidai de ko:: hashitteta wake yo. We went like this ‘by two’.

Like so is in the German examples, ko is uttered exactly at the moment when the speaker looks at her hands. It serves as an indexical that “points” to the context; it is also combined with the “look”, which selects the gesture as relevant contextual element. Ko simultaneously assigns the gesture an adverbial role: the gesture shows how something happens or is done. The latter function is also realized in the next example in which Satomi, the other participant, describes how she, when she had a car-accident, tried to keep her car on the road. She re-enacts how she turned the steering wheel to the left and to the right, in an attempt to keep her car going straight. Ko co-occurs with both motions (Figs. 5.9.1, 2). (Here, the speaker’s entire upper body is involved in the re-enactment, including the eyes, and there is therefore no gaze-shift to the hands.)



 Gesturecraft

5.9 Car Wrecks

Fig. 5.9.1

Nde de dooyu wake sugoi ko: • kirisugito n da yo ne (h) And somehow I had turned the steering wheel like this way too far

1

Satomi

2

Tomoio Nn. Mhm.

3

Satomi

4

Tomoio .hhhh .hhhh [ Satomi hh.h de kore wa taihen dattande ne hh .h and it was terrible and that’s why I [ Tomoio .h hh .h hh

5

6

De sugoi migi ni itte shimatte ake and I drove way over to the right

Fig. 5.9.2

7

Satomi

sugoi ko: • kondo: hantai ga ni kitta (wa)ke ne turn it way over to the other side, right?

What distinguishes ko from German so is that it can only be used adverbially, not adnominally or in other syntactic positions; the word is therefore not suited to direct attention to a gesture that depicts a thing. In Ilokano, a Philippine (Austronesian) language with a rich and productive verb-morphology, it is possible to make the demonstrative adverb kastoy, “like this”, the root of a verb. Like other verb-roots, it can be marked by affixes for tense,

Chapter 5. The turn to the hands 

aspect, person, causative, and other categories. In 5.10, three women in a small town in Northern Luzon discuss a traditional Ilokano healing practice administered to children. The remedy involves sprinkling salt into a fire and then cradling the child over the flame. As she checks whether she understands the procedure, Gallit says ikaskastoymon, “you’re like-this-ing-(it)-away-from-you”, as she gives a bodily rendition (Fig. 5.10). 5.10

Rumors

Fig. 5.10

Gallit

_______ | | Nga ikaskastoy• mon. That is, you’re “like-this-ing” it away from you.

The function of the demonstrative kastoy is not unlike that of so and ko. What differs between these examples are the grammatical patterns of the language and, accordingly, the grammatical behavior of the devices that incorporate gestures in the talk. In English, because of its rigid word-order, the placement possibilities for an indexical adverbial such as so or like this are highly restricted: a disadvantage is that like this cannot precede a verb, only follow a clause. Other languages probably provide yet other kinds of devices with different grammatical possibilities. This, in contrast, can only preface adjectival or nominal gestures, but no verbal ones.1 But a single language may also provide an array of different possibilities for making a gesture salient. In addition to deictic devices there are also tag-questions and interjections, forms corresponding to English phrases such as “you see” or “look”. These, certainly, are often used as purely rhetorical devices to appeal to the interlocutor’s understanding or to solicit a response. But they also appear in the context of gestures, to which they seem especially well-fitted. Segment 5.11 is from a French conversation; it includes the tag question tu vois (‘you see’, lines 2 and 3). The speaker is describing the clothes of a man whom she met at a party (the speaker in this examples keeps her gaze focused on the listener.)

 Gesturecraft

5.11 Mignon

1

2

3

4

‘tie’ ________|____________...... | Annie Et il avait en plus- Bon. Il avait une petite cravatte rouge, enAnd in addition he had- Well. He had a little red tie, with ‘dots’ _____|_______ | en points blancs, tu vois? with white polka-dots, you see? ‘suspenders’ ______|________ | | Et il avait deux bretêtes, tu vois? And he had two suspenders, you see? [ Céline Ah sympa! Oh, neat!

At issue here are visual phenomena; the speaker wants the listener to be able to imagine how this man looked. She depicts his clothes by drawing lines and dots on her chest, which evoke the neck-tie and suspenders. The tag tu vois literally asks for confirmation that the visual display has been successful: “do you see?”. Alternative tags that Annie might have used (for example, tu sais, ‘you know’) lack this visual profile and would seem less fitted to an utterance that includes a gestural component and therefore asks for visual attention. While the form tu vois in this example is a question that is fitted to a turn that requires the listener’s visual attention, the Japanese interjection hora in the following two segments constitutes a request for such attention. Tomoio issues it as she is turning her head to the left, facing the segment of space where she is about to make a gesture. In this example, hora solicits visual attention while the turning of the speaker’s head informs the listener where she should look, namely at the location of an imminent gesture. The gesture evokes the rounded end of a guardrail. (The details of the gesture become relevant later when Tomoio describes the shape into which her car was bent as a result of hitting this part of the guardrail.) 5.12 Car Wrecks 1

2

3

Tomoio Kuruma suiteta shi .h supiin shite watashi no ho: no seki ga The car was empty and it spun around, the seat on my side, [ Satomi Nn hn Mm hm. _____________________________ | | Tomoio ano hora (- - - - - - - - - - ) are aru ja na (.) gaadooreeru well look ( - - - - - - - - - -) there was a (.) guardrail.

Chapter 5. The turn to the hands 

In her subsequent account of the actual collision, Tomoio uses hora in this fashion several times. In 5.13 she describes a walkway beside the road where trucks were parked (Figs. 5.13.1, 2); the spatial relationships shown by the hands are important to her explanation of the causes of the accident. Notice the co-occurrence of hora and ko. 5.13 Car Wrecks

Fig. 5.13.1

1 2

3

________________ | ∼∼∼∼ | Tomoio ... michi no hora • ko ne? ... and besides the road, look, like this, right? Satomi

Hn Hm.

Fig. 5.13.2 _______________________________________ | ∼∼∼∼∼∼∼∼ Tomoio Hito ga de (.) ko to:ttari suru yoona • toko atto hora torakku There was this kind of a walkway and look a truck

_____

6

Satomi

| tsukureba parked there [ Nn Mhm.

 Gesturecraft

We see, then, that speakers whose utterances combine linguistic and visual representations – who perform descriptive gestures that are relevant to the understanding of their talk – , draw upon an array of devices to direct the listener’s attention to their hands. By looking at their hands at the onset of the gesture, they effectively “point” to it. By incorporating indexical forms (deictic adverbs, demonstratives, particles, tags, and interjections) in their utterances, they methodically request their recipients’ visual attention, at the same time as they incorporate the gesture into the grammatical structure of the talk. Depiction by gesture, in other words, is a highly methodical practice. It comprises more than the symbolic activity of the hands alone, involving other bodily as well as linguistic devices to produce a shared, intersubjective orientation to the gesture among the interaction participants.

The recipient’s orientation to gestures But how do recipients react to these methodical attempts to direct their attention to gestures? We have noted before that hand-gestures that are made within a certain range of proximity to the speaker’s face can be seen by the listener during eye-contact with the speaker; more fine-grained movements may require adjustments of focus, but unfortunately these are usually not recoverable from our video-recordings; neither are minor adjustments in gaze-direction. Moreover, the recipient’s understanding of a gesture is generally not independently revealed, as little as understandings of individual words in a spoken utterance are commonly displayed by recipients/next-speakers; it is the utterance as a whole that gets the response. Thus, we must not expect that a recipient’s attention and understanding are always overtly marked by concurrent or subsequent talk and behavior: absence of overt responses by the listener does not betray the absence of attention and understanding. However, the listener’s attention to a gesture does usually become visible when a gesture is made away from the line of regard. Listeners show their understanding of a gesture when they are requested to do so, for example, when the speaker is searching for a word and for a moment turns to the hands to convey meaning alone. In these kinds of contexts we can see that listeners, too, orchestrate their behavior by reference to salient gestures. The following two examples show how recipients adjust their gaze when a salient gesture is made outside their current visual field. In 5.6 (see above) the speaker rapidly brings his hands to eye-level as he makes a descriptive gesture. He also looks at his hands. The listener, who had been looking down at his own hands, playing with his fingers, immediately looks up at the gesture. (This is marked by a wave-line underneath line 3; Figs. 5.6.1, 2).

Chapter 5. The turn to the hands 

5.6 BMB

2

Fig. 5.6.1 ______________ | Dirk Arbeitsamt?oder (.) was das war. Da hatte ich ge- äh äh • Labor Office ot whatever that was. I had (ge-) uh uh

Fig. 5.6.2

______________________________________ 3

∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼ | mal son son son son • son son Merkblatt äh inner Hand? ∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼∼

one time this this this this this instruction sheet uh in my hand?

The speaker in the next extract (5.7), Christine, has made several gestures at eyelevel and then makes one in her lap (Fig. 5.7.1). The listener, who is looking at the speaker’s face at the beginning of the segment, redirects her gaze to the speaker’s hand (Fig. 5.7.2).

 Gesturecraft 5.7 Penthesilea

Fig. 5.6.2

9

Christine Sie stand • unten? She stood down below?

Fig. 5.6.2

____________ | | ∼∼∼∼∼∼∼∼∼∼∼∼∼∼

10

Mit som weißn • Rock nur? ∼∼∼∼∼∼∼∼∼

Had this white skirt on, only?

While the last two examples show how the recipient displays attention to a gesture, the following segment exemplifies how a recipient displays understanding of a gesture. In this case, the recipient’s analysis of the gesture is displayed by a “return-gesture”, that is, an identical or similar gesture by the recipient (de Fornel 1992; see also Kimbara 2006; Lerner 2002; Tabensky 2001). This is the example concerning the pediatric practice Heating a child. The speaker’s account (Fig. 5.10.1) meets with some incredulity by the listeners. Gallit, in an effort to make sure she has properly understood the description, re-performs its embodied part (Fig. 5.10.2).

Chapter 5. The turn to the hands  5.10 Rumors

1

Bato

2

Fig. 5.10.1 ________ | | Ket no ag-agranranutrotta asinen, irarabasabmo • diay ubing And when the salt is crackling like that, you’re heating the child,

nga kasdiay. like that.

3

Esmin Nakaitao? A living person?

4

Bato

5

6

Hmm. Anya payHmm. But what[ Esmin Ah. Ah. [ Gallit Nga ikaskas• toymon. That is you’re “like-this-ing” it (away from you). |______|

Fig. 5.10.1

The issue here is that both the child and its handler must be at the right distance from the flames in order not to get burned, and the child must be moved constantly. These details are more readily enacted than described. It is primarily within such circumstances – where a bodily enactment is more precise than spoken language might be – that gestures yield gestured responses.

 Gesturecraft

There is one type of conversational environment, however, in which verbal interpretations of gestures are called for on a more regular basis. These are wordsearches: routinely during word-searches, the speaker who cannot find a word makes a gesture and solicits the listener’s co-participation; if possible, the recipient provides the missing word, relying on his or her understanding of the gesture.

Gaze and gesture during searches for a word The organization of word-searches has been described in detail by C. and M. Goodwin (Goodwin 1986; Goodwin & Goodwin 1986). Typically, when speakers can not find a word, they first try to solve the problem on their own, and, more often than not, they are successful. Conversation analysts have have found that selfrepair is generally preferred to having one’s talk be repaired by others (cf. Schegloff, Jefferson & Sacks 1977). It is only when their own attempts fail that speakers invite their interlocutors’ collaboration and make their difficulties overt: by cutting off their utterance, restarting it, interrupting it again (as in examples 5.1 and 5.7); by repeating the article before the sought-for word (5.6); or by various other devices to be shown below. Word-searches thus can comprise two distinct stages: an initial stage in which self makes attempts to solve the problem, discouraging other’s participation; and, if the attempt fails, a second stage in which the recipient’s collaboration is sought. The different forms of interactional participation (listener does not/does collaborate) are brought about by the speaker’s differential use of gaze: during the first stage the speaker looks away from the interlocutor and makes self unavailable as a recipient of talk; talk by other is, by implication, discouraged. Then, moving to the second stage, the speaker turns to other and makes self available as recipient, and in this fashion invites the interlocutor to participate in the search. In the second stage of a search speakers can be seen to bring gesture into play, offering a visual representation which the interlocutor can use to infer the missing word. While they gesture, word-searching speakers shift their gaze again, this time, like all depicting gesturers, to their hands. In the following two examples, gesturally mediated word-searches fail or succeed only with great difficulty, but the effort to get one’s gesture understood is only more sustained and noticeable. In 5.14, the three Ilokanas in Luzon who have previously talked about childhood medicine discuss abortion methods; Gallit mentions a procedure that involves the use of an herb which grows in the mountains and can also be used to make massage oil or as an ointment for the scalp. But she cannot think of its name. It takes the women three trials to find it. Gallit tries to guide the others with pointing gestures; these mark the diverse locations where the substance is applied or found.

Chapter 5. The turn to the hands 

A search for the name is already underway at line 1 (Fig. 5.14.1). Gallit now raises her hand in apparent preparation of a gesture (Fig. 5.14.2). Then she drops her hand and abandons this construction-effort. Instead, she looks away. Demonstrably and with facially displayed earnestness she tries by herself to retrieve the name (Fig. 5.14.3). 5.14.1 Rumors

Fig. 5.14.1

Fig. 5.14.2

1

Gallit

Di kad- di kad • kankanayon nga agpapailot kenka, • Insan. Wasn’t it- wasn’t it the stuff she always used to give you a massage, Cousin.

2

Bato

Hm. Hm.

3

Esmin H m Hm. Gallit Ka- kankanayon nga- (.) ukinnana! Al- always that- shit of hers!

4

Fig. 5.14.3

5

(--•--)

Then Gallit specifies the sought-for item as the leaf of a plant (line 6) and at the same time invites the others to collaborate in the search: raising her hand, she turns to Esmin, lowers her head and applies rubbing motions with her finger to her scalp; Fig. 5.14.4). She calls the herb “the stuff you put up here” and goes on: “what is it called, the–”. Esmin proposes an herb, Mansinilya, but Gallit rejects it (lines 8–11).



Gesturecraft 5.14.2 6

7

8 9

10

Datay::: bulong ti::- ana! Tha::t leaf of the::- what!

Fig. 5.14.4 _____________________ | | Datay ikabkabil • ditoy, ana’t naganna’, ‘tayThe stuff you put up here, what’s it called, the[ Esmin Mansinilya. Mansinilya.

Gallit

Ah? Haan! Huh? No! [ Esmin Mansiniliya. Mansiniliya.

Again Gallit turns away, but now she focuses her gaze on some distant location and gestures in that same direction (Fig. 5.14.5). The action is coupled with the distal demonstrative datay (“that, away from us”); otherwise, the meaning of the gesture remains unspecified. Perhaps she is pointing to the mountains where the herb grows. After yet another rejected attempt (lines 12–13), Esmin finally finds the plant-name that Gallit has been searching for, Irbaka. 5.14.3

Fig. 5.14.5

11

Gallit

Haan! Da• tayNo! That[

Chapter 5. The turn to the hands

12

Esmin

13

Gallit

14 15

Herbabuena, ay!= Herbabuena, oh!

Haan! No! [ Esmin Irbaka! Irbaka!

Gallit

Irbaka! Irbaka

The gestures in example 5.14 are pointing gestures (although one is elaborated to allude to the process of rubbing a substance into the scalp). To make sense of these gestures – and use them as information towards a successful resolution of the word-search – the co-participants must apprehend them within a frame of reference that combines geographical, botanical, and medical knowledge. In this case, the gestures are indexical and only slightly depictive. But we see that, while gestures can aid the collaborative solution of speech-production problems such as problems in lexical access by facilitating a cooperative solution and thereby removing obstacles to intersubjectivity, they may also require background knowledge for their proper comprehension. The gestures in the following segment are enactments, that is, schematic versions of instrumental acts. These, too, require cultural knowledge for their recognition. In this conversation, a native Ilokano speaker who lives in Germany is talking to another and reports a telephone conversation that she had with another friend. This friend had complained that she had called numerous times, but could not reach her. At this point, the speaker, Elnora, is trying to cite her own response in that conversation: ipabasitko, “I had turned (it, the phone) down”. But she cannot find the proper verb, i-pa-basit (‘to make low’, i.e., ‘to turn down’). With two successive enactments of this action, she tries to get Sandra, her interlocutor, to find the word for her. The word-search begins at line 2. Elnora turns her gaze to her left hand and thus draws Sandra’s attention to the small gesture that it makes: the index finger is rapidly flexed and extended a few times, as if turning a small wheel (Fig. 5.15.1). This gesture is not very specific; it is a “low-resolution” gesture. To stand in for the verb that she is searching for, Elnora builds a complex verb-form around the morpheme kwa, which dictionaries translate as “whatchammacallit” (see Streeck 1996). Here, kwa is a placeholder for the verb-root, which is marked for instrument-focus, past (in-), and first person singular (-k). In-kwa-k, in other words, means sometthing ‘was- whatchammacallit-ed-with-by-me’ or, roughly, ‘Iwhatchammacallit-ed-it’. The affix in is repeated while the gesture is being made and the speaker is searching. Note that the search-process is marked here – and semantic profile is provided – in a manner that would not be possible in English;





Gesturecraft

there is no equivalent to inkwak. As she ends the gesture, still without the word, Elnora looks to Sandra. 5.15.1 Ilokano 1

Elnora

Inkabil ko ti telepono idiay kwa::- idiay sa:la, kunak a. I had put the telephone in the what- in the living-room, I said.

Fig. 5.15.1

2

3

______________________ | Sa in- kwak- in- • in- in- in- in- uhhAnd then I what-ed -ed -ed –ed uhh [ Sandra Hmm. Hmm.

Sandra indeed offers an interpretation of the gesture, a solution to the search: “you had closed the door” (line 4). Apparently she takes the gesture as showing the turning of a doorknob. While Sandra offers this solution, Elnora looks at her hand and sustains it in the position it had just reached, maintaining the handshape it had assumed as it did so; this is marked by a dotted line in the bracket indicating the gesture). This posture (Fig. 5.15.2) shows that she is listening but remains ready for another gesture. Elnora dismisses Sandra’s proposal (line 6). Again she then turns to her hand and now produces a different gesture: opposing index-finger and thumb, she performs a dialing motion (Fig. 5.15.3). This gesture is more specific than the prior one. Again Elnora turns to Sandra, as if waiting for an answer to her gesture. Sandra quickly begins another attempt at helping her (line 7), but she only gets as far as producing an article diay, and a hesitation marker, as Elnora is looking at her (Fig. 5.15.4). Elnora finally finds the word herself and concludes her gesticulation with a quick terminal stroke.

Chapter 5. The turn to the hands 5.15.2

Fig. 5.15.3

Elnora 4

5

__....................................

Sandra Insaram • didiay ruangan. You had closed the door?. [ Elnora Didiay- iniThe- this-

Fig. 5.15.3

......___________....................... ∼∼∼∼∼∼∼∼∼∼

6

7

Elnora

Haan! Didiay- i• ni:::::- sangad ngay:! No! The- thi:: knob, say! [ Sandra Sss- diay- Hnn. Sss- the- Hnn.

Fig. 5.15.4

______________ 8

Elnora

| Ket impabasit• ko, kunak. And I had turned it low, I said.





Gesturecraft

This example shows a speaker who not only orchestrates her bodily actions so that her gesture is seen and can be understood by the listener, but also re-designs the gesture when the recipient’s initial attempt at understanding it fails. During word-searches, hand-movements often carry the primary load of representation, and achieving visual intersubjectivity becomes a public and focal concern for the parties (cf. Schlegel 1998). All of these organizational contingencies come to bear upon the manufacturing of understanding in the last episode that I want to examine in this chapter. The topic in this episode is how a work of art was produced. We will see that how the speaker’s account of this process is understood depends upon whether or not the audience attends to her gestures: these depict the creation as an embodied process in a way which her words alone could not.

Attentional struggle The segment is a scene from the opening of an exhibition of contemporary German art around the time of Reunification, and it involves two state governors (the late Johannes Rau, who later became President of the Federal Republic of Germany, and Manfred Stolpe); a government minister; the artist; her work (an installation including a television set and a bathtub on the bottom of which a long list of names has been inscribed); and several press-photographers. When the politicians and their entourage arrive at the art-work during a tour of the exhibition, the photographers ask them and the artist, a young woman, to group themselves around the bathtub for a photo-shoot. The governors invite the artist (whose father they happen to know) to “meditate” on her work. The artist initially points to the bathtub, and the governors aim their gaze at the list at its bottom and keep it there, presumably reading the names (Fig. 16.1). The artist thus loses their visual attention for the gestures with which she tries to depict how the list of names that she was collecting became longer and longer, turning into a scroll (Fig. 16.2). But as noone is looking at her, she abandons these gestures. Now the artist leans in on her interlocutors; but she fails to regain their visual attention.

Chapter 5. The turn to the hands

5.16.1 The bathtub

Fig. 16.1

1 2

Artist Ja das sind- also das sind Na• men die ich mal- die ich also Yeah these are- well these are names that I once- that I ((

)) gefunden habe, sind aus dem Jahre 1932, have found, they’re from the year 1932

(two lines deleted)

Fig. 16.2

5 6

_________ | | Und äh- ich- habe die erst • ((inaudible)) And uh- I- have these at first und das is- das sind dann so wie n großes Brief geworden. and this has- they have turned out like a big letter

But the artist then discovers that someone else is looking at her, a minister in Mr. Stolpe’s party. She reorients her gaze and torso to this man and brings her hand in a position where he can see it, and then performs a descriptive gesture: she positions index finger and thumb so that they show the thickness of the letter (Fig. 16.3). Then she recruits her other hand in the making of another gesture with which she shows how she “put it together” (Fig. 16.4). 5.16.2 7

Artist Und dazu braucht ich ja noch ein’ Adressaten And you need an addressee for that.





Gesturecraft

Fig. 16.3

8

Fig. 16.4

___________________ | | und ich habe diesen • Brief zusammen• gestellt. and I put this letter together.

The artist’s gestures are responsive to the audience’s visual attention. When she fails to attract the intended audience’s gaze, she abandons her attempt to re-enact or depict (rather than merely describe) how she put the work together. But when she discovers that another participant is looking at her, she settles for the new audience and immediately resumes gesticulation. She shows with expansive, arched motions of her arms and hands how the list of names got longer and longer. Her gestures become meaningful components of her account of how the work came about. Only the looking audience will learn that she assembled a letter with these physical qualities. With these big gestures she finally regains the gaze of Mr. Rau (Figs. 16.5–6), who will eventually incorporate the shape and position of her hands in a return gesture of his own (Fig. 16.7), as he seeks to get her to confirm his understanding of her account (using the same method as Gallit in the discussion about how to properly heat a child.) 5.16.3

9

________________ | Artist Und äh- nach ner Weile halt, And uh- after a while

Fig. 16.5

Fig. 16.6

______________________________________ 10

| da wurden die Listen von Namen im• mer läng• er. the list of names got longer and longer.

Chapter 5. The turn to the hands

(10 lines deleted)

Fig. 16.5

20

Rau

Ach so, ‘n Aus• schnitt ( I see, a cut-out

)

The embodied account that the artist builds up in this fashion shows the development of her work as shapes emerging in space. Her own actions in the process are thus portrayed as embodied responses to spatial experience: she demonstrates that the work came about through a dialogue between her embodied self and the evolving physicality of her work. This aspect escapes those who, like the governors, fail to look at her gestures; to them, the purely verbal account that they receive may present a coherent, but rather more cerebral and dull, idea-driven conception. (They might misconstrue the piece as ”conceptual art”.) The example demonstrates that the gesticulatory activity of the hands can contribute salient meanings to conversational understanding and that speakers make concerted efforts to have their gestures seen and recognized. Mr. Rau and Mr. Stolpe cannot fully appreciate the artist’s account of her work without seeing her gestures, even though her talk will have appeared meaningful and coherent to them on its own terms. In other interactions that we have examined in this chapter, notably those involving speakers who were gesturing while they were searching for a word, no understanding is possible without attention to the gesture; the talk alone is noticeably incomplete. Gesture, in other words, is not only of interest to analysts, but is often relevant to interaction participants, and this relevance is most visible in the efforts that gesturing speakers make to direct their audience’s attention to their hands, in the methodical character of these efforts, and in the fact that natural languages provide devices that appear to be specifically designed for the purposes of anchoring gestures in talk. It is important to keep in mind, however, that this organization only covers one type of gestural praxis, namely the depiction of actions, objects, and events. Gestural depiction, in other words, is framed by a specific framework of participant attention. Using eye-tracking technology, Gullberg has confirmed that only depictive gestures are registered in this fashion. As her data also demonstrate, however, the turning of the eyes to depictive gesture appears to have social rather than cognitive reasons: focal gaze is not normally needed to recognize a gesture; glancing at a gesture may merely be a display that





Gesturecraft

one is aware of it (Gullberg 2003; Gullberg & Holmqvist 1999, 2003), a social act of confirmation. Having accounted for the methods by which speakers attract their listener’s visual attention to the gestures with which they depict the world, we must now investigate what methods they use in these depictions. That is the theme of the following chapter.

chapter 

Depiction

We have seen in the last chapter that interaction participants use systematic bodily procedures and dedicated language resources to make the gestural depictions they make demonstrably relevant to their talk. In this chapter I will investigate the logic of these pictorial representations, of the methods by which they are made. Questions about the nature of depiction in everyday gesture have received relatively little attention by gesture-researchers, although they have been intensively discussed by researchers of primary and alternate sign-languages. Otherwise, a widely held belief, shared by ordinary interactants and researchers, that there is a natural relationship (usually of similarity) between the gestures that our hands make and the visual features of the objects and events that they depict has precluded scrutiny of the methods by which such seeming resemblances come about. Mallery, among others, could therefore suggest that at the beginning of the evolution of sign-languages were “natural images” (Mallery 1978 (1880)), the building blocks of the gestural ur-language of humankind. Imagistic gestures appear to be more natural than other signs: we can understand them because we know what the world looks like and because we see its resemblance in the gestures. As Ekman and Friesen (1969) put it: an iconic gesture “looks in some way like what it means, its significant (1969: 60). But how is this possible? How can the movements of two hands “be like” or “look like” such diverse phenomena as car-accidents, the paintings of Franz Kline, and co-dependency, to name just a few examples I have come across? The philosopher Nelson Goodman has suggested that representation does not depend upon similarity between signifier and signified, but instead enables us to apprehend or imagine the denotatum in certain ways. Representation actively organizes the world. The making of a picture . . . participates in making what is to be pictured. . . . The object does not sit as a docile model with its attributes neatly separated and thrust out for us to admire and portray... Representing is a matter of classifying objects rather than of imitating them, of characterizing rather than of copying... A representation or description ... analyze[s] objects and organize[s] the world. (Goodman 1976 [1968]: 32)

 Gesturecraft

The relationship between a phenomenon or experience and its gestural representation must therefore be approached from the side of the signifier, not the signified, because it is the signifying method or practice that enables us to engage in pictorial-gestural communication. Some researchers have suggested that “gestural representations of concrete reality ... derive from the power of abstraction” (Kendon 1992a: 241). Arnheim (1969: 117) wrote that the portrayal of an object by a gesture rarely involves more than some one isolated quality or dimension, the large or small size of the thing, ... , the sharpness or indefiniteness of an outline. By the very nature of the medium gesture, the representation is highly abstract.

But gestures, like other kinds of images, do not copy previously (mentally or perceptually) abstracted features – they are tools that enable and accomplish the abstractions: There is a fallacy in the idea that reality contains such features as mountains and that, looking at one mountain after another, we slowly learn to generalize and to form the abstract idea of mountaneity. ... Neither in thought nor in perception do we learn to generalize. We learn to particularize, to articulate, to make distinction where before there was only an undifferentiated mass. (Gombrich 1961: 100–1)

Depictive gestures organize the world in their own fashion, which is fundamentally different from the way in which words organize the world. Thus, instead of taking reality as a given and moving from it to its representation in movements of the hands, we will take the actions of the hands – the making of the gestures – as our starting point and seek to explicate by which methods gestures depict, analyze, and evoke the world for purposes of communication. We want to find out “what is actually involved in the process of image making and image reading” (Gombrich 1961: 25). Depiction is always a matter of conventions, and this is as true for gestures as it is for paintings and drawings. Whether I recognize a cluster of paint particles or a sequence of motions of the limbs as a likeness of an object or not is a matter of the methods by which these images have been made, and whether these methods are part of the recipients’ cultural repertoire. But in the “natural attitude of everyday life” (Schütz 1967 [1932]), we treat the methods as if they were obvious, natural – found, not invented. For example, lines to us appear as features of visual reality itself: we have become so obedient to the artist’s suggestions that we respond with perfect ease to the notation in which black lines indicate both the distinction between ground and figure and the gradations of shading ... (Gombrich 1961: 42)

Chapter 6. Depiction

Gestural depiction abounds with similar conventions: straight movements with an extended index finger are often meant to be seen as lines, and this is a similar cultural convention as the line as representation of volume or light and shadow is: Just as objects are classified by means of .. various verbal labels, so also are objects classified under various pictorial labels. (Goodman 1976 [1968]: 30–31)

The gestural categorization of an object involves a distinctly manual analysis. Its vocabulary – a vocabulary of methods, not forms – is the subject of the present chapter. A small number of analyses and typologies of modes of gestural depiction have been published, notably by Kendon (2004), Müller (1998a, b), and Sowa (2006). (For a summary see Streeck 2008b; for a detailed overview of typologies of gestural representation methods Kendon 2004: Ch. 9.) A well developed framework for the analysis of techniques of representation in the gestural modality is the account of ‘iconic devices’ developed by Mandel (1977) for American Sign Language and adapted by Kendon (1980b) in his analysis of the semiotic properties found in a sign language observed in an Enga village in the Papua New Guinea highlands. Mandel and Kendon begin with an important distinction between the referent to which a sign refers and the base from which it is taken (Cohen, Namir & Schlesinger 1977). The base is the object or action from which the sign has been abstracted, for example the action of holding a container and bringing it to the mouth; the referent is the object or action or concept that the sign denotes in the current situation, for example, “bottle” or “water” or “something to drink”. There are multiple bases for every possible referent, and there are in turn multiple ways for depicting the base (so as to refer to a referent). Three questions arise: 1. What is the relationship between the action of the sign and the base? 2. What features of the base are selected for a sign? And 3. what is the relationship between the base and the referent? What matters for the present chapter is the answer to question 1, what Kendon calls the types of realization of a sign. Kendon distinguishes between presentation of object, pointing, and characterizing. The first two types correspond to some of the practices described in Chapter 4 (presentation of object being instantiated in the ‘head-light door’ episode, pointing being one variety of gestures of orientation) – “Characterizing” is what I call depicting. Mandel and Kendon distinguish these methods of characterizing: in an enactment some pattern of action is realized in order to characterize an action or an object involved in it; in body modeling the body or, more commonly, a body part is used as a representation of an object; whereas virtual depiction is defined as making “a virtual drawing of the base in space”, which can be done by sketching, stamping, and measuring (what Sowa aptly calls extent gestures; Sowa 2006). Even though the typology was originally developed for the analysis of signed languages



 Gesturecraft

and although different ecological conditions obtain for the understanding of gestures that are coupled with spoken languages, compared to sign, it is nevertheless a good framework to anchor our own investigations, which have yielded a largely overlapping, but somewhat differently motivated set of depictive practices. In the remainder of this chapter I will describe a slightly different subset of depiction methods than I have in Streeck (2008b), and some of them are presented here with a different slant. This is due to the somewhat different nature of the episodes studied: while those explored in Streeck (2008b) were single narrative turns, the research reported in this chapter centers on a few longer episodes in which the subjects build up imagery of a setting over extended sequences. I will summarize the entire set of practices I have found, including those not examined in this chapter, at the end. Note that hardly any of the depiction practices we will examine would work outside the contexts provided by spoken language within which they are firmly embedded. Talk – both its topical coherence and the local syntactic environment that it provides – narrowly constrains what recipients expect to see in a depictive gesture, and spoken language usually provides the greater part of imagery so that little is left to do for the hands. To fully explicate how gestures represent reality, we would need to explicate how it does so in conjunction with speech. My analysis is thus partial and limited in that it seeks to highlight the contributions that the hands as hands make, which may sometimes create the unintended impression as if there were no contributions to the understanding of the gesture made by speech.

Articulating gesture space The following examples are from three conversations: a narrative about the play, Penthesilea, in a modern adaptation by Robert Wilson; a series of spontaneous stories about their car-accidents by two Japanese speakers; and an extended description by an architect of a potential building-site and the kind of structure he imagines building there. In all three episodes, representations of settings are built over time, through successions of gestures. As she begins her narration of Penthesilea, the speaker initiates her first gesture by raising her right, supine hand, fingers adducted and configured as a horizontal plane, and holding it in place, first in a lower, than in a higher position (Fig. 6.1.1). The gesture is conspicuous for its stability and immutability over a sequence of utterance-parts. 6:1 Penthesilea Christine 1 Nja ‘ch wollt Dir ja erzähln von der Aufführung. Well, I wanted to tell you about the play.

Chapter 6. Depiction

Fig. 6.1

2 Also am Anfang ham se ganz • oben gestanden? Okay, in the beginning, they were standing all the way up [there]?

Christine’s gesture creates an enduring structure in gesture space: the flat, extended hand organizes it into horizontal levels and provides a structured image of the narrative setting; the reference of the depiction is provided by the word oben, “up there”. Once a minimally structured mock-up of the narrative setting – apparently a stage, surrounded by bleachers – is erected, actors and sub-events can then be located within it, and its structure further elaborated: Christine adds a lower level by repeating her initial gesture, a few inches below the previous one, as she introduces a trombone-player into the discourse. 6.2 Penthesilea 1 Christine dann blond- also total blonde Engelshaare und so und is then blonde well totally blonde angel-hair and stuff, and she’s 2

ziemlich also ( – – – ) macht Power die Frau, ne. pretty well she has power this woman, right. [ Hmm. Hmm

3 Susi “level” gesture | 4 Christine Und ( – ) äh dann hat der Posaunist erst gespielt. And uh then, the trombone player played at first. (–––––)

Christine then locates the main actress on the stage. She does this by an act of placing: bunching the fingers of a prone hand held in a posture that suggests she is holding an object. Then she lowers and opens the hand as if to deposit and release an object on a surface. The existence of an entity in the universe of discourse is thereby asserted. This is how small world-models are made. Through a few simple actions, the segment of space within which the speaker’s hands move is set up as a model for the setting within which the narrated event took place. Liddell (1995, 1998), following Fauconnier (1997), has described ways in which the “real space” of the interaction



 Gesturecraft

is used in American Sign Language as a model for the narrative setting. He conceives of these processes (for example, the depiction of reciprocal actions between protagonists through reciprocal actions of the two hands) as a “blending of mental spaces”. “Blending of spaces” means that the real space of the interaction, as it is structured by the actions of the speaker’s hands, is superimposed upon the imagined space of the narrative and thereby gives it structure. Spatial activity within the space of the interaction provides a configuration in terms of which the interlocutor can structure his or her imagination of the narrative scene. In Segments 6.1 and 6.2, spatial structuring is primarily created by capitalizing on the physical features of the hand: the fact that it can be configured as a plane makes it a good resource to show a level or plane or wall. On other occasions, it is not their physical properties but the actions of the hands that set up the stage or depict the setting. In the following, a vertical structure is erected, referring to a (non-existent) fence. The accident, Satomi seems to suggest, would not have happened if Japanese country-roads had fences or guard-rails. Satomi raises and lowers her hand twice so that its flat shape, enlarged by the vertical motion, effectively creates a flat, vertical appearance, such as could represent a wall or a fence (Fig. 6.3). The particle hora, “look”, draws attention to this gesture. 6.3 Car Wrecks 1 Satomi

2

ne (.) nde (.) h ano: (1.0) ·chotto ochita no ne? Hora inaka no well a bit plunged down look contryside’s Then, well, I plunged down a bit. Look, often the ____________ | michi tte iu nowa road so called so-called country-roads

3 Tomoio en en en en hm hm hm hm

____________________ 4 Satomi

Fig. 6.3

| yoko ni nani mo • nai ja nai saku toka ne= on the side nothing no isn’t it fence such as have nothing on the side, like a fence.

Chapter 6. Depiction

Here, too, a structure is assembled in gesture space and serves as a model for an imagined setting. Had a boundary as is shown by the gesture existed in the space of the accident, the car would not have gone off the road. That much can be seen. The hypothetical scenario that the hand evokes offers a tangible model of an alternative possible world: we can see the structure and imagine how it would have arrested the motion of a car. The hand, enlarged by motion, becomes a structure in a model-world. In the following segment, in contrast, a slope by the roadside is shown by virtual tactile action, the evocation of terrain by a gliding of the hand along an imaginary surface. In this case, the configuration of the hand does not imitate physical features of the referent; rather, these features are made visible by an imaginary action of “feeling”: the hand-shape results from the hand’s adaptation to the features of the terrain (Fig. 6.4). This exemplifies what I call scaping. 6.4 Car Wrecks

Fig. 6.4

_________________ | | 1 Satomi de soko ga ne:::· • yappari ‘hh inaka no michi de ‘hh de:: nani and there after all countryside road what And the place was, you know, a country road. And 2

sanju ssenchi ka yonju ssenchi gurai nara iin dakedo mo ne: (3.0) thirty cm or forty centimeters about if okay though if it had been thirty or forty centimeters, that would have been all right, but

3

nan me:toru san me:toru gurai atta no ne how many meters three meters about was how many meters was it? It was about three meters.

As her words provide background information, reminding Tomoio that she had been driving along a Japanese country road, Satomi raises her open hand, fingers extended and adducted, and moves it down to the right at a 45◦ angle; the hand “moulds” (Müller 1998b) or “sculpts” the terrain (Müller 1998a). Like Christine in her depiction of the stage, Satomi then populates the setting by placing imaginary objects within it. In her case, the focal object is a tree-stump –



 Gesturecraft

not an object which, in real life, could be freely placed and moved around. The tree-stump was located in the path of the car down the slope; this explains the severity of the accident. 6.5 Car Wrecks

Fig. 6.5

_______________ | | 1 Satomi nde h dakedo choudo • ki no kirikabu ga atte soko ni koo (1.0) and but just tree ‘s stump exist there to like this But there was a tree’s stump just right there, and like this, 2

tsuitotsu shita wake ne? crash ed I hit it you know?

Satomi shows the tree-stump by an act of virtual placing. Her right hand, in a grasping posture, is lowered as if she is setting down an object that can be held in the hand (Fig. 6.5). Notice that this action – the placing – has of course no counterpart in the narrated event; it is only the object so placed that does. The schematic act of placing is simply a narrative device; what exists is the object. In the prior representation of a fence, the structure of the flat hand enabled the speaker to erect a two-dimensional structure. Here the hand operates in a different fashion: not its visual features, but its ordinary, non-symbolic engagements – its ability to put, take, and handle – are enlisted. The hand places some generic object, specified by the moment’s talk, before the interlocutor, who is invited to include it in her visual imagination of the topical scene. The hand-movement can be understood, not in terms of its inherent visual features, but its results: now there is an object out there, which remains after the hand is retracted. Thus, with a few strokes of the hand, gesture space is articulated so that a rough blueprint emerges which the interlocutor can project onto the imagined setting of the narrated event. Several observations about the instances of gestural depiction that I have so far described are in order. The first gesture that I have identified – by which the speaker in Penthesilea depicted the layered set-up of the stage –, pointed up a fundamental ambiguity, namely that we can alternatively perceive the gesturing hand for its shape, its motion, its position in space, or perhaps for the action that it is performing. In the case of Christine’s space-structuring

Chapter 6. Depiction 

gesture, the features that mattered were the shape of the hand – both its native shape and its configuration – as well as its position in gesture space. Its motion was irrelevant to the depiction. However, hand-shape and position did not depict in isolation, but in the context of a series of gestures: the notion of level or layer is meaningful only if there are several, and Christine’s first layering gesture gains its significance in part from the second, differently positioned gesture that follows it. Often, it is not the single gestural act that depicts, but the series of such acts, which cumulatively build up an image.1 Thirdly, meaning – in gesture as much as in language-use – involves, in Langacker’s words (Langacker 1986) the imposing of a profile onto a base; without the existence and activation of a base-predication (e.g., the stage in the Penthesilea narrative) – a base that is often understood rather than expressed – a figure (e.g. the rows of bleachers) could not be recognized; the gesture would not achieve a figuration. In other words, meaning is not inherent in a form, but results from the articulation of ground and figure; both constitute one another. In the following episode, an architect depicts a building-site that he has just visited and for which he is just now beginning to design a house. He reports to a researcher a design problem that he will have to solve, namely to accommodate the power-lines that cross the site (lines 8, 9). His depiction of the power-lines is the first gesture in this episode. It is a rapid, sagittal motion across the gesture space by a flat open hand (Fig. 6.6.1), a “modeling” (Kendon 2004) which, by creating structure within gesture space, defines a basic spatial parameter of the site. By focusing gaze on his moving hand, the architect also positions himself vis-à-vis these structures and thus indicates the view-point from which the actual setting would be seen. 6.6.1 Architect 1 2 3 4 5 6 7 8

C So y’all went to a new space today. MJ (– – – – – ) We went to a- well. It’s gonna be ( . ) our ( . ) latest ( . ) job hopefully. C This is the Newman house is that the one? MJ ( – – )Ah- uh- Nieman. Nieman. Rick Nieman. C Nieman? MJ Yeah. So ( . ) we had to go check o:n

. See Enfield (2004).

 Gesturecraft

Fig. 6.6.1

9 10

how the • power-cabels come across becaus’ (–––––) thi- the initial idea I had for the- ( – – )

Fig. 6.6.2

11 12 C

the house is to actually do it in three terrace • levels.= = Mh hm.

The architect then uses a flat hand to show the terraces that he envisions: he holds the hand supine, then vertically, then supine, each time moving the hand a bit to the side and down, cutting steps into virtual terrain (Fig. 6.6.2, line 11). His gestures are unlike those of a sculptor, because they involve no interaction with resistant matter, but they erect an ephemeral structure nevertheless. Mandel (1977) might call this mode “sketching”, pointing out that it requires us to “see” the virtual trace that the motion leaves in the air. The architect repeats the depiction (13–16), but with a subtle, decisive difference. This time, there are no abrupt changes in movement direction (which mark the alternations between slope and terrace in the first series), but these angles are now shown with slightly rounder motions, the gesture overall is “softer”, and it evokes the existing terraces in the terrain, which are being “traced” by the motions of hands that “feel their way” along the surfaces (Fig. 6.6.3). The differences between the two gestural depictions are minimal and nearly imperceptible, but without doubt, these are actions of two different kinds: one, an act of erecting a spatial structure, the other one of discovering and thereby evoking an existing scape; one brings reality into being, the other discovers and discloses it. In the

Chapter 6. Depiction 

schematic versions of these genres of action, as performed in depictive gestures, the consequences are the same: a reality is made apparent. But in this context, where the architect is also explaining his conception of architecture, the contrasting features of the two acts also seem to instantiate the contrast of culture – the geometrical features of built form – and nature – the round, organic forms of a hilly terrain. Bringing these two kinds of structure into harmony by designing forms that adapt to the terrain is the foremost goal of this architect, who regards his task as being optimally responsive to the features of the site. He expresses as much when he formulates his appreciation of the site: “The side totally tells you how to put everything onto it. It’s incredible” (lines 18–21). What exactly he means by that is shown by the juxtaposition of the gesture variants in a depictive series. 6.6.2 13 MJ Because the side almost has this three ( . ) ter-

Fig. 6.6.3

14 15 16 17 18 19 20 21

It has two very disti:nct ( – • – ) terrace levels and then we’ll probably try’n pull one out of the very to:p. (––––) Absolutely fanta:stic. (– – – – – – – – – – – ) You- it- mean the the the side totally tells you how to put (–––) everythi:ng ( – – ) you know onto it You g- you wa:lk up it’s ( – – – ) incredible.

An interesting feature of this episode is that it blends depiction with imagination – or the depiction of reality with the depiction of imagination. In effect, the architect lets us in on his creative process, which is presently being mediated by his hands: the contrasting movement “tones” – angular and sinuous – are intermediate creative structurings. In the next segment, the architect describes the site as a visitor would experience it (“as you come down and begin to experience the site”, line 23).

 Gesturecraft 6.6.3 22 23 24 25 26 27 28 29

Has a natural grotto at the very bottom of the site so:( – – ) that as you as you come do:wn and start to experience the si::te on these different plateaus or these different terraces (––––) and thu- those different terraces ( – – ) are these natural rock outcroppings that just sort of ( – – – – – – – – ) tell you how thethe house should be formed around these things around this thing. And on this bottom one ( – – )

Fig. 6.6.4

30

there’s this natural • grotto ( – – – – – – – ) right down here.

Fig. 6.6.5

And it’s this very tiny, • intimateHmm. [ 33 MJ piece, with the with the water runoff coming down and 34 dripping. 35 So there’s this tiny little stelaphtite so [ 36 C Wo::w. 37 MJ all this beautiful ( – – – – ) mpt very delicate foilage that’s down 38 there31 32 C

In this segment, the architect makes several depictive gestures. He carves terraces (“outcroppings”) into the terrain by dragging his hand, with thumb and indexfinger in delimiting or “bounding” opposition, indicating width, in horizontal

Chapter 6. Depiction

curves. The width of the terraces is projected by the distance between the thumb and the fingers of the prehensile hand, while the sinuous, horizontal motions of the hand show how they cling to the hillside (lines 24–28). The last of these sinuous motions, which is repeated, then turns into a depiction of a grotto (Fig. 6.6.4, at line 30). Now the architect shifts the vantage point of the depiction. Before, the location of his gestures corresponded to an overall mapping-relationship between gesture space and the space or site of the narrated events. Accordingly, motions depicting the grotto at the bottom of the slope are made at the bottom of the gesture space. But now the architect brings his hands up to eye level and repeats the depiction of the oval-shaped grotto there (Fig. 6.6.5). This time, the proximity of the gesture to the eyes evokes the impression of him being surrounded by the grotto, and this corresponds to the location’s “intimate” quality that he emphasizes in his talk (line 31). This, then, is a depiction from the moving subject’s changing vantage point. These gestural practices are part of the very fabric of the architect’s creative process: they are among the vehicles of his spatial reasoning and imagination. Gestures shape the terrain, put structures in place, mold them. They anchor the design process by making it a physical, repetitive, kinesthetic experience; enlivened structure is interiorized through movements of the hands. By transposing the terrain or site into gesture space, the architect can apply the haptic skills of sculpting to the designing of a landscape. His visual imagination becomes fused with his “haptic system” (Gibson 1962). Sensorimotor imagery appears to be a central component of the design process. McCullough explains why this is so: Whereas the eyes stay fixed on the outer surface of things, hands have a way of getting inside, and so they contribute more to our belief in the reality of the world. Hands also discover. They have a life of their own that leads them into explorations. For example, a sculptor’s feel for a material will suggest actions to try, and places to cut. Learning through the hands shapes creativity itself. (McCullough 1996: 8)

Apart from the specific importance that creating in gesture or designing in gesture has for a professional architect, what is striking about the assorted methods of “model-world building” is how closely they resemble the narrativerepresentational strategies of users and makers of signed languages, of which Taub gives a vivid example at the beginning of her book on iconicity and metaphor in American Sign Language (Taub 2001): The signer is telling you about her kitchen. She sketches the four walls in space, then quickly identifies the appliances and furnishings. As she names each one – refrigerator, sink, cabinets, and so on – she places it within the sketched outline of the kitchen, punctuating each placement with a special head nod. Before long,





Gesturecraft

a virtual map of the room floats in the space between you. Now the signer is describing a conversation she had with her six-year old son. She names her son and points to a spot on her right. Her body shifts to face rightward and her signing angles down toward where a six-year old’s face and body would be, as she reports how she asked her son to get a towel; then her body shifts to face upward to the left as she gives his assent. The relative heights and locations of the signer and her son are clear to your mind’s eye. The woman goes on to describe how her son ran about the house to find the towel. Her index finger is extended upward from her fist, and she traces a complex path through the air with that handshake. The twists and turns of her hand sketch out for you the path her son took around the house. . . All of these features of the conversation are perfectly normal, conventional features of ASL. (Taub 2001: 1–2)

In all cases, a toy-world is built in gesture space, out of a few strokes, but stable enough to anchor the interaction participants’ visual and motoric imagination of the narrated places and events.

Motion: Real and fictive Gesture is motion and is therefore apt at depicting motion. But the relation between signifier and signified in motion gestures can be puzzling. For example, it is possible that a gesture shows an object in motion; in the following case, the hands refer to and perhaps classify the object that moves. A flat hand, held horizontally (Fig. 6.7), or a hand in a grip-shape, facing down, can serve as a token for a vehicle. But it is also possible that the moving hand serves as an abstraction device by which the manner or path of a movement can be shown. A greater degree of abstraction is involved. 6.7 Car Wrecks 1 Satomi nikaime wa bo:tto shitete ne (.) nde koo massugu hashitteta wakene was being lost like this straight was driving I was absent-minded and I was driving straight like this.

Fig. 6.7

Chapter 6. Depiction

2

_______ | | de magaranakucah nannakatta wake • yo (.) migi ni ne? and need to make a turn needed right to And I needed to make a turn, to the right, you know?

Is the car incorporated in the gesture? Gestures cannot be decomposed into constituent parts, but are produced and perceived as integral, structured wholes. McNeill (1992: 19–21) has called this feature of gesture “global-synthetic”: [The] gesture-symbol is global in that the whole is not composed out of separately meaningful parts. Rather, the parts gain meaning because of the meaning of the whole. ... The gesture also is synthetic. It combines different meaning elements ... into a single depiction.

However, whether all parts of the gesture are meaningful on any given occasion – or which parts are designed to be seen as components of a synthetic gestalt – is a matter of local context. And, more importantly, the gesture itself may be the vehicle that enables the abstraction in the first place. In other words, the gesture does not simply express an abstraction that “the mind” has independently arrived at, but it is itself the vehicle by which such abstraction operates. As I have written elsewhere (Streeck 2008c: 262) and elaborate in Chapter 7, I recommend that gestures be treated as abstractive actions, that is, as actions that take configurations or schemata that are readily at hand to give form to abstract senses ... Gestures classify in the first place by virtue of the acts that they are.. , it is the specific physical act itself – its particular, if underdetermined, grip, hold, push, etc. – that organizes the target.

This is the case with path gestures: there is no overtly visible path in most motion events, but the gesture of drawing a circle or a snake-line can abstract this feature from them. In the following segment, Satomi’s hands alternate between showing the path of her car and the terrain and obstacles along its path. She relies on gesture to convey the car’s motions and makes no attempt to describe it in words. The deictic adverb ko (“in this fashion”) identifies the role of the gesture in the overall, bi-modal representation: it shows how the car moved. Satomi says “several times like this” while her hand repeatedly moves to the left and right, indicating a snakelike path. 6.8.1 Car Wrecks snake-line ____|______ | | 1 Satomi nde nan kai ka ko: (– – – ) Then several times like this

Then, keeping the orientation and positioning of the hand, Satomi brings it to the right (Fig. 6.8). This move is made more rapidly and with greater effort and



 Gesturecraft

thus gains in granularity: we perceive it as showing not only a path, but also acceleration; it depicts how the car gained 6.8.2 Car Wrecks

Fig. 6.8 ______________ | | 1 Satomi ‘hh de nankai ka koo yatte de tsui ni ochite • shimatta no ne and several times like this tried finally fell couldn’t help And I tried several times this way and finally couldn’t help driving down (the slope).

These depictions show motion that actually occurred in the depicted event. However, just like languages know fictive motion (Talmy 2000), expressions in which motion is attributed to the land rather than the “trajector” (Langacker 1986) crossing it, in expressions such as “The road leads up the hill” or “The cliff steeply falls off to your left”, gestures often depict static features of the world by dynamic motion. The architect alternates in his site-description between using gesture to show real and fictive motion. He goes back and forth between depicting the roads that he envisions and movement along these roads and across the terrain. Visual imagery is thus integrated with kinetic imagery, which seems to be useful for architects who design buildings with a view to the dwellers’ changing, dynamic spatial experiences as they move through and about the site and building structure. 6.9 Architect 1 2 3

MJ There’s a little uh- there’s a little road that’s already been sort of cut into it ( – – – – – – – – – ) thatso the approa:ch?

Fig. 6.9.1

Fig. 6.9.2

Chapter 6. Depiction

4

( – – – • – – – ) as you as you come as you drive • down the road

Fig. 6.9.3

5 6 7 8 9 10 11 12 13

to it and come up • a hi:ll ( – – –) and then it kind of fa:lls off into this little valley ( – – – ) thing that I was talking about and there’s a little side road so you- you could get to it from the top (but) ( – – – ) we’ll come down this road and then we’ll actually loop arou::nd and then come into the site like this.

Fig. 6.9.4

14 15 16 17 18 19 C

So we’ll get an idea from the • to:p, around the side, and back around So we really play with the approach and how we come into it [ Wo::w!

When he first shows the “little road that’s already sort of cut into it” (line 1), the architect’s hand is flat and horizontally positioned, as in his previous acts of “sculpting” a road into the imagined terrain. Later, the architect shifts in his depiction from an objective vantage point to the visitor’s perspective, which is itself predicated upon motion (“approach”, line 3). He first brings his hand up to eye-level as if to align his gaze with the visual perspective of the would-be visitor (Fig. 6.9.1). Then, as he utters “drive down the road” (line 4), using his right hand,



 Gesturecraft

he again sketches a curved path in the air (Figs. 6.9.2–3). But his hand is upright, and this distinguishes the gesture as depiction of a moving vehicle or person from the scaping of the road upon which they travel, which is done by a prone, flat hand. When he utters come up a hill, the architect’s hand moves up and, in the process, is rotated so that the palm faces down when it reaches the hilltop: through this rotation, the reference of the hand also changes, from a representation of movement to a depiction of the terrain, the small plane at the top of the hill (Fig. 6.9.4). The hands are held still for a moment, before the right makes a rapid, sinuous downward movement, showing how it kind of falls off into the little valley. This is fictive motion. Stable features of the terrain are described by motion verbs that would describe what would happen to a person moving through it. The movement of the hand depicts the “falling off ” of the terrain. Throughout this depiction, the architect holds his left hand in the same position, continually marking the hill-top.

Drawing Drawing is a kind of depiction method that I define quite narrowly in analogy to drawing on paper – or better perhaps: in dust or sand. The index-finger is extended, and with its tip touching the surface, its movement leaves a trace, a line. In gesture, of course, the line does not physically exist; and yet we seem to see it. Mandel, who uses the term sketching to refer to a variety of methods that bear resemblance to drawing, emphasizes that, when any of these methods are used, the viewer must see the trace that the moving hand or finger appears to leave. There are different methods of sketching, because the instrument of the action – the hand – can be given different forms. A pointed instrument (pencil or fingertip) leaves a line as its trace. ... A linear implement (piece of chalk held flat, or a straight or bent finger) leaves a surface whose shape depends on the shape of the implement (straight or curved) and on the motion. ... A surface instrument (trowel or ... a whole hand held in various flat or curved surface shapes) can leave a trace shaped like itself, when moved parallel to itself ... or it can leave a solid trace perpendicular to itself, as if one indicated the height of a stack of papers by putting his hand palm down on the table and then raising it to the height of the stack. (Mandel 1977: 67)

When we think about these gestural representations and how they work, what it means that we “see” a line that “is not really there”, we must take into account that seeing is not a bodily process of perception, but an achievement (Coulter & Parsons 1990). We say: “Now I see the bird”, reporting a success. We “achieve a seeing” when we succeed in recognizing the target that we are looking for. Seeing involves the competent recognition of cultural “see-ables”. In this sense, we do not see the speaker move an extended index finger around, but we do see her drawing

Chapter 6. Depiction

lines. We report a culturally competent recognition of a culturally competent act when we say “I see her draw a line”, even if the line is “only in the imagination”. Thus, there is nothing mysterious about the appearance of a line in the air.. Drawing is undoubtedly a common method, but in the data that form the basis for this chapter, there is only one case, and it is atypical in that what is drawn is not the outline of an object, but a path – the path that Tomoio’s car took once it began to spin around. Tomoio has raised her hand and holds it with the index extended, and then draws three circles horizontally in the air (Figs. 6.10.1–2). She draws another two when she repeats that the car spun around (Fig. 6.10.3). 6.10 Car Wrecks

Fig. 6.10.1

_______ | 1 Tomoio supin • shite sa, spin did Spinning 2 Satomi

n:: Hmm.

Fig. 6.10.2 _______________________

| 3 Tomoio ·nikai ka san kai kurai • supin shite (.) hantai shasen denakatta kara twice or three times about spin did opposite line didn’t exceed because it spun two or three times, we were lucky because we were’t in



 Gesturecraft

4

5 Satomi

yokatta kedo sa (.) nichiyo:bi datta shi ne was okay though Sunday was also the opposite lane and it was on a Sunday. [ nn?n:::·, Hmm?

6 Tomoio kuruma suiteta shi h car wasn’t crowded also Traffic was light.

Fig. 6.10.3

5

________ | | supin • shite spin did It spun around.

The gesture shows a circular motion path and therefore overlaps with the genre of path gestures that I have introduced before. Not all path gestures incorporate techniques of drawing, which I define as the making of lines, but drawing is an effective technique to abstract paths from complex motion events.

Handling The most common mode of depicting objects, in this corpus, is the one I call handling. This practice involves gestures that are schematic versions of practical actions in the worlds, that are not used to depict action, however, but rather the object that is acted upon (or the tool that is used in the action). I thus distinguish “handling” from what I call “acting” or “mimesis”, which are gesture acts that are made to show other physical acts.2 In the case of “handling”, an object is characterized by an action that “goes with it” or is “structurally coupled” with it (Maturana & Varela 1992). The link between signifier and signified corresponds to the relationship between a human body and a class of material objects that the . Kendon (2004: 160) calls both modes “enactment” and treats the distinctions as differences resulting from application to different domains.

Chapter 6. Depiction 

body routinely handles, which is a relationship of adaptation and fit, not of imitation, as is characteristic of mimetic enactments, where the relationship between signified and signifier is one of original and copy. Seeing and understanding “handlings” as depictions of objects requires knowledge of the world: recipients must know what actions go with what things. Many of the action schemata that I have observed in this genre of depiction come from one of the most basic and generic domains of manual action, namely the transportation of things: part of the gestural vocabulary of handling is one of picking up and taking, holding, handing over and putting down; it is a language of simple motion signifiers and grasping postures which classify the object by some intrinsic criteria. In the least specific type of case, an object is treated as being able to be put, held, moved, and so on, solely by virtue of being an object at all. In other cases, the relationship is specific and distinct, coupling a complex gestural action pattern with an object endowed with very specific affordances. In the following case, the relationship between the form of the gesture – that is, the motor schema – and the object evoked is remote and unspecific: Satomi configures her hand as if she were holding a round object of a certain size and then lowers it as if placing it on the ground (Fig. 6.11). The gesture accompanies an utterance in which she asserts the existence of a tree-stump in the path of the car. The right hand forms a prehensile posture, and hand and arm move in a schematic act of placing something on a surface, of setting it down. Here, the form of the gestural action has no pictorial purpose; what matters is that something gets placed, put into existence, made salient relative to the narrated scene. What the gesture wants us to understand is that an object existed in a certain location. Within a representational world we can show or assert the presence of any object by putting it in a place. Thus, to understand the gesture, we must “see” the act of placing something, but at the same time, see through it: for nothing in the story corresponds to this act. We must disattend it as mere narrative method. 6.11 Car Wrecks

Fig. 6.11 _______________ | | 1 Satomi nde h dakedo • choudo ki no kirikabu ga atte soko ni koo (1.0) and but just tree ‘s stump exist there to like this But there was a tree’s stump just right there, and like this,

 Gesturecraft

2

tsuitotsu shita wake ne? crash ed

In the following example, in contrast, the coupling between hand-shape/actionschema and evoked object is very specific Satomi explains here that she was injured because she was not wearing a safety-belt. She evokes the belt by bringing her right hand to her left shoulder, configured in thumb-opposition, about three inches apart (Fig. 6.12.1), and then drags it diagonally down from her shoulder to her lap (Fig. 6.12.2). Viewers who have been shown this tape rarely missed the reference to seat-belts. There appear to be few, if any, other objects in their worlds that are handled in this fashion. 6.12 Car Wrecks

Fig. 6.12.1 Fig. 6.12.2 ____________________ | | 1 Satomi nde: (1.0) pata to yo: tte iu:nowa • ne kono • beruto o shime nakatta and suddenly in other words this seatbelt fasten didn’t And suddenly, I mean, I wasn’t wearing the seatbelt.

To quote Taub, here the “shape of the articulators’ path represents the shape of the referent” (Taub 2001: 77). Taub explains that ASL uses a number of conventional hand shapes as ‘tracers’. . . Index fingers trace out lines, flat hands with fingers together . . . trace out planes, curved hands .. trace out curved surfaces, and thumb-and-forefinger circles . . . trace out small cylinders. Just as with shape-for-shape iconicity, only certain parts of the articulators are understood as counting. (Taub 2003: 77)

When Tomoio responds with a car-accident story of her own, she too evokes the event with a gestural depiction in the mode of handling. She reports that, returning home from dinner with a group of friends, she was one of four passengers in her boyfriend’s car, and another friend drove in a car next to them when it started to rain. The driver of that car did not know what to do in such a situation and accelerated, and because his tires were worn out, they lost traction and started to twist very fast. The car then spun around and crashed into a gap in the guardrail by the road, which left a dent in the door. Tomoio was “sandwiched”, as she says, and

Chapter 6. Depiction

ended up in the hospital. When she describes how the tires lost traction and started to rotate aimlessly, Tomoio raises her dominant hand; the fingers are spread, but slightly bent and with the thumb oppsed, suggesting a grasping posture for a round object; roundness is independently conveyed by rotation of the hand at the wrist (Fig. 6.13). 6.13 Car Wrecks 1 Tomoio h de kou hashitteta no (.) ‘hh (shitara) sa ame ga furi hajiete like this was driving then rain rain started And we were driving like this, and it started to rain. ____________ | 2 sono kuruma ga sa: ‘hh akseru funde sa its car gas stepped The driver stepped on the gas and 3 Satomi

nng.= oh

Fig. 6.13

______________________________ | 4 Tomoio =subetta yo ta• iya ga bo:zu datta no slipped tires worn out was slipped. The tires were worn out. [ 5 Satomi hn::::: hm::: 6 Tomoio de karamawari shitta wake. de= and tire spun And the tires spun

The event that the gesture depicts is the rapid rotation (karamari) of the tires that have lost traction on the road. In the real-world event, in other words, it was the tires that behaved in a certain way. In the representing world (Norman 1993), Tomoio shows this behavior of the tires by rotating her hand back and forth at the wrist. In other words, she takes hold of the tire (or wheel) and turns it back and forth. An object and its behavior are evoked by schematic acts of the hand. In



 Gesturecraft

understanding the gesture, we abstract away from these features, however, seeing the wheel “turning by itself ”. The object is also “resized” so that it “fits” the hand: no human hand could take hold of the wheel of a car in the fashion which the gesture insinuates. The gestural action of turning, in other words, evokes the features “roundness” and “ability to rotate”, but not “size”.

Marking and self-marking Some of the gestural practices that I have investigated in Chapter 4 – practices by which the environment at hand or the world within sight are enhanced and structured for perception and comprehension – can also be used to depict an absent world, when the local environment is recruited as a figurative gestalt of which annotated parts are projected onto the world that is being depicted. For example, a wall at hand can serve to show where on another wall pictures hang. In these cases it is not the gestural action alone, but the incorporation of parts of the environment that creates the profile in terms of which the narrative setting is depicted. I call this practice “indexing” or “projective indexing”, because the marks made on one object are meant to be projected onto another, virtual one: the gesture selects parts of the present environment, marks them up, and the resulting figure is articulated with phenomena in the story space. This, too, is a “blending” of spaces – of the real space of the interaction and the imagined space that the narrative is about. But in this case, “real space” is populated by things and bodies that can be used as symbolic resources for representation. An object that is always ready to be marked up is the gesturer’s own body, which can be recruited as a mock-up for other bodies, for example that of a character in a story. I call this practice self-marking. In this segment from Penthesilea, the narrator raises her hand to her head, the fingers slightly flexed and abducted, and brings it to her hair (Fig. 6.14). In her talk, she refers to the character’s “blond angel hair”. 6.14 Penthesilea 1 Christine Nja ‘ch wollt Dir ja erzähln von der Aufführung. Also am Well, I wanted to tell you about the play. Okay, in 2

Anfang ham se ganz oben gestanden? ( – ) Und ( – – ) die Frau is the beginning, they were standing all the way up [there]? And this woman

Chapter 6. Depiction 

Fig. 6.14

3

4

5 Susi

_______________ | | dann blond- also total • blonde Engelshaare und so (like) blond- well, totally blond, angel-like hair and stuff makes fist _____|__________ | | ziemlich also ( – – – ) macht Power die Frau, ne. pretty well she has power this woman, right. [ Hmm. Hmm

The actress’ hair is thus depicted by a schematic action, directed to the speaker’s head, which, if executed fully, would involve the speaker’s hair. This is an indexical reference, once removed, and it is “iconic” in so far as an act of combing is suggested. In another instance, the narrator drags her hand quickly across her lap (Fig. 6.15). This, too, is an indexical reference once removed: an index to the speaker’s body is meant to signify an item on the character’s body. 6.15 Penthesilea 1 Christine Sie stand unten? She stood down below?

Fig. 6.15

 Gesturecraft ______________ | | ∼∼∼∼∼∼∼∼∼

2

Mit som • weißn Rock nur? ∼∼∼∼∼∼∼∼∼

Had this white skirt on, only?

A leotard, worn by a minor character, is similarly evoked by dragging her hand down the chest, in Segment 6.16. 6.16 Penthesilea “tracing” ______|_________ | | 1 Christine Ein son Typ im Gymnastikanzug, das sollte denn wahrscheinlich denOne guy in a leotard, he was probably supposed to 2

ihr- halt ihr- das Objekt ihrer Wünsche darstellen ( – – ) represent her- well her- the object of her desire

These practices also rely on cultural knowledge in so far as they presuppose conventional relationships between body-parts and pieces of clothing.

Mimesis: Depicting action Gesture is bodily action. Using it to depict bodily action thus appears to be an altogether natural and straightforward process. “Mimetic” gesturing – the performance of gestures to depict physical acts or behavior – is indeed a very common method of gestural depiction. But, we have seen that mimetic acts, while constituting schematic versions of “real life” acts, do not necessarily refer to such acts; in the mode I call handling, they are used to depict objects, or instruments, or accessories of action. We are thus dealing with two semiotically related, but otherwise very different modes of gestural representation, and I therefore treat them separately. I reserve the term mimesis for those enactments that are meant to be seen as representations of actions. Gestures which depict human action are very frequent, and their logic is simple: the body enacts a familiar motor-schema and thus produces an abstract, i.e. gestural, version of a real-life act. There are numerous examples of mimesis in the Penthesilea narrative. Because Robert Wilson’s theater is a theater of gesture, often consisting of slow, gestural motions without conventional reference and unsupported by spoken words, Christine frequently resorts to enactments, “doing” Robert Wilson as he prances about the stage, enacting a girls’ choir and the musicians, accompanying her gestures with sound-effects. During mimetic enactments, gaze is frequently part of the show – and not a “pointer” to the gesture, as it typically is during gestural depiction. In the Japanese accident stories one gesture of

Chapter 6. Depiction

this type appears: Satomi re-enacts how she tried to steer the car. Satomi is in the process of describing how she was driving along a straight road when she suddenly had to veer to the right. Because she was absent-minded, she turned the steering wheel too late and too much – ko (“like this”) – and so the car bore too far to the right; she therefore steered in the opposite direction (Fig. 6.17.1), but again too much. So she ended up steering the wheel back and forth in big turns (Fig. 6.17.2) rather than holding it straight, and the car moved to the right and to the left and to the right. Satomi thus describes the accident in several cause-and-effect sequences, shifting back and forth between re-enactments of her holding-on-to and turning the steering-wheel, and gestures depicting the car’s resulting direction through the path-gesture described above. 6.17 Car Wrecks 1

2

Satomi

de magaranakucah nannakatta wake yo (.) migi ni ne? and need to make a turn needed right to And I needed to make a turn, to the right, you know? [ Tomoio hhh hhh

3

Satomi

4

Tomio

de: mochiron magatta wake ne hh and of course turned And of course, I made a turn. [ hh hh

Fig. 6.17.1 _________ | | nde ne douiu wakeka sugoi • ko: kirrisugitan da yo ne h and somehow very like this truned too much And I don’t know why but I turned the steering wheel too much.

5

Satomi

6

Tomoio hm:? hm:?

7

Satomi

de sugoi migi ni itte shiamtte wake and very much right to gone And I went to the right too much.



 Gesturecraft

8

Tomoio =hhhhhh =hhhhhh

Fig. 6.17.2 _______ | | 9 Satomi hh h de kore wa taihan dattende ne (.) sugoi • ko: kondo: this dangerous said very much like this this time So, I thought this is trouble so that this time [ 10 Tomoio hhh hhh

11 Satomi

hantai gawa ni kitta wake ne to sugoi hidari ni kicchatta no ne hhh opposite side to turned and very much left to happened to turn I turned the steering wheel the opposite way and went to the left too much. [ [ 12 Tomoio hh hhhh hh hhhh 13 Satomi

de mata taihen dattsun de kitta wake ne (.) nde souiu baai to ne: h and again dangerous said then turned and like that case And again, because I thought it’s no good I turned the steering wheel again.

Satomi’s hands are in prehensile postures that suggest she is enclosing a round object in them (as in a power-grip). She has them raised, one on the left, the other on the right, with a distance of perhaps 20 inches between them. She raises and lowers her hands alternatingly. There are certainly many ways in which this action could be seen. And yet, when this video is shown to viewers unfamiliar with Japanese, they still tend to see someone steering a steering wheel. Apparently, ‘turning a wheel’ is a ‘good gestalt’ by which to read the figuration of the two hands, and for this figure, there is a ready and unique real-world reference: driving a car. There is only one wheel that, in most people’s lived experience, is handled in this fashion. Ordinary conversational re-enactments bear the seed of performance art, of stagecraft. A speaker can characterize another person by one or two mimetic strokes, delivering just as much as is needed to produce the characterization that the story, at this point, needs. Enactments can also be elaborated into panomime and caricature, and they can be subjected to dramaturgical treatment, especially when people in interaction are being characterized. Like all works of art, enactments enable us to see the world in new, more educated ways. They organize

Chapter 6. Depiction 

experience, imagination, thought. But in contrast to descriptive gestures which organize experience by “blending spaces” (Liddell 2000), that is, by projecting a hand-made gestalt onto a different domain of experience, enactments organize experience by enacting, exaggerating, embellishing, and modulating patterns made from the same stuff from which their denotata are made. The most common way in which replayings approximate art is as a form of conversational entertainment: a bit of reanimated dialogue, complete with gestures and facial contortions, that is not so much geared towards informing but entertaining the audience. When a speaker changes the capacity in which she speaks, from that of “author” of her own words to that of “animator” of someone else’s words (and/or alters her body’s role to that of animator of another body’s behaviors), the capacity in which listeners participate in the interaction changes as well. In contrast to “ordinary” conversational action – Goffman calls it “fresh talk” (Goffman 1981) – replayings do not seek “the shout of responsive action ... but murmurings – the clucks and tsks and aspirated breaths, the goshes and gollies and wows – which testify that the listener has been stirred, stirred by what is being replayed for him. ... The response we often seek is not an answer to a question or a compliance with a request but an appreciation of a show put on” (Goffman 1974: 541, 546; see also Bergson 1899). Skilled story-tellers take care to subject their enactments and re-enactments of social life to dramaturgical control: skillfully timed sequencing of reanimated words and reanimated actions and single-stroke characterizations of characters and their interactions will keep the audience engrossed. For a few lines, the audience can sustain the brief illusion of being witness to the actual event. Brief, simple and conventional re-enactments or quasi-re-enactments, such as those by which one’s surprise or astonishment in a narrated interaction is re-displayed, are often anchored in the talk by quotative verbs, e.g. to be like or to go (Golato 2000; Streeck 2002a). Re-enactments are used during the accident narratives to convey the experience of the crash as it was subjectively being experienced. Satomi re-enacts how her forehead was hit by the rear-view mirror, and her gesture shows the mirror slowly moving towards her as she is staring at it. She slowly brings her left hand, the palm facing up, before her face, freezes the motion, and intently looks at her hand (Fig. 6.18.1). Her entire upper body thus becomes involved in an enactment of herself in the car, staring at the rearview-mirror that is coming her way. Then she lightly hits her forehead with the bottom of her hand, enacting the impact of the mirror (Fig. 6.18.2). Viewers witness a gestural re-enactment “in the first-person” and are enabled to vicariously experience the freezing of time that is a characteristic experience during many accidents and that is evoked here by the freezing of the gestural motion.

 Gesturecraft

6.18 Car Wrecks

Fig. 6.18.1 __________ _________ | || | 1 Satomi: ‘hh are • de ko:: (.)• butsukatte that with like this hit Like this, it hit me.

Fig. 6.18.2

2 Tomoio ara yada bad Oh, how awful.

The heterogeneity of representational practices I have described seven different kinds of depiction practices in this chapter, some of which are themselves diverse sets (e.g. practices of shaping and sculpting). What is most striking about these practices is their heterogeneity: they constitute anything other than an easily subdivided set. I have tried to act on Gombrich’s dictum that “making comes before matching”: representation by gesture must be approached from the signifier’s side – as constructions or fabrications – , not as abstractions from given perceptual realities. Gestures make abstractions that would not be possible without them: it is the gesture (not the mind) that effectively abstracts the path from the motion event. The practices that I have identified are the following: First, there are practices that create spatial structure by relying on the hand’s spatial features and on possibilities for enhancing them. Then there are indexical methods, including marking, self-marking, and drawing. In each of these, some contact between the fingertip and a surface is involved, whose shape varies with the hand-configuration. Secondly, there are depiction practices that center around grasp as the mode of connection between hand and denotatum. This set includes all the varieties of handling. There are also the narrowly mimetic practices in which the hand – or the entire body – simply enacts what some body did or could do on another occasion; these practices are predicated on an original-copy relationship between signfied and signifier.

Chapter 6. Depiction 

Upon closer analysis, we recognize in these manual practices of depiction reflections of some of the most basic and distinct ways in which human hands operate, and of the roles that the hands play in the physical world. On the one hand are gestures that are predicated on the hands’ haptic engagements, engagements mediated by grasp and involving things, mobile physical objects. These practices come from our hands’ roles in our use and transportation of objects. Here, the hand operates in its default role, and the depiction method derived from it is extremely frequent. On the other hand are methods that are predicated on tactile engagement, practices involving surface-contact. The drawing methods correspond to this mode. Most of the exploratory actions described in Chapter 3 are also tactile, rather than haptic. Tactile in a broader sense are acts of molding and scaping, but these should more likely be considered a mode of their own, defined by the resistances and malleabilities of the world’s substances, like clay or sand; we usually operate with flat or cupped hands when we mold them into shapes. These real-world practices, too, have provided us with forms for use in gesture. Of particular interest are gestures of making, because these incorporate the most distinctly human use of the hands (Streeck 2008b). Making routinely involves holding, and thus grasp. It can take the form of putting together parts or of holding and molding objects into shape; in that case, it is like sculpting (Müller 1998a). In each case, artefacts are made. Gestural depiction in this mode means: making representational artefacts. Gestural artefacts, of course, are only virtual, just like the lines drawn in gestural drawings. But the logic of making representational artefacts in the medium of gesture is the same as making them in the medium of wood. “Model-world making”, finally, comprises a host of practices, but as a whole, the logic of the process comes closest to that of fictional play; the semiotic foundation is similar, in that a small piece of space is turned into a scenery, populated by agents and props. We have thus moved a long way away from our initial question, how it is possible that motions of the hands produce pictures of the world. We have gradually subverted the terms of this question which, as we can now see, has unconvincingly favored the visual mode. Because gestures are visual phenomena for interlocutors and are often looked at and seen by the people making them, it is often falsely assumed that gesture is a medium that transforms visual experience into visual representations. Instead, gesture, as a medium of human understanding, incorporates haptic epistemology, more than others: it presents the world not as visible, but as handle-able. It is shaped by the body’s practical acquaintance with a tangible environment which it has forever explored, lived in, and modified. But the beholder, the recipient of conversational gestures, also draws upon this undisclosed background of haptic understandings; otherwise, he or she would not be able to recognize the action-patterns that the gestures instantiate nor the equipment and objects that go with them.

 Gesturecraft

We have moved away from a visual account of gestural representation to an understanding that incorporates our physical actions and bodily “being-in-theworld” (Heidegger 1979 (1926)): gestures do not usually represent what we see, but what we know about something. Making gestures does not always mean making something so that the other can see and understand with their eyes. It does not in the first place mean making pictures. It often simply means: doing what the hands might do under certain circumstances, confronting certain objects or tools, but doing so schematically, virtually, “in the air”. What do our hands usually do with a bucket? They lift it, hold it, put it down. By performing such basic actions schematically we evoke the world that we know in common.

chapter 

Thinking by hand

Human conversation abounds with “imagistic gestures” (Beattie 2004). Not all of these are made to depict something, however. Many simply “construe” or conceptualize a thematic object, and not every conceptualization is a depiction. This distinction is a subtle and difficult one, and not always does a gesture event neatly fall into one category or the other. An example of a non-depictive use of an imagistic gesture, analyzed below, is a tracing gesture that visualizes a stem, as a speaker says Selbständigkeit, “autonomy”, literally the ability to stand on one’s own. The gesture appears to construe autonomy as comparable to the rootedness and stability of a tree. But it does not depict autonomy (nor a tree). How we can understand conceptualization (or construal) by gesture – what I call ceiving (see below) – and, more generally, thinking in the medium of gesture, will be developed throughout this chapter. The chapter is anchored by several sequences from a psychotherapeutic conversation, in which a patient conceptualizes aspects of his emotional life in part by motions of the hands. After examining some of these examples, we will make sense of them in light of the conception of embodied meaning that cognitive linguists have proposed. Usually no distinction is made between gestural depiction and gesture as conceptual action. Both modes of gesticulating are commonly subsumed under the term “iconic gesture” (Ekman & Friesen 1969), “representational gesture” (Holler & Beattie 2003), or “imagistic gesture” (Beattie 2004). The distinction I make between the two modes, depiction and ceiving (or ception), is grounded in the observation of a fundamental difference in the behavioral and attentional orientation that speakers show during the two modes. During depiction, as we have seen, gesturers orient to their own hands. Ceiving, in contrast, is usually a background activity, taking place without the speaker or interlocutor attending to it. Rather than deliberately using their hands to give their interlocutors a visual account of some reality, speakers appear to unwittingly rely on their bodies to find an image schema (Johnson 1987) to construe a phenomenon, a bit of content, an experience. When they “think with their hands”, speaker rely on their bodies to provide conceptual structure. The term ceiving (and ception, cept) that I have coined and occasionally use to label this mode of gesturing (see Streeck 2005) refers to conceiving as a manual activity and highlights the fact that many of our con-cepts provide



Gesturecraft

manipulation-based imagery for abstract content. Verbal concepts often express haptic understandings that are so fundamental to human life.

Gestures of emotion The differences and overlap between the two modes of gesturing, depiction and conceptualization, can be gleaned from the following segments from a psychotherapeutic conversation between a therapist and a patient, a man in his early forties, suffering from bulimia and panic attacks. The segments1 show changing orientations by the speaker to his gestures. In the talk leading up to the first segment the patient has been remembering his childhood and youth and how the adults in his life took advantage of him and oppressed him. All that he really wanted, he says, was to be accepted and to be given some security and warmth. He uses the term Geborgenheit to denote the desired state. The word literally denotes a state of being rescued, but in its daily usage means the comfort and protection that children feel among loving adults. In preparation for uttering the term, the speaker cups his hands, keeping them a few inches apart and moving them as if they were encircling a spherical object (Fig. 7.1): the gesture depicts a womb-like container and thus gives sensory reality to the state of Geborgenheit, which would be the state one experiences inside such a container. 7.1 Angst 1

2

und ich war:, ja und ich wollte einfach nur anjenommn and I was well I wanted to simply be accepted

Fig. 7.1 ___________________________........ | werden- (-).hh ich wollte einfach nur n • bisschen I wanted simply just a little bit of

. See also Egbert & Bergmann (2004).

Chapter 7. Thinking by hand _________________________________________ 3 4

| Geborgenheit und (bei) diesen ganzen Ängsten- Ängsten security and comfort, with all these anxieties- anxieties die ich (jetz) so hat(te) nich that I (now) had, right

Throughout this utterance, the patient keeps his gaze focused on the therapist’s face. His gesture reveals the spatial and bodily schema in terms of which he understands ‘Geborgenheit’. The patient then describes how his illness first broke out after he got married, when his first son was born and he realized that he now had to be independent. The German word for “independent” is selbständig, which literally denotes the ability to stand on one’s own. The patient accompanies his utterance with a tracing gesture, not entirely unlike Satomi’s tree-stump and Tomoio’s guardrail gestures, in which he rapidly lowers his right, cupped hand, which is facing up, as if pulling a long cylindrical object, a pole, a tree (line 3). When he utters bewusst wahrgenommen, “consciously noticed”, he points to his head, the seat of consciousness.

7.2 Angst 1 P: .hh und dann wie ich: wie ich: dann verheiratet war and then when I when I was married 2

3

m mein erster SOHn geborn wurde my first son was born

Fig. 7.2.1 ________________________ | | und ich denn auf einmal selb•stständig sein musste,? and I then suddenly had to be independent?



 Gesturecraft

4 5 6

Fig. 7.2.2 _____________ | | wo ich das dann be•wusst wahrgenommen habe where I consciously realized that

.hhhh tch ja und da bin ich ja krank geworden hhh well and that’s when I fell ill da is das dann ausjebrochen= that’s when this broke out

These gestures communicate – enact and display – images associated with concepts. In McNeill’s words, with these kind of gestures people unwittingly display their inner thoughts and ways of understanding events of the world. These gestures are the person’s memories and thoughts rendered visible. Gestures are like thoughts themselves. They belong . . . to the inside world of memory, thought, and mental images [and exhibit] . . . thinking that is instantaneous, imagistic, and global. (McNeill 1982)

This description applies to the patient. The patient expresses his thoughts and uses concepts such as “standing-on-one’s-own” (independence) or “conscious”, whose inherent imagery is revealed by the gestures; but the patient does not depict spatial realities here. The gestures, rather, support and give kinesthetic form to the imagistic representation of word meanings. At another moment during the therapy, the patient wonders whether the various circumstances that he and the therapist just revisited may together have contributed to his illness. The patient is trying to come up with a word which would denote the joint influence of several forces, a word like confluence, for which there is, however, no German equivalent.

Chapter 7. Thinking by hand

7.3 Angst

Fig. 7.3

1

Ich weiss nich wie weit das zusamm- • also einfliesst in diese Erkrankung I don’t know how far this together- well influences this illness

The speaker moves his hands, which are slightly cupped, towards one another until they become joined – a clear, embodied image of a concept such as confluence. Note how the gesture mediates the speaker’s thought process: it provides a transitory, kinesthetic organization that “holds” and represents a concept (Freedman 1977; Kita 2000), but it does not represent a linguistic concept, because there is no corresponding word in the German lexicon. The gesture corresponds to a possible word: there is a word in German, einfliessen, “flow in”, which is commonly used to refer to the impersonal influence of some force in a process, experience, or decision. There is another word, zusammenfliessen, “flow together”, which is not very common and used mainly literally, as in reference to two rivers that merge into one. The speaker seems to be on track to say zusammenfliessen, recognizes the inadequacy, and then uses a periphrastic, non-idiomatic construction: zusammen einfliessen. It is also not uncommon that the hands provide a different – perhaps a more fitting – image than the speech, so that there is a discrepancy between gestural and verbal image. This is the case in the following segment in which the verbal image chosen is an einem Strang ziehen, “to pull on the same rope”, while the hands provide an image of togetherness and solidarity that is much like the gesture in the first and last example of this series, the cupped hands marking a spherical object or container by modeling it. 7.4 Angst 1

Aber ich will wenigstens n n Partner in dem Sinne haben But I want at least have a partner in the sense that

2

der mir ( - - ) das Gefühl gibt ( - - ) (s)he gives me the feeling



 Gesturecraft

Fig. 7.4

3

dass ma ( - ) dass ma zusammen • an einem Strang ziehen that one that one together pulls on one rope

“Cupped hands” appears to be this patient’s signature gesture during this session, which a psychoanalyst interested in such matters might take as a clue to his core conflict or concern: the dialectic of the desire for closeness and the fear of annihilation. The gesture certainly is not congruent with the verbal gestalt, and the discrepancy is neither corrected nor is there otherwise indication that the patient is cognizant of the symbolic potential of what he is doing with his hands. GoldinMeadow (2003) has found that among children learning mathematical concepts, those whose gestures show what she calls a “mismatch” between verbal and gestural concepts are most likely to learn the new concept: as a “cognitive tool” (Norman 1993), the gesture facilitates the re-organization of thought (Kita 2000; see also Kirsh 1995). It is thus possible that this patient’s gesture foreshadows later insight into the overwhelming importance of Geborgenheit. During these segments as well, the patient’s gaze remains focused on the therapist. His unattended gestural efforts are all devoted to attempts at finding fitting conceptual images for his experiences, desires, or some state of affairs that he is talking about. They clearly do not serve the purpose of depiction. As we have seen, depiction is carried out within a different framework of bodily and cognitive orientation, which is manifest in the patterns of gaze as well as in the ways in which gestures are anchored in talk. This patient organizes his talk and behavior in much the same way as the subjects described in Chapter 5 when he gives a description. One example is the following extract in which describes how he sometimes goes for a drive near the clinic grounds and sits down on a bench near a young oak tree, and gradually calms down. 7.5 Angst 1

Wissen Se das- ich ich erleb das ja wenn ich wenn ich You know that - I I experience this when I when I

2

da oben zur Friedenseiche fahre drive to the Peace Oak up there

3

hinter dem ICE behind the ICE (= high speed train)

Chapter 7. Thinking by hand

Fig. 7.5.1

4 5

hinter der ICE-Strecke da • oben ne? Saibershausen da oben zur Feldmark. Saiberhausen up there towards the Feldmark.

Fig. 7.5.2

6

Da steht • so ne Eiche There’s this oak-tree

7

nja die mag vielleicht zehn Jahre alt sein oder so well it may be perhaps ten years old or so

8

und äh da is ja nun ne Bank and uh there’s this bench

9

da setz ich mich denn ab und zu mal drauf that’s where I sometimes sit down

Fig. 7.5.3

10

wenn’s mir einigermassen auch so vom • Feeling her geht when I’m doing somewhat allright as far as feeling is concerned





Gesturecraft

Fig. 7.5.4

11

und wenn ich dann in diese • Runde gucke so and when I look around like this

12

man kann denn wirklich dieses ganze Tal durchblicken one can really see across the entire valley

13

die ganzen Berge ringsum all the mountains around

14

und denke mir mal eben die ICE-Strecke weg hhhhh and sometime I imagine that the ICE teacks aren’t there

15

dann schau ich nach Hause then I look towards home

The speaker begins the narrative with a pointing gesture (1), which is coupled with the deictic “up there”; we can assume that he seeks to point in the approximate direction of the space that he is about to describe (across the train-tracks). His subsequent gestures depict the setting and his experiences from the character’s first-person perspective. He depicts an oak-tree with a tracing gesture that shows the stem (2), and he looks up in re-enactment of his own looking up the tree. He presents himself in a landscape. The next gesture is made outside this spatial frame of reference and constitutes an example of ceiving: keeping his gaze focused on the therapist, the patient brings an open, slightly cupped hand, the palm facing up, near his heart, as he utters feeling (3). The gesture does not so much depict as concretize: it provides a metonymic reference to feelings. Then the patient returns to the depictive mode. Making a wide, horizontal arc in mid-air, first with a flat, than a pointing hand, he shows the panorama – die Runde (4) – and “all the mountains around”, taking them in with his eyes as he draws them. Gaze direction, but also the anchoring of the gestures in his descriptive talk, notably the adverb so, “like this”, make it clear that these gestures are meant to depict, that they serve to provide the interlocutor, not with semantic construals of experience, but with a concrete depiction of a setting. It does not matter how the parts of the landscape are conceptualized, as long as they can be imagined. But then later the patient depicts his “inner” emotional life in much the same way as he does the scenery. This seems to confuse our distinction between depic-

Chapter 7. Thinking by hand 

tion and conceptualization – how could someone’s emotions be literally depicted? Emotions are internal processes, not physical objects or acts. But clearly, emotions can also be depicted as if they were events in space, and so can any number of abstract realities. The criterion for “depiction” is not whether something exists as an external, physical reality, but rather if it is treated as such by the speaker in his representation. Here, too, the patient depicts what is surrounding him, but in this case, these surroundings are not tangible but construed, not “physical” but “psychological”. The patient says that he sees something “almost like a grey bell-jar (or cheese cover; Käseglocke) around him”, and he depicts this jar with motions of the hands. He raises his hands and then brings them down in a wide, sinuous motion (Fig. 7.6). He makes the same gesture two more times, the third time cupping his hands as if to pull something with them; this installment is coupled with the descriptor drückt, ‘presses (down)’. 7.6 Angst 1

Ha’ ich da gesessen I was sitting there

2

hab ich gesacht I said

3

pfff hat sich überhaupt nichts geändert pff nothing at all has changed

4

fühlst genau den- wie wie vorher hier you feel exactly the- liike like before here

Fig. 7.6

5

.hhhh wie- hab ich so sowas gesehen wie sone • graue Käseglocke fast um mich rum like I saw something like such a grey bell-jar almost around myself

6

also nur ganz weich well only totally soft

7

( - - - - - ) so ne? like that right?

8

Und dass mich was drückt irgendwo and that this presses down on me somewhere

9

was lastet. It’s a load

 Gesturecraft

Note that an external spatial depiction of the emotion is fully adequate to this emotion: the patient experiences it as coming from without and holding him captive. Again we witness how the gesture mediates the patient’s thinking. “Thinking” in this case means: making sense of and explaining emotions by finding adequate conceptual imagery for them. The “gray glass jar” (die graue Käseglocke) and the gesture accompanying it are concrete images for a diffuse emotional state. The patient conveys: “this is how it was”. If he were to later use the term Käseglocke – or the gesture with which it was introduced – referentially, in a context like “when I felt the glass jar feeling again, I did. . .”, he could be said to be using an imagistic gesture – a cept – in non-depictive fashion.

Gesture as conceptualization My view of gesture as conceptual action is not fundamentally different from the view of conceptualization that cognitive linguists have espoused, except that I give this theory, in its application to gesture, a non-intellectualist interpretation, as I have indicated before: I regard gestures as conceptual acts, not as expressions of conceptual acts that take place elsewhere, “in the mind” (see Streeck 2008c). For a more elaborate treatment of gesture semantics than I can offer, we can turn to the work of Calbris. Calbris’ shift from a more traditional, structural framework of semantic analysis (Calbris 1990) to a dynamic, action-based framework that is consistent with cognitive linguistics (Calbris 2003) is illustrative of the impact that the discovery of the active, intelligent body had on the sciences concerned with symbolic behavior. In Semiotics of French Gesture (1990), Calbris sought “to establish the structural semantics of coverbal gesturing through a semiotic analysis of the French gestural system” (xv). From the outset, she assumed that gestures are amenable to structural analysis (analysis in terms of contrasts or distinctions of form), that gesture constitutes a system, and that this system belongs to a language community (French). Her objective was to identify the parallels between formal contrasts in the gestural signifiers on the one hand and semantic contrasts in the meanings conveyed by them on the other. Thus, she established contrasts between types of movement-path (the “directional axis carrying the intention”; 42), namely straight-line/curved, forward/backward, and outward/inward; plane (hand in profile/palm inward, etc.); repetition; symmetry; and body segment (arm, hand, fingers). Using these descriptive criteria, Calbris compiled a small “dictionary” that listed various meanings that gestures with certain features of form predictably convey.

Chapter 7. Thinking by hand

An example of a “dictionary entry” is the list of meanings conveyed by variations of circular gestures (50–52). Circles can be represented either statically (“by two elements joined in a circle: the thumb and forefinger or middle finger..., or the two rounded hands facing eath other”) or dynamically (“by drawing a circle with the forefinger or the hand”). In the horizontal plane, with the fingers pointing down, a circular gesture can signify “a disk, a traffic circle, a round table, etc. The significations are numerous” (51). People use dynamic circular gestures in the horizontal plane with utterances such “Yes, [gesture] so the turntable will start” and “People were talking about it [gesture] all around Bologna” (51). When used in a transferred, metaphorically descriptive fashion, “the movement .. may refer to the zone ... thus enclosed, and hence for example a cliquish group” (51). This framework is based on the assumption that gestures – or, more generally, forms (including shapes; Arnheim 1969) – are inherently meaningful, and that meanings can be stated in terms of universal patterns of contrast. But the meaning that is actually communicated by the making of a gesture is a result of the interaction of the forms with a specific domain of application. Thus the multiple senses conveyed by a horizontal circle, or any other conceptual gesture, for that matter. In her recent work, Calbris reintroduces the body in a more active capacity and describes her mode of analysis as an investigation of bodily schemata out of which gestural conceptualization arises. She now analyzes these conceptualizations using documented samples of naturally occurring interaction. The title of an article exemplifies her program: “From cutting an object to a clear cut analysis: Gesture as the representation of a preconceptual schema linking concrete actions to abstract notions” (Calbris 2003). Like the present author, Calbris assumes that routinized gestures often are in some way derived from practical actions of the hands; they represent concepts that are themselves inherent in these acts. Gesture represents a preconceptual schema, an intermediary between the concrete world and abstract notions, and it evokes either equally well... [For example,] the schema of cutting underlies numerous and various notions: separation, cutting into elements, division into two halves, blockage, refusal, . . ., measurement, categorization . . . Gesture represents the visual and proprioceptive operational schema and, through it, the two extremes of the semantic continuum stretching from the concrete to the abstract: it represents the cutting of a real object into pieces in the same way that it represents the work of analysis (20).

The particular strength of Calbris’ analysis lies in her close attention to the nuances of meaning that emerge, on the one hand, from the subtle interplay between gestural prototype and semantic and pragmatic contexts of application, and to the variations of form such as the variable orientations of the palm of the hand. Construal or conceptualization by actions of the hands, as in this example offered by Calbris, is what I call ceiving. Ceiving is gestural conceptualization: the construal of content by means of manual schemata for purposes of



 Gesturecraft

communication. It is best explained in light of the theory of conceptualization that has been developed by cognitive linguists, notably Langacker (1988, 1990) and Johnson (1987). Conceptualization is the imposition of structure on an experience (Langacker 1987: 105), for example the “interpretation of novel experience with reference to previous experience” (105), which can be schematized experience as it is encoded in the established symbolic resources of a language. Conceptualization means projecting a “profile” onto a “base”, an “image” onto a “domain”. “The profile”, writes Langacker, quoting Susan Lindner, “‘stands out in bas-relief ’ against the base” (Langacker 1987: 183). The semantic value of an expression resides in neither the base nor the profile alone, but only in their combination (183).

Gestural conceptualization (“ception”) means that a person in a situation of communication draws upon and enacts for symbolic purposes bodily schemata and thereby gives visible and haptic form to content: experiences in real or fictional worlds (in the case of narrative), ideational realms (in talk about what we call “abstract matters”), or the interaction itself, as in the examples in the next chapter. But the methods of conceptualization available in gesture – that is, the specific ways in which embodied, enactive forms are created to organize and represent content – are diverse. Gesture and natural languages differ in the degree of conventionalization, elaboration and systematicity of schemata and in their combinatorial possibilities. Gestures are primarily single units - simple or compound - which can be brought into serial orders; but there is no gesture-equivalent to the sentence. On the other hand, gesture and language reflect the same basic principles of symbolic representation and sense-making: both provide us with schematic imagery with which to make sense. Gesture is not only visual symbolism; it makes use of kinesthetic and haptic experiences and schemata as well. Often the gesture communicates by indexing a familiar kinesthetic or haptic experience. Cognitive linguists attribute a central role in the making of meaning to schematized (repetitive) bodily experiences. In Johnson’s view, structures arising out of our bodily experience in the world provide patterns that are inherently meaningful to us, shape our imagination and reasoning, and serve as means to make other, more abstract kinds of experience meaningful. These experiential structures arise in our active engagements with the world, and they are largely enactive structures or schemata. In other words, our perceptual interactions and bodily movements within our environment generate ... schematic structures that make it possible for us to experience, understand, and reason about our world. (Johnson 1987: 19)

Chapter 7. Thinking by hand 

And “meaning in natural language begins in figurative, multivalent patterns [that] are embodied” (5), that is, in “patterns of embodied experience and ... sensibility (14)”. These embodied patterns do not remain private or peculiar to the person who experiences them. ... They become shared cultural modes of experience and help to determine the nature of our meaningful, coherent understanding of our ‘world’ (14).

Embodied schemata are “constantly operating in our perception, movement through space, and physical manipulation of objects” (23). They are therefore familiar to us and enable us to easily organize related and unrelated types of experience. Image-schemata are categories of experience gained not from contemplation, but from being physically engaged with and “being in the world” (Dreyfus 1991). It is apparent that gesture often operates at exactly this level of organization and thus gives an impressive demonstration of the reality of “image-schematic” cognitive organization. Gestural depictions are almost always schematic in that they involve a measure of abstraction from the specific contexts of physical life: a hand-shape configured for a gesture of grasping needs less specificity (in terms of hand-posture and forces applied) than a hand getting ready to grasp an object of a particular size, shape, weight, and texture in the context of a specific task. A gestural concept – a “cept” – is akin to what Goodman calls a “pictorial label” (Goodman 1976 (1968): 30): a gestalt-like, embodied schema by which an experience is classified. Thus, in light of Johnson’s theory, gestural concepts construe the world as it appears to the hands – as an array of tangible objects and fields of action or as something that can be given shape by actions of the hands. Whatever comes under gesture’s sway is in some fashion construed so that it fits human hands.

Further examples of ceiving Gesture can conceptualize thematic objects with greater or lesser specificity. The first gesture in 7.7 (not shown) is entirely concrete. The car-mechanic describes to the researcher a problem that one of his customers had with the engine of her car. It would not start. The gesture is coupled with the word flooded and figures flooding: the supine hand is opened in the direction of the interlocutor and moved forward and down in an arc. The gesture is not a depiction of the actual process (the ‘flooding’ of a car, i.e. the drenching of the ignition system in petrol, has entirely different properties), but an enactive representation of the concept: an available manual schema, not unlike that for giving or offering (see Chapter 8), is enacted and gives imagistic and sensory quality to the concept ‘flooding’.

 Gesturecraft

7.7 Hi-Tech Automotive 1 Hussein When she came to start the car, the car flooded her.

Fig. 7.7

2

She crank•in’. ( - - ) She misdescribed to me.

With the truncated clause she crankin in line 2 (presumably signifying that “her car was cranking”) the speaker indexes a state-of-affairs that he has established during his diagnosis of the customer’s car, namely that the starter was in fact working (and the engine ‘cranked’) when the customer first called him to report that the car would not start. In that sense, she “misdescribed” the problem to him over the phone. As he utters ‘cranking’, Hussein performs a gesture that he also made several times when he interacted with this customer and that represents a person recognizing that the starter is cranking: the hand is positioned near the ear, raised eye-brows demonstrate attentiveness and recognition, and the extended index-finger is rotating, showing the turning of the starter that produces the sound. 7.8 Hi-Tech Automotive 1 Hussein Try to start the car. (5.0) 3 Hussein When you call me it use to do this. 4 Nancy M hm. “cranking” ___|____ | | 5 Hussein It did that noise? 6 Nancy Yeah. “cranking” ___|____ | | 7 Hussein You hear the engine crankin like this? Try again. Try again. (1.0) 8 Hussein You hear the engine, crankin?

Uses of the gesture in these data are always coupled with the word “cranking”; the pair consisting of word and gesture thus appears to be stored as a single, bimodal construction, available for use whenever the concept ‘cranking’ is in play. The connection between experience and gesture in the cranking-gesture is indirect

Chapter 7. Thinking by hand 

and mediated by factual knowledge: the link between the turning of the starter and the noise produced by it is the result of cognitive synthesis and not directly experienced. Combined with the facial expression that accompanies it, the gesture visualizes how an expert, relying on accumulated knowledge of the world, interprets an auditory impression. At the same time, it simply is a visible display, useful like the rest of Hussein’s repertoire for communication under noisy conditions. In the following two examples Hussein gives manual shape to abstract concepts, ‘equality’ and ‘fairness’. Both segments are taken from an interaction with a potential buyer of a used car, a customer he has known for years. The first interaction occurs when Hussein is getting out of a convertible that he has just driven from the shop-floor to the front of the compound; as he is bringing the car to a stop the customer drives up in her own convertible right next to him, and the two end up getting out of their nearly identical cars in perfect synchrony. Hussein formulates this situation by uttering equal and making a two-handed “steering wheel” gesture (no transcript or image is included here, but compare example 6.19): he raises both hands, configured in power grips, and alternatingly raises and lowers them once, very rapidly. Here, too, he produces a bi-modal construction: “drive + equal”, or “equal (drive)”, that is to say: when it comes to cars and driving, we are presently equal. An almost identical gesture is made at the very end of this episode of negotiation, in connection with the term fair. Hussein is complaining that the counteroffer that the customer has made in response to his asking price is far too low. We need to go very fair, he says. Here, too, he raises and lowers his hands in alternation, but the hands are configured differently, prone and slightly cupped. This enactment projects the prototypes of see-saw and scale. The Gestalt could apply to the process of negotiation, figuring the alternation of requests and offers, moving towards equilibrium over time; or it could be a conventional metaphor for the abstract concept of fairness, represented as the balance of a scale. 7.9 Hi-Tech Automotive 1 Hussein There is no way for deal if you- I’m asking sixty-five and you say “balance” ______|______ | | 2 forty-five. We need to go ( - - - - - - - - - - - ) very fair, and I’ll be 3 very fair with you and you can take it anywhere you like

Johnson has used the image-schema of balance that is instantiated by the gesture as an example of the general process of metaphor. Metaphor results from activity: Balancing is a preconceptual bodily activity. ... The meaning of balance begins to emerge through our acts of balancing and through our experience of systemic processes and states within our bodies. (Johnson 1987: 75)

 Gesturecraft

The image-schema “balance” structures experiences of several kinds, one of them, according to Johnson, involving “cases where we carry an equal load in each of our hands”(96). This pattern seems to underlie Hussein’s two gestures above. The gestures exemplify that “metaphorical activity ... is an actual structuring operation in our experience” (80–1). Conceptual gestures can serve thinking-for-speaking in a variety of ways. In the following segment from a conversation between two young Turkish women and two linguists in Germany, an acoustic phenomenon is spatialized. The speaker is talking about the many French loanwords in Turkish, and she points out that their pronunciation is different from the French original. She has trouble finding the German word Aussprache (“pronunciation”, literally “out-speech”) and makes a gesture, with the hand held in a precision grip, that figures an act of pulling something away from her mouth. 7.10 Zuhal 1

Zuhal

die ham halt da in Frankreich och- gelebt und so und denn they used to live in France as well and so on and then

2

hamse diese ganze Sprache, Fachsprache und so ham die alles they brought this whole language, professional language and so on

3

mitgebracht they brought it all with them

4

Cornelia Ja. Ja. Yeah yeah

5

Zuhal

6

oder so ne? oder Gendarmerie or something right? Or gendarmerie [ Cornelia Hmm hmm. Hmm hmm

7

nach Türkei. Und da sind sehr viele Wörter da an- Gendarma to Turkey. Und there are very many words then there an- gendarm

Fig. 7.10

8

Zuhal

Natürlich da- hört sich des- ( - - - - ) also vom ( - • - ) Naturally there this sounds well in terms of [

Chapter 7. Thinking by hand 

9

Cornelia

von der n terms of

10 Zuhal

von der Aussprache her anders an aberin terms of pronunciation different but [ 11 Cornelia Aussprache pronunciation

In this example, the aspects of the actual metonymic meaning of the targeted word – ‘out-speech’ – is given concrete, enactive, visual form. The gesture is not a depiction of the process at issue – pronunciation – but it represents the imagistic gestalt of the meaning inherent in the linguistic term. In the following example, a more abstract phenomenon – the status hierarchy of Turkish society – is given concrete spatial expression, in a fashion that is similar to the way in which Christine in Penthesilea showed the different layers of a stage. Behiçe, the speaker, visualizes the social hierarchy that is built into the morphology of the Turkish language. Positioning her open, prone hand mid-high in gesture space, she then lowers it in several steps, thus suggesting levels. 7.11 Zuhal 1 Behiçe Und da wird“level” And there is___|___ [ | | 2 Vielleicht da wird die Rangordnung gleich mit ( - - ) in diePerhaps the hierarchy gets right along into the [ 3 Zuhal in der in the 4 Sprache language [ 5 Behiçe in der Sprache... in the language

Behiçe thus fabricates a concrete image of layers; projected onto the topic of the sentence, it spatializes social hierarchy.

Models for theories Gesture action allows speakers to construct model-worlds which can instantiate and concretize abstract, intangible matter, and to make abstract concepts tangible by translating them into haptic or kinesthetic forms. Before, we have seen a panic patient create gestural models of the desired state of Geborgenheit. A professional gestural model of psychic and interpersonal life is sketched

 Gesturecraft

in the following segments from a lecture by a psychologist; he enacts a gestural model of co-dependency. The lecturer’s manual displays provide concrete schemata with which audience members can form images associated with the concept codependency and learn how one can get in and out of relationships characterized by it. Raising his hands, the speaker first wiggles his fingers. In this way he shows selves without boundaries (1–3). Then he folds his hands with interlacing fingers in what he calls a Chinese finger-puzzle. This depicts a state of enmeshment (3, 4), believed to be characteristic of co-dependent relationships. (Enmeshment later breeds the desire for autonomy, which can only assert itself by complete separation, embodied by a forceful separation of the hands. These are held far apart, and fists represent rigidly separated egos. The loneliness of these egos creates renewed desire for the finger-puzzle of enmeshment. The alternative goal state of intimacy between autonomous selves is represented by two cupped hands, not facing one another but, positioned side by side, facing in the same direction.) 7.12 Co-Dependency 1 2 3

The core of the co-dependency is really i- the loss of our own identity. It’s like if you don’t have a sense of yourself, so you’re this amorphous, whatever, floating around the world.

Fig. 7.12.1

4 5 6

( - - ) Uh no sense of • boundary, you have no idea where you stop and other people begin. But when you get into a relationship

Fig. 7.12.2

7

you tend to get into a relationship like thi:s.

Chapter 7. Thinking by hand 

Fig. 7.12.3

8 9 10 11 12

And this is called en•meshment. You loose self. You don’t know where you stop and other people begin You don’t kno:w which finger belongs to which ha:nd, and you struggle ‘n you pull

Fig. 7.12.4

13 14 15

and:: • it’s ha:rd to get out once you’re in a co-dependent relationship. In fact, that’s the analogy I know, it’s like the Chinese finger-puzzle

Fig. 7.12.5

16

the harder you pull, the • tighter you’re in.

What is so striking about this metaphorical play is the skill with which the speaker makes use of the affordances of the hand, the shapes that hands can create, the actions that they can perform, and the haptic and tactile experiences that they can cause each other. A cultural topos of the late 1980s and early 1990s, the concept of co-dependency in relationships is condensed into a handful of tableaux

 Gesturecraft

made from hand-shapes and manipulations. The lack of ego-boundaries is made visible by fingers that keep wiggling so that no clear and firm gestalt can appear; “self ” – the bounded self – is conceptualized by a fist, a closed, saturated gestalt. Co-dependency is translated as “enmeshment”, which is visualized by the interlocking of all fingers of the two hands, and the entrapment in enmeshment is made tangible by the “Chinese finger-puzzle”, that is, the experience that the fingers lock whenever they are pulled apart. This schema, of course, can be replicated by anyone in the audience. And so can the experience of sudden “freedom” when the fingers let go of one another – which, in this speaker’s scenario, leads to renewed isolation. This is obviously a scripted performance, demonstrating skillful and wellrehearsed use of visual-haptic aids, but the patterns of metonymic metaphorization that are used so consciously here are also common when lay people construct models for their theories of psychic life. We have seen this in the panic patient above and can illustrate it with two more segments from his therapy. In them he performs gestures that make use of the affordances and physical structure of the hands in ways that are very much like the psychologist’s methods: he figures the self by two hands facing one another and enclosing a sphere, and the feeling of dissolution of the self by a moving apart and down of the hands, which at this moment seem to model a liquid that “flows apart” (zerfliessen), which is a common German metaphor for the loss of a sense of self, for example in moments of intense grief. (While there is no German word that matches confluence, there is no English word to match its German opposite, zerfliessen, “to flow apart, to disperse (said of liquid)”. 7.13 Angst 1

2

Weil vorher is das dann wirklich so ne Because before that is really like right

“disperse” ______|______ | | dann hat man wirklich das Gefühl man man dann man würde zerfliessen then you really have the feeling as if you were dispersing

3

gefühlsmässig so ganz fürchterlich emotionally like really terrible

4

als würde man sich auflösen as if one disolved

The patient first raises his hands and looks at them as they face one another, and then lets them drop and move laterally apart as he describes the feeling of dispersing (zerfliessen, “flow apart”); he produces another version of the gesture when he offers another, comparable term, auflösen, “dissolving”. The theory of the self as a contained entity, clearly bounded and separated from others and the world around it, is also given gestural shape in the following segment, the last to be addressed in this chapter; it occurs right after the patient

Chapter 7. Thinking by hand

describes his feelings of elation and autonomy when he sits on a bench next to the oak-tree and looks at the valley and mountains around him. Again he brings his hands to his chest, the palms facing one another, and he looks at them as if he could see his own self in them. This happens when he refers to himself as Mensch (“human being”), Individuum (“individual”). 7.14 Angst 1

Wissen Se das is einfach You know this is simply

Fig. 7.14

2

da spür ich mich • alleine nur, selber that’s when I only sense myself

3

als als Mensch als Individuum hier in dieser Natur as as human being as individual here in this nature

4

auf dieser Erde on this earth

If it is true, as Beattie (2004: 1) has said, that gestures provide glimpses of “hidden, unarticulated thought”, then this patient seems to envision self as a dialogue, like Mead, not as a self-contained unit, like the psychologist in the example above. What is obvious in these last examples in which gestures construct a tangible, spatial model which in turn interprets and illustrates a theory of psychic life is the role of gestures as “psychological tools” (Vygotsky), as mediating, enacted structures that are not only visible to interlocutors but also available to their makers, particularly through kinesthetic perception, but often also visually, and in terms of which thought processes can be organized and reorganized. Gestures thus occupy a unique position in human behavior: they are bodily actions, but they are also cognitive actions. Their makers can feel them, but can also bring them before their eyes, thus exchanging “experiential” for “reflective cognition” (Norman 1993). Often it is not the visual image that communicates, but the fact that the action indexes a familiar kinesthetic (haptic) experience, and it is this kinesthetic experience that aids the gesturer’s thought process. By way of gesture, relationships cannot only be seen, but also felt as if they were a Chinese finger puzzle.



 Gesturecraft

Gesture and thought revisited Gesture is said to make thought visible. This view has been succinctly articulated by Geoffrey Beattie. He writes about gestures: These movements of the hands and arms reflect thinking, like language itself but in a completely different manner. . . . Such behaviors provide us with a glimpse of our hidden unarticulated thoughts. Movements of the hands and arms act as a window on the human mind; they make thought visible. (Beattie 2003: 1)

That the human psyche and mind reveal themselves in the motions of the body is not an idea that originated only with modern psychology. It is embedded in the broad trend toward individualism that is characteristic of Western thought, and has been articulated by thinkers from Cicero to Leonardo da Vinci. New in modern psychology is the effort to identify causal and measurable relations and interactions between bodily behavior and hypothesized internal processes and to explain these within embracing and detailed theories of kinetic, communicative, cognitive, and symbolic systems. These efforts began with Wilhelm Wundt. Wundt regarded the formation of signs as a process that takes place in the individual psyche, and disregarded social factors such as cooperation and compassion that had been so important to other theorists, especially enlightenment philosophers like Condillac. “In the majority of signs”, he wrote, “the question of origin can only be understood in a psychological sense” (Wundt 1975 [1911]: 163). Signs are “natural developmental products of expression movements” (236) that initially involve no communicative intent at all. Affect and expression are two sides of the same coin, connected by “psychophysiological parallelism”: “primitive expressions are psychic and bodily processes at once” (48). There are at least two alternative views: one, that gesture mediates thought; the other that gesture is thought. The latter notion that gesture is a form of thinking (and not just an expression of it) follows naturally from the conception of bodily action as a form of knowing that phenomenological philosophers have offered. Heidegger thought of manipulation as a firmer foundation of knowledge than vision, given the distance between subject and object that vision entails. Whenever we are “intervolved” with each other and the world before us, in our quotidian care, we acquire experience and interiorize situations and their coordinates; our bodies, as they become embedded in their world, also embed these worlds within themselves. Bodily actions are mindful in that they incorporate an understanding of worlds. It would be almost impossible for anyone who approaches bodily action in the vein of phenomenologists not to conceive of gesture as a form of thought, simply because we believe that the entire person – not just the brain or mind – is the subject and agent of thought. Thus, in seeing a person gesture, we see that

Chapter 7. Thinking by hand 

person at thought in the same way we do when, say, we see a blacksmith bending an iron rod just at the right temperature. But of course gesture is a unique type of bodily praxis because it involves bodily signs, that is, bodily actions or productions that indicate, depict, or symbolize meaning and that are intended to be understood as acts of meaning, rather than as practical utility. Thus, we must assume that gesture mediates thought not only by virtue of being bodily activity, but by being symbolic activity: what goes on in the cognitive system of a blacksmith who has bodily cognitions and acts upon them and a conversationalist whose thinking-while-speaking is partly gestural is not the same. I will resume this issue below. One way of conceiving the cognitive role of gesture more specifically is to think about it as an organizational resource, as a set of dynamic structures in terms of which cognitive content can be organized and re-organized: cognitive content that needs to be processed can be off-loaded to gesture where it can be retained while the cognitive task is being solved. Even though the analogy is only half plausible, we may for a moment think of gestures as a kind of artifact, a cognitive artifact, that enables flexible, dynamic, and multimodal representations which operate both inter-psychologically, that is, in the intersubjective, public realm, and intra-psychologically, that is, in the realm of individual imagination and thought. Cognitive artifacts – or psychological tools (Vygotsky 1978) – mediate cognition. Some psychological processes or cognitive operations are possible without the help of public representations, but nevertheless facilitated by them (for example, adding three-digit numbers is facilitated by, but not contingent on written representations), whereas others are impossible for most people without “external” psychological tools (for example, composing a symphony). While Vygotsky and the “socio-historical school” are primarily interested in psychological tools produced by societies (linguistic categories, writing systems, number systems, etc.), when I call gestures cognitive artifacts I mean to draw attention to the fact that gestures are human productions – makings – , which is something they have in common with artifacts. And occasionally they are also made to endure – like artifacts – at least for a little longer than gestures normally are, and it is this enduring quality that gives these gestures some of their capacity, for example to hold meanings that are offloaded to them. Sometimes gesture is like a manufacture by which speakers fashion their own cognitive tools. There has been quite a considerable amount of research on gesture that roughly takes a Vygotskyan position, although people’s understandings of what constitutes a Vygostkyan perspective naturally differ. McNeill (1985, 1992) has argued that gesture is an integral part of language whose contribution is to express instantaneous, imagistic and global-synthetic thinking (1992: 11), in contrast to the analytic-categorical meanings that are encoded in speech. Gestures help constitute thought (1992: 245), they are material carriers of thinking for speaking (McNeill &

 Gesturecraft

Duncan 2000). “Thinking-for-speaking”, which is what is at issue here, has been defined in this fashion: the expression of experience in linguistic form constitutes . . . a special form of thinking that is mobilized for communication. . . . ‘Thinking for speaking’ involves picking those characteristics of objects and events that (a) fit some conceptualization of the event and (b) are readily encodeable in the language. (Slobin 1996: 76)

Kita (Kita 2000) has conducted experimental research to clarify how representational gestures may aid thinking and, consequently, speaking. He suggests that, because spatio-motoric thinking underlies representational gestures, producing representational gestures “helps speakers organize rich spatio-motoric information into packages suitable for speaking” (Kita 2000: 163). Spatio-motoric thinking and analytic thinking utilize different organizations of information, even though these are typically brought into alignment during speech. Given the limited options provided by a natural language, for example to represent motion in verbs, gesture facilitates the search for a compromise between the “raw material” of cognition and the “encoding possibilities” of the language, by temporarily encoding content within the alternative, global-synthetic medium of gesture (2000: 168). Krauss and his co-workers have suggested that gestures aid the speaker in retrieving lexical items during speech by holding “the conceptual properties of the sought-for lexical entry in memory during lexical search” (Rauscher, Krauss & Chen 1996: 421; see also Hadar, Dar, & Teitelman 2001). This is not altogether different from a prior theory, proposed by Freedman (1977), according to which gestures cement the link between psychic image and linguistic symbol. In a study of gesture during psychotherapeutic conversations, Freedman suggested that patients alternate between “body-focused” hand-movements (self-touch) and “object-focused” movements (representational gestures) and that these two types of action coincide with different modes of talk. He concluded that auto-focused movements help the speaker focus, by creating proprioceptive experiences and simultaneously diminishing sensory input from the surrounds, while representational gestures, which are directed towards the interlocutor, serve to anchor and solidify the symbolization of the experience as the linguistic representation is being constructed. Similarly, de Ruiter (de Ruiter 2000) proposes that spatial features in short-term memory are kept in a state of activation by representational gestures, until an appropriate linguistic representation is found. Goldin-Meadow (Goldin-Meadow 2003) has investigated a wider range of conditions under which gesture may aid in thinking, including mathematics classrooms and experiments in which children’s cognitive development was tested, and her research shows that, across contexts, gesture increases available cognitive resources: “gesture offers a route . . . through which new information can be brought into the [cognitive, J.S.] system” (167). Gestures can also facilitate thought in an

Chapter 7. Thinking by hand

indirect fashion, by giving others – for example teachers – access to a person’s thought processes, stimulating them to provide thought-scaffolding feedback. Here, the function of gesture is indirect, or mediated by the inter-psychological realm of interaction – which is commonly in play when people make sense together through gesture.2 Gesture activates haptic and enactive schemata but uncouples these from material contexts so that they become free to be creatively used in organizing cognitive (and discursive) content. Gestures supply the manipulative components of multimodal concepts. While a concept like “bucket” has visual and functional components, the gesture can supply a “transport” component; a bucket is an object that can be handled in certain ways, and it is stored modally (Barsalou 1999), that is, as a kinesthetic, enactive pattern. It is this schema in terms of which this bit of content (i.e. the object or class that the speaker is seeking to refer to) is organized. In this sense, gestures are modal representations. Since they exist both as “external”, visible “facts” and as internal, kinesthetic patterns, it is easy to see how they can organize conversational understanding and individual thought at the same time. The important point is that, from a Vygostkyan perspective, such gestures not only “show the speaker’s train of thought”, as Efron (1942: 96) said about what he called ideographs, but help the speaker have – or rather make – that train of thought. They are conceptual acts in terms of which the thought process progresses (McNeill 2005). In recent years, the phenomenological conception of the indivisible bodymind has received a lot of support in the form of evidence from unexpected quarters. Cognitive science – initially a thoroughly Cartesian and rationalist enterprise, quite immune to the lure of bodily experience – and neurobiology – a thoroughly materialist enterprise – in their attempts to come together in an explanation of . Gesture has also become increasingly relevant as a topic for research on first- and secondlanguage acquisition. Not much is known about how children of different ages understand gestures, but there is ample evidence that the ability to gesture and the complexity of gesticulation develops over time, along with language. How language socialization and the development of gesture interface is one of the important topics for developmental research; for example Goldin-Meadow, Alibali, & Church 1993; Goldin-Meadow & Butcher 2003; Goldin-Meadow & McNeill 1999. Some researchers study children’s developing abilities to gesture within the broader development of social and interactional competence and take an interest in the roles that gestures play, beginning with pointing and greeting gestures, in the ways in which young children construct their social world with others (Jones & Zimmerman 2003). There is also research on the roles of gesture in second-language speaking and uses the semantic structures of speechaccompanying gestures as evidence for the learner’s transition from thinking-for-speaking in the categories of the native language to those of the target language (Gullberg 1998; Stam 2005). The learner’s hand gestures while speaking in the target language reflect the extent to which thinking-for-speaking has moved in the direction of the target language.



 Gesturecraft

the evolution of the human brain, now recognize the incarnate character of the mind and the cognitive nature of the body. Today, embodied cognition and embodied mind are at the very center of a great deal of work across the disciplines that study the nature, logic, and evolution of the human mind. At the center of the new work on embodied cognition is the rejection of the traditional cognitive science model of the human mind as a processor of information that is stored in the form of abstract symbols (a language of thought; Fodor 1975), or the abstract terms of an innate universal grammar (Chomsky 1965), or some other language that can be subjected to algebraic calculations. That model is usually coupled with the assumption that the mind consists of a number of semi-autonomous “modules” (e.g. a syntax-module, a motion-generation module, a perception-module, etc.). Wilson has summarized the new trend in cognitive science as a growing commitment to the idea that the mind must be understood in the context of its relationship to a physical body that interacts with the world. . . . We have evolved from creatures whose neural resources were devoted primarily to perceptual and motoric processing, and whose cognitive activity consisted largely of immediate, on-line interaction with the environment. Hence human cognition, rather than being centralized [and] abstract, . . . may instead have deep roots in sensorimotor processing. (Wilson 2002: 625)

Even “abstract” mental activities – those that are “decoupled from the environment . . . [are] grounded in mechanisms that evolved for interaction with the environment – that is, mechanisms of sensory processing and motor control” (626). Wilson articulates a conception of embodied cognition that couples well with the view of gestures as “thinking tools” that I am proposing. She writes about the mediation of abstract thought by sensorimotor schemata: Many . . . allegedly abstract cognitive activities may in fact make use of sensorimotor functions. . . . . Mental structures that originally evolved for perception or action appear to be co-opted and run ‘off-line’, decoupled from the physical inputs and outputs that were their original purpose, to assist in thinking and knowing (626).

While Wilson also refers to the covert activation and operation of motor processes,3 it would seem that hand-gestures, which involve the overt enactment of motor schemata, facilitate processes of thinking and knowing in a similar fashion. Gestures, too, can be decoupled from the environment, and conceptual gestures in conversational environments usually are. But rather than conceiving of motor action (including gesture) as an externalization of processes that are, by their own . Recent research on language processing suggests that the motor-system is activated when we listen to texts that include reference to motion events or use movement imagery (fictive motion); Bergen 2003, 2005.

Chapter 7. Thinking by hand 

nature, “internal”, many neuroscientists now regard abstract thought as what is left of “fully embodied”, physically enacted thought once the motor component is inhibited: “pure mentation” is inhibited body motion. Narayanan (1997) and Gallese and Lakoff (2005) have suggested that parameters of motor action that are specified in the pre-motor cortex are uncoupled from the motor cortex to serve new functions. “Sensory-motor brain mechanisms . . . serve new roles in reason and language, while retaining their original function as well. . . . Circuitry across brain regions links modalities, infusing each with properties of others. The sensorymotor system of the brain is . . . ‘multimodal’ rather than modular” (Gallese & Lakoff 2005: 456). That there are strong connections between body movement and cognition has been shown by research on mental and cognitive health; some studies suggest that an immobile lifestyle increases the risk of Alzheimer’s and other cognitive disorders by as much as 50%. Usually the connections are explained in terms of the production of endorphines during body-movement, but a deeper explanation is that the very purpose of brains is to enable and control movement: a brain in an immobile body looses its reason for being. The neuroscientist Llinàs (2001) approaches thinking from the perspective of the evolution of neural networks, beginning with the single cell and ending with what he calls “the mindness state”, the state in which humans have conscious phenomenal experiences, either perceptual or virtual (remembered, imaginary or conceptual). Only animals, he points out, have brains; and only animals move by their own wits. The function of brains simply is to enable and control movement. And in order to do this work, brains need to make predictions (about, say, the effects of placing a foot a bit more to the left or the right). The brain is the helmsman at the front of a vessel who predicts its next location and constantly adjusts motion to changing “probable next locations”. Thus, during evolution, the property of motricity is being internalized – the beast is literally pulling itself up by the bootstraps! . . . The system takes properties from the outside and pulls them . . . inside. Through intrinsic oscillatory properties and electrical coupling [i.e., through emerging neural networks, J.S.], these properties are pulled up the neuraxis and into the encephalization of the brain. So what do we have? The ability to think, which arises from the internalization of movement . . . . (Llinàs 2001: 62)

And so Llinàs ends up with a view that is fully compatible with the view of gesture (or rather, some uses of gesture) as a manual thinking method and concept manufacture. Gesture is one realization of bodily thinking among others. Thinking ultimately represents movement, not just of body parts or of objects in the external world, but of perceptions and complex ideas as well.(Llinàs 2001: 62)

chapter 

Speech-handling

A domain that is routinely structured and mediated by conceptual gestures is the process of speaking, communication, and interaction itself. I will refer to this genre of gesticulation as pragmatic (Kendon 1995b, 2004) or meta-pragmatic gesture: the gestures are pragmatic when they themselves enact a communicative function (for example, when a raised hand, palm facing the interlocutor, admonishes him to wait his turn), meta-pragmatic when they embody and visualize (and potentially enact) a communicative function or illocutionary force that is simultaneously performed by the spoken utterance that they accompany. Usually, this difference does not need to concern us, and I will just use the label “pragmatic”. Other terms used for some versions or subset of this mode of gesturing are “interactive” (Bavelas, Chovil, Lawrie & Wade 1992) and “illocutionary” and “discourse-structure markers” (Kendon 1995b). Pragmatic gestures and cepts, such as the gestures of emotion made by the patient in the last chapter, appear to use similar semiotic strategies, and there are only subtle differences in the ecological parameters of their operation: the conceptual gestures that we have analyzed in the last chapter were coordinated with thematic units of speech, typically predications. Pragmatic gestures are coupled with interaction units such as turns, turn-construction units, speech acts, and speech act sequences. They are about the process of communication and acts of speech.

Kendon’s pragmatic-gesture families Kendon (2004: 158–9) has offered a succinct classification of modes or functional varieties of pragmatic gesturing. Pragmatic gestures can contribute to the significance of the utterance of which they are part in several different ways. They can show the kind of move being undertaken by a turn at talk, indicating, for example, that it is a question, a denial, or an offer. In these cases the gestures have . . . performative functions. On other occasions, however, . . . the speaker may mark up some feature of the discourse structure, and in such cases the gestures . . . have parsing functions. . . . Gestures that operate on the verbal component of the utterance . . . can function in ways similar to negative particles, but . . . can also serve as intensifiers. . . . In such cases . . . the gestures have modal

 Gesturecraft

functions. The ways in which these gestures achieve these results is partly by combining with the verbal or contextual meaning of the utterance, but also . . . by reaching beyond the bounds of the current utterance to operate in relation to the implied dialogue within which the utterance is embedded. (Kendon 2004: 281–2)

Kendon treats interactive or interpersonal functions of gesture as a separate category, but would probably agree that the categories overlap, not least because a single gesture can blend several functions. However, Kendon also points out that this is “a typology of functions, not of gestures. Any given gestural form may, according to context, function now in one way, now in another” (p. 225). Kendon has conducted a number of “context of use studies” (ibid.) of pragmatic gestures, investigating British and Southern Italian materials. This research has led him to distinguish a number of “gesture families” (2004: 225–9; see also Müller 2003). Families of gestures are “groupings of gestural expressions that have in common one or more kinesic or formational characteristics” (p. 227). Thus, using “a recurrent gestural form . . . as a starting point”, Kendon found that each of the gestural forms . . . has its own semantic theme – and this theme, being introduced as it is in different ways in different contexts, through the way it interacts with the . . . meaning of the spoken component of the utterance, contributes to the creation of a highly specific local meaning. (p. 226)

Four different families were thus identified (without any claim, however, that these exhaust the range of pragmatic gestures). These families include two that are made with the hand configured in a precision grip. These are the “G-family”, identified in data from Southern Italy, which comprises gestures in which “the digits . . . [are] drawn together so that they are in contact with one another at their tips” (p. 229) – in a shape that is called grappolo in Italy (roughly, “grape”). Four movement patterns occur with this hand-shape: when the hand is being closed to form the grappolo, a topic is “nominated for consideration” (p. 230); when the grappolo “is held on a supine forearm and is moved upwards and sometimes somewhat toward the speaker several times” (p. 231), the topic is simultaneously marked as salient; when the bunched hand is being opened, a topic-comment structure is generated: the grappolo marks the topic, and the subsequent opening of the hand accompanies the comment that is being made about it.1 In the fourth variety, the hand is moved down, and in this case the gesture marks the central part of the proposition. The hand-shape that defines the second family is the “ring”, in which thumb and index finger are positioned so that their tips touch one another. Pragmatic gestures made with this hand-shape convey the semantic theme of specificity or . This resembles the use of the pistol hand in Iranian conversations, as described by Seyfeddinipur (2003).

Chapter 8. Speech-handling

exactness (p. 240). Members of the family differ by motion pattern: when the hand is first formed as a ring and then opened, a topic-comment structure is displayed, with the difference (in comparison to the grappolo) that the topic here is a highly specific one: “something is being singled out, something is being made precise, something specific is being stated, but always in a context in which such making precise . . . is called for as a move of needed clarification” (p. 247). The other two gesture families involve open hands, in which the fingers are adducted (i.e., not spread). When the palm is positioned horizontally (facing down), negation is being expressed: “spoken language provides what is being presupposed and the gestures act on these presuppositions. . . . [They] may be regarded as operator gestures” (p. 263). These gestures, however, can also be made in the company of positive assessments, in which case they “express an implied negative . . ., the implication that nothing else is as good as the evaluated object” (p. 259): it is as if they conjure up counter responses that might be evoked by what is said, and then operate in relation to that. They enrich the utterances of which they are a part, by making reference to the dialogue that any assertion implies. They simultaneously evoke this dialogue and make a move within it. (p. 264)

“Palm up” gestures, in contrast, figure “offering or giving or . . . show. . . readiness to receive something” (264). In these cases, the hand is moved forward, towards the interlocutor, or down. When the hands are laterally moved apart, in contrast, the semantic theme of the gesture is the speaker’s “inability or unwillingness to act, . . . to offer any suggestions or solutions, to provide meaning or an interpretation of something” (275) – to take a stance. These gestures often occur in the context of shrugs (see below). Overall, pragmatic gestures are an unruly bunch: speakers show all manner of idiosyncrasies in making them – some people gesture abundantly, others hardly at all; little constructional effort is usually invested in them, and it is sometimes difficult to identify meaningful or recurrent forms. It is conceivable that societies differ in the degree to which they develop conventional repertoires of pragmatic or rhetorical gestures – for some cultural groups, for example Italians in the area around Naples and Salerno (DeJorio, 1832; Kendon 1995b, 2004), extensive sets of commonly used and understood enactive forms have been described (see also Poggi 2002), but we have relatively little information about the number, kinds, and functions of pragmatic gestures in most other societies. We are just beginning to find patches of orderliness and motivated relationship between form and function in this wilderness.



 Gesturecraft

Pragmatic gestures, turns at talk, and kinesthetic feedback The research reported in the remainder of this chapter was conducted independently of Kendon’s and on the basis of materials collected in the U.S.A. and the Philippines,2 and yet it has yielded a number of overlapping results, an indication, perhaps, that the degree of cultural distinctness of gestures is sometimes overestimated. Like Kendon, I have isolated a small number of forms – hand-shapes, hand-orientations, and motion patterns. However, I examine these with respect to the specific sequential locations at which they are made. Different points within the unfolding of utterances and utterance-sequences pose different interactional tasks for the interaction participants. Chief among these points (or “places”) is the context of turn-transition itself, the “transition place” (Sacks, Schegloff & Jefferson 1974). I explore a subset of these positions and concentrate on a handful of gestures, apparently routinized, occurring in them, most of which involve a flat, open hand: 1. “palm-up/open-hand” gestures (Müller 2003; Streeck & Hartge 1992) that are made during turn-completion and in mid-turn; 2. shrugs; 3. gestures with which something is pushed back or to the side; 4. gestures that figure an interruption of the conduit between speaker and listener; and 5. open-handed gestures at or before turn-beginning. I then discuss the shared conceptual pattern in which these gestures participate, which construes talk in interaction as a transaction with physical objects (a concept known as the conduit metaphor; Reddy 1979; McNeill 1985). This discussion will lead us to briefly examine some linguistic construals of communicative action, which reveal themselves to be similarly derived from the conduit metaphor. In order to fully appreciate the roles of pragmatic gestures in conversational interaction, it is important to take into consideration that they are kinesthetic phenomena: they not only furnish recipients with visual displays of pragmatic aspects of communicative acts, but at the same time give the speaker feedback about his or her own communicative intentions. I address this issue at the end of the chapter.

Open hands and turn-completions Open-handed, unilateral or bilateral pragmatic gestures in which the palm faces up (Kendon calls them Open Hand Supine, OHS; Kendon 2004: 264–275) are extremely common (Streeck 2007b). Let us first examine enactments of such gestures in the context of turn-completion. In Segments 1 and 2 the speakers make the following gesture at the end of their respective turns: both hands are raised, open and . See the Appendix for the sources of these materials.

Chapter 8. Speech-handling 

supine, and, in the stroke, moved down and forward toward the addressee. The action pattern is that of “handing over” or “giving”, or, when the hand is somewhat differently oriented, “presenting” or “offering”. In Segment 8.1, Hussein, the auto-mechanic, is in the process of explaining to the researcher the nature of the trouble that a customer had with her car. Coming out of a depictive gesture, his hands are raised, supine, and open, the fingers slightly flexed. Then the speaker counts out two points he wishes to make: he uses the counting method of first flexing all fingers and then extending them one by one, beginning with the index finger. Finally, he performs a large and ceremonial act of handing over: he opens and lowers both hands to me as he makes the third, concluding point: “no power”. This is the pragmatic gesture in focus. 8.1 Hi-Tech Automotive 1 Hussein all the car smoke 2 wasting gas

Fig. 8.1

3

no po•wer.

4 Jürgen

Oh.

In the second segment a woman in the Philippines, in conversation with two others, sums up a points that she has been making about the fate of young Filipinas, even those from good families, who seek work abroad: they end up as maids. As she utters the word imbot (“maid”), she “thrusts” her open, supine hands to the addressee. The listener shows that she has got the point: she repeats the word. 8.2 Rumors 1 Esmin Naistoriya ni Rachel langenen. Naistoirya ni Rachel. Rachel’s story alone. Rachel’s story. [ 2 Gallit ( ) 3 Esmin Apo! Lord! 4 Gallit

Napan? They went?

 Gesturecraft

5 Gallit

Dagita mapmapan ( - - - - - ) iti sabali nga luglugar, nga mapan Those who are going to foreign places, who go to “push” _____|____ | | agbirbirok: ( - - ) Imbot! work Maids!

6 Batto

Imbot. Maids.

Turn-completive open hand/palm up gestures – or “offerings” – can thus facilitate or solicit the forthcoming of response (for further examples see Streeck 2007). In these segments, one can observe a certain “fit” between embodied, figurative actions and the transactional, conversational contexts in which they are made. The social moment where these gestures appear is a transition: whatever “message” was to be given to the other, it is now complete and therefore being handed over, and so is the right to speak. In the motions of the hands, perhaps motivated by our silent attention to turn-taking matters, we see a familiar gestalt, an action: a giving or offering. Our socialized bodies respond with an inclination to take and give back.

Giving and receiving; waiting to receive We perceive pragmatic gestures in the company of speech, and grammatical frames and the content of speech influence how we see them. The downward and forward trajectory of the open hand that occurs during turn-completion – or the completion and “handing over” of smaller packages of speech (e.g., “idea units”; Kendon 1980) – is a plausible form to figure the “handing over” of the turn. But what happens next? Time passes, and the meaning of the open hand changes: frozen in position, it gradually transforms into a receptacle: it appears to be waiting for something to be given in return (Kendon 2004: 264; Streeck 2007b). This dual function of presentation platform and receptacle is given to the hand in acts of object transfer. Open hand-gestures can become solicitations, appeals. Co-participants seem to respond to turn-transitional “offering” gestures in this fashion. In segment 8.3, the speaker performs the gesture in such a way that her hand comes close to the interlocutor, and she leaves it there, the palm turned up. The listener expresses agreement: after a micro-pause, she says “I agree”. The speaker retracts her hand.

Chapter 8. Speech-handling

8.3 Girlfriends

Fig. 8.3 ___________________ | 1 Rani I mean- Andy • McDowell is terrible. (-) ___ | 2 Bev I agree.

With only a little flexing of the fingers, the open hand can become a beckoning: the passage of time and the micro-movement conspire to create the effect of an upgraded solicitation. By performing such mundane but utterly familiar actions during turn-transitions, interaction participants construe their conversation as a give-and-take: within this model, utterances are produced for the benefit of the other to whom they are given, and they may require that something – a response, a token of reception – is given in return. Listeners may take the “giving” gesture as a sign that the turn is now complete. In Segment 8.4 Bev and Rani are discussing movies. Bev portrays the longing of a lonely female character. As she says “dead time, dead time”, she lowers her prone, open hands twice, in an apparent gesture of depression. Then, during a brief gap in her utterance, she rotates her forearm so that the palm faces up, and makes a handing-over gesture. With this, she completes a three-part list. Then she retracts her hands and continues to talk. Rani produces an appreciative response-token in immediate juxtaposition to this offering, which shows that she analyzes this point as a transition-relevance place, giving her license to take the turn. She produces further talk, but so does the other speaker, and for a moment the two talk in overlap.



 Gesturecraft

8.4 Girlfriends

Fig. 8.4 _______ _________ _____________ | | | | | | 1 Bev dead time, dead time ( - - - ) someone • come- ( - - ) you know [ 2 Ranajana Ahh, 3 that’s really good [ 4 Bev and then she’s like- doing 5 Rani Did you see her

We can glean from this segment that the recipient analyzes an offering gesture as a turn-completer. In 8.5, it is the recipient of a report (about the state of a relationship) who brings a “palm up/open hand” (Müller 2003) to the speaker, as she asks for a piece of background-information. Before Bev answers, she opens her mouth in preparation. As she sees this, Rani closes her hand and retracts it, apparently satisfied that a response is coming.

8.5 Girlfriends 1 Bev

He’s been up there like a couple of times.

Fig. 8.5 _________________ | 2 Rani So they’re broken up • now. ______ | 3 Bev [opens mouth] No, they’re still together.

Chapter 8. Speech-handling 

Open-handed gestures are well-adapted to the tasks of turn-taking and speakerchange. We are all familiar with manual acts of giving, offering, and solicitation. We know how to give and take things into and out of each other’s hands. And we are also familiar with the gesturalization of these acts in ordinary acts of objecttransfer: children learn early to recognize the open hands of their parents as requests for objects that they have in their hands. The ambiguity of the appearance of open hands before us congrues with the dual nature of turn-transition: while the listener is the recipient of whatever the speaker’s turn is conveying, he or she is also called upon to give something in return. The meaning of the open hand changes with the passing of shared time in conversation: the more time passes after turn-completion, the more pressing is the constraint upon the recipient to produce a turn in response (Streeck 2007b). McNeill (McNeill 1992), following Reddy (Reddy 1979), has called those gestures that figure the exchange of talk during conversation as object-transfer “conduit gestures”: they picture and conceptualize the production and reception of talk as movement of physical objects along conduits between the participants. Different kinds of gesture can be coherently explained by the conduit model of symbolic object-transfer. However, gestures of speaking deal with abstract, intangible objects, and they often involve less than a full expenditure of constructional efforts by the speaker. Not always can we therefore recognize a clear gestalt.

Mid-turn offerings “Offering” gestures not only occur during turn-completion and transition, but also in mid-turn. But even in mid-turn they appear to “transfer” an object, for example, a central idea, new information, or some distinguishable utterance part (see Bavelas, Chovil, Lawrie & Wade 1992; Kendon 1996). In Segment 8.6, three items on a list are “handed over” by three consecutive installments of the gesture.

 Gesturecraft

8.6 Girlfriends

Fig. 8.6 _________________________ __________________ ____ | || || 1 Bev He marries this French woman, he moves to Paris, he opens • a ____ | 2 salon.

What exactly is being offered can be marked by the beginning and end-points of the gesture-stroke. In 8.7, the speaker (the talk show-host Conan O’Brien) times the gesture so that it coincides with the name Dan Quayle, the core-element in an ironic characterization of someone (Dan Quayle kind of appeal). Thus, in addition to displaying the transfer-act, the gesture also marks the focus of the utterance. 8.7 Conan O’Brien

C

Fig. 8.7 ___________ | | the Dan • Quay:le kind of appeal

When a motion pattern can be understood in its contextual frame as a “giving” or “offering”, then the manner in which the motion is carried out can also be perceived as meaningful. For example, it can mark the relative importance of the item offered or the attitude that the speaker adopts toward it. What I call weak offerings – open-handed, palm-up gestures that are neither effortful nor clearly articulated – sometimes frame peripheral turn-components such as parenthetical inserts, as in Segment 8.8; the insert well, I mean, he knew this as a Christian is framed by a

Chapter 8. Speech-handling 

“weak offering” and the turn-unit is thus set off from the continuing narrative, which is accompanied by depictive gestures. 8.8 Satan

Fig. 8.8 ____________________________ | | 1 A is in fact a:- Well, I mean he knew this as a • Christian but- a uh 2 master deceiver, Satan

“Weak offerings” can also display a “weak stance”, serve as a marker of the diminished importance of a proposition or of the speaker’s less-than-full commitment to it. In segment 8.9 the speaker casually lifts her open hand without extending it towards the recipient, as she comments on her own prior talk with the clause I don’t really know. 8.9 Heather “offering” _____|_____ | | 1 A I don’t really know, I just make it up.

Shrugs Open hands are also a central component of shrugs, which are complex and varied displays of great communicative importance (and perhaps a candidate gesture universal.) A prototypical shrug involves several body-parts, the eye-brows (which are being raised), the hands (which are turned so that the palms face up), the forearms (which may be lifted), and the shoulders (which are also raised). In addition, the head may be tilted. Thus, shrugs are compound enactments. The components can also be enacted successively, in response to recipients’ changing responses (or lack thereof). Shrugs are actions in which paired body-parts – eye-brows, fingers, hands, forearms, shoulders – are raised and thereby disengaged from any forward movement, any line of action, any active involvement with things. The

 Gesturecraft

components of shrugs – the actions that these components constitute – each display disengagement. The body displays a distanced, less than committed stance, and we usually read this as an expression of the person’s attitude towards a proposition, a state of affairs, or an illocutionary act. When a shrug is performed during turn-completion, it can demonstrate that no particular fashion in which the talk should be taken is proposed. The shrug places the burden of finding an appropriate attitude toward the proposition on the interlocutor. In segment 11.15 we see the components of a compound shrug distributed across a succession of positions in the transit through the transition space, in the pursuit of response. 8.10 Girlfriends

Fig. 8.10

_______________ | 1 Rani ___ Just more honest. ( - ) You know what I • mean? | (-----) 2 But I ...

The speaker’s hands are involved in a turn-final offering gesture. When no response is forthcoming, she appends the response-solicitor “You know what I mean?” to her turn, along with a shrug. When there still is no response, she adds further shrug-components, a head-tilt and spread lips. Then, giving up, she continues to talk. In 8.11 we see shrug-components produced in succession during a gap between two utterances of the same speaker. The speaker, a televangelist, formulates his lack of understanding of another preacher’s criticism of himself. As in the previous example, it is possible that the successive production responds to the lack of uptake or alignment by the listener.

Chapter 8. Speech-handling 8.11 PTL Club

Fig. 8.11.1 _____________........ | 1 A ... and he’s coming back • in this. 2 (----)

3

Fig. 8.11.2 ______________________________....____ I mean this is what he • teaches and preaches, __________ | this is what I teach.

The speaker produces three shrug-installments: a full shrug before the beginning, which displays the stance he is going to take; an upgrade (the hands are raised higher, the speaker shakes his head), and then another full performance, when he adds the clause this is what I teach. Shrugs, then, can be distributed across several positions and varied in conspicuousness and magnitude. The fashion in which this widespread and complex gesture is performed appears motivated by the logic of our bodies’ engagement with the world. In shrugs, our bodies withdraw and retract from possible engagements. This may be the reason why we understand shrugs as displays of distancing and disengagement. Kendon, who discusses shrugs under the term “PL” (palm lateral) gestures and found five contexts in which shrugs are used, concluded that “in all cases . . . the lateral movement of the Open Hand Supine indicates that the speaker is not going to take any action with regard to whatever may be the focus of the moment” (Kendon 2004: 275).



 Gesturecraft

Moving things aside The treatment of talking as object-transfer can also be identified in gestures that appear to move something in other directions, besides the recipient, for example to the side. These gestures serve to structure discourse, by marking the talk that is being completed as unrelated or peripheral to the story-line and thereby “making room” for something else to follow. In the following example, the hand figuratively moves the prior topic – which has caused laughter – aside so that the old topical path can be resumed. A lateral movement by a cupped hand, oriented outward, is coupled with the discourse marker anyway. It appears in the midst of the shared laughter. 8.12 Girlfriends

Fig. 8.12 ______ | | 1 Rani Any• way.

In 8.13. Rani offers a piece of information with an offering gesture, but then rotates her hand and figuratively moves it aside with a similar lateral movement, as she explains that “this is another story”, i.e., not relevant to the topic at hand. 8.13 Girlfriends 1 Rani Uhhm, although his last name is not (

2 3 Bev

Fig. 8.13 ________________ | | But that’s another • lo:ng story.= =Okay.

).

Chapter 8. Speech-handling 

Throwing back (re-jecting) Rejection gestures imply a similar practical logic. For example, the hand can be raised (and perhaps thrust) toward the interlocutor, the palm facing away from the speaker. This gestalt can be projected onto many discourse domains and convey – beyond rejection – repulsion, stopping, refusal, objection, and negation, as Calbris (1990: 137) has suggested. Segment 8.14 is a good example. Bev is about to tell Rani about a movie which is “really good”, but Rani does not want to hear about it. She thrusts both hands toward Bev, the palms facing her. Kendon, who calls these gestures “VP” (“vertical palm”) gestures, similarly notes that thse gestures “are used in contexts where the speaker indicates an intention to halt his or her current line of action . . . or a wish that what is being done by the interlocutor should be halted” (Kendon 2004: 251). 8.14 Girlfriends 1 Bev

It’s really good.

Fig. 8.14 __________ | | 2 Rani Don’t • tell me.

In 8.15, the same type of enactment accompanies a speech act that is quoted: “All of a sudden he’s like ‘No!”’. The palm faces the interlocutor who is thereby given the role of the quoted speaker’s interlocutor: the recipient – or the prior talk that came from her – is figuratively held at bay. 8.15 Girlfriends

1

Fig. 8.15 _____ | | And then all of a sudden, he’s like “• No:!”

 Gesturecraft

This gesture of rejection is conventional enough that it can replace a verbal display of rejection altogether. In 8.16, the gesture sums up (in a re-enactment) an entire act of rejection. 8.16 Heather “push back” ___|______ | | 1 Heather I was j’s- ( - - - - - - - ). 2 I mean the other girl was giving a talk.

Negations Negations, even negations that are implicit in prefixes such as un and in, can be conveyed by gestures of rejection. In 8.17 a gesture of rejection is coupled with a strongly positive assessment term, incredible, and it figures the negative component in it, i.e. in-, that is, ‘not’. 8.17 Evangelists “negation” ___|___ | | 1 A is incredible

Other common gestures of negation are made in the frontal plane between speaker and hearer. Figuratively, they interrupt the conduit between the interlocutors. They indicate that a certain proposition is not being transferred. In Segment 8.18, the speakers moves his two hands, which are prone and open, quickly laterally apart, “cutting off the conduit”. Kendon interprets the common gestalt of these gestures in a somewhat similar vein: Open Hand Prone gestures . . . share the semantic theme of stopping or interrupting a line of action that is in progress, [where] . . . ‘line of action’ [can] . . . mean . . . any project that someone might be engaged in, whether this involves physical action or . . . assuming a certain mental attitude toward something. (Kendon 2004: 248–9)

Chapter 8. Speech-handling  8.18 Satan

Fig. 8.18

____ | | 1 A which in fact it is • not.

The conduit between speaker and audience can also be interrupted by a line, the trace left behind by two separating hands: in 8.19, the speaker’s two index fingers are being held upright in the center, then abruptly moved apart. Then the gesture is repeated as the negation is intensified by the quantor whatsoever. 8.19 Satan

Fig. 8.19 ____________ ______ | | | | 1 A It has nothing • ( - ) whatsoever to do with...

All of these gestures are embodied practices that model conversation as a physical conduit governed by the dynamics of physical forces, a conceptual model that is also pervasive in natural languages (Talmy 1988). In language, semantic prototypes that correspond to conduit-models include the bending of an oncoming object’s trajectory (e.g., deflect a question), shielding oneself from it (e.g., dodge a question), and moving objects (e.g., setting aside a question). Diller (1991: 226), writing about Indonesian speech act verbs, has remarked that cognitive models which conceptualize specific types of physical action and object manipulation are systematically used for the conceptualization of discourse situations. Moreover, the function that these physical actions fulfill with respect to external objects in the spatial environment is also transferred to the conversational domain.

 Gesturecraft

The open hand before tellings In some of the pragmatic gestures that we have so far examined, the open hand has figured as a container and receptacle; talking in interaction is modeled after the transfer of objects, and the hands enact this transfer: they offer, give, reject and receive. Another way in which an open hand is used during moments of possible or actual speaker-transition is that the palm is oriented so that it faces the speaker (faces up), who may or may not then look at it. We find this movement at the beginning or during the pre-beginning of tellings. Phenomenally, the open hand creates a plane in gesture space that can engage the speaker’s attention, allowing for a moment of self-intimacy of the mind. Here, while still a container of symbolic objects, the open hand supplies imagery of a different kind: it no longer holds or contains a thing that is being transferred, but marks a universe for the speaker’s attention. The palm-up/open-hand gesture appears to be a conventional method by which Ilokano speakers in the Philippines pre-figure an imminent telling, or another installment of an extended telling (Streeck & Hartge 1992). It is not clear how widely distributed the gesture is, whether it is an “Ilokano gesture”, or is more widespread in Northern Luzon, the Philippines, or Southeast Asia. In our data, it is exclusive to older female speakers, and my consultants reported having seen it “when older people tell stories”. The sense of the gesture as a preface to – and thus announcement of – a telling can be gleaned from segment 8.21, in which the speaker is intermittently talking in English to the author. (A is an Ilokano interjection.) 8.20 Conchita

Conchita

Fig. 8.20 A! I will • tell it, a!

In 8.21 the speaker initiates the gesture, and thereby discloses her upcoming tellership, before she actually begins to speak. Then she deals with questions by her co-conversationalists which pertain to the talk that she has just completed, all the while holding her hand in its palm-up position (Fig. 8.21). Finally, as she takes up her story with the word “ma-istoria-kon”, “I can tell you the story now”, she relinquishes the gesture (which is thus held from lines 2 through the beginning of

Chapter 8. Speech-handling 

line 7). The gesture embodies not only its maker’s claim to the floor, but also the kind of action for which the floor is requested: a telling. 8.21 Rumors 1 Esmin Kasta met iti naimbag, kunana. So that’s the news, she said. 2 Gallit

Hm hm. Hm hm.

Fig. 8.22 ______________ | 3 Gallit Asinno ‘ng Inyong • data. Who is that Inyong? ____________________________ 4 Esmin Datay kakabsatda ngay nga lallake. Their brothers. _____ 5 Gallit (a e i) (a e i ) _______________ ( - - - - - - - - - - - - -) ______________ 6 Batto Ni Manang Maria? Is it Manang (“Big Sister”) Maria? _______________ | 7 Esmin We- . Maistoriyakon, Iket. Kunana ta adda gapo-, adda puonan, Ye- . Now I can tell you the story, Auntie. She said there’s a cause, there’s

8

kunana. a root cause, she said.

One of the advantages, especially for intending next speakers, of producing a gesture during moments of turn-transition is that visual displays are readily noticed and recognized by someone who is talking. A gesture does not interfere with ongoing talk by others and can yet be noticed and taken into account by them. By “flagging” an imminent telling, the intending speaker can provide other would-be speakers, including the current one, with a motive to relinquish or forego the floor.

 Gesturecraft

The palm is a plane on which small objects can be placed to be inspected. It is perfectly common for us to concentrate our visual attention on what is in the palm of our hands. It is thus apt to show that the speaker is “collecting her thoughts”, demonstrably remembering, and thereby “preparing for a telling”, by an extended look at the hand that is held like an open book. But the gesture is not exclusive to the Philippines. We find similar figurations – a look at the open palm formulating the would-be speaker’s engagement with a universe of discourse – in American, German, and other conversations. It is not uncommon that they constitute the first installment of a complex, sequential enactment of “counting” which is wellenough adapted to the linguistic activity “recounting”, but also to other types of list-constructions. In segment 8.22 the speaker opens his hand, silently looks at the palm for a moment, and then uses the left hand to grab, first the thumb, then the index-finger of his right hand as he lists new-age trends that undermine the teachings of the Christian faith. 8.22 Satan palm up/open hand ___|__ | 1 B ... trend in the church of- ( - - - ) counts on fingers ________|____________________ 2 you have your Christian psychology, _________________________________________ 3 ( - - - - - ) you have your positive-thinking messages, _______ 4 we have...

Segment 8.23 is constructed in the same fashion. As he prefaces an extended turn, the speaker turns his hand so that the palm faces up and looks at it. Then, as he lists hypothetical circumstances (incorrect teachings of the gospel), he uses his right index-finger to point, first, to the pinkie, then the ring-finger of the left, counting the circumstances. 8.23 PTL Club palm up/open hand ____|_____ | | 1 A The question is this. ( - - - - ) counts ‘1’ ______________|_______________ | | 2 Suppose Kenneth Copeland is wrong.

Chapter 8. Speech-handling 

3

4

counts ‘2’ ________|_______ | | Suppose I’m wrong. counts ‘3’ _____|__ | Suppose...

Implicit objects It is difficult at times to find a verb which precisely describes the action-schema or motoric gestalt of a (meta-) pragmatic gesture; not always is it possible to unequivocally assign the gesture to an action schema such as giving, receiving, pushing aside, or throwing back. The fuzzy, under-determined appearance of the gesture can make such description problematic. What is not doubtful, however, is that the schematic action would not be a complete gestalt without the implicit presence of a thing: an object, instrument, or accessory to the action. The movements of the hands would appear neither coherent nor meaningful if they were not perceived as intentional actions, abstract to be sure, but nevertheless object-oriented in their practical logic. The implicit engagements with physical objects that are inherent in the motion-patterns of these gestures cluster in a coherent model that equates conversational interaction to the transfer of things. In this respect, the pragmatic gestures in this chapter require the same logic of perception that we have found to operate in those depictive gestures in which the motion of the hand evokes a physical world by implicitly indexing physical objects. The domains are blended, the logic of manipulation projected onto the domain of speech, and the physical world as we know it is the tacit background of our recognition of the enactive meanings inherent in pragmatic gestures

Verbs of speaking Talk is also conceptualized as object-transfer in the lexical patterns of diverse human languages, notably in verbs of speaking. Many verbs of speaking are derived from verbs denoting practical, intentional actions of the hands. The following table shows a number of German verbs of speaking (on the right) and their roots (on the left). We can see that the roots all denote actions that are done with the hands, they are manipulations: to reach for, to pound, to present, to put, to take, and so on. When combined with various verb-prefixes (only some of which translate into English), including ver, which does not, vor (“before”), and dar

 Gesturecraft

(“there”), the verbs denote acts of speaking. Sometimes, the metaphor is transparent: we can understand why the language would construe an act the way it does, e.g., that a proposal is something that is put or thrust before someone. At other times, the metaphor is the contingent, opaque result of language history, as in sich beschweren, “to put weight on oneself ”, which means “to complain”. langen schlagen bieten geben tragen fügen stellen legen nehmen lassen drücken

to long, to reach to pound to present, offer to give to carry to put together to put (in an upright position) to put (in a lying position) to take to release, let to press

verlangen vorschlagen verbieten angeben

to ‘ver’-long, demand to pound before, propose to ‘ver’-offer, prohibit to at-give, brag

verfügen vorstellen

to ver-fugue, decree to fore-put, introduce

darlegen to

there-lay, point out

vernehmen zulassen ausdrücken

to ver-take, interrogate to to-let, allow to out-press, express

Other roots of speech act verbs in German add the imagery of an instrument: schneiden schliessen zeichnen

‘to cut’ ‘to close’ ‘to draw’

anschneiden beschließen bezeichnen

to cut into, raise an issue to be-close, decide (together) to be-sign, signify

Finally, there is the ubiquitous root handeln itself, which literally means “to handle”, but also “to barter”, in which case it figures bartering as handling of one another’s goods, as well as “to act”, “to take action”. Verhandeln means “to negotiate”, and speech act theory is Sprechhandlungstheorie, “speech handling’s theory”. While the conceptual imagery of many English speech act verbs such as those below is opaque to most of the language’s users, because they are derived from Latin and have come into the English language via French, the imagery can nevertheless be recovered from the Latin roots. It is the same kind of imagery. These are some Latin common roots and their elaborations in English speech act verbs. fer tra ten pon met cep pret

bear pull hold put put take press

to bear together, confer, to pull together, contract to hand-hold, maintain to put before, propose to put under, submit to in-take, accept to out-press, express

Chapter 8. Speech-handling 

There is an interesting historical fact about these English verbs. As Olson has explained in his book The World on Paper (Olson 1994: Ch. 7), many English speech-act verbs were borrowed into the English language at around the time when the printing press began to be widely used and written specimens of communication became commonplace, enabling transactions that were previously conducted face-to-face. Olson suggests that the verbs were borrowed and reinvented because, through them, writing could come into its own: it became an imprint not just of content, but of interaction. The newly coined speech act-verbs incorporated representations of the gestural component of oral communication in the language of print. It thus enabled, among other new social achievements, to seal contracts with a signature which previously had required bodily gestures. What this means, then, is that gestures are not so much concomitant expressions of the same underlying model that is also expressed in speech act verbs, but that conventional gestures may have served as the models for the verbs.

Conclusion A number of different gestures thus cohere as a cluster of conduit gestures, and a cognitive linguist might say that these gestures, individually and collectively, express a conceptual metaphor, “talking is transfer of objects” (or some such equation). This is an intellectualist interpretation of embodied “image-schemata”: it is assumed that there is an independently existing, prior cultural or individual thought formation, expressed in the conceptual metaphor, which is articulated in the meta-communicative lexicon of languages (e.g. in verbs of speaking) as much as in symbolic motor-acts, i.e. gestures. Another way of making sense of the same phenomena is to treat the gestures as practical metaphorizations (or construals): what is given to the speaker is a fluid repertoire of abstract, schematic actions of the hands – actions that are “uncoupled from real-world consequences” and thus available for symbolic use. These schematic actions are inherently meaningful to the extent that they are related to full-fledged actions or action components that are familiar to the interactants. Not all postural and enactive schemata used in gesture are meaningful in this way, but it appears that they are frequent enough in the vocabulary of forms used in pragmatic gesturing. The motor patterns are meaningful not by what they express, but by what they are; it is their own meaningfulness that is projected onto the moment’s communication. For this interpretation – we can call it the kinesthetic or motor-cognition hypothesis – , the fact that the gestures are available to their makers as kinesthetic or proprioceptive motor-patterns is crucial. For, it allows us to hypothesize that, from the gesturer-speaker’s point of view, pragmatic gestures are a feedback mechanism that in effect tells the speaker what he or she is up to:

 Gesturecraft

gestures provide experiential frames – prehensile and motor frames – in terms of which our own communicative actions are tacitly made meaningful for us. To the extent to which our gestures share patterns with “real-life” motor acts, the neuronal patterns that shape our experience while gesturing may “remind us” of our non-symbolic, practical acts, and, in this fashion, make sense for us (in an entirely implicit, tacit way). It is a sense that comes from the body, rather than being just expressed by it.

chapter 

A sustainable art

In this book I have conceived of gesture neither as a sign-system nor as a part of language (or a “body language” onto itself) nor as expressive behavior (behavior that reveals what goes on in the person’s mind or pysche), but as a mode of communicative praxis and craft, comprising skills, methods, and techniques. Agreeing with the anthropologist Jackson, I contend that, insofar as the body tends to be defined as a medium of expression or communication, it is not only reduced to the status of a sign; it is also made into an object of purely mental operations. . . . [But] thinking and communicating through the body precede and to a great extent always remain beyond speech. (Jackson 1989: 122–3)

Gesture is an embodied “art which . . . cannot be transmitted by prescription, since no prescription for it exists” (Polanyi 1958: 53).1 We can assume that it is acquired through observation, imitation, and practice, as well as variation and invention by individual human bodies, who develop gestural signature or style in the process. Gesture is a personal set of skills, and it synthesizes practical, communicative, and cognitive functions: being able to index the surroundings or to depict a world by motions of the hands are also ways of knowing these worlds and of structuring them in meaningful ways so that others can reckon with them. Human beings do not inhabit a world that is distant and to which they relate in the manner of observation, but one that constantly requires and enables an agent’s active grasp. It is known in the first place through acts of physical manipulation. The criteria for knowing the world do not reside in the ability to adequately represesent it, but in the skillfullness with which it is handled. Many of our gestures reveal not how things look to us, but what we do with them, and they are connected to “everyday skills, discriminations, and practices into which we are socialized [and which] provide the conditions necessary for people to pick out objects, to understand themselves as subjects, and ... to make sense of the world and of their lives” (Dreyfus 1991: 4). Varela, Thompson, and Rosch (1991: 9) write in . There have, of course, been attempts to codify gestures and train orators and actors in proper gesticulation; see, for example, Quintilian (1922 [100]) and Barnett (1987); it is difficult to assess whether these regulations had any impact on everyday gesticulation.

 Gesturecraft

The Embodied Mind that cognition is “not the representation of a pregiven world by a pregiven mind but is rather the enactment of a world and a mind on the basis of a history of the variety of actions that a being in the world performs”. Our gestures show the world not as it is, but as we understand it by virtue of our practical engagements: ”perception and action are . . . inseparable in lived cognition” (173). Bourdieu uses the term hexis to pinpoint the connections between bodies, life-worlds, and praxis. He says that hexis “speaks directly to the motor function, in the form of a pattern of postures that is both individual and systematic, because linked to a whole system of techniques involving the body and tools, and charged with a host of social meanings and values” (Bourdieu 1977: 87). Noting that “most cultural knowledge is transmitted by way of the body (rather than words)”, the Danish anthropologist Hastrup emphasizes that it is not the biological body, but the body-in-life . . . or as we might wish to call it: the incarnate subject, . . . that is the locus of experience. (Hastrup 1995: 3)

My aim in this book has been to describe embodied subjects that inhabit life-worlds as the producers of gesture, and I have therefore described gesture as a craft, comprised of practices and skills. Of course, the focus in gesture is rarely on creativity or beauty or skill itself; gesture is about practicalities of situated communication, and not much else. The craftsman’s standards apply to gesture only in a loosened fashion; refinement is not measured, only situated success. And yet conceiving of gesture as a craft – or and art or techné – seems to be more empirically adequate than to construe it as a code or a part of language. McCullough gives a definition of craft that is particularly applicable to gesture: Craft . . . is habitual skilled practice with particular tools, materials, or media . . . applied toward practical ends. Craft is the application of personal knowledge to the giving of form. It is the condition in which the inherent qualities and economies of the media are encouraged to shape both process and products. It is not about standardized artifacts, however. . . . It is about the individually prepared artifact . . . and .. involves[s] the skilled hand. (McCullough 1996: 22)

Because there is no prescription for gesture, it is a flexible, sustainable craft, always replenishing itself by new solutions to new or familiar understanding tasks, in an incessant dialogue between rehearsed patterns and unplanned situated invention. More even than others forms of craft, gesture “focuses on the intimate connection between hand and head” (Sennett 2008: 9). Every good craftsman conducts a dialogue between concrete practices and thinking; this dialogue evolves into sustaining habits, and these habits establish a rhythm between problem solving and problem finding (9).

This is how G. H. Mead understood the evolution of gestures (and symbols and mind): as an ongoing production of solutions to interpersonal action and co-

Chapter 9. A sustainable art 

ordination problems; just like an animal’s communication repertoire, a human language and a person’s gesture repertoire are storehouses of standard solutions to recurrent interpersonal problems, especially problems of representation and interaction management. Gesture must have begun when our ancestors discovered that they could uncouple some of their actions from their environmental consequences, and that the resulting schematic component actions or action components were well suited to abstract various features from the environment for communicative purposes, for purposes of together understanding the world: gestures can abstract the path or acceleration from the perceived motion of a vehicle, the circumference from a rod, and the texture from a clutch (see Chapter 4). Abstracted motion can be used for an infinite variety of analytic or instructional purposes. We know, of course, that not all gesturing is “goal-directed” or “deliberate” or “conscious” to the same degree: it appears that depictions are usually performed with greater self-awareness – right in the focus of the eyes – than ceivings. Ceiving often is a form of thinking out loud by hand. It is communicative thinking alright – or thinking while speaking – , but these gestures are not deliberately made. Rather, they sometimes emerge when the speaker is able to relinquish control over the body and “let it speak”. I have distinguished six genres of gesture praxis by the ecological relations between self, other, speech, and world that are realized in each: gesturing about the world at hand; gesturing about the world in sight; gesturing about virtual worlds (depiction); gesture about the gesturer’s communicative acts, and gestures about the acts of the other. There surely must be other meaningful ways to slice up the cake, but this one has served us well and is clear-cut, notwithstanding the fact that, obviously, any given gesture event may have characteristics of more than one genre: I can gesturalize the completion of my turn and, by the same token of the hand, hand the turn over to you. I can also annotate the world at hand by a depiction (as Cita did in Chapter 5). And I can control your action by a pointing gesture. In general, whatever specific problem they address, gestures mediate the relationship between the individual, others, and the inhabited world, and they do so in a number of fundamentally different ways. This is the reason for explicating gesture ecologies. These are in part ecologies of perception, constraining who can perceive what at what time. “Ecology” also encompasses the precise co-production in time of words and gestures: for example, we can use gesture to prepare the ground for speech, to adumbrate meaning that is yet to fully be conveyed. For five of the genres/ecologies, I have offered examples. These, in all their heterogeneity, indicate that human societies have evolved numerous methods or routinized practices to deal with generic understanding and management tasks, and that these have developed along diverse paths, so that we may sometimes question the use of lumping them all together as “gestures”. They are things one can do with the hands in order to make a moment’s understanding. We found practices

 Gesturecraft

such as tracing, dis-assembling, and exploratory procedures; gestures of orientation, making use of distinct hand shapes; depiction methods; and motor acts by which pragmatic aspects of speech and speech action are conveyed. If we could count the practices that an average lively individual knows how to use, taking practices such as those described in Chapters 4, 6, and 7 as units, we would likely come up with a number in the hundreds, although some of these are problably rarely used. It will always be impossible by definition to give a complete account of gestural practices: new practices are evolving all the time, and there is no natural way to separate “common” from “specialized” ones, so as to describe the “standard”. The list of practices that I have studied should be taken as an invitation for further research. I have already indicated various lacunae in my study, and here are some more: –

–

–

I have only sampled a very small corner of the huge and vastly diverse domain of pragmatic gesticulation; for example, there has been work on topiccomment structures being marked by gesture (Kendon 1995b, 2004: 233–6; Seyfeddinipur 2003), on gesture practices used to mark cohesion, and other pragmatic dimensions of talk that are handled by gesture; in the chapter on depiction (Chapter 6), I have made very little of the fact that a great deal of gestural depiction is sequential depiction, imagery being built up over time. This gesture methodology – imagery building upon prior imagery – requires much more systematic analysis; I have almost entirely neglected how gestures are situated in and contribute to the intercorporeal dimension of social interaction: how gestures organize co-presence. Not only have I not studied touch and its gesturalization, I have also neglected the intercorporeal dimension of human life in my account of the corporeal and experiential foundations of gesture (and the treatment of the hand in Chapter 3). In other words, I have described the hand as if it only ever handled dead matter, but never other bodies such as those of lovers and children. This is a serious neglect, but it is, unfortunately, one that is characteristic of our entire field, from “embodied cognition” to cognitive linguistics to micro-ethnography: the paradigmatic importance of intercorporeality – of physical contact, care, love, and sexuality – for all human interaction systems has not even begun to be recognized, and it is not clear how our framework would change if, for example, it were possible to study human sexual interaction. Clearly, it represents a form of of corporeal intersubjectivity (“intercorporeité” in Merleau-Ponty’s term; Merleau-Ponty 1962) that is unlike other forms of embodied relatedness. It is the paradigmatic reciprocal sharing of bodily experience.

In the accessible outdoor world of gesturecraft, the gesture “codes” of certain professions make for particularly interesting subject-matter, allowing us to observe

Chapter 9. A sustainable art 

how this near-forgotten praxis comes into its own when communities need to represent and analyze bodily experience. When they talk about their work or their hobbies or other routine activities that require acquired physical skills, many professionals are apt to give abstract, gestural renditions of these skills during conversations. Their gesturing is almost an enactment of the very sensorimotor patterns that they also enact at work, but with some notable modifications: the absence of recalcitrant objects modifies the actions of the hand, compared to their “real life” involvements with matter. Practical skills gesturally displayed are communicative, not practical acts. And yet, the display would not be possible if the body giving the performance were not in possession of the skills. Sauer (2003) has analyzed the hand gestures that coal miners produce when they talk about hazardous situations and accidents. Being an experienced miner entails that one has developed embodied sensory knowledge of site-specific cognitive practices, which miners refer to as pit sense (219). It is this form of personal knowledge that is conveyed by the miners’ gestures and that cannot be articulated in so many words. Central to the miners’ gestural conveying of their lived experience of dangerous situations in the mines is the availability of both “first person” and “third person” – or character-viewpoint and observer-viewpoint – methods for depicting events (McNeill 1992). Sauer reports that miners use their entire bodies to mimetically portray characters and to re-enact events as they have subjectively experienced them. They use third-person (analytic) strategies to “represent abstract processes, objects, and conditions that would be literally beyond the reach of human perception” (Sauer 2003: 221). When individuals assume a mimetic viewpoint, they reenact events and conditions as if they were characters inside the spaces they describe. When they assume an analytic viewpoint, they assume a position outside of, above, or at a distance from the events, actions, and situations they observe. This distance is displayed by the speaker’s body: mimetic gestures frequently take place in the plane of the body, i.e., laterally. Analytic gestures take place at arm’s length from the observer, as if the hands were detached from the body, self-sufficient organs of representation (p. 221). Miners routinely alternate between the two viewpoints in their narratives of danger and accidents. This enables them, on the one hand, to re-enact and reexperience their own intense emotional involvement in situations of danger, and to give an empathetic display of their fearful, confused, or heroic actions, and, on the other, to analyze these same actions within an objectively construed context. Sauer points out that, “to assess risk, workers in risky occupations must actively move outside of their own embodied experience” (239), but to make risk palpable, for example to novice miners, it is also important that they be able to communicate their subjective experience. Shifting between both viewpoints enables speakers to re-analyze and modify their original actions and experiences and thus derive

 Gesturecraft

knowledge from it. Gesture allows speakers to distance themselves from their own experiences and to work through them, not only the mode of reflective thinking, but through concrete sensori-motor activity. It is the physicality of the gestures and the fact that they are experienced in kinesthetic perception by their makers – and not their cognitive content alone – that make this possible. We have thus moved a long distance from some of our initial questions, for example (in Chapter 6), how it is possible that motions of the hands produce pictures of the world. We have gradually subverted the terms of this question which, as we can now see, had unconvincingly favored the visual mode. Because gestures are visual phenomena for interlocutors and are often looked at and seen by the people making them, it is often falsely assumed that gesture is a medium which transforms visual experience into visual representations. Rather, as a medium of understanding, gesture incorporates haptic epistemology: it is driven by the body’s practical acquaintance with a tangible environment that it has forever explored, lived in, and modified. The beholder, the recipient of conversational gesture, also draws upon an undisclosed background of haptic understandings, couplings of motorschemata and things in the world; otherwise, they would be unable to recognize the action patterns that the gestures instantiate. We acquire our understanding of the world in large measure through actions of the hands, and this understanding includes knowledge that we rarely represent to ourselves and consciously reflect upon. It is this tacit knowledge that informs our gestures. Moreoever, this knowledge also informs our visual perception: our eyes see things in terms of how we can handle them, they see weight, texture, fragility, and so on (even though, of course, they are occasionally deceived). And it equips us with manipulation-schemata with which we can evoke the shared world for others as well. Gesture is also creative action. Gestures are the products of processes that are “a form of dialogue between a plastic subject and an equally plastic environment” (Belgrad 1998: 136). In other words, the form of the gesture emerges from a dialogue between the situated subject and the material patterns that the body subject suggests. The body is a locus of knowing that reveals its “recalcritant materiality” in “patterns of personal remembered physical co-ordinations” (Merce Cunningham, cited after Belgrad 1998: 162), in “rote combinations of movement conditioned by everyday life” (ibid.). The body talks back to us. In gesture “we realize (verwirklichen) ourselves, [and] we recognize the self that is being realized (Joas 1992: 122). Gesture can become a vehicle of spontaneous self-discovery. The best example of this possibility is the anxiety patient from Chapter 7, who is struggling to find adequate imagery for his psychic life, as this psychic life presents itself to himself when he reflects upon it. At one point, this patient complains about the pressure he is under.

Chapter 9. A sustainable art 

9. Angst 1

2

P

Da: will ich hin That’s where I want to go points backwards over shoulder __________|_________.... | Ich will wieder ohne diesen Druck den ichI want again without this pressure that I.......................

3

4 5

Naja, sehn Se? Well, you see? ..__________ | Ich zeig nach hinten. I’m pointing to the back. Mein Rücken ha- den ich hinten verspüre My back ha- that I feel in the back.

The speaker initiates gesture apparently unawares but then notices that he is making it as he is making it: he interrupts the gesture and inserts a solicitation of attention (“you see”) and a self-report (“I’m pointing to the back”) into his topical talk. This seems to be a case of proprioception reaching the threshold of awareness. In the gesture that he thus notices the patient recognizes himself: he owns up to the figuration, takes it as evidence of his inner state. This is gesturally mediated meta-cognition: the patient recognizes in his gesture an idea that he had, a conceptualization. As in word-searches, the gesture represents to the speaker what he knows, even though he may not be able to say it (Koriat 2000; Koriat & Levy-Sadot 2000). Gesture is creative action, a mixture of traditional forms, personal habit, and spontaneous invention. Inspecting gesture, we can also understand something about creativity in language. Humboldt wrote: “We must look at language, not as a dead product, but far more as a producing” (Humboldt 1836: 48). “Language [in itself] is no product (ergon), but an activity (energeia)” (49); and “linguistic form” means “method of language-making” (52). We can add: gesture form means method of forming a gesture. Speaking, the mode of existence of language, is energeia; speaking is activity that makes use of existing forms (ergon), sustaining these but also modifying them in the process. Speaking and gesturing are activities that always make their own resources – grammar, phonemes, words, and so on. These are products of communication, and then of of repetition and imitation and

 Gesturecraft

erosion and renewal. Gestures – those standard forms that can be “used” (because they already have been “made”) – are the sustained products of situated action. Gesture practices, then, are improvisational practices, practices that employ methods and use pre-made parts, but are neither guided by rules nor built from fixed vocabularies of form-meaning pairs. As many examples, beginning with Captain Cook’s encounters in Tannna, have shown, gestural communication is spontaneous and often fabricates its own devices spontaneously. In this respect gesture is indeed a prime example of energeia. We are beginning to understand how conventional signs in signed languages evolve from “home-signs” (Kegl, Senghas & Coppola 1999), that is, the originally spontaneous, subsequently habitualized signs used in families that have a deaf member. We also understand the nature of the process by which linguistic inventions (for example, metaphors) slowly become “grammaticalized” (Hopper & Traugott 1993). While utterances – and gestures – are more or less under the control of their makers – they are intentional products –, the words, idioms, constructions, and so on, that we also produce as a collective by-product of our speaking and interaction, are subject to processes and law-like trends that are beyond any speaker’s control. Keller (1990) has called these processes, following Adam Smith (1776), “invisible-hand” processes or “phenomena of the third kind”: A phenomenon of the third kind is the causal consequence of a multitude of individual intentional actions. (Keller 1990: 93)

Natural languages arise and change in part as a result of invisible hand processes. We must assume that gesture does, too: that, for example, culturally shared repertoires have evolved as a result of processes that result from the fact that individuals use similar forms for similar functions and that these forms, once they become part of shared repertoires, are subject to drift: increased frequency of use and factors of economy may lead to the erosion of forms, not unlike the erosion of forms that is characteristic as part of the grammaticalization of lexical forms in spoken and signed languages (Hopper & Traugott 1993; Liddell 2003). Apart from a few 19th century studies of metaphor in gesture and gestural contact-codes (de Jorio 2000 [1832]; Mallery 1978 [1880]), we know little about the processes by which gesture inventions become gesture conventions (but see Kendon 1988a). In his study of “the infinite art of improvisation”, Thinking in Jazz, Paul Berliner (1994) describes “fluent speakers of jazz” who “refer to the discrete patterns in their repertoire storehouses as vocabulary, ideas, licks, tricks, pet patterns, crips, clichés, and . . . things you can do” (p. 102). Among the repertoire are things that “derive from the common language of jazz” and are “the kinds of things that everybody plays” (ibid.). Others are signature licks adopted from mentors, yet others personal inventions. “Learners must cultivate the ability to grasp precise gestures within the contours of fluent improvisations” (Berliner 1994: 104).

Chapter 9. A sustainable art

This is a good metaphor for the situation of gesturers and their medium. What makes it metaphorical is the fact that gesture is not literally an art: it is not meant to be appreciated for its esthetic value, and adult gesturers rarely study other gesturers to expand their own repertoire. But children do. When they are two and a half they begin to perform certain gestures that they have observed, often with demonstrated awareness of the performative character of their actions. They learn about types of communicative acts and their own intentions by recognizing and reflecting on their own gestures, as well as those of others. This is one of the affordances of our arms and hands: they let us see ourselves gesture. An adult repertoire always comprises culturally shared forms, local (family etc.) conventions, and idiosyncratic things one can do. The “jazz situation”, as we can call it, is the situation of the creatively communicating individual, “under the pressures of thinking in motion” (Berliner 1994: 205) and interacting with the medium: picking up what tradition offers, adapting it to local circumstance, noticing how it comes out, and only realizing in retrospect what one was up to. Of course the focus on gesture is rarely on creativity itself, or on beauty; gesture is about utility. And yet, we have seen moments when people ostensibly come face-to-face with themselves through their gestures: they discover their implicit communicative intent. Gesturecraft is not a closed system, it is not governed by rules and largely left alone in everyday life. Its creativity lives almost entirely underground: visible, yet undisclosed. In this fashion, improvisational gesture jazz will continue to provide insight and understanding and be of service to the growth of knowledge, just like it was for Captain Cook and Georg Forster and the rest of their party, when they met the gesturer Fanokko on the remote island of Tanna (Forster 1777: 287–8): It happened that my father and Captain Cook, on comparing their vocabularies, discovered that each had collected a different word to signify the sky; they appealed to Fannokko, to know which of the two expressions was right; he presently held out one hand, and applied it to one of the words, then moving the other hand under it, he pronounced the second word; intimating that the upper was properly the sky, and the lower the clouds which moved under it.





Data and transcript conventions

These are the data-sets, roughly in the order in which they were collected: –

–

–

–

– –

dyadic conversations, recorded in West Berlin 1986-87, mainly in the author’s or his students’ apartments, and including conversations in German (Penthesile and BWB), French (Mignon), Japanese (Car Wrecks), Thai (Thai), Turkish/German (TRK), and the Philippine language Ilokano (ILO). Cornelia Müller and Ulrike Hartge made some of these recordings; Elnora Held, Cornelia Müller, Satomi Ogawa, Janchai Yingprayoon, and the author produced transcripts and/or translations of the foreign-language data. The work was supported by a grant from the Freie Universität Berlin to the author. field recordings in Nueva Vizcaya province, island of Luzon, Philippines, in the course of linguistic research on language typology, gesture, and interaction in the language Ilokano, including Cita & Cito, a conversation between a ricefarmer and a visitor to her farm about irrigation and Rumors, a conversation between three older women in front of their house. The recordings were facilitated by the Alindayu family in Bayombong/Nueva Vizcaya and Bonifacio Ramos of St.Mary’s University, Bayombong, and transcribed and translated by Elnora Held, Teodora Cordova, and Julius Ordoñez; the research was made possible by a grant from the German Science Foundation (DFG) to the author. dyadic conversations among US-American students in Austin/Texas (Girlfriends and Heather); the recordings were made by Peggy Yang and the author and facilitated by a small grant from the University of Texas. Hi-Tech Auto, an 11-hour video-recording of one day in the working life of the Lebanese-born owner of a car-repair shop in Austin/Texas, made by the author; Vernissage, an incidental recording of the visit of two German state-governors to an exhibition of modern art, in Potsdam/Germany, made by the author; Angst, an in-patient short-term psychotherapy session with a man suffering from bulimia and panic attacks, recorded as part of the research grant Soziale Ängste, by Ulrich Streeck, Medical Director of the psychotherapeutic hospital Tiefenbrunn in Göttingen/Germany; the data were studied when the author was a fellow at the Center for Interdisciplinary Research (ZiF) at the Universität Bielefeld/Germany and participated in a research group on the

 Gesturecraft

–

–

communicative expression and clinical representation of anxiety during Summer 2004; data collection was made possible through a grant from the German Science Foundation to Ulrich Streeck; Architect, ethnographic video-recordings in the studio of, and interviews with, the architect M. J. Neal in Austin/Texas; the recordings were made by Chris Koenig and Siri Mehus; and segments from The Tonight Show, PTL Club, and a self-help video on codependency.

With some exceptions (e.g. the psychotherapy session), the names in the transcripts are the participants’s real names; they did not request that they be changed.

Transcription conventions In this book, the system for transcribing talk developed by Gail Jefferson for conversation analysis is used with some additions to represent gesture and gaze. These additions vary according to the degree of detail that is required to explain a given segment: when the exact coordination of gesture and speech is at issue, more detail is required than when we want to know about and explain the succession of different gestures over a longer segment of discourse. These are the conventions used in the first kind of case: bold face ::: (--) [ ___|___ | |

__........__ | ∼∼∼∼

·

indicates stress; colons signify vowel-elongation; bracketed dashes indicate pauses in tenths of seconds; square brackets mark the beginning of simultaneous talk (overlap); a dash indicates that a word or syllable is cut off; horizontal square brackets above an utterance indicate (the duration of) a gesture; where labels are included they identify either the type of motion or the assumed meaning-in- context of the gesture; dots in a gesture-bracket indicate that a movement is temporarily ‘frozen’; indicates that a participant’s gaze is focused upon the speaker’s hands (unless otherwise indicated, the person gazing is the speaker herself); a dot in a line of transcript marks the exact moment that is represented by the drawing that accompanied the transcript.

The drawings were made by Jürgen Grothues (Berlin) and Jacob Villanueva (Austin).

Bibliography

Armstrong, D. F., W. C. Stokoe, & S. E. Wilcox (1995). Gesture and the Nature of Language. Cambridge: Cambridge University Press. Armstrong, D. F., & S. E. Wilcox (2007). The Gestural Origin of Language. Oxford: Oxford University Press. Arnheim, R. (1969). Visual Thinking. Berkeley: University of California Press. Ashley, R. (2005). A neo-Gricean framework for understanding music conducting. Paper presented at the Second Conference of the International Society for Gesture Studies, Lyon, France. Banks, M., & H. Morphy (1997). Rethinking visual anthropology. New Haven: Yale University Press. Barnett, D. (1987). The Art of Gesture. The Practices and Principles of 18th Century Acting. Heidelberg: Carl Winter University Press. Barresi, J., & Moore, C. (1996). Intentional relations and social understanding. Behavior and Brain Sciences, 19, 107–154. Barakat, R. A. (1972). Cistercian Sign Language. In D. J. Umiker-Sebeok & T. Sebeok (Eds.), Monastic Sign Languages (pp. 67–322). Berlin: Mouton de Gruyter. Barsalou, L. W. (1999). Perceptual symbol systems. Behavior and Brain Sciences, 22 (4), 577–609. Bateson, G. (1949). Bali: The value system of a steady state. In M. Fortes (Ed.), Social structure: Studies presented to A. R. Radcliffe-Brown (pp. 33–53). Oxford: Clarendon Press. Bateson, G. (1956). The message ‘this is play’. In B. Schaffner (Ed.), Group Processes: Transactions of the Second Conference (pp. 145–242). New York: Josia Macy, Jr. Foundation. Bateson, G. (1971). Communication. In N. McQuown (Ed.), The natural history of an interview. Chicago: University of Chicago Microfilm Library Microfilm. Bateson, G. (1972 [1956]). A theory of play and fantasy. In Steps to an Ethology of Mind (pp. 177–193). New York: Ballantine. Bateson, G., & M. Mead (1942). Balinese Character. A Photographic Analysis. New York: New York Academy of Sciences. Bavelas, J., N. Chovil, D. A. Lawrie & A. Wade (1992). Interactive gestures. Discourse Processes, 15, 469–489. Beattie, G. (2004). Visible Thought: The New Psychology of Body Language. London: Routledge. Belgrad, D. (1998). The Culture of Spontaneity. Improvisation and the Arts in Postwar America. Chicago: University of Chicago Press. Bell, C. (1885). The Hand: Its Mechanism and Vital Endowments. London: George Bell & Sons. Bergen, B., S. Narayan, & J. Feldman (2003). Embodied verbal semantics: Evidence from an image-verb matching task. In R. Alterman & D. Kirsh (Eds.), Proceedings of the TwentyFifth Annual Conference of the Cognitive Science Society (pp. ??–??). Mahway, NJ: Lawrence Erlbaum.

 Gesturecraft

Bergen, B., & K. Wheeler (2005). Sentence understanding engages motor processes. In B. G. Bara, L. Barsalou, & M. Bucciarelli (Eds.), Proceedings of the Twenty-Seventh Annual Conference of the Cognitive Science Society. Mahway, NJ: Lawrence Erlbaum. Bergson, H. (1956 [1899]). Laughter. Baltimore: Johns Hopkins University Press. Berliner, P. F. (1994). Thinking in Jazz. Chicago: Chicago University Press. Birdwhistell, R. (1972). A kinesic-linguistic exercise. In J. J. Gumperz & D. Hymes (Eds.), Directions in Sociolinguistics (pp. 381–404). New York: Holt, Rinehart, and Winston. Birdwhistell, R. L. (1979). Toward analyzing American movement. In S. Weitz (Ed.), Nonverbal Communication: Readings with Commentary. New York: Oxford University Press. Bolden, G. (2003). Multiple modalities in collaborative turn-sequences. Gesture, 3, 187–211. Boonzajer Flaes, R. M., & D. A. Harper (1993). Eyes across the water, II: Essays on visual anthropology and sociology. Amsterdam: Spinhuis. Bourdieu, P. (1977). Outline of a Theory of Practice. Cambridge: Cambridge University Press. Brookes, H. J. (2001). O clever ‘He’s streetwise’. When gestures become quotable: The case of the clever gesture. Gesture, 1, 167–184. Brookes, H. J. (2004). A repertoire of South African quotable gestures. Journal of Linguistic Anthropology, 14 (2), 186–224. Brookes, H. J. (2005). What gestures do: Some communicative functions of quotable gestures in conversations among Black urban South Africans. Paper presented at the Second Conference of the International Society for Gesture Studies, Lyon, France. Bruner, J. (1990). Acts of Meaning. Cambridge, MA: Harvard University Press. Bühler, K. (1978 [1934]). Sprachtheorie. Frankfurt/M.: Ullstein. Calbris, G. (1990). The Semiotics of French Gestures. Bloomington: Indiana University Press. Calbris, G. (2003). From cutting an object to a clear cut analysis: Gesture as the representation of a preconceptual schema linking concrete actions to abstract notions. Gesture, 3, 19–46. Chaiklin, S., & J. Lave (Eds.). (1993). Understanding Practice. Perspectives on Activity and Context. Cambridge: Cambridge University Press. Chomsky, N. (1965). Aspects of the Theory of Syntax. Cambridge, MA: MIT Press. Clack, J. A. (2002). Gaining Ground: The Origin and Evolution of Tetrapods. Bloomington, IN: Indiana University Press. Clark, H. H., & S. Brennan (1991). Grounding in communication. In L. B. Resnick, J. M. Levine & S. D. Teasley (Eds.), Perspectives on Socially Shared Cognition. Washington, DC: American Psychological Association. Cole, M., Y. Engeström, & O. Vasques (1997). Mind, Culture, and Activity. Cambridge: Cambridge University Press. Cohen, E., L. Namir, & I. M. Schlesinger (1977). A New Dictionary of Sign Language. The Hague: Mouton. Collier, J. J., & M. Collier (1986). Visual Anthropology. 2nd edition. Albuquerque: University of New Mexico Press. Comolli, A. (1991). Les Gestes du Savoir. 2nd edition. Paris: Formation des Recherches Cinématographiques, U.E.R. des Sciences Sociales, Université Paris X, Nanterre. Comolli, A. (1993). Using film to investigate the transmission of learning. Visual Anthropology, 6 , 45–64. Comolli, A. (2003). Elements de methode en anthropologie filmique. In A. Comolli & C. de France (Eds.),Travaux en anthropologie filmique (pp. 5–44). Paris: Publidix. Comolli, A., & C. de France (Eds.). (2005). Corps filme, corps filmant. Paris: Publidix. Connerton, P. (1989). How Societies Remember. Cambridge: Cambridge University Press.

Bibliography 

Corballis, M. C. (2002). From Hand to Mouth: The Origins of Language. Princeton: Princeton University Press. Coulter, J., & E. D. Parsons (1990). The praxiology of perception: Visual orientations and practical action. Inquiry, 33 (3), 251–272. de Fornel, M. (1992). The return gesture: Some remarks on context, inference, and iconic gesture. In P. Auer & A. di Luzio (Eds.), The Contextualization of Language. Amsterdam: John Benjamins. de France, C. (1977). Foundations for a filmic anthropology. Paper presented at the Symposium on Visual Anthropology. Xth International Congress of Anthropological and Ethnological Sciences, Delhi. de France, C. (1983). L’analyse praxéologique: Composition, ordre et articulation d’un procès. Paris: Editions de la Maison des Sciences de l’Homme. de France, C. (1989). Cinéma et Anthropologie (2nd edition). Paris: Editions de la Maison des Sciences de l’Homme. DeJorio, A. (2000 [1832]). Gesture in Naples and Gesture in Classical Antiquity (Translated and edited by A. Kendon). Bloomington, IN: Indiana University Press. de Ruiter, J. P. (2000). The production of gesture and speech. In D. McNeill (Ed.), Language and Gesture (pp. 284–311). Cambridge: Cambridge University Press. Diller, A.-M. (1991). The conceptual structure of linguistic action verbs in Bahasa Indonesia. Cognitive Linguistics, 2 (3), 225–245. Ditton, J. (1980). The View from Goffman. New York: St. Martin’s Press. Donald, M. (1991). Origins of the Modern Mind. Cambridge, MA: Harvard University Press. Drew, P., & A. Wootton (Eds.). (1988). Erving Goffman: Studying the Interaction Order. Oxford: Polity Press. Dreyfus, H. L. (1991). Being-in-the-World. A Commentary on Heidegger’s “Being and Time”. Cambridge, MA: The MIT Press. Efron, D. (1972 [1941]). Gesture, race and culture. The Hague: Mouton. Egbert, M., & J. Bergmann (2004). Angst – Von der Phänomenologie zur Interaktion. Psychotherapie und Sozialwissenschaft, 6, 227–242. Ekman, P., & W. V. Friesen (1969). The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1, 49–98. Enfield, N. J. (2001). ‘Lip-pointing’: A discussion of form and function with reference to data from Laos. Gesture, 1, 185–212. Enfield, N. J. (2004). On linear segmentation and combinatorics in co-speech gesture: A symmetry-dominance construction in Lao fish trap descriptions. Semiotica, 149 (1), 57– 123. Engeström, Y. (1987). Learning by Expanding: An Activity-Theoretical Approach to Developmental Research. Helsinki: Orienta-Konsultit Oy. Erickson, F. (1971). The cycle of situational frames: A model for microethnography. Paper presented at the Midwest Anthroplogy Meeting, Detroit, MN. Erickson, F. (2004). Talk and Social Theory. Cambridge: Polity Press. Fauconnier, G. (1997). Mappings in Thought and Language. Cambridge: Cambridge University Press. Fine, G. A., & G. W. H. Smith (2000). Erving Goffman. London: Sage. Foçillon, H. (1992 [1934]). The Life of Forms in Art. New York: Zone Books. Fodor, J. A. (1975). The Language of Thought. Cambridge, MA: Harvard University Press. Forster, G. (1777). A Voyage round the World. London.

 Gesturecraft

Freedman, N. (1977). Hand, word, and mind: On the structuralization of body movement and the capacity for verbal representation. In N. Freedman & S. Grand (Eds.), Communicative Structures and Psychic Structures. A Psychoanalytic Interpretation of Communication (pp. 109–133). New York: Plenum Press. Gallese, V., & G. Lakoff (2005). The brain’s concepts: The role of the sensory-motor system in conceptual knowledge. Cognitive Neuropsychology, 22 (3/4), 455–479. Gardner, H. (1985). The Mind’s New Science. New York: Basic Books. Garfinkel, H. (1967). Studies in Ethnomethodology. Englewood Cliffs, NJ: Prentice-Hall. Gehlen, A. (1988 [1958]). Man. His Nature and Place in the World. New York: Columbia University Press. Gibson, J. J. (1962). Observations on active touch. Psychological Review, 69 , 477–491. Gibson, J. J. (1966). The Senses Considered as Perceptual Systems. Boston: Houghton Mifflin. Gibson, J. J. (1986). The Ecological Approach to Visual Perception. Hillsdale, NJ: Lawrence Erlbaum. Goethe, J. W. von (1953). Joseph Bossi über Leonard da Vincis Abendmahl. In Goethes Werke (Vol. XII, pp. 164–168). Hamburg: Christian Wegner. Goffman, E. (1959). The Presentation of Self in Everyday Life. Garden City, NY: Doubleday. Goffman, E. (1961). Encounters. Indianapolis, IN: Bobbs-Merrill. Goffman, E. (1974). Frame Analysis. New York: Harper & Row. Goffman, E. (1981). Forms of talk. Oxford: Blackwell. Goffman, E. (1983). The interaction order. American Sociological Review, 48, 1–17. Golato, A. (2000). Und ich so / und er so [and I’m like / and he’s like]: An innovative German quotative for reporting on embodied actions. Journal of Pragmatics, 32 (1), 29–54. Goldin-Meadow, S. (2003). Hearing gesture: How our hands help us think. Cambridge, MA: Belknap Press of Harvard University Press. Goldin-Meadow, S., M. W. Alibali, & R. B. Church (1993). Transitions in concept acquisition: Using the hand to read the mind. Psychological Review, 100, 279–297. Goldin-Meadow, S., & C. Butcher (2003). Pointing toward two-word speech in young children. In S. Kita (Ed.), Pointing: Where Language, Culture and Cognition Meet (pp. 85–108). Mahwah, NJ: Lawrence Erlbaum. Goldin-Meadow, S., & D. McNeill (1999). The role of gesture and mimetic representation in making language the province of speech. In M. C. Corballis & S. Lea (Eds.), The Descent of Mind (pp. 155–172). Oxford: Oxford University Press. Gombrich, E. H. (1961). Art and Illusion: A Study in the Psychology of Pictorial Representation (2 ed.). Princeton NJ: Princeton University Press. Goodman, N. (1976 [1968]). Languages of Art (2 ed.). Indianapolis: Hackett. Goodman, N. (1978). Ways of Worldmaking. Indianapolis: Hackett. Goodwin, C. (1980). Restarts, pauses, and the achievement of a state of mutual gaze at turnbeginning. Sociological Inquiry, 50, 272–302. Goodwin, C. (1981). Conversational Organization: Interaction between Speakers and Hearers. New York: Academic Press. Goodwin, C. (1986). Gesture as a resource for the organization of mutual orientation. Semiotica, 62(1–2), 29–49. Goodwin, C. (1993). Recording Human Interaction in Natural Settings. Pragmatics, 3 (2), 181– 209. Goodwin, C. (1994). Professional Vision. American Anthropologist, 96 (3), 606–633. Goodwin, C. (1995). Co-constructing meaning in conversations with an aphasic man. Research on Language and Social Interaction, 28 (3), 233–260.

Bibliography 

Goodwin, C. (2000a). Gesture, aphasia, and interaction. In D. McNeill (Ed.), Language and Gesture (pp. 84–98). Cambridge: Cambridge University Press. Goodwin, C. (2000b). Practices of seeing. Visual analysis: An ethnomethodological approach. In T. Van Leeuwen & C. Jewitt (Eds.), Handbook of Visual Analysis. London: Sage. Goodwin, C. (2002). Multi-Modal Gesture. Paper presented at the First Conference of the International Society for Gesture Studies, Austin, TX. Goodwin, C. (2003). The body in action. In J. Coupland & R. Gwyn (Eds.), Discourses of the Body (pp. 19–42). New York: Palgrave/Macmillan. Goodwin, C. (2004). A competent speaker who can’t speak: The social life of aphasia. Journal of Linguistic Anthropology, 14 (2), 151–170. Goodwin, C. (2007). Environmentally coupled gestures. In S. D. Duncan, J. Cassell & E. T. Levy (Eds.), Gesture and the dynamic dimension of language: Essays in honor of David McNeill (pp. 195–212). Amsterdam: John Benjamins. Goodwin, C., & M. H. Goodwin (1986). Gesture and coparticipation in the activity of searching for a word. Semiotica, 62 (1–2), 51–75. Gullberg, M. (1998). Gesture as a Communication Strategy in Second Language Discourse. Lund: Lund University Press. Gullberg, M. (2003). Eye movements and gestures in human face-to-face interaction. In J. Hyönä, R. Radach & D. H. (Eds.), The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research. London: North-Holland/Elsevier. Gullberg, M., & K. Holmqvist (1999). Keeping an eye on gestures: Visual perception of gestures in face-to-face communication. Pragmatics and Cognition, 7, 35–63. Gullberg, M., & K. Holmqvist (2003). What speakers do and what addressees look at. Visual attention to gestures in human interaction live and on video. Pragmatics and Cognition, 14, 53–83. Hadar, U., R. Dar, & A. Teitelman (2001). Gesture during speech in first and second language: Implication for lexical retrieval. Gesture, 1, 151–166. Hanks, W. F. (1990). Referential Practice: Language and Lived Space among the Maya. Chicago: University of Chicago Press. Hanks, W. F. (1996a). Language and Communicative Practices. Boulder, CO: Westview Press. Hanks, W. F. (1996b). Language form and communicative practices. In J. J. Gumperz & S. C. Levinson (Eds.), Rethinking Linguistic Relativity (pp. 232–270). Cambridge: Cambridge University Press. Harper, D. (1987). Working Knowledge. Skill and Community in a Small Shop. Chicago: University of Chicago Press. Hastrup, K. (1995). Incorporated Knowledge. Mime Journal, 17, 2–9. Haviland, J. B. (1993). Anchoring, iconicity, and orientation in Guugu Yimidhirr pointing gestures. Journal of Linguistic Anthropology, 3 , 3–45. Haviland, J. B. (1996). Projections, transpositions, and relativity. In J. J. Gumperz & S. C. Levinson (Eds.), Rethinking Linguistic Relativity (pp. 271–323). Cambridge: Cambridge University Press. Haviland, J. B. (2000). Pointing, gesture spaces, and mental maps. In D. McNeill (Ed.), Language and Gesture (pp. 13–46). Cambridge: Cambridge University Press. Haviland, J. B. (2002). How to point in Zinancantan. In S. Kita (Ed.), Pointing: Where Language, Culture, and Cognition Meet (pp. 39–70). Mahwah, NJ: Lawrence Erlbaum. Heath, C. (1982). The display of recipiency: An instance of a sequential relationship in speech and body movement. Semiotica, 42 (2–4), 147–167.

 Gesturecraft

Heath, C. (1984). Talk and recipiency: Sequential organization in speech and body movement. In M. Atkinson & J. Heritage (Eds.), Structures of Social Action (pp. 247–266). Cambridge: Cambridge University Press. Heath, C. (1986). Body Movement and Speech in Medical Interaction. Cambridge: Cambridge University Press. Heath, C. (1992). Gesture’s discreet tasks: Multiple relevancies in visual conduct and in the contextualisation of language. In P. Auer & A. diLuzio (Eds.), The Contextualization of Language (pp. 101–128). Amsterdam: John Benjamins. Heidegger, M. (1962 [1926]). Being and Time. New York: Harper and Row. Heine, B., & T. Kuteva (2002). World Lexicon of Grammaticalization. Cambridge: Cambridge University Press. Henderson, K. (1999). On line and on paper: Visual representations, visual culture, and computer graphics in design engineering. Cambridge, MA: MIT Press. Heritage, J. (1984). Garfinkel and Ethnomethodology. Cambridge: Polity Press. Hewes, G. (1974). Gesture language in culture contact. Sign Language Studies, 4 , 1–34. Hockings, P. (1995). Principles of visual anthropology (2nd ed.). Berlin: Mouton de Gruyter. Holler, J., & G. Beattie (2002). A micro-analytic investigation of how iconic gestures and speech represent the core semantic features in talk. Semiotica, 142(1/4), 31–69. Hopper, P. J., & E. C. Traugott (1993). Grammaticalization. Cambridge: Cambridge University Press. Humboldt, W. von (1988 [1836]). On Language. Cambridge: Cambridge University Press. Hutchins, E. (2006). The distributed cognition perspective on human interaction. In N. J. Enfield & S. C. Levinson (Eds.), Roots of Human Sociality (pp. 375–398). London: Berg. Hutchins, E., & L. Palen (1997). Constructing meaning from space, gesture, and speech. In L. Resnick, R. Säljö, C. Pontecorvo & B. Burge (Eds.), Discourse, Tools, and Reasoning (pp. 23–40). New York: Springer Verlag. Huxley, J. (1966). Ritualization of behaviour in animals and man. Paper presented at the Proceedings of the Royal Society of London. Ingold, T. (2000). The Perception of the Environment: Essays on Livelihood, Dwelling and Skill. London: Routledge. Jackson, M. (1989). Paths Toward a Clearing. Radical Empricism and Ethnographic Inquiry. Bloomington, IN: Indiana University Press. Jeannerod, M. (1984). The timing of natural prehension movements. Journal of Motor Behavior, 16 (3), 235–254. Jeannerod, M. (1997). The Cognitive Neuroscience of Action. Oxford: Blackwell. Joas, H. (1992). Die Kreativität des Handelns. Frankfurt/M.: Suhrkamp. Johnson, M. (1987). The Body in the Mind. Chicago: University of Chicago Press. Jones, S. E., & D. H. Zimmerman (2003). A child’s point and the achievement of intentionality. Gesture, 3, 155–186. Katz, D. (1925). Der Aufbau der Tastwelt. Leipzig: Barth. Kegl, J., A. Senghas, & M. Coppola (1999). Creation through contact: Sign language emergence and sign language change in Nicaragua. In M. de Graff (Ed.), Language Creation and Language Change: Creolization, Diachrony and Development (pp. 179–237). Cambridge, MA: MIT Press. Keller, C. M., & J. D. Keller (1996). Cognition and Tool Use: The Blacksmith at Work. Cambridge: Cambridge University Press.

Bibliography 

Keller, R. (1990). Sprachwandel: Von der unsichtbaren Hand in der Sprache. Tübingen: UTB Francke. Kendon, A. (1967). Some functions of gaze direction in two-person conversation. Acta Psychologica, 26, 22–63. Kendon, A. (1970). Movement coordination in social interaction: Some examples described. Acta Psychologica, 32 , 100–125. Kendon, A. (1972). Some relationships between body motion and speech. In A. Seigmann (Ed.), Studies in Dyadic Communication (pp. 177–210). Elmsford, NY: Pergamon Press. Kendon, A. (1979a). Some emerging features of face-to-face interaction studies. Sign language studies, 22 , 7–22. Kendon, A. (1979b). Some theoretical and methodological aspects of the use of film in the study of social interaction. In G. P. Ginsburg (Ed.), Emerging Strategies in Social Psychological Research. New York: Wiley. Kendon, A. (1980a). A description of a deaf-mute sign language from the Enga Province of Papua New Guinea with some comparative discussion. Part II: The semiotic functioning of Enga signs. Semiotica, 32 , 81–117. Kendon, A. (1980b). A description of a deaf-mute sign language from the Enga Province of Papua New Guinea with some comparative discussion. Part III: Aspects of utterance construction. Semiotica, 32 , 245–313. Kendon, A. (1980c). A description of a deaf-mute sign language from the Enga Province of Papua New Guinea with some comparative discussion. Part III: Aspects of utterance construction. Semiotica, 32, 245–313. [AdL: what is the difference between this and 1980b?] Kendon, A. (1980d). Gesticulation and speech: Two aspects of the process of utterance. In M. R. Kay (Ed.), The Relationship between Verbal and Nonverbal Behavior. The Hague: Mouton. Kendon, A. (1985). Uses of gesture. In D. Tannen & M. Saville-Troike (Eds.), Perspectives on Silence. Norwood, NJ: Ablex. Kendon, A. (1988a). How gestures can become like words. In F. Poyatos (Ed.), Crosscultural Perspectives in Nonverbal Communication (pp. 131–141). Toronto: C.J. Hogrefe. Kendon, A. (1988b). Sign Languages of Aboriginal Australia. Cambridge: Cambridge University Press. Kendon, A. (1988c). Goffman’s approach to face-to-face interaction. In P. Drew & A. Wootton (Eds.), Erving Goffman: Exploring the Interaction Order (pp. 15–40). Cambridge: Polity Press. Kendon, A. (1989). The place of kinesic components in an oration at a Te in the Enga province of Papua New Guinea. Paper presented at the Annual Meetings of the American Anthropological Association, Washington, D.C. Kendon, A. (1990). Conducting Interaction. Cambridge: Cambridge University Press. Kendon, A. (1992a). Abstraction in gesture. Semiotica, 90 (3–4), 225–250. Kendon, A. (1992b). The negotiation of context in face-to-face interaction. In A. Duranti & C. Goodwin (Eds.), Rethinking Context: Language as an Interactive phenomenon (pp. 323– 334). Cambridge: Cambridge University Press. Kendon, A. (1995a). Andrea de Jorio – The first ethnographer of gesture? Visual Anthropology, 7, 371–390. Kendon, A. (1995b). Gestures as illocutionary and discourse structure markers in Southern Italian conversation. Journal of Pragmatics, 23 (3), 247–279. Kendon, A. (2004). Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. Kendon, A., & S. J. Sigman (1996). Ray L. Birdwhistell 1918–1994. Semiotica, 112 (3/4), 231–261.

 Gesturecraft

Kendon, A., & L. Versante (2003). Pointing by hand in “Neapolitan”. In S. Kita (Ed.), Pointing: Where Language, Culture, and Cognition Meet (pp. 109–138). Mawah, NJ: Lawrence Erlbaum. Kimbara, I. (2006). On gestural mimicry. Gesture, 6, 39–62. Kirsh, D. (1995). Complementary strategies: Why we use our hands when we think. In J. D. Moore & J. F. Lehman (Eds.) Proceedings of the Seventeenth Annual Conference of the Cognitive Science Society (pp. 212–217). Mahwah, NJ: Lawrence Erlbaum. Kita, S. (1993). Language and Thought Interface: A Study of Spontaneous Gestures and Japanese Mimetics. Ph. D. Dissertation, University of Chicago. Kita, S. (2000). How representational gestures help speaking. In D. McNeill (Ed.), Language and Gesture: Window into Thought, Language, and Action (pp. 162–185). Cambridge: Cambridge University Press. Kita, S., I. van Gijn, & H. van der Hulst (1998). Movement Phases in signs and co-speech gestures, and their transcription by human coders. In I. Wachsmuth & M. Fröhlich (Eds.), Gesture and sign language in human-computer interaction (pp. 23–35). Berlin: Springer. Klima, E., & U. Bellugi (1979). The Signs of Language. Cambridge, MA: Harvard University Press. Knoblich, G., I. M. Thornton, M. Grosjean, & M. Shiffrar (Eds.). (2006). Human Body Perception from the Inside Out. Oxford: Oxford University Press. Koriat, A. (2000). The feeling of knowing: Some metatheoretical implications for consciousness and control. Consciousness and Cognition, 9, 149–171. Koriat, A., & R. Levy-Sadot (2000). Conscious and unconscious metacognition: A rejoinder. Cosciousness and Cognition, 9, 193–202. Koschmann, T., & C. D. LeBaron (2002). Learner articulation as interactional achievement: Studying the conversation of gesture. Cognition and Instruction, 20 (2), 249–282. Koschmann, T., C. D. LeBaron, C. Goodwin, A. Zemel, & G. Dunnington (2007). Formulating the triangle of doom. Gesture, 7, 97–117. Kuklick, H. (2008). A New History of Anthropology. Oxford: Blackwell. Lakoff, G., & M. Johnson (1999). Philosophy in the Flesh. Chicago: University of Chicago Press. Lakoff, G., & R. E. Nunez (2000). Where mathematics comes from: How the embodied mind brings mathematics into being. New York: Basic Books. Langacker, R. W. (1986). An introduction to cognitive grammar. Cognitive Science, 10, 1–40. Langacker, R. W. (1987). Foundations of Cognitive Grammar. Vol.1. Theoretical Prerequisites. Standford, CA: Stanford University Press. Langacker, R. W. (1988). An overview of cognitive grammar. In B. Rudzka-Ostyn (Ed.), Topics in Cognitive Linguistics (pp. 3–48). Amsterdam: John Benjamins. Langacker, R. W. (1990). Concept, Image, and Symbol. The Cognitive Basis of Grammar. Berlin: Mouton de Gruyter. Lave, J. (1988). Cognition in Practice. Cambridge: Cambridge University Press. Lave, J. (1991). Situating learning in communities of practice. In L. B. Resnick, J. M. Levine & S. D. Teasley (Eds.), Perspectives on Socially Shared Cognition (pp. 63–83). Washington, DC: American Psychological Association. LeBaron, C. D., & J. Streeck (2000). Gestures, knowledge, and the world. In D. McNeill (Ed.), Language and Gesture (pp. 118–138). Cambridge: Cambridge University Press. Lederman, S. J., & R. L. Klatzky (1987). Hand movements: A window into haptic object recognition. Cognitive Psychology, 19, 342–368. Leeds-Hurwitz, W. (1987). The social history of The Natural History of an Interview: A multidisciplinary investigation of social communication. Research on Language and Social Interaction, 20, 1–51.

Bibliography 

Leeds-Hurwitz, W. (2005). The natural history approach: A Bateson legacy. Cybernetics and Human Knowing, 12(1–2), 137–146. Lerner, G. (2002). Turn-sharing: The choral co-production of talk-in-interaction. In C. Ford, B. Fox & S. A. Thompson (Eds.), The Language of Turn and Sequence (pp. 225–256). Oxford: Oxford University Press. Lessing, G. E. (1962 [1766]). Laocoon: An Essay on the Limits of Painting and Poetry (E. A. McCormick, Trans.). New York: Bobbs-Merrill. Levinson, S. C. (1996). Language and space. Annual Review of Anthropology, 25, 353–382. Levinson, S. C. (1997). From outer to inner space: Linguistic categories and nonlinguistic thinking. In J. Nuyts & E. Pederson (Eds.), Language and Conceptualization (pp. 13–45). Cambridge: Cambridge University Press. Levinson, S. C. (2003). Space in language and cognition: Explorations in cognitive diversity. Cambridge: Cambridge University Press. Liddell, S. (1995). Real, surrogate, and token space: Grammatical consequences in ASL. In K. Emmory & J. Reilly (Eds.), Language, Gesture, and Space (pp. 19–41). Hillsdale, NJ: Erlbaum. Liddell, S. (1998). Blends with real space: Cognitive structures in signed and spoken discourse. In S. Santi, I. Guaitella, C. Cavé & G. Konopzynski (Eds.), Oralité et Gestualité (pp. 107–112). Paris: L’Harmattan. Liddell, S. (2000). Blended spaces and deixis in sign language discourse. In D. McNeill (Ed.), Language and Gesture (pp. 331–357). Cambridge: Cambridge University Press. Liddell, S. (2003). Grammar, Gesture, and Meaning in American Sign Language. Cambridge: Cambridge University Press. Llinas, R. R. (2001). I of the Vortex. Cambridge, MA: MIT Press. MacKenzie, C. L., & T. Iberall (1994). The Grasping Hand. Amsterdam: North-Holland. Mallery, G. (1978 [1880]). Introduction to the study of sign language among the North American Indians as illustrating the gesture speech of mankind. In D. J. Umiker-Sebeok & T. A. Sebeok (Eds.), Aboriginal Sign Languages of the Americas and Australia (pp. 291–310). New York: Plenum Press. Mandel, M. (1977). Iconic devices in American Sign Language. In L. A. Friedman (Ed.), On the Other Hand: New Perspectives on American Sign Language (pp. 57–108). New York: Academic Press. Manning, P. (1992). Erving Goffman and modern sociology. Stanford, CA: Stanford University Press. Marzke, M. (1997). Precision grips, hand morphology, and tools. American Journal of Physical Anthropology, 102, 91–110. Mauss, M. (1968). Sociologie et Anthropologie. Paris: Presses Universitaires de France. Mauss, M. (1973 [1935]). The techniques of the body. Economy and Society, 2 (1), 70–88. McCullough, M. (1996). Abstracting Craft. The Practiced Digital Hand. Cambridge, MA: The MIT Press. McDermott, R., K. Gospodinoff, & J. Aron (1978). Criteria for an ethnographical adequate description of concerted activities and their contexts. Semiotica, 24(3/4), 245–276. McDermott, R. P., & D. Roth (1978). Social organization of behavior: Interactional approaches. Annual Review of Anthropology, 7, 321–345. McNeill, D. (1985). So you think gestures are nonverbal? Psychological Review, 92 (3), 350–371. McNeill, D. (1992). Hand and Mind: What Gestures Reveal about Thought. Chicago: The University of Chicago Press. McNeill, D. (2005). Gesture and thought. Chicago: University of Chicago Press.

 Gesturecraft

McNeill, D., & S. Duncan (2000). Growth points in thinking-for-speaking. In D. McNeill (Ed.), Language and Gesture (pp. 141–161). Cambridge: Cambridge University Press. McQuown, N. (Ed.). (1971). The natural history of an interview (Vol. 95 – Series XV). Chicago: University of Chicago Library. Mead, G. H. (1909). Social psychology as a counterpart to physiological psychology. Psychological Bulletin, 6, 401–408. Mead, G. H. (1934). Mind, Self and Society. Chicago: University of Chicago Press. Meissner, M., & S. B. Philpott (1975). The sign language of sawmill workers in British Columbia. Journal of Sign Language, 9 , 291–308. Merleau-Ponty. (1962). Phenomenology of Perception. London: Routledge. Mohn, P. E. (2002). Filming Culture: Spielarten des Dokumentierens nach der Repräsentationskrise. Stuttgart: Lucius & Lucius. Mondada, L. (2007). Multimodal resources for turn-taking: Pointing and the emergence of possible next speakers. Discourse Studies, 9 (2), 194–225. Müller, C. (1998a). Redebegleitende Gesten. Kulturgeschichte – Theorie – Sprachvergleich. Berlin: Arno Spitz. Müller, C. (1998b). Iconicity and gesture. In S. Santi, I. Guaitella, C. Cave & G. Konopczynski (Eds.), Oralité et Gestualité. (pp. 321–328). Paris: L’Harmattan. Müller, C. (2003). Forms and uses of the Palm Up Open Hand. In C. Müller & R. Posner (Eds.), The Semantics and Pragmatics of Everyday Gestures. The Berlin Conference (pp. 234–256). Berlin: Weidler. Mulder, A. (1996). Hand gestures for HCI. Simon Fraser University: Hand Centered Studies of Human Movement Project. Technical Report 96-1. Muybridge, E. (1955a [1887]). Animals in Motion. New York: Dover. Muybridge, E. (1955b [1887]). The Human Figure in Motion. New York: Dover. Napier, J. (1956). The prehensile movements of the human hand. Journal of Bone and Joint Surgery, 38B , 902–913. Napier, J. (1980). Hands. New York: Pantheon. Neisser, U. (1976). Cognition and Reality. San Francisco: Freeman. Norman, D. (1993). Things that Make Us Smart. Reading, MA: Addison Wesley. Obeyesekere, G. (1997). The Apotheosis of Captain Cook: European Mythmaking in the Pacific. Princeton, NJ: Princeton University Press. Obeyesekere, G. (2005). Cannibal Talk: The Man-Eating Myth and Human Sacrifice in the South Seas. Berkeley: University of California Press. Ochs, E., P. Gonzales, & S. Jacoby (1996). “When I come down I’m in the domain state”: Grammar and graphic representation in the interpretive activity of physicists. In E. Ochs, E. A. Schegloff & S. Thompson (Eds.), Interaction and Grammar. Cambridge: Cambridge University Press. Olson, D. (1994). The World on Paper. Cambridge: Cambridge University Press. Palagyi, M. (1924). Naturphilosophische Vorlesungen über die Grundprobleme des Bewusstseins und des Lebens. Leipzig: Barth. Phillips, C. G. (1986). Movements of the Hand. Liverpool: Liverpool University Press. Pink, S. (2006). The future of visual anthropology: Engaging the senses. London: Routledge. Poggi, I. (2002). Symbolic gestures: The case of the Italian gestionary. Gesture, 2 , 71–98. Polanyi, M. (1958). Personal Knowledge. Towards a Post-Critical Philosophy. New York: Harper Torchbooks. Polanyi, M. (1966). The Tacit Dimension. Garden City, NY: Doubleday. Quine, W. V. O. (1960). Word and Object. Cambridge, MA: The MIT Press.

Bibliography 

Quintilianus, M. F. (1922 (100)). The Institutio Oratoria of Quintilian (H. E. Butler, Trans. Vol. IV). London: Heinemann. Rauscher, F. H., R. M. Krauss, & Y. Chen (1996). Gesture, speech, and lexical access: The role of lexical movements in speech production. Psychological Science, 7 (4), 226–231. Reddy, M. (1979). The conduit metaphor: A case of frame conflict in our language about language. In A. Ortony (Ed.), Metaphor and thought (pp. 284–324). Cambridge: Cambridge University Press. Reynolds, P. C. (1994). The complementary theory of language and tool use. In K. R. Gibson & T. Ingold (Eds.), Tools, Language and Cognition in Human Evolution (pp. 407–428). Cambridge: Cambridge University Press. Riggins, S. H. (1990). Beyond Goffman: Studies on communication, institution, and social interaction. Berlin: Mouton de Gruyter. Rizzolatti, G., & L. Craighero (2004). The mirror-neuron system. Annual Review of Neuroscience, 27, 169–192. Rouch, J. (2003). Cine-Ethnography. Minneapolis: University of Minnesota Press. Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 50 , 696–735. Sarles, H. B. (1975). A human ethological approach to communication: Ideas in transit around the Cartesian impasse. In A. Kendon (Ed.), Organization of Behavior in Face-to-Face Interaction (pp. 19–46). The Hague: Mouton. Sauer, B. (2003). The Rhetoric of Risk. Technical Documentation in Hazardous Environments. Mahwah, NJ: Lawrence Erlbaum. Scheflen, A. (1964). The significance of posture in human communication systems. Psychiatry, 27, 316–321. Scheflen, A. (1973). Communicational Structure. Bloomington, IN: Indiana University Press. Scheflen, A. (1974). How Behavior Means. Garden City, NY: Anchor Press. Scheflen, A. (1975). Models and epistemologies in the study of interaction. In A. Kendon et al. (Ed.), Organization of behavior in face-to-face interaction. The Hague: Mouton. Schegloff, E. A. (1984). On some gestures’ relation to talk. In J. M. Atkinson & J. Heritage (Eds.), Structures of Social Action (pp. 266–295). Cambridge: Cambridge University Press. Schegloff, E. A. (1995). Discourse as an interactional achievement III: The omnirelevance of action. Research on Language and Social Interaction, 28(3), 185–211. Schegloff, E. A. (2007). Sequence Organization in Interaction: A Primer in Conversation Analysis. Cambridge: Cambridge University Press. Schlegel, J. (1998). Finding words, finding meanings: Collaborative learning and distributed cognition. In S. Hoyle & C. T. Adger (Eds.), Kids talk: strategic language use in later childhood. New York: Oxford University Press. Schütz, A. (1967 [1932]). The Phenomenology of the Social World. Evanston, IL. Northwestern University Press. Schütz, A. (1982). Collected Papers (Vol. 1–3). The Hague: Martinus Nijhof. Sennett, R. (2008). The Craftsman. New Haven: Yale University Press. Seyfeddinipur, M. (2003). Meta-discursive gestures from Iran: Some uses of the ‘Pistol Hand’. In C. Müller & R. Posner (Eds.), The Semantics and Pragmatics of Everyday Gestures: The Berlin Conference (pp. 205–216). Berlin: Weidler. Sennett, R. (2008). The Craftsman. New Haven: Yale University Press. Seyfeddinipur, M. (2006). Disfluency: Interrupting Speech and Gesture .. Nijmegen: Max Planck Institute for Psycholinguistics.

 Gesturecraft

Sherzer, J. (1972). Verbal and nonverbal deixis: The pointed lip gesture among the San Blas Cuna. Language in Society, 2 (1), 117–131. Sherzer, J. (1991). The Brazilian thumbs-up gesture. Journal of Linguistic Anthropology, 12 , 189– 197. Slobin, D. I. (1996). From “thought and language” to ”thinking for speaking”. In J. J. Gumperz & S. Levinson (Eds.), Rethinking Linguistic Relativity (pp. 97–114). Cambridge: Cambridge University Press. Smith, A. (1776). Inquiry into the Nature and Causes of the Wealth of Nations. London. Sowa, T. (2006). Understanding Coverbal Iconic Gestures in Shape Descriptions. Berlin: Akademische Verlagsgesellschaft. Stam, G. (2005). Gesture interlanguage. Paper presented at the Second Conference of the International Society for Gesture Studies, Lyon, France. Strathern, A. (1996). Body Thoughts. Ann Arbor: University of Michigan Press. Streeck, J. (1988). The significance of gesture: How it is established. Papers in Pragmatics, 2 (1/2), 60–83. Streeck, J. (1993). Gesture as communication I: Its coordination with gaze and speech. Communication Monographs, 60 (December 1993), 275–299. Streeck, J. (1995). On projection. In E. Goody (Ed.), Social Intelligence and Interaction (pp. 87– 110). Cambridge: Cambridge University Press. Streeck, J. (2002a). Grammars, words, and embodied meanings: On the evolution and uses of so and like. Journal of Communication, 52 (3), 581–596. Streeck, J. (2002b). A body and its gestures. Gesture, 2, 19–44. Streeck, J. (2003). Balinese Hands. [Online] http://www.utexas.edu/coc/cms/faculty/streeck/ bali/. Streeck, J. (2005). Pragmatic aspects of gesture. In K. Brown (Ed.), The Encyclopedia of Language and Linguistics (Vol. 5, Pragmatics, pp. 71–76.). Oxford: Elsevier. Streeck, J. (2007). Geste und verstreichende Zeit. Innehalten und Bedeutungswandel der “bietenden Hand”. In H. Hausendorf (Ed.), Gespräch als Prozess (pp. 157–180). Tübingen: Gunter Narr. Streeck, J. (2008a). Laborious intersubjectivity: Attentional struggle and embodied communication in an auto-shop. In I. Wachsmuth, M. Lenzen & G. Knoblich (Eds.), Embodied Communication in Humans and Machines. Oxford: Oxford University Press. Streeck, J. (2008b). Depicting by gestures. Gesture, 8, 3, 285–301. Streeck, J. (2008c). Metaphor and gesture: A view from the microanalysis of interaction. In A. Cienki & C. Mueller (Eds.), Metaphor and Gesture (pp. 259–264). Amsterdam: John Benjamins. Streeck, J. (2008d). Gesture in Political Communication: A Case Study of the Democratic Presidential Candidates During the 2004 Primary Campaign. Research on Language & Social Interaction, 41 (2), 154–186. Streeck, J. (2009a). Forward-gesturing. Discourse Processes, 46, 2–3, 161–179. Streeck, J. (2009b). Depicting gestures. Examples of the pictorial analysis of bodily communication in the arts of the West. Gesture, 9, 1, 1–34. Streeck, J., & U. Hartge (1992). Previews: Gestures at the transition place. In P. Auer & A. di Luzio (Eds.), The Contextualization of Language (pp. 138–158). Amsterdam: John Benjamins. Streeck, J., & W. Kallmeyer (2001). Interaction by inscription. Journal of Pragmatics, 33 (4), 465–490.

Bibliography 

Streeck, J., & S. Mehus (2004). Microethnography: The study of practices. In K. L. Fitch & R. E. Sanders (Eds.), Handbook of Language and Social Interaction (pp. 381–405). Mawah, NJ: Lawrence Erlbaum. Strickberger, M. W. (2000 (1990)). Evolution. 3rd Edition. Sudbury, MA: Jones and Bartlett. Suchman, L. (1993). Artificial intelligence as craftwork. In S. Chaiklin & J. Lave (Eds.), Understanding Practice (pp. 144–178). Cambridge: Cambridge University Press. Sudnow, D. (1978). Ways of the Hand. The Organization of Improvised Conduct. Cambridge, MA: Harvard University Press. Tabensky, A. (2001). Gesture and speech rephrasings in conversation. Gesture, 1, 213–235. Talmy, L. (1988a). The relation of grammar to cognition. In B. Rudzka-Ostyn (Ed.), Topics in Cognitive Linguistics (pp. 165–205). Amsterdam: John Benjamins. Talmy, L. (1988b). Force dynamics in language and cognition. Cognitive Science, 12, 49–100. Talmy, L. (2000). Toward a cognitive semantics. Cambridge, MA: MIT Press. Taub, S. F. (2001). Language from the Body. Iconicity and Metaphor in American Sign Language. Cambridge: Cambridge University Press. Tettamanti, M., G. Buccino, M. C. Saccuman, V. Gallese, M. Danna, P. Scifo, et al. (2005). Listening to action-related sentences activates fronto-parietal motor circuits. Journal of Cognitive Neuroscience, 17 (2), 273–281. Tylor, E. B. (1871). Primitive Culture. London: J. Murray. Varela, F. J., E. Thompson, & E. Rosch (1991). The Embodied Mind: Cognitive Science and Human Experience. Cambridge, MA: The MIT Press. Vygotsky, L. (1978). Mind in Society. Cambridge: Harvard University Press. Wertsch, J. V. (1981a). Voices of the Mind. A Sociocultural Approach to Mediated Action. Cambridge, MA: Harvard University Press. Wertsch, J. V. (1981b). The Concept of activity in Soviet psychology. Armonk, NY: M.E. Sharpe. Wertsch, J. V., P. del Rio, & A. Alvarez (Eds.) (1995). Sociocultural studies of mind. Cambridge: Cambridge University Press. Wilkins, D. (2003). Why pointing with the index finger is not a universal (in sociocultural and semiotic terms). In S. Kita (Ed.), Pointing: Where language, culture, and cognition meet (pp. 171–216). Mahwah, NJ: Lawrence Erlbaum. Wilson, F. (1998). The Hand. New York: Pantheon. Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin and Review, 9, 625– 636. Wilson, M., & G. Knoblich (2005). The case for motor involvement in perceiving conspecifics. Psychological Bulletin, 131 (3), 460–473. Wittgenstein, L. (1953). Philosophical Investigations: Blackwells. Wundt, W. (1975 [1911]). Völkerpsychologie. Eine Untersuchung der Entwicklungsgesetze von Sprache, Mythus und Sitte. Aalen: Scientia Verlag.

Person index

A Alibali, M.  Alvarez, A.  Armstrong, D. F.  Arnheim, R.  Aron, J.  Ashley, R.  B Banks, M.  Barnett, D.  Barresi, J.  Barakat, R. A.  Barsalou, L. W.  Bateson, G. , –,  Bavelas, J. ,  Beattie, G. , ,  Belgrad, D.  Bergen, B. ,  Bergmann, J.  Bergson, H.  Berliner, P. F. – Birdwhistell, R. L. ,  Boas, F.  Bolden, G.  Boonzajer Flaes, R. M.  Bourdieu, P. ,  Brookes, H. J.  Brosin, H. W.  Bruner, J.  Bühler, K.  Butcher, C.  C Calbris, G. –,  Carson, J. – Chaiklin, S.  Chen, Y.  Chomsky, N. ,  Chovil, N. ,  Church, R. B.  Cicero, M. T. 

Clack, J. A.  Clark, H. H.  Cole, M.  Cohen, E.  Collier, J. J.  Collier, M.  Comolli, A. , –,  Condillac, B. de  Connerton, P. ,  Cook, J. –,  Coppola, M. ,  Corballis, M. C.  Coulter, J.  Cunningham, M.  D Dar, R.  de Fornel, M.  de France, C. , – Darwin, C.  DeJorio, A. –, ,  de Ruiter, J. P.  Diller, A.-M.  Ditton, J.  Donald, M.  Dreyfus, H. L. , , , , , ,  Duncan, S.  Dunnington, G.  E Efron, D. ,  Egbert, M.  Ekman, P. ,  Enfield, N. J. ,  Engeström, Y.  Erickson, F.  F Fauconnier, G.  Fine, G. A.  Foçillon, H. 

Fodor, J. A.  Forster, G. –,  Freedman, N. ,  Friesen, W. V. ,  G Gallese, V. ,  Gardner, H.  Garfinkel, H.  Gehlen, A.  Gibson, J. J. , , , , ,  Goethe, J. W. von  Goffman, E. , –, , ,  Golato, A.  Goldin-Meadow, S. ,  Gombrich, E. H. ,  Gonzales, P. , ,  Goodman, N. , ,  Goodwin, C. , , , –, , , , , , , –,  Goodwin, M. H. –,  Gospodinoff, K.  Grosjean, M.  Gullberg, M. –,  Gumperz, J. J.  H Hadar, U.  Hanks, W. F. ,  Harper, D. ,  Hartge, U. , , , ,  Hastrup, K. ,  Haviland, J. B. ,  Heath, C. , , – Heidegger, M. , , , , , , , ,  Heine, B.  Henderson, K.  Heritage, J. 

 Gesturecraft Hewes, G.  Hockett, C.  Hockings, P.  Holler, J.  Holmqvist, K.  Hopper, P. J.  Humboldt, W. von  Hutchins, E.  Huxley, J.  I Iberall, T. , –, ,  Ingold, T. , , , ,  J Jackson, M.  Jacoby, S. , ,  Jeannerod, M. –,  Jefferson, G. , ,  Jenni, R. – Joas, H.  Johnson, M. , , –, – Jones, S. E.  K Kallmeyer, W.  Kegl, J. ,  Keller, C. M. ,  Keller, J. D. ,  Keller, R.  Kendon, A. , , , , , , , , –, , –, , –, , , , –, , , , ,  Kimbara, I.  Kirsh, D. ,  Kita, S. , , ,  Klatzky, R. L. – Kleist, H. v. , Kline, F.  Knoblich, G.  Koenig, C.  Koriat, A.  Koschmann, T.  Krauss, R. M.  Kuklick, H.  Kuteva, T.  L Lakoff, G. , ,  Langacker, R. W. , , 

Lave, J. , , ,  Lawrie, D. A. ,  LeBaron, C. D.  Lederman, S. J. – Leeds-Hurwitz, W.  Leonardo da Vinci  Lerner, G. ,  Lessing, G. E.  Levinson, S. C.  Levy-Sadot, R.  Liddell, S. ,  Lindner, S.  Llinàs, R. R. , , ,  Luff, P.  M MacKenzie, C. L. , –,  Mallery, G. ,  Mandel, M. , ,  Manning, P.  Marzke, M. –,  Maturana, H. , ,  Mauss, M. ,  McCullough, M. , , , , ,  McDermott, R.  McNeill, D. , , , , , , , ,  McQuown, N. ,  Mead, G. H. –, , ,  Mead, M. –, , ,  Mehus, S. ,  Meissner, M.  Merleau-Ponty, M. , , , , ,  Mohn, P. E. ,  Mondada, L. , ,  Moore, C.  Müller, C. , , , , , ,  Mulder, A. ,  Murphy, K.  Muybridge, E.  N Namir, L.  Napier, J. , , ,  Narayanan, S.  Neisser, U.  Norman, D. ,  Nuñez, R. E. 

O Obeyesekere, G.  O’Brien, C.  Ochs, E. , ,  Olson, D.  P Palagyi, M.  Palen, L.  Parsons, E. D.  Phillips, C. G.  Philpott, S. B.  Pink, S.  Poggi, I.  Polanyi, M. , , , , , ,  Q Quayle, D.  Quine, W. V. O.  Quintilianus, M. F.  R Râio, P. d.  Rau, J. ,  Rauscher, F. H.  Reddy, M. ,  Reynolds, P. C. – Riggins, S. H.  Rosch, E.  Roth, D.  Rouch, J.  S Sacks, H. , ,  Sarles, H. B.  Sauer, B.  Scheflen, A. , , , –,  Schegloff, E. A. ,  ,, –,  Schlegel, J.  Schlesinger, I. M.  Schmitt, J.-C.  Schütz, A. ,  Senghas, A. ,  Sennett, R. –, ,  Seyfeddinipur, M. , ,  Sherzer, J. –,  Shiffrar, M.  Sigman, S. J. 

Person index  Slobin, D. I.  Smith, A.  Smith, H. L.  Sowa, T.  Stam, G.  Stokoe, W. C.  Stolpe, M. ,  Strathern, A.  Streeck, J. , , , , , , , , , , , , , , , , , , , ,  Streeck, U. – Strickberger, M. W.  Suchman, L. ,  Sudnow, D. 

T Tabensky, A.  Talmy, L. ,  Taub, S. F. –,  Tettamanti, M.  Teitelman, A.  Thornton, I. M.  Thompson, E.  Trager, G. L.  Traugott, E. C.  Tylor, E. B. – V van Gijn, I.  van der Hulst, H.  Varela, F. J. , , ,  Vazques, O. 

Vygotsky, L. , ,  W Wade, A. ,  Wertsch, J. V.  Wilcox, S. E.  Wilkins, D.  Wilson, F.  Wilson, M. ,  Wilson, R. ,  Wheeler, K.  Wittgenstein, L. ,  Wundt, W. –,  Z Zemel, A.  Zimmerman, D. H. 

Subject index

A act, action abstractive  cognitive  communicative , , ,  epistemic  exploratory , –,  instrumental , ,  manual –,  practical , , , , , ,  physical  action formation  action-perception loop , ,  activity theory  anthropology cinematographic  visual ,  art – attention , , , , –, , – augmented action  reality , – B ballistic movement  bracchiation – brain , – bricolage  C care  ceiving , , , , –,  cept , ,  ception , , ,  ceptive  cinematography – classifier 

clearing –, – cognitive linguistics ,  cognitive science  cognitive tool , ,  cohesion  community of practice , – concept , –, , ,  body-based  multimodal  conceptualization  conduit , , –,  context , –, –, , , , ,  context analysis , , , –, ,  contiguity – conversation analysis , , –, ,  convention – coordination gesture-speech ,  hand-eye  coupling, structural , ,  craft ,  cultural skill  culture, cultures , , –,  D depiction , , –, , , , ,  deictic expressions  disassembling , –,  drawing –,  dwelling  E ecology of gesture –, , , 

embodied , , –, , , , ,  enactment , , ,  enculturation  ethnography –, ,  ethnomethodology ,  exploratory procedure , , – F figure-ground – film – functional perspective  G gaze , , , , –, ,  to gesture , – genre ,  gestalt psychology  gesture as abstractive action , ,  as early part of an act  as psychological/thinking tool ,  as significant symbol  of emotion – of making  of negation – of offering –, –,  of orientation , – of rejection – conduit  conventional  conversational , ,  depictive , – environmentally coupled , ,  iconic ,  illocutionary 

 Gesturecraft imagistic , ,  interactive – meta-pragmatic  operator  pragmatic , , –, , ,  model function of – parsing function of  performative function of  pronominal  quotable  referential ,  representational ,  gesture family – gesture phrase  gesture space , – gesture unit  grammaticalization  grappolo – grasp, grasping , –, – dynamic  static  grip ,  power ,  precision 

I imagery  kinetic  visual  image schema ,  improvisation  indexical practive  indexical sign – indexing  individualism  individuation  intentional relation  interactionism – interaction order  intercorporeality  interjection  intersubjectivity , , , ,  invisible hand  J jazz – K kinesthetic –, , –, ,  kinetic style  L landscape , 

H habitus  hand – as cognitive organ ,  as compromise organ  as data-gathering device  as organ of action  as organ of perception  enactive and sensory skills of  evolution of – posture  shape , , , ,  structure – handling , –, ,  precision  haptic epistemology ,  haptic system –,  hexis  hybridity, hybridization , 

M marking – self-marking – materiality of body  meaning , –, , , , , ,  meta-cognition  meta-communication  metonymic, metonymy ,  metaphor , , , , , ,  micro-analysis , , – micro-ethnography ,  mimesis , , , ,  modality , ,  modeling  model-world making , ,  moulding  multisensory 

N narrative , ,  natural history method , , – neuroscience , 

O opposition ,  pad – palm – side – opposition space –, 

P participation framework ,  phenomenology , ,  photography  placing  pointing , , , , –, , –, , , , , ,  lip  polylith –, ,  posture configuration  practice, practices , , – praxeology , –,  predication  pre-enactment  prehensility ,  prehensile posture ,  prehension  primates – proprioception –, 

R repair , , ,  replaying  representation , , –, , ,  ritualized behavior 

S scape ,  scaping  sequence, sequencing , , , ,  of gaze –

Subject index  sequentiality  shrug – sign language , , – simultaneity of gesture and speech  sketching ,  Sorge (care)  spatial organization  spatial representation  stone-tool manufacture 

T tactile  tag question – texture  thinking, thought , –, , , , , , –,  touch active/dynamic , , ,  tracing , –, ,  turn 

turn-completion , , , ,  turn-taking/transition , , , , , ,  V virtual finger – verbs of speaking – W word-search , , – world-model –

In the series Gesture Studies the following titles have been published thus far or are scheduled for publication: 3 2 1

Cienki, Alan and Cornelia Müller (eds.): Metaphor and Gesture. 2008. ix, 306 pp. Streeck, Jürgen: Gesturecraft. The manu-facture of meaning. 2009. xii, 236 pp. Duncan, Susan D., Justine Cassell and Elena T. Levy (eds.): Gesture and the Dynamic Dimension of Language. Essays in honor of David McNeill. 2007. vi, 328 pp.