Product Experience

PREFACE

‘With this collection of chapters,’ say Hendrik Schifferstein and Paul Hekkert, ‘we hope to lay down the basics for the field of product experience research’. With these words, Schifferstein and Hekkert set forth a daunting challenge, but a necessary and important endeavor. Although industrial design has long been concerned with products, the emphasis has been on the physical structures of the devices, the forms and shapes, the materials, and the manufacturing processes to be followed. These are all important, to be sure, but today the focus has shifted from objects to the experiences that result from interacting with them. To understand experience we need to go beyond shape and form, even beyond simple ergonomics. We need to understand how psychology, the social sciences, communication, and business shape a person’s experience. This book provides an important first step. Design is hot: It is the topic of the day. Moreover, design is now coming to be understood in the broader sense, not just in the styling that provides a pretty face to products. Design is now recognized as a process applicable to all phases of product development. Design is, moreover, a multidisciplinary endeavor, requiring knowledge of a wide range of topics. The result is that the field of product design is entering a renaissance period. It’s a new world. It’s a challenge. And, of course, challenges are opportunities. But where to get all this new knowledge? Product Experience provides a powerful start. Schifferstein and Hekkert have packed an incredible array of information into the 27 chapters of the book, introducing the reader to multiple perspectives of product experience. From the senses to shopping, the chapters cover a rich gamut of issues. Schifferstein and Hekkert set the stage well with their introduction that reviews the many chapters,

xix

xx

PREFACE

providing structure and guidance. Then comes the meat of the book, the 27 chapters by recognized authorities, followed by Schifferstein and Hekkert’s closing reflections, neatly tying things together. The result encompasses many of the varied components and perspectives involved in understanding how products shape experience. Product Experience, the book, provides an essential reference work for students, instructors, and practicing designers. How do people experience products? What is the nature of the interaction, that dynamic interplay between the perceptions, actions, and understanding of the person experiencing the product? For the answers, start with this book. Don Norman1 Palo Alto, California Evanston, Illinois

1

Donald A. Norman. www.jnd.org [email protected] Nielsen Norman group and Northwestern University

LIST OF CONTRIBUTORS

The chapter(s) contributed to are indicated by the numbers in parentheses. Katja Battarbee, IDEO, Palo Alto, CA 94301, USA (19) Christina Bodin Danielsson, School of Architecture and Built Environment, The Royal Institute of Technology, 10044 Stockholm, Sweden (26) Stella Boess, Department of Industrial Design, Faculty of Industrial Design Engineering, Delft University of Technology, 2628 CE Delft, The Netherlands (12) Ünal Ö. Boya, Marketing Department, Appalachian State University, Boone NC 28608, USA (25) Reinhart Butter, Department of Industrial, Interior and Visual Communication Design, Ohio State University, Columbus, Ohio 43210, USA (14) Armand V. Cardello, Office of the Technical Director, US Army Natick Soldier Center, Natick, MA 01760-5020, USA (4) John M. Carroll, School of Information Sciences and Technology, The Pennsylvania State University, University Park PA 16802, USA (21) John Clarkson, Department of Engineering, University of Cambridge, Cambridge CB2 1PZ, United Kingdom (6) Gerald C. Cupchik, Department of Psychology, University of Toronto at Scarborough, Department of Life Sciences, Scarborough, ON, Canada M1C 1A4 (9) Michiel P. de Looze, TNO, 2130 AS Hoofddorp, The Netherlands (18) Pieter M.A. Desmet, Department of Industrial Design, Faculty of Industrial Design Engineering, Delft University of Technology, 2628 CE Delft, The Netherlands (15)

xxi

xxii

List of contributors

Jörgen Eklund, Department of Mechanical Engineering, Division of Quality and Human Systems Engineering, Linköping University, SE-58183 Linköping, Sweden (20) Ann Marie Fiore, Department of Textiles and Clothing, Iowa State University, Ames IA 50011, USA (27) Lawrence L. Garber, Department of Business Administration, Elon University, Elon, NC 27244, USA (25) Marc Hassenzahl, Economic Psychology and Human–Computer Interaction, University of Koblenz-Landau, Universität Campus Landau, 76829 Landau, Germany (11) Paul Hekkert, Department of Industrial Design, Faculty of Industrial Design Engineering, Delft University of Technology, 2628 CE Delft, The Netherlands (10) Michelle C. Hilscher, Department of Psychology, University of Toronto at Scarborough, Department of Life Sciences, Scarborough, ON, Canada M1C 1A4 (9) Eva M. Hyatt, Marketing Department, Appalachian State University, Boone NC 28608, USA (25) Shigekazu Ishihara, Department of Kansei Ergonomics, Hiroshima International University, Hiroshima, Japan (20) Jeroen Jansz, Amsterdam School of Communications Research (ASCoR), University of Amsterdam, 1012 CX Amsterdam, The Netherlands (23) Heimrich Kanis, Department of Industrial Design, Faculty of Industrial Design Engineering, Delft University of Technology, 2628 CE Delft, The Netherlands (12) David V. Keyson, Department of Industrial Design, Faculty of Industrial Design Ingeneering, Delft University of Technology, 2628 CE Delft, The Netherlands (22) Ilpo Koskinen, University of Art and Design Helsinki, 00560 Helsinki, Finland (19) Klaus Krippendorff, The Annenberg School for Communication, University of Pennsylvania, Philadelphia, PA 19104-6220, USA (14) Helmut Leder, Faculty of Psychology, University of Vienna, 1010 Wien, Austria (10) Herbert L. Meiselman, US Army Natick Soldier Center, Natick MA 01760-5020, USA, (24) Helena M. Mentis, School of Information Sciences and Technology, The Pennsylvania State University, University Park PA 16802, USA (21) Ruth Mugge, Department of Product Innovation Management, Faculty of Industrial Design Engineering, Delft University of Technology, 2628 CE Delft, The Netherlands (17) Mitsuo Nagamachi, User Science Institute, Kyushu University, Fukuoka 815-8540 Japan (20) Harold T Nefs, School of Psychology, University of St. Andrews, St. Mary’s College, St. Andrews, KY16 9JP, United Kingdom (1) Marsha L. Richins, College of Business, University of Missouri, Columbia, MO 65211, USA (16) Axel Roesler, Division of Design, University of Washington, Seattle, WA 98195, USA (7, 8)

List of contributors

xxiii

Hendrik N.J. Schifferstein, Department of Industrial Design, Faculty of Industrial Design Engineering, Delft University of Technology, 2628 CE Delft, The Netherlands (2, 5, 17) Jan P.L. Schoormans, Department of Product Innovation Management, Faculty of Industrial Design Engineering, Delft University of Technology, 2628 CE Delft, The Netherlands (17) Simon Schütte, Department of Mechanical Engineering, Division of Quality and Human Systems Engineering, Linköping University, SE-58183 Linköping, Sweden (20) Marieke H. Sonneveld, Department of Industrial Design, Faculty of Industrial Design Engineering, Delft University of Technology, 2628 CE Delft, The Netherlands (2) Charles Spence, Department of Experimental Psychology, University of Oxford, Oxford, OX1 3UD, United Kingdom (5) Ed S. Tan, Amsterdam School of Communications Research (ASCoR), University of Amsterdam, 1012 CX Amsterdam, The Netherlands (23) René van Egmond, Department of Industrial Design Faculty of Industrial Design Engineering, Delft University of Technology, 2628 CE Delft, The Netherlands (3) Thomas J.L. van Rompay, Department of Marketing Communication and Consumer Psychology, Faculty of Behavioral Sciences, Twente University 7500 AE Enschede, the Netherlands (13) Peter Vink, TNO, 2130 AS Hoofddorp, the Netherlands (18) Paul M. Wise, Monell Chemical Senses Center, Philadelphia, PA 19104-3308, USA (4) David Woods, Institute for Ergonomics, Ohio State University, 210 Baker Systems, Columbus OH 43210, USA (7, 8)

INTRODUCING PRODUCT EXPERIENCE PAUL HEKKERT AND HENDRIK N.J. SCHIFFERSTEIN

People live in a world in which they are surrounded by designed artifacts and services, products that were created by (other) people to serve some purpose: To get from one place to another; to clean the house; to cook; to feed and protect oneself; to contact someone; to have fun; to retrieve information; and so on. As a result, we have cars and bicycles, tissues and vacuum cleaners, shopping malls and living rooms, mobile phones and weblogs, computer games and festivals, archive systems and SMS messaging, and to use these products we need to interact with them. Notwithstanding the fact that the way in which people interact with a product is clearly product-dependent, they always use their senses to perceive it, they use their motor system and their knowledge to operate or communicate with it, and during the interaction they process the information they perceive, they may experience one or more emotions, and they are likely to form an affective evaluation of the product. Thus, although the interaction may be product-specific, the processes that are activated during the interaction are similar over products. As a consequence, it should be possible to develop an overall theoretical framework that guides the study of how people experience products. In the current book, we bring together existing work from several different research areas, and make a first attempt to integrate this work, as the starting point for the development of an overall, encompassing framework of product experience.

Theoretical views on product experience Before we can develop a conceptual framework, we first need to define the concept that we are interested in. In this book, we define the field of ‘product experience’ as the research area that develops an understanding of people’s subjective experiences that result from interacting with products. By focusing on products we restrict ourselves to physical objects or non-physical designs that have a utilitarian function, thereby excluding works of art and other non-utilitarian artifacts. Building on our previous definitions Product Experience Copyright © 2008 Elsevier Ltd.

1

2

Product Experience

(Hekkert, 2006; Schifferstein and Cleiren, 2005), we define the subjective product experience as the awareness of the psychological effects elicited by the interaction with a product, including the degree to which all our senses are stimulated, the meanings and values we attach to the product, and the feelings and emotions that are elicited. It can be debated whether these psychological consequences are always affective in nature. According to Russell (2003), core affect – the combination of pleasure and arousal – is, to varying degrees, ‘involved in most psychological events’ [p. 148, emphasis added]. Not all authors in this book embrace our definition of product experience. Some hold a different view and explain in their chapters where their definition originates from, and why they think it fits their research purposes better. These different perspectives from various fields of scientific endeavor add to the richness of our understanding of the concept. Our definition of product experience is quite broad compared to the definitions adopted by other authors and, thereby, generally encompasses their definitions. Therefore, in our view, all these authors investigate some aspects or antecedents of ‘product experience’, even though their reference frame may be narrower than ours. For example, Carroll and Mentis (Chapter 21) give an overview of how research on user experiences that emerged in the field of human–computer interaction yielded new insights that enriched research on product usability in the area of human factors. Battarbee and Koskinen (Chapter 19) use a designer’s perspective on product experience and stress the importance of social interaction in the formation of experiences. Cupchik and Hilscher (Chapter 9) explicitly limit the term ‘experience’ to special life events. They state that ‘experience refers to uniquely meaningful life events with both cognitive and affective qualities’ [emphasis added]. The contrast between Cupchik and Hilscher’s definition and ours becomes explicit in the two ways in which the term ‘experience’ can be translated into German or Dutch. All experiences, including very common, day-to-day experiences, are referred to with the German word ‘Erfahrung’ (Dutch: ‘ervaring’), whereas experiences of special, memorable events are better captured by the word ‘Erlebnis’ (Dutch: ‘belevenis’). The Erlebnis definition is closest to the views found in business literature on the Experience Economy (e.g. Pine and Gilmore, 1998). According to Pine and Gilmore, ‘an experience occurs when a company intentionally uses services as the stage, and goods as props, to engage individual consumers in a way that creates a memorable event’ (p. 98). Pine and Gilmore suggest that producers and retailers need to create special experiences for their customers, beyond simply offering good products. Their notion of experience thus resembles the uniquely meaningful events Cupchik and Hilscher refer to. Such experiences can occasionally be bound to individual products, but will more likely arise from designed spaces, installations, and attractions typically encountered in exhibitions and amusement parks. The design of these kinds of overwhelming and compelling events is often denoted as ‘experience design’. Designers and marketers may wonder whether it is always necessary to create an ‘Erlebnis’, or whether it will be sufficient to deliberately create, alter or improve the ‘Erfahrung’ that people have when they interact with products. According to Schmitt (1999), for example, the ultimate goal of experiential marketing is to create a desirable, coherent, and consistent customer impression that enhances the brand image. For companies it may be worthwhile to create possibilities for potential consumers to explore their products in a specific context, as part of a branding strategy. When people repeatedly encounter a particular brand within the context of a pleasant experience, they are more likely to develop a favourable attitude towards this brand. In addition, characteristics of the atmosphere during the experience (e.g. modern, fresh, impressive) may become associated with the brand.

3

Introducing product experience

The approaches sketched above, by Pine and Gilmore and Schmitt, were developed from a marketing perspective, in which the purchase and consumption of the product plays a central role. This deviates from the user perspective that is employed in most chapters of the current book. Many everyday experiences involve people who simply use and enjoy products. These everyday experiences are central to the book: The desire when seeing the new Apple iPod; the comfortable feel of a screwdriver in your hand; the awareness of the friendly whistle of a teapot; the frustration over the poor quality of an online help system; the delicious smell of a freshly baked apple-pie; the relief of smoothly parking your car in a narrow space. Understanding such experiences will allow designers and companies to ‘design for experience’.

Human–product interaction As argued at the beginning of this chapter, a product experience or ‘Erfahrung’ always results from some interaction the user has with a product. This interaction is not necessarily restricted to instrumental or non-instrumental (e.g. playing with your pen) physical action, but may also consist of passive (often visual) perception, or even remembering or thinking of a product (e.g. anticipating using your new stereo) (Desmet and Hekkert, 2007). Furthermore, the experience does not only result from the interaction, but also accompanies and guides the interaction, and thus affects the interaction. In sum, experience and interaction are fully intertwined and in order to explore people’s experiences of products, we need to thoroughly understand the constituents or building blocks of human–product interaction. Figure 1 provides a model of human–product interaction. In trying to obtain an overview of the literature covering the field of product experience, we got the impression that most scholars tend to approach human–product interaction from one of three possible perspectives. These three perspectives form the basic structure of the present book: The human beings with their systems and skills (Part I); the interaction itself with its different components (Part II); or a product (domain) with its specific properties (Part III). Independent from their surroundings and social context, humans are biologically equipped with a number of systems that make it possible for them to interact with their environment: A motor system to act upon the environment; sensory systems to perceive changes in the environment; and a cognitive system to make sense of the environment and to plan actions. Products are part of this environment. The motor capacities are needed to explore products, interact with them, and operate them. Sensory systems allow people to perceive a product and assess what kind of product it is. They provide feedback on people’s actions. Furthermore, they ‘tell’ a person whether a sensation (visual, auditory, tactual, olfactory, or gustatory) is pleasurable or should be avoided. Cognitive

HUMAN Motor system

PRODUCT

Motor skills

Sensory systems

Sensitivity

Cognitive system

Cognitive skills

Instincts

Concerns

INTERACTION

FIGURE 1 Model of human–product interaction.

Sensory properties

Structural properties

Possibilities for behavior

Composition

Functionality

Labels

Materials Technology

4

Product Experience

capacities link perceived information to stored knowledge to interpret the incoming information; they elicit memories of previous usage and evoke associations with other products. Finally, people are born with a limited set of instincts, which make them explore the world to try to satisfy basic needs. Through interaction with an environment, all these human capacities gradually develop into skills, expertise, and concerns (such as goals, intentions, and preferences). Skills and concerns can only be defined in relation to an outside world: We have skills in something, sensitivity for something, a preference for something, and our goals and intentions are directed towards something. The chapters in Part I of this book start out from one or more of these systems or skills, and try to explain experiential effects (and limitations) of human–product interaction on the basis of the specific qualities of these human capacities or characteristics. Analogously, a product perceived in isolation has (only) a number of structural or formal properties, such as its size, weight, and shape. Physical products are made of materials with specific technical characteristics (e.g. chemical constitution, heat conductivity, elasticity). Furthermore, products have a composition relating the various constituent parts to the whole. In addition, more and more products make use of embedded technology (electronics, sensors, and other digital components) to operate them. Finally, all kinds of labels (e.g. brand name, usage information, price information) may be attached to the product. It is, however, in the interaction with people that products obtain their meaning: On the basis of what is perceived sensorially (e.g. softness, freshness, loudness) products reveal cues of how to use them, and they reveal their function. Perceived product properties will only be of interest to an individual if they are somehow instrumental in fulfilling needs; only in relation to people we can determine what behaviour a product allows for, and what its primary or secondary functions might be. Studies that start out from the product (Part III) typically focus on one particular (set of) products and on how their product- or domain-specific characteristics affect people’s experiences. Often they include case studies of, for instance, how people experience household equipment, office interiors, or computer games. Product experiences depend on the way in which a person interacts with a product. Although phenomenologically experienced as a whole, at least three major components can be distinguished in product experiences (Desmet and Hekkert, 2007; Hekkert, 2006). The aesthetic response is characterized by feelings of (dis)pleasure that are based on the sensory perception of the object; the object looks beautiful, feels pleasurable, or sounds nice. In addition, people try to understand how a product must be operated or which actions it affords, and people attribute all kinds of expressive, semantic, symbolic, or other connotative meanings to it. The interactions with a product can help a person to reach a goal or can obstruct him or her in attaining that goal, and thereby lead to various emotional responses. Together these components shape the overall product experience. The chapters in Part II focus on one of these components of the experience, or they study a specific experiential state typified by a term like (dis)comfort, engagement, quality, or attachment. Finally, the way an interaction unfolds depends on the context in which this interaction takes place. This context can vary from the physical circumstances literally surrounding the interaction, such as the lighting conditions under which a product is perceived or the qualities of the space in which a meal is consumed, to the activities or experiences that have preceded the interaction at stake, and to the broader cultural and social situation that determines how people interact with and experience products (see e.g. Chapters 14, 24, and 26). It is clear that driving the same car may be experienced

Introducing product experience

5

completely differently when on holiday, in a hurry for an important meeting, testing it in order to make a purchase decision, or with the love of your life sitting next to you.

Empirical approaches to studying product experiences Given that product experience is defined as the awareness of the psychological effects elicited by the interaction with a product, it is not surprising that most research in this area typically assesses people’s subjective reports of their experiences with products. In the majority of studies, products or product parts are varied or manipulated under naturalistic conditions, and the effect of the manipulation on the subjective reports is assessed. In experimental studies the manipulations are typically done in a more or less systematic way, in order to isolate various underlying factors. In case studies and design projects, on the other hand, the manipulations are determined by wishes, demands, and limitations given by the product and its usage context, company goals, and designer capabilities. These two approaches yield complementary outcomes that both contribute to our knowledge on product experiences. The subjective reports may consist of either qualitative (e.g. in-depth interviews, diaries) or quantitative (e.g. responses on rating scales, preference rank orders) data. The ensemble of approaches used in these studies assures the ecological validity of the outcomes. However, subjective responses can only be obtained through introspection, and cannot be assessed objectively. Therefore, in several chapters self-reports are related to more objective measures, such as physical characteristics or instrumental measures of products, measures of information processing or decision making, consumer choice data, or psychophysiological measures and brain imaging data (see Chapters 1, 3, 18 and 20). Adding more objective measures and analyzing the relationships with subjective measures enables an internal validity-check on the subjective reports.

An overview of contributing scientific disciplines Research on product experience is situated at the intersection of several scientific (sub)disciplines. Because product experience research focuses on subjective experiences of people, all product experience research falls within the discipline of psychology. However, due to its multi-faceted nature, experience research crosses borders between several sub-disciplines of psychology that are usually distinguished, such as psychology of perception, cognitive psychology, and psychology of emotion. Acknowledging these different sub-disciplines, many questions can be asked concerning the interaction between a person and a product. How do people use their senses in experiencing products? How do people understand how to use a product? Why are people attracted to some products and not to others? On what grounds do people perceive a product as smart, solid, stupid, or splendid? Which memories, associations, and emotions does a product evoke? Why do people develop a bond with a product? These are the kinds of questions that will be addressed in this book. Although the contents of some chapters are clearly linked to one of these subdisciplines of psychology (e.g. Chapters 1–5 to perception; Chapters 7, 8, and 12 to cognition; and Chapters 15 and 16 to emotion), other chapters build on all sub-disciplines, and cover an applied field of knowledge. Most of these applied disciplines have their tradition in the social and behavioural sciences, such as psychological aesthetics, human factors, marketing, and consumer science. Others, however, have their roots in the technical sciences, such as mechanical and material engineering, and human–computer interaction (HCI). Analogously to how all these disciplines together contribute to and define

6

Product Experience

FIGURE 2 Disciplines contributing to the field of product experience.

the multi-disciplinary field of industrial or product design, they also converge in this relatively new domain of research: Product experience (see Figure 2). We now discuss each of these contributing disciplines briefly, and we indicate which (often new) areas or fields of study within each discipline ‘move’ into the emerging domain of product experience. Philosophers and psychologists have extensively studied people’s responses to works of art. From the field of philosophical aesthetics, the work of John Dewey has been most influential to the domain of product experience. In his ‘Art as Experience’, Dewey (1934) analyzes people’s involvement with artworks from a phenomenological perspective (see Chapters 9, 13, and 19). Ever since psychology was founded as a field of science, psychologists have had a keen interest in aesthetic experiences and evaluations of ‘objects’, such as simple patterns, faces, paintings, and landscapes (see e.g. Berlyne, 1971; Fechner, 1876). In their studies they generally looked for the principles governing people’s perception and appreciation of these manifestations by applying more general theories of perception, motivation, cognition, and emotion. More recently, psychologists have discovered products as an interesting subject area to investigate these aesthetic or pleasure principles (e.g. Crozier, 1994). Most of this work fits perfectly in the domain of product experience (see Chapters 10 and 11). The discipline of ergonomics or human factors traditionally focuses on the usability of products (in itself already an experiential goal). For a long time, the discipline limited itself to the perceptual and cognitive processes involved in product understanding, and to the physical or motor skills and processes enabling (or limiting) product use. In the current book, the perceptual systems and the way they operate in interaction with products are covered in Chapters 1–6, while the cognitive abilities and skills and their

Introducing product experience

7

effects on the interaction with products are covered in Chapters 6–8; human motor abilities and skills affecting interaction are predominantly discussed in Chapter 6. One of the ways in which products can be made easy to use is by making it self-evident how a product should be operated (see Chapter 12). However, this perspective has been expanded. At the end of the twentieth century the ergonomic discipline increasingly began to focus on other subjective experiences arising from the use of products, including research on satisfaction, pleasure (e.g. Jordan and Green, 2002), and comfort and convenience (Vink, 2005, see Chapter 18). Mechanical and material engineering have developed from having a singular focus on the technical/physical properties of artifacts and their effects on the durability, reliability, production, and (technical) performance of products, to studying, quantifying, and modeling the relationship between such properties and sensorial and other subjective responses in terms of meaning and aesthetics (e.g. Ashby and Johnson, 2002). This movement is most prominently seen in Kansei Engineering, a branch of engineering that started in Japan in the 1970s (see Chapter 20 for an extensive account of this approach). Technology-driven research focuses on how products can be created with new technologies that may be beneficial to potential users. The majority of this research consists of studies on the application of digital or smart technologies in human–computer interaction. Designers are interested in exploring new functionalities and interaction possibilities that can be created with these new technologies (see e.g. Moggridge, 2007). Because these technologies are applied in more and more everyday products, the findings obtained in the field of human–computer interaction become increasingly relevant for the entire product spectrum. In addition, within the HCI field we also see a shift from usability research to user experience research, variously looking at experiences such as presence, fun, trust, or engagement (e.g. Blythe, Overbeeke, Monk, and Wright, 2003). Various issues of user-product interaction with respect to digital technology are discussed in Chapters 7 and 8. In addition, the experience of functional interfaces (Chapter 21), intelligent products (Chapter 22), and computer games (Chapter 23) is extensively treated in Part III of this volume. The field of marketing studies how products find their way to customers. Traditionally, the marketer can make use of the four instruments in the marketing mix to bring the product to market in a profitable way: product, price, promotion, and distribution (Kotler, 1984). Research on product experience will typically be focused on the subjective evaluation of a physical product or service. This may concern a first encounter with the product in a store (Chapters 25 and 27), or a repeated encounter during product usage (Chapters 16 and 17). Also in the field of consumer research, research attention has shifted from information processing approaches with a focus on utilitarian value and price, to the emotional experiences associated with product consumption (e.g. Holbrook and Hirschman, 1982). In summary, the field of product experience research encompasses research from each of these disciplines. However, to fully understand human product experience, we need to use approaches that allow us to build bridges between these various fields of expertise. In our final ‘reflections’ we will further elaborate this view on the future of the research domain. People’s experiences with products are by definition subjective; probably no two experiences are alike. Individual differences in terms of, for instance, gender, age, expertise, and (cultural) background may account for the variety in people’s experiences, and some of these differences are discussed in this book. Finally, this book is not only about understanding product experience: We want to understand this phenomenon in order to contribute to ‘design for experience’. Hence, many authors address the design implications of their theories and findings, and they provide directions or guidelines to

8

Product Experience

increase the likelihood that a particular designed interaction will lead to the intended experience. Given that there are many scientific areas relevant for the study of product experiences, it is no surprise that the authors who contributed to this book differ considerably with respect to their scientific background and expertise. To preserve the coherence in the book, we have stimulated all authors to link their views and research to other related areas, to combine theoretical overviews with empirical studies, to provide concise descriptions of the methods they used, and to describe how their results may be used in design practice. It has proven to be a hard task for many authors to leave the field of expertise in which they feel comfortable, to explore largely unknown, but nonetheless fascinating, neighboring areas. This additional effort has made this book a unique collection of chapters that try to build bridges, for example, from neurophysiology and traditional psychophysics to product semantics, from universal aesthetic principles to design applications, and from human physical capabilities to product and consumption emotions. With this collection of chapters we hope to lay down the basics for the field of product experience research.

REFERENCES Ashby, M. and Johnson, K. (2002). Materials and design: the art and science of material selection in product design. Oxford, UK: Butterworth-Heinemann. Berlyne, D. E. (1971). Aesthetics and psychobiology. New York: Appleton-Century-Crofts. Blythe, M. A., Overbeeke, K., Monk, A. F. and Wright, P. C. (Eds.) (2003). Funology: from usability to enjoyment. Dordrecht: Kluwer Academic Publishers. Crozier, W. R. (1994). Manufactured pleasures: psychological responses to design. Manchester: Manchester University Press. Desmet, P. M. A. and Hekkert, P. (2007). Framework of product experience. International Journal of Design, 1, 57–66. Dewey, J. (1934). Art as experience. New York: Berkley Publishing Group. Fechner, G. T. (1876). Vorschule der Ästhetik. Leipzig: Breitkopf und Härtel. Hekkert, P. (2006). Design aesthetics: principles of pleasure in design. Psychology Science, 48, 157–172. Holbrook, M. B. and Hirschman, E. C. (1982). The experiential aspects of consumption: consumer fantasies, feelings, and fun. Journal of Consumer Research, 9, 132–140. Jordan, P. W. and Green, W. S. (2002). Pleasure with products: beyond usability. London: Taylor and Francis. Kotler, P. (1984). Marketing management: analysis, planning and control. London: Prentice-Hall. Moggridge, B. (2007). Designing interactions. Cambridge, MA: MIT Press. Pine, B. J. and Gilmore, J. H. (1998). Welcome to the experience economy. Harvard Business Review, 76(July/ August), 97–105. Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110, 145–172. Schifferstein, H. N. J. and Cleiren, M. P. H. D. (2005). Capturing product experiences: a split-modality approach. Acta Psychologica, 118, 293–318. Schmitt, B. (1999). Experiental marketing. Journal of Marketing Management, 15, 53–67. Vink, P. (2005). Comfort and design – Principles and good practice. Boca Raton: CRC Press.

1

ON THE VISUAL APPEARANCE OF OBJECTS HAROLD T. NEFS University of St. Andrews, Scotland, UK

1. ON VISUAL APPEARANCE 1.1. Introduction When looking around in the world, humans have to figure out what it is they see, what the shapes of things are, what kind of materials they are made of, and what their potential uses or dangers are. In addition to meaning, aesthetic value or affordance, the visual system also extracts some more fundamental object properties. Examples include the shape, size, glossiness, and lightness of an object. Counter-intuitively maybe, these aspects of the percept are not objective qualities because they are constructed in and by the human mind. Some of them have obvious correlates in the physical world, but this is often not the case. For example, objects might look flatter when they are made from a different material (e.g. Khang, Koenderink and Kappers, 2003; 2004). The position, that perceived object shape is determined exclusively by the objects’ physical shape, is therefore clearly untenable. Likewise, perceived material and perceived illumination characteristics are also not determined exclusively by their respective physical counterparts. The problem of what objects look like is certainly not a trivial problem. It determines whether people approach an object carefully or not, if and how they pick it up, etc. In this chapter we examine the difficulties that humans face when looking at the real world. In particular we will examine the perceived shape, material properties, and illumination of objects with special emphasis on the perceived shape. A question that has inspired and frustrated many scholars over the last few millennia is why objects look the way they do. In the ancient Greek tradition it was thought Product Experience Copyright © 2008 Elsevier Ltd.

11

12

Product Experience

that objects send out exact little copies of themselves to the eyes where they are mysteriously inspected by the mind (Howard, 2002). Aristotle, however, rejected the idea that objects emitted substance to the eyes, and he clearly had the idea of the rectilinear propagation of light. Although this seems quite obvious from our modern vantage point, for a very long time it was not properly understood how image formation in the eye was accomplished. In our modern understanding we know that an image of the world is projected onto the back of the eyes where it is further processed and transported to the brain through the optic nerve. The visual image is a two-dimensional projection of the threedimensional world on the retina. In spite of the fact that the depth dimension is lost in the process, humans report a clear impression of a three-dimensional world. Somehow the brain reconstructs the world from what is only available in a degenerate two-dimensional projection. The main problem for vision lies in the fact that for any visual image there are an infinite number of possible objects that give rise to the same image. This ambiguity not only lies in the fact that the depth dimension is lost in the projection onto the twodimensional retina, but also in the materials that the object is made of and how the object is illuminated. At first sight, the reconstruction of the world in the mind is thus an impossible task. However, if the visual system were to use some assumptions about the world, and constraints in the way it reconstructs the world, the task becomes much more manageable. Some of the assumptions that one can make about the world are very powerful in constraining the number of possible solutions, while still maintaining a high predictive power in the physical world. For example, it is reasonable to assume that what is smoothly connected in the visual image is also smoothly connected in physical space. It is a highly coincidental situation indeed that two unconnected surface patches still form a smoothly connected patch in the visual image. It is more likely that an area of constant luminance, or a smoothly changing area, corresponds to a single, smoothly connected surface in physical space and that an area with a sharp transition indicates two separate surfaces. Obviously, the more assumptions one makes, the further the class of possible surface solutions is restricted. However, the chance that the reconstructed object is not accurate also increases with each additional assumption. Before we can begin to understand how objects are perceived, we need to know about the physical behavior of objects. How they reflect light, and what the descriptors of shape are that can be used. In other words, what kind of information about the depth dimension is preserved in the projection from a three-dimensional object to a two-dimensional retinal image? A pattern in the retinal image can only be an effective cue to the threedimensional shape if it has a causal relationship with the real shape of the physical object. In order to understand how the physical world is represented in the mind, one thus needs to study the physical properties of objects under projection. From our knowledge of the physical behavior of objects we might be able to derive potential cues for the visual system to derive a three-dimensional reconstruction of the world. The idea that the observer is inextricably linked with its environment was very fashionable at times in the twentieth century. Gibson (1966), for example, stated that an understanding of the physical properties of the world is essential for understanding perception. Once the physics of object properties is sufficiently clear for its present purpose, the question can be raised of what physical correlates perceived object properties have. It is clear that the perceived shape of things must correspond sufficiently reliably to the physical shape otherwise the percept would be quite useless. However, the physical shape in itself is not what the visual system processes. It rather processes light and dark distributions and their changes over time in the visual image. It is well known from the literature, and from everyday life as well, that shapes might look different if they are illuminated

On the visual appearance of objects

13

from a different direction (e.g. Nefs, Koenderink and Kappers, 2005) or when you paint them in a different color. In other words, we want to know what object properties make objects look big, three-dimensional, rough, shiny, etc. As will be clarified later, there are physical relationships between shape, material and illumination once certain simple assumptions have been made. A further question emerges then. Namely, what is the internal coherence of the percept of an object as a whole? For example, one can come up with an algorithm to reconstruct the shape of an object from the shading pattern once an assumption is made on what the material and illumination conditions are. Changing these assumptions should change the reconstructed shape in systematic ways. However, is that actually the case? Does the visual system construct a coherent representation of the world, or are the percepts of shape, material and illumination essentially independent? The question that is raised is thus whether and how physical relationships between shape, material, and illumination are duplicated in perceptual relationships. In order to answer the questions that I raised above, I will first give in Section 2 the necessary theoretical background on the physical properties and relationships between shape, material, and illumination of objects in projection. Secondly, in Section 3, the physical correlates for perceived object properties are discussed. Thirdly, in Section 4, I will discuss perceptual organization and whether perceived object properties conform to physical constraints; that is, whether they form a coherent set. In addition I will discuss how additional cues can be used to form a coherent percept.

2. THE PHYSICAL WORLD 2.1. Shape Before starting a discussion on perceived shape, it is useful to agree on a measure to determine the extent to which two different object shapes are different or the same. The first step is to restrict shape to the ecological realm that can actually be seen. Every natural object is thoroughly fractured at the microscale and unstable, whereas at the macroscale it may be quite smooth and unchanging. The microscale is, however, not very interesting for visual perception. Likewise, the fact that the earth is round at the macroscale is not of any importance to perception since we are more concerned with smaller things. Given the restriction on the ecological realm, a transformational approach can be used to describe how two object shapes are different. For example, two objects might be the same in every way that can be discriminated by the spatial resolution of the visual system, except for their position or orientation in three-dimensional space. We can therefore say that these two objects are related by a translation and/or a rotation. All object shapes can be transformed into each other by some mathematical function. A prerequisite is that it is known which points on one object correspond to which points on the other object. Although this is not a trivial problem, let us for now assume that we do know. I think it is reasonable to say that two different sugar cubes that lie next to each other on a table and have the same global shapes are the same except for their position and orientation. That is, a sugar particle in the upper left corner of the first object corresponds to a sugar particle on one of the corners of the second cube, within a reasonable spatial tolerance, but not to a random particle somewhere in the middle. At the end of the nineteenth century, the German mathematician Felix Klein proposed to order transformations according to the properties that they leave invariant. This ordering has become known as the Klein umbrella. The transformations that are grouped

14

Product Experience

under the Klein umbrella are not the only possible transformations one can make from one object to another, there are many more, less restrictive, geometries possible, such as a topology. However, the Klein transformations have some important properties that are highly relevant for perception. For example, they all preserve connectivity and smoothness. That is, all points on a surface that are smoothly connected to another point are still smoothly connected to the same point after transformation. Therefore, this kind of transformation does not tear a surface apart, nor does it make holes in it. Cederberg (2001) gives a relatively accessible text on modern geometries. The transformations in the Klein umbrella are illustrated in Figure 1.1. Although we do not do so here for illustration reasons, the images of Figure 1.1, and the definition in Equation 1.1, can be expanded into three spatial dimensions. In the remainder of the chapter we are interested in three-dimensional shape rather than two-dimensional shape. The most restrictive transformation is when two different objects are the same in every way except for their position. These two objects are related by a translation or a rotation. We say that these objects are isometries of each other. An isometry transformation preserves the absolute distance between all points in an object. Isometries are illustrated in Figure 1.1 in the left most column. A more general transformation than the isometry preserves only the ratios of distances between points of an object and some point in the field. This is a called a similarity transformation. Dilation is a similarity transformation, but not an isometry. Note that all isometries are also similarities, because if absolute distances are preserved between all points, so are their ratios. Similarity transformations are illustrated in Figure 1.1 in the second column from the left. Storing a representation in memory in a similarity geometry instead of in a full Euclidean format has distinct advantages. For example, objects can be recognized irrespective of their size in either physical space or on the retina. That is, object representation and object recognition are size-invariant. The third group of transformations preserves parallelity but not necessarily position, distance or ratios of distances. All that is parallel in the original object is still parallel in Projectivity Affinity Similarity Isometry Translation

Rotation

Dilation

Strain

Collineation

Shear

FIGURE 1.1 The Klein umbrella. The black figure is in each case the original figure and the gray image is the transformed figure. From left to right isometries, similarities, affinities and projectivities are illustrated.

On the visual appearance of objects

15

the transformed object. These transformations are called affinities. Straining and a shearing are both affinities, but not similarities. Straining is increasing or decreasing the size of the object in one dimension only, whereas the dimension at right angles with the first one remains unchanged. Shearing translates the points in the image in one dimension as a function of the second dimension. For example, the more you go to the right, the more the points are translated upwards. It is like holding on to left side of the object while pulling it up on the right side. Straining and shearing are illustrated in the third column from the left in Figure 1.1. All similarities are also affinities, because they also preserve parallelity. The fourth group preserves collinearity. All points that are collinear, that is on a single straight line, in the original object are also collinear in the transformed object. Since areas of zero curvature have at least three collinear points, these areas thus remain invariant as well. These transformations do not add extra bumps or dimples on a surface. Transformations that preserve collinearity are called projectivities. All affinities are also projectivities. An interesting property is that the cross-ratio of four distinct collinear points is invariant. The cross-ratio can be interpreted as the ratio of ratios of distances. More complex, but also invariant is a five point cross-ratio in which the points do not have to be collinear. Storing mental object representations in a projective geometry can have distinct advantages for object recognition. Objects that are stored in a projective format are invariant for changes in viewpoint. For example, the fact that the cross-ratio of any set of four collinear points, or the five-point cross-ratio, on an object remains invariant after a projective transformation can be exploited in machine vision for automated object recognition. The transformations that are shown in Figure 1.1 look complicated at first glance, but they can be described by a simple matrix multiplication as in Equation 1.1. ⎡ xi ⎤ ⎡ a r u ⎤ ⎡ xi ⎤ ⎢ ⎥ ⎥ ⎢ ⎥ ⎢ ′ Equation 1.1: X ⫽ ⎢ yi ⎥ ⫽ A ⋅ X ⫽ ⎢ s b v ⎥ ⋅ ⎢ yi ⎥ ⎢ ⎥ ⎢ p q 1⎥ ⎢ ⎥ ⎢⎣ hi ⎥⎦ ⎥⎦ ⎢⎣ hi ⎥⎦ ⎢⎣ In this equation, X is the set of points of the first object and X⬘ is the set of points on the transformed object. The points in the sets X and X⬘ have been put in the same order. That is, point Xi corresponds to point X⬘i. The two objects are two-dimensional, xi and yi are the coordinates of point i. We added a third dimension, h. The triplets {xi, yi, hi} are homogeneous coordinates. Cederberg (2001) describes the convenience and use of homogenous coordinates in more detail. The thing to remember about homogeneous coordinates is that all {xi, yi, hi} are equivalent to {xi/hi, yi/hi, 1}. The Matrix A is the transformation matrix. The parameters a and b strain the object, s and r shear the object in the image plane. Rotations can be obtained by specific combinations of a, b, s, and r. The parameters u and v translate the object in the image plane, and p and q shear the images in the extra dimension h; the latter induces a projective transformation in the image plane.

2.2. Material For the perception of material properties it is important to know how the reflection of objects can be described. The restriction is to the ecological realm that can actually be seen. Furthermore, objects that emit light themselves, such as lamps, or objects that store energy and emit it over a prolonged period of time, such as ‘glow-in-the-dark’ paint, are not considered here.

16

Product Experience

FIGURE 1.2 Lambert’s Law. Light beam B is spread out over a larger surface area than beam A. Therefore, the illumination per surface unit area is less in the case of beam B than in the case of beam A.

When light falls on a surface, it is reflected, absorbed, or passes through. The total amount of reflected light as a proportion of the light that falls on the surface is called the albedo or ‘whiteness’. Obviously an object cannot reflect more light than what falls on its surface. This has an interesting consequence. Namely, if light falls on a surface under an angle, the incident light is smeared over a larger area than when it hits the surface perpendicularly (see Figure 1.2). Therefore, the amount of reflected light per surface unit area depends on the angle of illumination relative to the surface. The apparent darkening of a surface depending on the illumination angle is called shading. Shading is a primary cue for the visual system for shape perception. The relationship between luminance and the angle of incidence is known as Lambert’s Law. Lambert’s Law can be expressed as in Equation 1.2: Equation 1.2: λ(n, i) ⫽ i ⋅ n ⫽ Iin * Cos(β ) In this equation, λ is the surface luminance. Luminance is thus the amount of light that is reflected from the object to the eye, whereas illuminance is the amount of light that shines on the object. The surface normal is denoted by n, and i is the vector describing the direction of illumination. The length of i is the illumination intensity Iin. The angle of incidence, that is the angle between the surface normal, n, and the direction of the incoming light, i, is denoted by β. These notions are illustrated in Figure 1.3. Each surface has its own reflection characteristics on top of Lambert’s Law. These characteristics are known as the Bidirectional Reflection Distribution Function, or BRDF. In general, the BRDF depends both on the angle of incidence and also on the viewing angle. The observed luminance can then be described as the product of the intensity of the illuminant, the dot product of the surface normal and the direction vector of the illuminant, and the BRDF as in Equation 1.3. Equation 1.3: λ(n, i, p) ⫽ i ⋅ n * BRDF(n, i, p) The vector p is the viewing direction. Nicodemus et al. (1977) give a good description of reflectance properties.

On the visual appearance of objects

17

FIGURE 1.3 The surface normal, n, is the vector that is perpendicular to the surface. The illumination direction is indicated with i. Vector p indicates the viewing direction. The angle α that is spanned between n and p is the viewing angle. The angle β that is spanned between n and i is the angle of incidence.

FIGURE 1.4 Computer renderings of a slightly bumpy sphere. The objects have, from left to right, a Lambertian, specular, backscatter and asperity BRDF. The two rows depict the same objects and BRDFs, but illuminated from different directions. The top row is illuminated from the viewpoint of the observer. The bottom row is illuminated from an area to the right of and above the observer.

In the special case that the BRDF equals one for all incidence and viewing angles, we say that the surface is a Lambertian surface. In that case, the viewing direction, p, does not appear on the right-hand side of Equation 1.3. Therefore, the luminance of a Lambertian surface is independent of the viewing direction. The incoming light is scattered from the surface in all directions irrespective of the angle of incidence. Many of the less-sophisticated computer graphics algorithms render objects using Lambertian surfaces since the luminance computation is easy, fast, and gives a reasonably realistic threedimensional impression. There are, however, no known materials that actually reflect light in a Lambertian fashion even though many materials come close. Figure 1.4 shows images of the same slightly bumpy sphere that are rendered with different BRDFs. From left to right in Figure 1.4 we used a Lambertian, specular, backscatter and asperity BRDF. A nearly Lambertian BRDF can, for example, be found on objects carved from chalk. In the case of specular reflection, incident light is mostly

18

Product Experience

reflected in the direction where the angle of incidence is mirrored in the surface normal (i.e. α ⫽ β in Figure 1.3). All mirrors, including Christmas balls, have a specular BRDF. Objects often show a combination of a Lambertian and specular BRDF. These objects have a Lambertian-like shading, but with added specular highlights, and are perceived as being ‘glossy’. Specular highlights are the bright spots on a glossy object where the illumination direction and the viewing direction are exactly mirrored in the surface normal. Examples of these BRDFs include glossy fruit, plastic, and porcelain. In the case of a backscatter BRDF, incident light is mostly reflected back into the direction of the light source. Backscatter BRDFs can be found on the retroreflectors on bicycles. Retroreflectors cast the light back towards the headlights of an approaching car. An asperity BRDF can frequently be found on materials with short hairs such as peaches and velvet. An asperity BRDF makes the areas near the contour of the object brighter. The two horizontal rows in Figure 1.4 depict the objects when illuminated from 0 and 45 degrees away from viewing direction. For illustration purposes, the illuminant is made to be stronger for the backscatter and asperity BRDFs than for the Lambertian and specular BRDFs (see also Pont and Te Pas, 2006). Some materials are partly translucent. That is, incoming light penetrates the surface and is scattered or reflected inside the object. This light is not absorbed, but eventually leaves the object again. A most-clear example is the reflectance of a diamond. The light that falls on a diamond penetrates the surface and is then reflected again and again in the facets inside the diamond until it finally finds a way out. The particular way a diamond is shaped causes the light to bounce so many times that the point where the light eventually leaves the diamond is unpredictable and highly unstable with small changes in the illumination direction. This gives a diamond its distinctive fire. Another example of a translucent material is human skin. Incoming light is scattered below the surface through the skin layers. The translucency of the skin gives it its slightly reddish appearance and softens the edges of cast shadows on the body. I will not discuss translucency in this chapter any further. Nicodemus et al. (1977) describe an extended form of the BRDF that includes subsurface scattering as well.

2.3. Illumination A beam of light has intensity, direction and a wavelength spectrum. It always propagates in a straight line unless it is reflected. The illumination in a scene can all come from one direction, such as a scene that is directly illuminated by the sun. The illumination is in this case parallel. Common forms of illumination are illustrated in Figure 1.5. In the case of a spotlight or point light source that is relatively close to the scene, the illumination is convergent or divergent. Usually, however, there is light coming from multiple directions. Illumination that comes from all directions is called diffuse or ambient. In the case that illumination comes from only one half of all possible directions, it is called hemispherically diffuse. For example, in the case of an overcast sky, the majority of illumination is coming from all directions in the upper hemisphere (the sky) and very little from the ground. Of interest is to note that ambient light in combination with parallel light softens cast shadows. Parallel light gives hard cast shadows that hide many features of the object whereas it enhances surface imperfections (tiny bumps and dimples). Photographers often make use of ambient light from a relatively small range of directions with a ‘soft box’ or an umbrella to make the image esthetically more appealing. This kind of light is much more ‘forgiving’ of surface imperfections.

On the visual appearance of objects

19

FIGURE 1.5 Illumination conditions. Illumination can be parallel, divergent or convergent. Often light comes from all directions which is called ambient or diffuse. In the case light comes from only half of all possible directions, it is called hemispherically diffuse.

2.4. Color Of special interest in the appearance of objects is their color as it can set the objects apart from the environment and might signal their use or condition. For example, a slightly brownish tint on an apple signals that is rotten. Law-Smith et al. (2006) found that there are visual cues in female faces that are indicative of high estrogen levels and which are associated with ratings of femininity, attractiveness, and healthiness. Law-Smith et al.’s facial images suggest that a slightly reddish tint is one of those cues. Humans have four different types of photoreceptors that allow them to see light with wavelengths between roughly 350 nm (violet) and 750 nm (red). Three photoreceptors are associated with color vision whereas the fourth photoreceptor is associated with ‘night vision’. Wyszecki and Stiles (1982/2000) give an elaborate overview on color science, as well as a wealth of quantitative data on the subject. Objects reflect light for different wavelengths to different amounts and thereby create different colors. Elementary theory on color mixing can be found in any undergraduate textbook on sensory psychology. There are many systems available to order and describe colors. At the heart of all systems are the so-called CIE color matching functions. CIE stands for ‘Commission Internationale de l’Eclairage’ (International Committee on Illumination). The CIE color matching functions were first reported in 1931 and have served since then as the standard definition of color, although other color matching functions have been measured as well. The color matching functions describe the intensities of three light sources with different spectra that, when they are combined, look the same as a light source of a single wavelength. Light that is composed of only a single wavelength is called monochromatic. Monochromatic light is, for example, used in, typically yellowish, sodium streetlights because they are particularly energy efficient. Usually, natural light consists of a spectrum of different wavelengths.

20

Product Experience

The three light sources that were used to measure the CIE color matching functions were quite arbitrarily chosen. In fact one can perform the same procedure with three different light sources as long as they are sufficiently far apart in wavelength or color. It is relatively straightforward to calculate the correspondence between sets of color matching functions made with different light sources. I have plotted the CIE color matching functions in Figure 1.6A. The points of monochromatic lights form a continuous curve in the space spanned by the three light sources. A ‘color cone’ can be constructed by connecting the origin with all monochromatic points. All possible colors that can be perceived have a corresponding coordinate within the color cone. Usually, a cross-section is made through the color cone at which the intensity of the light is perceived as constant, as in Figure 1.6B. In Figure 1.6B all data points lay on a plane. This plane forms the CIE (x, y)-chromaticity diagram that might be familiar to the reader. Colors can be described with their coordinates in this plane, along with their intensity. Note that some colors, namely the purples, are not on the color-matching curve. There are no wavelengths that correspond to this color, because they can only result from a combination of a ‘reddish’ and a ‘bluish’ wavelength. Purples can be found on what is called the purple plane, the

B

A

Blueish

Blueish

Gr

ee

ish

dd Re Gre

nis

h

enis

dish

Red

h

C

h dis

Greenish

d

Re

FIGURE 1.6 Colors and object colors. (A) The color cone in the CIE (x, y, z) color space. (B) The positioning of CIE (x, y) chromaticity diagram in the color space. (C) All object colors in the CIE (x, y, z) color space that can be made with illuminant CIE-D65.

On the visual appearance of objects

21

flat plane in Figure 1.6A, or purple line, the straight line in Figure 1.6B, in case of a chromaticity diagram. This plane closes the gap between the red side and blue side of the color cone. The chromaticity coordinates for any light source can easily be found by multiplying the color matching functions with the spectrum of the light source. The color matching functions can, for example, be obtained from the Color and Vision Laboratories of the Institute of Ophthalmology at the University College London (http://www.cvrl.org). It is possible that different spectra yield the same chromaticity coordinates. The colors of these light sources are thus the same. This phenomenon is known as metamerism. Usually, objects with metamer colors can easily be distinguished by looking at them through a colored filter or by shining light of a different spectrum on them. For example, a trick that is sometimes employed by the colorblind to distinguish red and green is to view them through a red filter. Red and green are metamer colors for the colorblind. Viewing a red object through a red lens does not change its intensity, but a green light that is viewed through a red filter turns to black. One particular shortcoming of the CIE (x,y)-chromaticity diagram is that it does not give a quantification of how well two different colors can be discriminated. Furthermore, how far two different colors have to be apart in the chromaticity diagram in order to be seen as distinct depends on the color as well. Additional transformations can be made to rectify this problem, as is for example done in the CIE (x⬘,y⬘) chromaticity diagram and the CIE (L * u * v) and (L * a * b) color spaces. Not all colors can be made given a certain illumination spectrum. If there is no light in the red part of the spectrum, it is impossible to make an object look red, whatever its spectral reflection function. For example, the color of an object is very different in daylight than under the illumination of yellow streetlights. In fact, since sodium streetlights are monochromatic, there are no colors any more! All light that comes into the eye is of the same wavelength, irrespective of the material that has reflected it to the eye; only the intensity of the light differs. The casual observer might not even notice the absence of color under these conditions though, because there is still a lot of variation in light intensity. Given a certain illumination spectrum, it is possible to calculate all possible colors that can be made from shining this light on a surface. The resulting colors are called object colors. An object color is the color that the observer identifies as the color of an object. Object colors can thus, depending on the spectral composition of the illumination, take on different values. The object colors for a given illumination spectrum fill only a part of the color cone. The set of object colors thus fills a different volume, often called ‘color solid’, for each specific illumination spectrum. The volume is filled with all colors that can result from all possible reflectance functions and given a certain illumination spectrum. I have created such a color solid in Figure 1.6C. I have taken a standard illuminant, namely ‘CIE-D65’, and calculated all colors that can result under that illumination. The CIE has specified several standard illuminants for this purpose. There are many systems to order different object colors. I will briefly mention one particularly popular model, namely the Munsell Color system. The Munsell system orders object colors on three dimensions namely: Luminance; hue; and saturation. Luminance is a value between black and white. Hue can be described as ‘red’, ‘blue’, ‘purple’, and so on. Saturation is the amount of hue as opposed to being gray. Pastel colors, for example, have a low saturation. The system is composed of a set of differently colored chips that can be arranged in a solid, much like in Figure 1.6C. The chips are chosen such that their discriminability increases in equal steps. The Munsell color system does not take into account how colors of materials change over viewing, illumination angle, and spectral composition of the light source. Calibrated chip sets for the Munsell are commercially

22

Product Experience

available. More detail on the CIE color matching functions, the Munsell color system, etc., can be found in Wyszecki and Stiles (1982/2000). Object colors as discussed so far do not form the entire story though. There are a few restrictions. First, the BRDF is dependent on the wavelength of illumination. A striking example is the Compact Disc. Light with the same angle of incidence but different wavelengths is reflected in different directions on a Compact Disc. A full description of the color of an object therefore involves the viewing and illumination directions as well. Secondly, the perceived color of objects also depends on the color of the environment. The chromaticity coordinates and the Munsell system is only valid when the object or light is viewed against a gray background. I will address this further in Section 3.2. Obviously, this has important consequences for the product designer, as the typical illumination and viewing conditions in which the product is going to be sold or used must be taken into account.

2.5. The SMI triangle It can be seen in Equation 1.3 that the luminance of a surface patch, as seen from the eye, depends on three components, namely (a) the surface normal; (b) the BRDF; and (c) the illumination. All that the visual system is presented with, however, is the object’s luminance in the direction from the surface to the eye. This gives rise to an interdependency of object properties which I will denote here as the Shape–Material–Illumination Triangle, or SMI triangle for short. The SMI triangle has two degrees of freedom. That is, you can choose any two vertices of the SMI triangle to be whatever you want, but the third one is then fixed. When reconstructing the world, all three vertices of the SMI triangle are a priori unknown. This does not mean, however, that one cannot make an educated guess. However, if one wants to find, for example, a solution for an object’s shape, one has to make assumptions on what the material and illumination conditions are. Ikeuchi and Horn (1981), for example, describe an iterative algorithm to reconstruct shape from shading. This algorithm assumes the BRDF and the illumination conditions to be known. One of the interesting questions that can be raised is what the resulting ambiguities are, given some reasonable assumptions about the real world. Let us first assume that every pair of neighboring points that are smoothly connected in the image are also smoothly connected in three-dimensional space. Vice versa, every pair of neighboring points that are not smoothly connected in the image are also not smoothly connected in three-dimensional space, that is, connected but not smoothly, or not connected at all. Furthermore we assume that illumination and material are homogeneous over the object. These two assumptions already reject trivial solutions such as paintings on canvas or highly unlikely light sources on flat surfaces such as computer beamer projections. Consequently, paintings and projections are perceived just as if they were three-dimensional scenes. The SMI triangle is best illustrated by looking at a specific example of an ambiguity in more detail. Although this example concerns the ambiguity of shape, similar examples can be made for illumination and material properties. One of the best-known ambiguities in constructing the surface of an object is the so-called bas-relief ambiguity, which translates from French as ‘low sculpture’. Bas-reliefs are relatively flat sculptures that still give a very good impression of depth. Bas-reliefs can be found throughout the centuries. In spite of the fact that the bas-relief ambiguity has been known for a very long time, its mathematical structure and relationship with the illumination direction and intensity has been derived only recently (Belhumeur, Kriegman and Yuille, 1999). The bas-relief ambiguity is shown in Figure 1.7. The ambiguity lies in the fact that the same image can be obtained from objects that are scaled or sheared in the depth dimension when the illumination direction and intensity is changed at the same time.

On the visual appearance of objects

23

FIGURE 1.7 The bas-relief ambiguity. The top row shows an untransformed globular object from a frontal viewpoint, and the same object as seen from the top. The other objects show a transformed object with a transformed illumination direction. In these images, albedo is kept constant. The object in the second row from the top is sheared with a value of 0.3. The third row from the top is strained in depth in the direction towards the observer with a factor of 0.7. The bottom row shows the object when it is both strained (1.2) and sheared (⫺0.3). Despite the large differences in 3D shape, the images in the left column are nearly indistinguishable.

24

Product Experience

The angle of incidence can be expressed as the difference between the directions of the surface normal and the direction of illumination. An affine transformation changes the direction of the surface normals in a linear manner. This gives us an intuitive understanding of the bas-relief ambiguity: If the surface normal is changed to the same extent as the illumination direction, the same luminance results and the images are therefore indistinguishable. Cast and body shadows are preserved exactly. However, in order to keep the shading pattern exactly the same in the image, a transformation in albedo is required as well. For example, when an object is flattened the contrast in the image is lowered and the object is then typically perceived as being flatter. In order to correct for this effect, the albedo of the surface has to be changed locally such that the surface becomes darker the more the surface normal turns away from the illumination direction. However, the necessity to transform the albedo becomes only apparent in very strong bas-reliefs. In Figure 1.7, I did not correct for albedo. Interestingly, introducing an ambient component in the illumination affects the contrast in the image as well; ambient light illuminates all parts equally irrespective of the surface normal, and therefore lowers the contrast within the image. Adding an ambient term can thus counterbalance the effects of straining and shearing. When an ambient component is added, the contrast in the image is reduced which leads to a flattening effect and, when the ambient term is lowered, it leads to higher contrast and to an enhanced impression of depth in the object.

3. OBJECT APPEARANCE 3.1. Shape Several methods have been used in the past to measure the perceived shape of an object. These measures involve local depth comparisons, cross-sections, depth–width ratios, and surface attitude probes. For example, Koenderink, Van Doorn and Kappers (1992) introduced a popular method based on measuring the surface attitude. The method involves a surface attitude probe or a so-called ‘gauge figure’. The probe typically consists of a small circle with a little stick that is superimposed on the image. It resembles the shape of a pushpin. The aspect ratio and orientation of the gauge figure can be changed such that it looks as if the probe rotates in three-dimensional space. The probe is located on the image of an object and the observer sets the probe such that it looks like a small circle that is painted on the surface, with the little stick perpendicular to the surface. Across trials, the probe is placed at numerous points all over the image. The setting of the gauge figure is usually, but not necessarily, measured in slant and tilt. Slant is the angle away from the line of sight and tilt is the angle around the line of sight. Slant and tilt are illustrated in Figure 1.8. An example of the gauge figure task is shown in Figure 1.9. Figure 1.9A shows what the observer typically sees in an experiment during a single trial. The featured object in this case is an image of a globular virtual object that was created from a computer algorithm to simulate a three-dimensional shape with a glossy BRDF that was illuminated from a specific direction. Usually this kind of research is done on images of objects on a computer monitor in a darkened room rather than with real three-dimensional objects, for practical reasons. These images might be photographs taken of real objects or renderings of virtual objects that are created from a specific computer algorithm. In Figure 1.9B, I have accumulated all probe settings in one image. From the probe settings a surface can be recreated as in Figure 1.9C.

On the visual appearance of objects

25

FIGURE 1.8 Slant and tilt. A number of different circular patches are shown on a sphere. The slant of a patch is the angle at which the surface patch is turned away from the line of sight. Tilt is the angle around the line of sight. For example, patch A has a slant of 67 degrees and a tilt of 45 degrees. Patch B has a slant of 45 degrees and a tilt of 135 degrees.

FIGURE 1.9 The gauge figure task. (A) shows what the observer typically sees. The pushpin-like gauge figure can be manipulated by the observer and has to be oriented such that it looks like a circle painted on the object’s surface with the little stick as the outward surface normal. All observer settings are accumulated in (B). The settings are integrated into a ‘perceived’ surface in (C), for illustration purposes seen from a different viewing angle.

It should come as no surprise that the perceived surfaces of shaded objects show likeness with the ground truth up to a certain point. Human observers are also consistent in perceiving the three-dimensional shape of shaded objects, although perceived surfaces vary a lot in straining and shearing in depth when the procedure is repeated over different observers. Some observers set the gauge figures consistently to larger values than other observers. This might be interpreted as though these observers see more depth than others. However, it might also mean that observers map the aspect ratio of the gauge figure differently to a surface gradient in the mental world. There is somewhat less variance in straining and shearing when the procedure is repeated for the same observer over different sessions than when repeated over different observers. It has been found that most of the variance in perceived surface shape between observers or between tasks can be accounted for by affine transformations (Koenderink et al., 2001).

26

Product Experience

We can now address the question of the effects of material and illumination conditions on perceived shape. A change in illumination direction can have significant effects on perceived object shape. Curran and Johnston (1996), for example, found that the perceived curvature of spheres is affected by the position of the light source. Perceived curvature decreased as light source tilt (rotation in the image plane) increased and as light source slant (rotation away from the line of sight) decreased. Nefs, Koenderink and Kappers (2005; 2006) also found significant effects of illumination direction on the perceived depth and on the overall attitude of the object. The effective cue seems to be the overall contrast in the image. Less contrast in the visual image seems to be interpreted as the object being flatter. Increasing the level of ambient light makes the surfaces thus appear flatter, since contrast is reduced. In addition to straining and shearing the perceived surface, a change in illumination direction also induces deformations in the perceived surface that cannot be described by affine transformations. Koenderink, Van Doorn, Christou and Lappin (1996a,b) for example found systematic distortions of the perceived shape of human torso replicas with changes in illumination direction. For all observers in their study, those parts in the image that had a higher luminance were consistently interpreted as being nearer. Christou and Koenderink (1997) also concluded that even with images of simple ovoid shapes, subjects’ settings of surface attitude were systematically biased in the illumination direction, and were consistent with a regression to image luminance gradients. That is, dark parts are perceived as deeper and brighter parts of the objects are perceived as being closer. Nefs et al. (2005) also found that there are non-linear changes in perceived surface relief when the direction of illumination is changed. They described the change that was induced by changing the illumination direction as a shift in the position of the parabolic curves in the perceived surface. Parabolic curves are the sets of points on the object where the surface goes from being convex or concave to being saddle-shaped. In general, these points form closed curves on the surface. The shift in the convex, saddle-shaped and concave areas in Nefs et al.’s experiment is shown for one observer in Figure 1.10. The dark gray part represents a convex area, the black part is saddle-shaped and the light gray part is convex. The parabolic curves in the perceived surface reconstruction are located on the transitions from convex to saddle-shaped and from saddle-shaped to concave. The white jagged lines indicate where the parabolic curves of the physical shape are located. The knowledge of the effects of different BRDFs on perceived surface shape is somewhat less advanced. However, it is reasonable to conclude from Figure 1.4 that the BRDF clearly does have an effect. Most research has focused on the presence of specular highlights. Figure 1.9 shows an example of an object with three distinct specular highlights. There are good reasons for investigating specular highlights. First is a practical reason, specular highlights are easy to make in computer graphics and therefore easily available for researchers to play with. Secondly, specular highlights are not easily confused with Lambertian shading because of their high intensity. And thirdly, it has been argued that specular highlights are potentially informative of shape: Specular highlights are in general found on specific areas on an object (e.g. Longuet-Higgins, 1960; Oren and Nayar, 1996). Specular highlights have, for example, the tendency to cling to areas of high curvature and move rapidly over areas of low curvature when the illumination direction or the viewing direction is changed. This is also the reason why specular highlights follow predictable paths over the surface with changes in illumination or viewing direction. Specular highlights tend to move over the parabolic curves of an object. In the case of purely specular objects in complex environments we see a compression of the reflection of the world at areas where the object is strongly curved, and an expansion at areas of low curvature (e.g. Flemming, Dror and Adelson, 2003; Flemming, Torralba

On the visual appearance of objects

27

FIGURE 1.10 The top row shows the stimulus as used in Nefs et al. (2005). Observers made gauge figure judgments on these two images. In the bottom row, two diagrams are shown that indicate the convex, hyperbolic and concave areas in the perceived surface. The dark gray areas are convex, the black areas are hyperbolic and the light gray areas are concave. The jagged white lines show the positions of the parabolic curves of the real, physical shape are, which were the same for both objects. Reprinted by permission from Pion, London: Perception, 34, 275–287, copyright 2005.

and Adelson, 2004). For example, the reflection of the world in a Christmas ball makes everything look small, whereas the same world viewed in a flat mirror appears to have its original size. On objects that have both flat and curved areas, the world is thus compressed to a different extent depending on the surface curvature. The scientific evidence that the addition of specular highlights to otherwise Lambertian reflecting objects induces differences in perceived surface shape is contradictory though. On the one hand, there are studies that support an effect of specular highlights. Todd and Mingolla (1983) reported that the presence of specular highlights on cylindrical surfaces enhances the perception of surface curvature. Norman, Todd and Orban (2004) showed that shape discrimination was improved when specular highlights were added. Todd, Norman, Koenderink and Kappers (1997) reported that observers’ judgments were significantly more reliable when specular highlights were added to Lambertian surfaces when objects were viewed stereoscopically. Blake and Bülthoff (1990; 1991) showed that specular highlights can be used by observers to disambiguate convex/concave surfaces in stereoscopic vision. On the other hand, there are studies that do not report significant effects of the presence of specular highlights. Mingolla and Todd (1986) found that changing the illumination direction influenced the judged shape of ellipsoid surfaces, but the presence or absence of specular highlights did not have a significant effect. Nefs et al. (2006)

28

Product Experience

also reported effects of illumination, but not of specular highlights, on shape perception for a range of globular objects. Norman, Todd and Phillips (1995) reported that observer settings were no less accurate for objects containing just specular highlights than for Lambertian objects or objects having both Lambertian shading and specular highlights. There are a couple of explanations for the different conclusions in the literature. An important reason is probably the specific shape of the objects that are used. For example, in the case of cylinders, specular highlights form a straight line over the cylinder. And in the case of an object with a large curvature contrast, the specular highlights may trace ridges and parabolic curves over the object. Khang, Koenderink and Kappers (2004) looked into the effects of Lambertian, specular, asperity, and backscatter BRDFs on perceived surface shape. They found that spheres and ellipsoids that were rendered with asperity or backscatter BRDFs were perceived as being flatter than those rendered with a Lambertian or specular BRDF. Hue has no measurable effect on perceived shape (e.g. Claessen, 1996). Troscianko et al. (1991) also showed that iso-luminant gradients from red to green do not affect the perceived slant of a surface. An iso-luminant gradient from red to gray (saturation), however, did have an effect. There is a small interesting effect of hue on the perception of depth though when an object is viewed with two eyes. This effect is best seen when a red object is displayed in front of a blue background. The red part is sometimes seen to stand in front of the blue background. This effect is known as chromostereopsis. It is believed that this effect is caused by the different refraction indices for red and blue light in the cornea of the eye, which results in a binocular disparity between the red and blue parts of the image. Usually the effect is small though and has little consequence for visual perception. You can buy special ‘ChromaDepth™ 3-D’ glasses that enhance chromostereopsis by separating the refraction indices for red and blue further (Chromatek®).

3.2. Material A reduction in illumination makes an object on a gray background look as if it has a lower albedo when the luminance of the background is kept constant. If the background luminance is reduced, that is the contrast between the object and the background remains constant, the perceived albedo of the object is fairly constant over a large range. This is known as luminance constancy. Likewise there is color constancy. The color of an object does not look much different in the evening compared to midday, especially for the colors that have a low saturation. Nevertheless, the spectrum of light entering the eye at the beginning of the evening is quite different from the spectrum at midday. If the evening and midday colors of the object would be shown simultaneously against a gray background, the difference would be striking. What is important for perception is not so much the light intensity or its spectrum, but the luminance or color contrast with the environment. In many ways the human observer has no respect for absolute intensities, perception is foremost concerned with relative intensities. This principle comes back in many different forms in sensory psychology. For example, in luminance and color contrast illusions, but also in motion perception as in the Duncker illusion. The Duncker illusion is best illustrated by the apparent motion of the moon rushing against the moving clouds. Three contrast illusions, namely luminance, curvature and size, are illustrated in Figure 1.11. When predicting or evaluating the appearance of objects in terms of albedo, color, size, motion etc., it is thus advisable to take the environment into account. The effects of luminance constancy and color constancy have also been described elsewhere (e.g. Palmer, 1999). A large part of the literature on material perception is devoted to the perception of lightness/albedo. Todd, Norman and Mingolla (2004) investigated the lightness of objects

29

On the visual appearance of objects

(A)

Luminance contrast

(B)

Curvature contrast

(C)

Size contrast

FIGURE 1.11 Contrast illusions. (A) Luminance contrast. The gray patch in the white circle is perceived as being darker than the gray patch in the black circle, even though they have the same gray value. (B) Curvature contrast. The physically straight line appears to bend. (C) The black disk is perceived as larger when it is surrounded by small circles than when it is surrounded by large circles, although the physical size is the same.

as a function of the presence of specular highlights. Lightness is here defined as the fraction of the amount of reflected light that is reflected in a Lambertian manner over the total amount of incident light. Todd et al. concluded that observers were quite able to discount the luminance intensity of specular highlights when judging lightness. An implication of this is that a mere glossy finish of a product does not change the perceived albedo of the underlying material. That is, objects do not look whiter (as opposed to gray) with a layer of varnish. Khang et al. (2003) investigated whether the perception of surface albedo is influenced by the mode of lighting in 3D geometrical shapes. They rotated a Platonic solid (in this a case a cube (six faces), a dodecahedron (12 faces) or an icosahedron (20 faces)) on a computer monitor. One of the faces of the solid was rendered with an albedo different from the other faces. Observers had to indicate whether there was a face with a different albedo from the other faces present in the object or not. It was found that observers performed best when the illumination was ambient, then hemispherically diffuse, and they performed worst when the illumination was parallel. It was also found that the number of faces that could be seen at any time was important. Observers performed best with the icosahedron, then with the dodecahedron, and they performed the least well with the cube. The symmetry of the object was found not to be an important variable in identifying the faces of polyhedrons with different albedo. The identification accuracy was the same whether the polyhedron was regular or irregular. Of practical use is that parallel light can hide slight variations in albedo better than other illumination modes. In case the aim is to detect imperfections of albedo on the surface it is best to use an ambient illumination.

3.3. Illumination Pentland (1982) proposed that the direction of illumination in smoothly shaded convex objects could be estimated from the first-order derivative of the image intensity and demonstrated that estimates of the illumination direction that are based on this idea correspond

30

Product Experience

well with estimates made by human observers. This study shows that observers have an idea where the light in a scene is coming from, which might in turn be important for shape and material perception. Khang et al. (2006) investigated the effects of material on perceived illumination direction. They showed that the type of BRDF influenced the accuracy of illumination settings. They concluded that observers might not take the effects of BRDF into account when judging the illumination direction, despite the large changes in the image intensity that it brings about. It has been suggested that observers are sensitive to the color of specular highlights (Yang and Maloney, 2001). Although not strictly true for all materials, the spectral composition of specular highlights is for many materials more similar to the light source than to the spectral composition of Lambertian shading of the object. This effect is often seen for varnished or wet materials. For example, when shining a white light on a glossy red plastic ball, there will be a white, rather than a red, specular highlight on the ball. On the other hand, the specular reflection of metals, such as copper and gold, is not color neutral to the light source; the specular reflection of gold is distinctly yellow for white illumination.

4. PERCEPTION 4.1. Perceptual organization The visual system is not a passive machine, but actively tries to make sense of the surrounding world. It reconstructs the world on the basis of the visual input into meaningful units. Gibson (1979) formulates this notion as: ‘Perceiving is an achievement of the individual, not an appearance in the theatre of his consciousness’. Several organizing principles have been identified in the past and are now commonly known as the Gestalt Laws. Central to Gestalt theory is the idea of ‘Prägnanz’. The Law of Prägnanz can be formulated as: ‘The psychological organization will be as “good” as the prevailing conditions allow’ (Koffka, 1935). The objective of the visual system is to achieve an organization that is as simple and regular as possible. In the context of shape, it means that a visual image is organized into the simplest and most regular shape possible given the visual image. This is a rather vague definition, because what exactly defines ‘good’ and what is a ‘good shape’? Several factors, or laws, have been described that contribute to perceptual organization, or to a good shape. The most important factors are: Good continuation; Proximity; Equality; Closure; Symmetry; and Common fate. The Gestalt factors are illustrated in Figure 1.12. Good continuation means that curves and surfaces are perceptually continued when they appear interrupted. The visual system identifies curves and surfaces that are good continuations of each other and combines them into a single unit. The factor of proximity means that points, lines, etc. that are close to each other are grouped together. Equality refers to the tendency of the visual system to group parts of the visual system together that share some feature, such as color or shape. Closure means that parts of the image that form a closed area or volume tend to be seen as such. Symmetry in this context means a figure is the same as its mirror image. Finally, dot, lines, etc. that move in a common direction and speed are seen together. This is known as common fate. Although we did not go into that in this chapter since all our objects are stationary, structure-from-motion, that is common fate, is often an important cue for shape perception. For example, a well-camouflaged bird may go unnoticed as long as it keeps perfectly still. As soon as the bird starts to move,

On the visual appearance of objects

31

FIGURE 1.12 The Gestalt factors: Good continuation, Proximity, Equality, Closure, Symmetry and Common fate. The arrows in the bottom right picture indicate a motion in that direction.

camouflage is broken, because the observer sees a coherent motion in the bush. A fact that is generally underappreciated in textbooks, but which should be mentioned for good measure, is that many demonstrations of the Gestalt laws in the literature are subject to the time that is given to view them. Like any other process, perceptual organization takes time. The more time that is given to view the scene, the more complex and detailed the resulting organization may become. Irregularities that are at first overlooked are incorporated at later stages. A fairly irregular shape may be perceived as regular when it is shown very briefly, but at longer durations the irregularities become clear. The Gestalt principles are highly important in visual perception since the visual system tries to order the world accordingly. Sometimes however, they lead to incorrect or bistable interpretations. Many visual illusions are therefore associated with the Gestalt laws. A classic example of bistability is the well-known Rubin face/vase illusion where the perceptual organization into a foreground and a background switches between two states. In this illusion the observer flips between a percept of two faces or a vase in the foreground. More interesting, artistic illustrations of bistability than the face/vase illusion can be found in the ‘Slave market with a disappearing bust of Voltaire’ by Salvador Dali, and in ‘Vanity’ by Charles Allen Gilbert. ‘Vanity’ was recently given new life in an advertisement for Dior that won an award for best European advertising photography in the Epica 2002 contest. Anderson and Winamer (2005) recently published one of the most fascinating new illusions. They showed that because of organizing an image into a foreground and background, the lightness of the depicted objects is strongly modified. This illusion is shown in Figure 1.13. The discs on the right of Figure 1.13A are perceived as black, while partly occluded by a white mist. The discs on the left of Figure 1.13A are perceived as white, while occluded by a dark fog. The images of all discs are exactly the same though. Likewise, the images of the two sets of chess pieces in Figure 1.13B are exactly the same, despite their strikingly different appearance in terms of albedo. Anderson and Winamer showed that this illusion is not a mere luminance contrast illusion. They showed this by

32

Product Experience

FIGURE 1.13 An illustration of the Anderson and Winamer (2005) illusion. In (A) the discs in the two images are exactly the same; in (B) the two sets of chess pieces are identical. Reprinted by permission from Macmillan: Nature, 434, 79–83, copyright 2005.

rotating the ‘mist’ over 90 degrees. This manipulation destroys the illusion in spite of the fact that the contrast in the image is unchanged. One of the problems with Gestalt theory is to quantify the contribution of each Gestalt law and to describe exactly what the contribution means. What exactly is a good shape and why is a square a better shape than a rectangle? The key is believed to be the simplicity of the resulting organization: The simpler the organization, the better. Simplicity can be described as the number of information bits that are necessary to describe the scene. Koffka (1935) gives an early example of this idea. Koffka suggested that a circle is more likely to be seen as a mere line than a triangle, because every element of the circle contains the principle of the whole. That is not the case for the triangle. That is, it requires more bits of information to describe a triangle than a circle. Yet another way to put it is that the circle has a higher level of symmetry than a triangle. Several coding systems have been proposed in order to evaluate the simplicity of the structure (e.g. Leeuwenberg, 1971; Van der Helm and Leeuwenberg, 1996).

On the visual appearance of objects

33

Important concepts in these approaches are regularity and symmetry. The effects of different kinds of symmetry in Same/Different tasks have, for example, been investigated by Leeuwenberg and Van Lier (2005). In addition to geometrical considerations for perceptual organization it may be argued that the visual system makes assumptions on what is statistically likely in the real world. For example, the majority of light usually comes from above, rather than from the floor. It makes sense to be biased towards light from above if the situation is ambiguous. In addition, Hill and Bruce (1993; 1994) found that people have a preference to see objects as convex rather than concave. It is often unclear whether a perceptual bias is based on geometric considerations, or on statistical likelihood. Regularity in the visual image, for example, is a very unlikely coincidence if it does not exist in three-dimensional space as well. Consider for example a sphere. From any viewpoint the sphere will appear as a circle in the visual image; that is, spheres yield stable projections. Ellipsoids (egg-shaped objects) project in general to ellipses in the visual image but in accidental viewpoints they project as circles; these projections are unstable. That is, if you see a circle it is more likely projected from a sphere than from an ovoid. A circle in the visual image is just too good to be incidental. Gibson (1979) takes the idea of perceptual organization one step further in his theory of information pick-up. According to Gibson not perceptual qualities, such as color and intensity, are perceived, but rather the information that is provided in their changes over space and time. Mere perception of color or curvature is unsatisfactory to Gibson. True perception is all about seeing the structure in the variation of the world and the structures that remain invariant over time. Gibson suggested that perception is thus not concerned with positions but with places, with objects rather than 2D-shapes and with substances (materials) rather than luminance. This is very much what we are after when discussing the appearance of objects: Object appearance is not about the perception of luminance, it is all about inferred shape, albedo, BRDF, illumination, etc. of the object. What observers are often particularly interested in is what you can do with an object. Observers specifically look for information in the visual image that allows them to infer what it is they can do with an object. A prominent feature in Gibson’s theory is therefore the inferred function, called affordance, of an object. The affordance of an object translates roughly into ‘this object is for … (raking, hammering, throwing, etc.)’. The affordance of an object is not only a property of the object, but also of the observer. For example, a big tree trunk does not afford throwing for a child, but it does for an adult interested in caber tossing.

4.2. The SMI triangle Koffka (1935) writes that prior to his time it was believed that the visual system could extract properties of the visual image independently of each other. For example, color could be extracted independently from motion. Koffka himself, however, believed that this was impossible. More recently, however, evidence has been published that suggests that his position is not quite tenable. The mere fact that you need some light in order to make a pattern move over the retina does not mean that you cannot perceive motion without being able to see light. There are, for example, patients who can see light but not movement (e.g. Zihl, Voncramon and Mai, 1983), and there are also cortically blind patients with relatively intact motion perception (e.g. Azzopardi and Cowey, 2001). Although evidence from patient studies has several problems, it suggests that the coupling of motion and light perception is not a simple fact. The problem of whether perceived object qualities have an inherent relationship goes one step further. These object

34

Product Experience

properties are not qualities of the visual image, but of the representation that is created from that. There are good reasons to expect a relationship between object properties just as there are good reasons not to expect one. A good reason for perceived properties to be coupled is of course the SMI triangle. Gregory (1973) showed a relationship between perceived convexity and illumination direction. He showed that both a mask and a mould of a face look like convex faces, but that the light seems to come from a different direction. That is, a convex mask that is illuminated from above looks like a convex face that is illuminated from above, whereas a concave mould of a face that is illuminated from above looks like a convex face illuminated from below. Johnston, Hill and Carmen (1992) showed that this also holds for faces turned upside down and for hollow potatoes as well. In these cases illumination and shape perception seem to be coupled. It is a common mistake to consider the perceived illumination direction as the reason why the hollow face is seen as convex. The more likely reason is that there is a prior for convexity (Langer and Bülthoff, 2001). Note also that the hollow face illusion is never bistable. Mingolla and Todd (1986) also showed a correlation between perceived shape and perceived illumination direction when observers viewed relatively flat egg-like objects, but not when they saw egg-like objects of greater depth. Te Pas and Pont (2005) found that the perception of BRDF and illumination on spherical shapes was in many cases confounded. In many of the objects that they tested, a change in BRDF was instead perceived as a change in illumination conditions. They did not report whether the perception of BRDF and illumination were correlated on a trial-by-trial basis. Interestingly, when the task was to compare the BRDF of two objects, each with a different BRDF but illuminated the same, observers consistently reported the BRDF to be different. When the task was about the illumination direction, and the objects differed in illumination but the BRDF was the same, observers were less consistent in reporting the illumination to be different. This suggests that the processes underlying material and illumination perception are at least partly decoupled.

4.3. Additional cues Apart from shading there are usually more cues to shape, material and illumination available that are largely independent of the SMI triangle. The shape of the object in the visual image is a very good cue for three-dimensional shape. For example, Mamassian and Kersten (1996) concluded that observers used the occluding contour of the object rather than shading to estimate the local surface orientation of croissant-shaped objects. The occluding contour is where the object curves around to occlude itself from sight. That is, the occluding contour is the edge of the object’s silhouette. Erens, Kappers and Koenderink (1993) showed that the presence of an occluding contour is necessary for the perception of shape-from-shading. The absence of a well-defined contour renders the sign of the surface shape (convex/concave) essentially ambiguous. Furthermore, even in images of objects that lack any shading, such as silhouettes, there are sufficient cues to the three-dimensional structure, such as the occluding contour, in order to get to a reasonably accurate three-dimensional impression. Tse (2002), for example, argues convincingly that the three-dimensional shape of objects can be quite accurately predicted when unlikely combinations of object shape and viewpoints are rejected. Several authors have shown that the perception of material properties is influenced by perceived shape. The perception of shape is in these cases induced by contours and suggestive groupings of surfaces. Knill and Kersten (1991) showed that perceived surface curvature influences perceived albedo. Perceptions of surface attitude and curvature

On the visual appearance of objects

35

shape were induced by suggestive contours. Gilchrist (1971) showed that perceived lightness depends on the relationship of the luminance of surfaces that lie in the same depth plane, but not on surfaces that are merely adjacent in the retinal image. Soranzo and Agostini (2006) showed that increasing perceptual belongingness of surface patches into larger units improved the perception of lightness constancy. Object surface texture deforms in specific ways, depending on the surface attitude relative to the viewing direction. Also, because of linear perspective, the density of the texture increases with increasing distance. Note that the texture in the visual image strains in one direction because of a change in surface attitude of the object, but shows dilation, when the distance is changed. Here we make the assumption that texture is uniform over the object and without a specific direction, but there is a dependency of texture and shape, much like we have seen for shape, material, and illumination. Curran and Johnston (1996) have shown that the effect that the position of the light source has on perceived shape is reduced when texture cues are added. Pont and Koenderink (2003) showed that the direction of illumination is accurately predicted from the direction of the texture in the visual image. That is, bumps and pits in the surface create systematic distortions in the texture in the visual image, because they cast small shadows on the surface that tend to elongate in the direction of illumination. They also showed that illumination direction could be estimated in a similar way in complex scenes. Koenderink et al. (2003) showed that observers are good at estimating the tilt of the illumination direction, but are less proficient at estimating the slant of the illumination direction. Te Pas and Pont (2005) reported also that observers are more reliable in their estimation of illumination direction when three-dimensional texture was present on the object. Three-dimensional texture did not prove helpful in discriminating objects with different BRDFs though. The effects of three-dimensional texture on shape perception have not been evaluated systematically yet. In cases where a higher accuracy of illumination perception is required it makes good sense to add three-dimensional texture to the objects in the scene. For example, when one wants to use the information on the illumination direction to estimate shape and material properties of other nearby objects that are not textured but are only shaded. Stereoscopic vision is another frequently cited cue to three-dimensional shape. Because the two eyes look at the world from slightly different viewpoints, the images in the two eyes are slightly shifted relative to each other. This shift is known as binocular disparity. In general, however, it is not the solution to unambiguous perception. First, binocular disparity works best only in a limited range of distances, and secondly, there is a considerable part (about 30%) of the general population that does not have the stereoscopic resolution that one would wish for (Richards, 1970). Thirdly, there is considerable uncertainty in the perceived distance of the object, which causes unpredictable effects on depth scaling (e.g. Drga and Harris, 2005). And fourthly, stereoscopic vision works only for relatively high spatial frequencies such as edges, lines, texture, etc. For example, Arndt, Mallot and Bülthoff (1995) showed that a reversal of the left and right half images of a stereoscopic image of a shaded sphere does not turn a convex sphere into a concave bowl. Only when texture is added does the sphere turn inside out, but still only reluctantly. Apparently, the prior for convexity is much stronger than the cue provided by stereoscopic viewing of shading. Several models have been proposed of how different cues can be integrated to resolve ambiguities in shape perception (e.g. Landy et al., 1995). Cue accumulation models do not in general take material and illumination ambiguities into account. They do not tell whether, for example, shape perception becomes more stable when cues to illumination are specified. There are no formal models that incorporate all these cues. The question of

36

Product Experience

whether reduction of ambiguity in one of the vertices of the SMI triangle leads to more perceptual stability in the other two vertices, and how that might best be achieved is still largely an open question, though with obvious consequences for disambiguating environments where shape and material information of objects is crucial for picking them up in the correct way, manipulating or recognizing them. For example, conditions that allow for making a good estimate of the BRDF might provide the observer with a clue to the type of material an object is made from, such as metal or cloth, which determines how hard you can squeeze it when picking it up. Other important applications are in virtual or augmented reality applications where the number of visual cues is often limited, and as a result shape or material might be become ambiguous.

5. CONCLUSION In this chapter I have described the roles of shape, material, and illumination for object appearance at three different levels, namely: At the physical level, the relationship between the physical world and the mental world, and finally at the level of the mind. First I discussed the physics and mathematics of shape, material, illumination and color. I made clear that there are inherent ambiguities in visual object perception. For example, the color of an object is not only determined by its reflectance function, but also by its illumination. This is important in conditions where objects have to stand out of the environment irrespective of illumination, such as traffic signs or the clothes of road workers. Traffic signs have to be clearly visible, also at night when the only illumination comes from the sodium streetlights that we discussed earlier. A retro-reflective property of the traffic sign or the clothes of road workers that reflects the headlights of cars back to the source, as well as takes advantage of the more complex light spectrum, is clearly a desirable property at night. Another area in which metamerism is important is where objects have to blend in with the environment, such as in military camouflage. It would make sense to be well aware of the possibility that metamere colors are not metamere under different illumination conditions, when the observer uses a color filter, or even when the observer looks from a different direction. I have also illustrated that ambiguities of shape must fall within a certain class once some assumptions have been made. For example, we have seen how the bas-relief ambiguity is restricted to an affine class of shapes. In the second part, I discussed how physical parameters affect perception. I discussed a method to measure perceived shape. Perceived shape can be significantly altered by changes in material and illumination. An important factor in the perception of material, color and albedo, is contrast. The visual system operates mostly on contrast in luminance, color, etc., rather than its absolute value. For example, when a product needs to draw attention, it is better to increase the contrast in luminance or in the density of the surface texture or in the color, of the product with the environment rather than increasing the overall illumination levels. The luminance, color, etc. of the environment must therefore always be taken into account if one wants to predict what a product will look like, especially where it is important that objects are presented favorably, such as in shop windows and on supermarket shelves. Contrast is also an important factor for improving visibility of products for people with low vision. Rather than increasing the overall luminance, although that might also be beneficial in many cases, increasing the color contrast or luminance contrast with the environment will make the product more visible. In the third part, I discussed how the visual system tries to organize the visual image into a sensible impression of the world. The Gestalt laws describe several factors that

On the visual appearance of objects

37

contribute to a perceptual organization. Proper structuring of the environment can disambiguate perception, overcome luminance contrast illusions, or just as well result in a visual illusion. In the design situation, ambiguity in the perceptual organization might, for example, be exploited in order to create an interesting artistic product, but the designer might also want to avoid ambiguity as much as possible depending on the application in mind. I discussed how the perception of shape, material and illumination might be coupled. This is important because it is a potential way to disambiguate the visual scene. It might be possible to make the material that an object is made of less ambiguous by providing cues to the shape, such as a clearly visible contour and surface texture. Or, observers might be able to recognize that an object is less flat when it is clear to them that the illumination is highly ambient. For example, in interactive virtual reality or augmented reality applications where accurate perception of depth and distance is often required, the presence of visual cues is often limited which may result in a high level of ambiguity under these conditions. However, the ambiguity might be resolved by adding more visual cues. Adding additional independent visual cues to shape, material and illumination such as surface texture and 3D texture, binocular disparity, specular highlights, sharp contours, coherent motion, and regularity in the spatial layout, either in a real or a virtual environment, may all contribute to a more accurate visual perception of the product.

ACKNOWLEDGMENTS I wish to thank the members of the Vision Lab at the University of St. Andrews, and Samira Bouzit of the School of Psychology at the University of Southampton for providing invaluable feedback on the draft version of this chapter.

REFERENCES Anderson, B. L. and Winamer, J. (2005). Image segmentation and lightness perception. Nature, 434, 79–83. Arndt, P. A., Mallot, H. A. and Bülthoff, H. H. (1995). Human stereovision without localized image features. Biological Cybernetics, 72, 279–293. Azzopardi, P. and Cowey, A. (2001). Motion discrimination in cortically blind patients. Brain, 124, 30–46. Belhumeur, P. N., Kriegman, D. J. and Yuille, A. L. (1999). The bas-relief ambiguity. International Journal of Computer Vision, 35(1), 33–44. Blake, A. and Bülthoff, H. H. (1990). Does the brain know the physics of specular reflection? Nature, 343, 165–168. Blake, A. and Bülthoff, H. H. (1991). Shape from specularity: Computations and psychophysics. Philosophical Transactions of the Royal Society of London: Series B: Biological Sciences, 331(1260), 237–252. Cederberg, J. N. (2001). A course in modern geometries. NY: Springer. Christou, C. G. and Koenderink, J. J. (1997). Light source dependency in shape from shading. Vision Research, 37(11), 1441–1449. Claessen, J. P. (1996). Shaped by colour. Doctoral thesis, University of Delft, The Netherlands. Curran, W. and Johnston, A. (1996). The effect of illuminant position on perceived curvature. Vision Research, 36(10), 1399–1410. Drga, V. F. and Harris, J. M. (2005). Distance perception during binocular viewing via stereo-goggles at different display distances. Perception, 34 (Suppl.), 189–189. Erens, R. G. F., Kappers, A. M. L. and Koenderink, J. J. (1993). Perception of local shape from shading. Perception and Psychophysics, 54, 145–157. Flemming, R. W., Dror, R. O. and Adelson, E. H. (2003). Real-world illumination and the perception of surface reflectance properties. Journal of Vision, 3, 347–368. Flemming, R. W., Torralba, A. and Adelson, E. H. (2004). Specular reflections and the perception of shape. Journal of Vision, 4, 798–820. Gibson, J. J. (1966, 1983). The senses considered as perceptual systems. Westport, CT: Greenwood Press.

38

Product Experience

Gibson, J. J. (1979, 1986). The ecological approach to visual perception. Hillsdale (NJ), London: Lawrence Erlbaum Associates. Gilchrist, A. L. (1971). Perceived lightness depends on perceived spatial arrangements. Science, 195, 185–187. Gregory, R. L. (1973). The confounded eye. In: R. L. Gregory and E. H. Gombrich (Eds). Illusion in nature and art, pp. 49–96. London: Duckworth. Hill, H. and Bruce, V. (1993). Independent effects of lighting, orientation, and stereopsis on the hollow-face illusion. Perception, 22, 887–897. Hill, H. and Bruce, V. (1994). A comparison between the hollow-face and ‘hollow potato’ illusions. Perception, 23, 1335–1337. Howard, I. P. (2002). Seeing in depth. Volume 1, Basic mechanisms. Toronto: I. Porteous. Ikeuchi, K. and Horn, B. K. P. (1981). Numerical shape from shading and occluding boundaries. Artificial Intelligence, 17, 141–184. Johnston, A., Hill, H. and Carmen, N. (1992). Recognising faces: effects of lighting direction, inversion and brightness reversal. Perception, 21, 365–375. Khang, B. G., Koenderink, J. J. and Kappers, A. M. L. (2003). Perception of surface reflectance of 3D geometrical shapes: influence of the lighting mode. Perception, 32, 1311–1324. Khang, B. G., Koenderink, J. J. and Kappers, A. M. L. (2004). Shape constancy does not hold for images rendered with different types of material surfaces. Journal of Vision, 4(8), 79–79. Khang, B. G., Koenderink, J. J. and Kappers, A. M. L. (2006). Perception of illumination direction in images of 3-D convex objects: influence of surface materials and light fields. Perception, 35, 625–645. Knill, D. C. and Kersten, D. (1991). Apparent surface curvature affects lightness perception. Nature, 351, 228–230. Koenderink, J. J., Van Doorn, A. J., Christou, C. G. and Lappin, J. S. (1996a). Shape constancy in pictorial relief. Perception, 25, 155–164. Koenderink, J. J., Van Doorn, A. J., Christou, C. G. and Lappin, J. S. (1996b). A perturbation study of shading in pictures. Perception, 25, 1009–1026. Koenderink, J. J., Van Doorn, A. J. and Kappers, A. M. L. (1992). Surface perception in pictures. Perception and Psychophysics, 52(5), 487–496. Koenderink, J. J., Van Doorn, A. J., Kappers, A. M. L., Te Pas, S. F. and Pont, S. C. (2003). Illumination direction from texture shading. Journal of the Optical Society of America, 20(6), 987–995. Koenderink, J. J., Van Doorn, A. J., Kappers, A. M. L. and Todd, J. T. (2001). Ambiguity and the ‘mental eye’ in pictorial relief. Perception, 30, 431–448. Koffka, K. (1935). Principles of Gestalt psychology. London: Kegan Paul, Trench, Trubner. Landy, M. S., Maloney, L. T., Johnston, E. B. and Young, M. (1995). Measurement and modeling of depth cue combination: in defense of weak fusion. Vision Research, 35(3), 389–412. Langer, M. S. and Bülthoff, H. H. (2001). A prior for global convexity in local shape-from-shading. Perception, 30, 403–410. Law-Smith, M. J., Perrett, D. I., Jones, B. C., et al. (2006). Facial appearance is a cue to oestrogen levels in woman. Proceedings of the Royal Society of London B, 273, 135–140. Leeuwenberg, E. L. J. (1971). A perceptual coding language for visual and auditory patterns. American Journal of Psychology, 84(3), 307–349. Leeuwenberg, E. L. J. and Van Lier, R. (2005). Symmetry cues for matching mirrored objects. Spatial Vision, 18(1), 1–23. Longuet-Higgins, M. S. (1960). Reflection and refraction at a random moving surface. I. Pattern and paths of specular points. Journal of the Optical Society of America, 50(9), 838–844. Mamassian, P. and Kersten, D. (1996). Illumination, shading and the perception of local orientation. Vision Research, 36(15), 2351–2367. Mingolla, E. and Todd, J. T. (1986). Perception of solid shape from shading. Biological Cybernetics, 53(3), 137–151. Nefs, H. T., Koenderink, J. J. and Kappers, A. M. L. (2005). The influence of illumination direction on the pictorial reliefs of Lambertian surfaces. Perception, 34(3), 275–287. Nefs, H. T. Koenderink, J. J. and Kappers, A. M. L. (2006). Shape-from-shading for matte and glossy objects. Acta Psychologica, 121(3), 297–316. Nicodemus, F. E., Richmond, J. C., Hsia, J. J., Ginsberg, I. W. and Limperis, T. (1977). Geometrical considerations and nomenclature for reflectance #NBS MN_160. Washington: National Bureau of Standards, Department of Commerce. Norman, J. F., Todd, J. T. and Orban, G. A. (2004). Perception of 3D shape from specular highlights, deformations of shading, and other types of visual information. Psychological Science, 15(8), 655–570. Norman, J. F., Todd, J. T. and Phillips. F. (1995). The perception of surface orientation from multiple sources of optical information. Perception and Psychophysics, 57(5), 629–636.

On the visual appearance of objects

39

Oren, M. and Nayar, S. (1996). A theory of specular surface geometry. International Journal of Computer Vision, 24(2), 105–124. Palmer, S. E. (1999). Vision science: Photons to phenomenology. Cambridge, MA: MIT Press. Pentland, A. P. (1982). Finding the illumination direction. Journal of the Optical Society of America, 72, 448–455. Pont, S. C. and Koenderink, J. J. (2003). Illuminance flow. Computer analysis of images and patterns. Lecture Notes in Computer Science, 2756, 90–97. Pont, S. C. and Te Pas, S. F. (2006). Material–illumination ambiguities and the perception of solid objects. Perception, 35, 1331–1350. Richards, W. (1970). Stereopsis and stereoblindness. Experimental Brain Research, 10, 380–388. Soranzo, A. and Agostini, T. (2006). Does perceptual belongingness affect lightness constancy? Perception, 35, 185–192. Te Pas, S. F. and Pont, S. C. (2005). A comparison of material and illumination discrimination performance for real rough, real smooth and computer generated smooth spheres. Proceedings APGV 2005, ACM SIGGRAPH, 75–81. Todd, J. T. and Mingolla, E. (1983). Perception of surface curvature and direction of illumination from patterns of shading. Journal of Experimental Psychology: Human Perception and Performance, 9(4), 583–595. Todd, J. T., Norman, J. F., Koenderink, J. J. and Kappers, A. M. L. (1997). Effects of texture, illumination and surface reflectance on stereoscopic shape perception. Perception, 26, 806–822. Todd, J. T., Norman, J. F. and Mingolla, E. (2004). Lightness constancy in the presence of highlights. Psychological Science, 15(1), 33–39. Troscianko, T., Montagnon, R., Le Clerc, J., Malbert, E. and Chanteau, P. L. (1991). The role of colour as a monocular depth cue. Vision Research, 31(11), 1923–1930. Tse, P. U. (2002). A contour propagation approach to surface filling-in and volume formation. Psychological Review, 109(1), 91–11. Van der Helm, P. A. and Leeuwenberg, E. L. J. (1996). Goodness of visual regularity: a nontransformational approach. Psychological Review, 103(3), 429–456. Wyszecki, G. and Stiles, W. S. (1982/2000). Color science: Concepts and methods, quantitative data and formulae. New York: John Wiley. Yang, J. M. and Maloney, L. T. (2001). Illuminant cues in surface color perception: tests of three candidate cues. Vision Research, 41, 2581–2600. Zihl, D., Von Cramon, D. and Mai, N. (1983). Selective disturbance of movement after bilateral brain damage. Brain, 106, 313–340.

2

THE TACTUAL EXPERIENCE OF OBJECTS MARIEKE H. SONNEVELD Delft University of Technology, Delft, The Netherlands

HENDRIK N.J. SCHIFFERSTEIN Delft University of Technology, Delft, The Netherlands

1. INTRODUCTION A child’s first tactual experiences with objects mostly involve being touched, for example by the latex gloves of the midwife, the towels she cleans the baby with, and the textiles of the clothes that separate the baby’s skin from its mother. Once children grow old enough to reach out and touch what surrounds them, their tactual experiences become active. They hold, squeeze, and swing whatever comes into reach. Their world becomes an exciting environment, in which they learn to develop their physical skills through manipulating balls, dolls, grandpa’s spectacles, bicycles, skates, and in which they learn how to avoid unpleasant encounters, such as with the sharp edges of the table. When tired of exploring and playing, a cuddly toy may wait for them to keep them company. And although children are aware that they are the active agent in kicking the ball and riding the bicycle, it is not always clear whether they are cuddling the toy or whether the toy is cuddling them; touching becomes interactive. This unavoidable reciprocity is characteristic for the senses of touch. Seeing does not imply being seen, neither does hearing imply being heard. But touching implies being touched simultaneously. Touching and being touched are integrated into one phenomenon; the tactual experience. In this chapter, we provide an overview of the different domains from which tactual experiences can be described and explored. These different domains are summarized in the Tactual Experience Guide that is discussed below (Sonneveld, 2007). A leading motive during the construction of this guide was to enable the description of the aesthetic aspects of tactual experience: What are the influential factors that make objects pleasant or unpleasant to touch? Product Experience Copyright © 2008 Elsevier Ltd.

41

42

Product Experience

Tactual experiences are common to everyday life. Nevertheless, it seems as if people hardly ever talk about them, and they appear to find it very difficult to talk about how objects feel. Suppose that you ask a person to describe how the pen he usually writes with feels. He would probably pick up the pen and start to manipulate it, turn it over, hold it in different positions, consider the different surface textures of the pen by stroking it, swing it between his fingers, possibly slightly hit the table with it, and put down some lines on a piece of paper. Obviously, this manipulation is different from the interaction when actually writing with it, or when carrying it around. Sooner or later the person will become aware of the complexity of the question and will reply: ‘What do you mean, how it feels? When I write with it, play with it, carry it, or what?’ We cannot simply ask people how an object feels, we need to take into account that the answer depends on the nature and context of the interaction with the object. In addition, people seem to lack the vocabulary to describe their tactual experiences of interacting with objects. To support designers as well as users in describing and aesthetically assessing the experience of interacting with objects, Sonneveld (2007) developed the Tactual Experience Guide. This guide helps people to describe their tactual experiences with objects, by offering a consistent framework of the different aspects of tactual experience (the content of the tool), and by offering a format that guides people through this experience (the design of the tool). The framework is based on a qualitative analysis of personal descriptions of pleasant and unpleasant tactual experiences with objects. These descriptions were collected through a printed questionnaire completed by 46 participants, 24 men and 22 women, varying in age from 18 to 67 years. The data analysis resulted in a set of themes that characterize the different aspects of tactual experience. These themes were grouped along the different domains of tactual experience that form the basis of the structure of the Tactual Experience Guide (Figure 2.1) and will be used as the outline of the current chapter: Moving with the object (Section 3); perceiving tactual

movements tactual properties

tactual experience affective behavior gut feelings

sensations

FIGURE 2.1 Overview of the five domains of tactual experience presented as a mind-map structure.

The tactual experience of objects

43

properties of the object (Section 4); sensing physical sensations (Section 5); experiencing the affective behavior of the object (Section 6); and experiencing the gut feelings involved (Section 7). Throughout this chapter, the descriptions of the different aspects of tactual experience will be illustrated with examples derived from this study. This chapter starts out with a description of the meaning of touch, based on a literature overview approaching touch from different perspectives. Next, the five domains of tactual experience are described, complete with background information for each specific domain. The chapter concludes with a section on the development of product designers’ aesthetic sensitivity for the tactual experience in human–product interaction, and recommendations for future research in the field of tactual aesthetics.

2. THE MEANING OF TOUCH 2.1. Touch: Physical encounters and awareness of oneself Physical interaction with the world is not limited to the hands, it involves the whole body. Physical engagement with the world, the awareness of touching and being touched, makes people aware of being a physical body themselves, sharing the physical world with other physical objects. It is within this embodied encounter that the ‘I’ experiences itself and its surrounding world simultaneously, making this encounter the basis for selfawareness (see Bermudez, Marcel and Eilan, 1995, for an overview). Although people can see their body, they need to sense their body to be aware of themselves. Touch allows sensing one’s own body, sensing the borders between the self and the outside world, and the interaction between the two. The neurologist Oliver Sacks emphasized this aspect of the physical experience of the body as the foundation for self-awareness in his descriptions of patients with disturbed self-perceptions. For example, a patient who did not experience his leg as part of himself any more tried to throw the alien leg out of the bed (Sacks, 1984; 1987). Such situations of disturbed tactual sensations may also occur in non-pathological cases such as when waking up with a numb arm or having an anaesthetized cheek at the dentist. When touching these body parts, they feel alien, as ‘not part of me’; they are experienced as ‘dead’ matter. These findings could be summarized in the statement ‘you are what you feel’ (Bergsma, 1987). A world in which touch is poorly addressed is likely to weaken the feeling of being in contact with the world, which may lead to a disappearing feeling of self-awareness. Paradoxically, touch may also blur the boundary between the experienced self and the world. The rubber hand illusion provides a striking example; when one is watching a rubber hand being stroked, while one’s own unseen hand is synchronously stroked, one may attribute the rubber hand to one’s own body and ‘feel like it’s my hand’ (Tsakiris and Haggard, 2005). Another phenomenon that blurs the experience of the border between the self and the outside world is people’s capacity to feel through objects, by ‘incorporating’ these objects as part of their own body (Polanyi, 1967). For example, a blind person feels the world at the end of his white stick, and a carpenter feels the wood through his saw.

2.2. Touch: A foundation for knowledge of the material world In his philosophy on education, Dewey states that the material world people live in and through forms the basis of learning and personal growth, because it is the basis of their

44

Product Experience

‘experiential world’. According to Dewey, one only learns and grows through and from experience, in physical interaction with a material world (Dewey, 1997 [1938]). Physical interaction puts the body, and thereby the tactual senses, back into experience as the foundation of knowledge. People can see the shapes and colors of the physical world, hear the events that take place, smell it, but it is only through touch that people learn about its materiality. The sight and sound of a frog, for example, give some clues for suppositions about its tactual properties, but it is only through touch that one learns about its physicality: Its weight, temperature, wetness, the texture and elasticity of its skin, its force and movements, and so on. The experience of touching a frog (and being touched by it) embodies what learning through touch is about. From this point of view, people actually need touch to get to know and understand the world. Likewise, people need touch to know and understand the man-made objects they are manipulating to grasp their meaning (Lakoff and Johnson, 1999). This knowing through bodily experiencing is different from the knowledge gained through thinking as deduction from theory.

2.3. Touch: A foundation for feelings and emotions Touching is being in physical contact and, as such, is the basis for the feeling of being in contact. Within this contact, touch is a strong basis for the development of feelings of affection and intimacy (Fields, 2003; Montagu, 1971) and is necessary for physical and mental development. Touch during the first phases of life has to be loving and protecting in order for a person to develop into a healthy, empathic human being. It is through touch that one experiences that one is safe and cared for. Experiments with rats have shown that touch-deprivation leads to growth retardation and withering (Montagu, 1971). This need for touch seems so primordial that monkey infants deprived from their mother prefer a terry-cloth surrogate mother without milk to hang on to, over a wire surrogate mother with milk. This is confirmed for people by observations of children who grew up in Romanian orphanages, where touch was infrequent due to understaffed situations. The need for loving touch remains throughout people’s lives, a phenomenon referred to as touch hunger (Fields, 2003). These insights on affective and emotional aspects of touch are based on studies of people touching people, and not on human–product interactions. However, some observations suggest that people’s affective and emotional development and well-being may also be affected by the way they are touched by objects. For example, the experiment with the monkey infants showed that being touched by a non-living terry-cloth surrogate mother contributed to the infant’s well-being. Furthermore, transitional objects such as a blanket or a teddy bear, described by Winnicott (1964) as objects that allow the child to feel safe in a world where the mother is temporarily absent, are illustrations of this affective meaning of touch embodied by objects. Another example is the cuddly walls developed for demented elderly to promote emotional well-being.

2.4. Touch: A communication channel for affection Touch implies contact and thus bodily involvement, whereas seeing and hearing are distant senses and thus are more apt to create distance and ‘objectification’ in social contact. Touch is therefore often considered as our most social sense. Interpersonal touch tells us whether we are safe, cared for and have value (Fields, 2003; Finnegan, 2002). Fagan (1998) suggested that touch is the first language that children learn to communicate

The tactual experience of objects

45

interpersonal affection. Fagan distinguished between various kinds of touch: Ritual touch (e.g. a handshake), athletic touch (e.g. in contact sports), punishing touch, nurturing touch, intimacy-evoking touch and sexual touch. These categories are not mutually exclusive; a formal handshake in a greeting ritual, for example, may feel intimate if it feels softer or lasts longer than expected. To be able to function properly in social contacts, one has to be able to understand (and express) the language of touch properly. Thayer (1982) established a taxonomy of touch based on and illustrated by the relationships people have: A functional–professional type (the doctor touching his patient for examination), social–polite type (the handshake), the friendship–warmth type, the love– intimacy type, and the sexual type. This sequence of categories can be characterized by increasing intimacy: More body parts get involved, longer and more frequent instances of touching, and an increasing variety of the types of touch involved. These social aspects of interpersonal touch that contribute to the meaning of touch may also apply to our relationships with products. The way in which people interact with products may express different kinds of affective relationships, ranging from the functional–professional to the intimate.

3. TACTUAL INTERACTION 3.1. Active and passive touch In human–object interactions, we can distinguish between touching an object and being touched by an object. These two distinct phenomena are referred to as active and passive touch, respectively (Gibson, 1962). Active touch produces a perception of the object being touched: One is exploring the object’s properties (objective pole). On the other hand, passive touch with the same object (being touched by the object) gives an internal sensation: One experiences the sensations in the body, what is being done to the body (subjective pole) (Gibson, 1962). To illustrate this distinction, imagine picking up a glass of wine, handling it in your hands, gently turning it to move the wine: You perceive its shape, its temperature, its fragility, and the movement of the liquid. On the other hand, imagine lying on the bench of a masseur who is putting hot stones on your back: You sense the pressure on your back, the warming of your skin, but you do not sense the shape and the size of the stone. In the active mode, if you focus on your hand you are able to feel the pressure of the glass against your skin, and in the passive mode you may perceive the weight of the stones on your back. Thus touching and being touched occur simultaneously in a physical encounter. Apparently, in actively reaching out to manipulate and touch the world your attention is directed towards the object, whereas in being touched your attention is directed towards the sensations caused by that object. But, in interaction, one can be made aware of both. The difference between touching or being touched is not only a matter of attention and being active or passive. It is also related to the body parts involved in the interaction, because the skin of the different body parts differs in the suitability for active or passive touch. The skin of the palm of the hands and the soles of the feet seems especially suited for touching, whereas the hairy skin, covering the rest of the body, is better equipped for signaling the locus of events touching the body (Bolanowski, 2004). Touching and being touched is not limited to the contact between the body and an object. People have the capacity to touch the environment through other objects (Burton, 1993). Some of these objects are our own non-neural extensions, such as nails, teeth, and

46

Product Experience

hair, in anatomy and physiology referred to as accessory organs (Saladin, 2001). But our interest here focuses on the man-made objects people touch through: People touch the bread through the knife they cut with, they touch the road through the bicycle they are riding on, the tennis ball through the racket they hit with, and the paper through the pen they are writing with. Although touching through intermediate objects may be different from direct touch (Lederman and Klatzky, 1999), it forms an interesting part of people’s everyday tactual experiences. In the following paragraphs, we first discuss the properties one perceives when actively touching objects, followed by the sensations perceived when being touched by objects.

3.2. Exploration strategies When you start to think about it, it is an astounding achievement of tactual perception that we perceive and recognize an object as one object. For example, when holding a glass of wine the skin is touched at different places with different pressure intensities. Joints, muscles and tendons have specific positions and apply specific forces to different parts of the glass. Therefore, it is remarkable that all these impressions are integrated into the perception of one glass of wine. This capacity of identifying an object as one object can be fooled: Aristotle discovered that rolling a pen over two crossed fingers with one’s eyes closed gives the impression of rolling two pens over the fingers. Nevertheless, generally when people use only their sense of touch, they identify the integrity of objects correctly. When people are presented with an object, they typically first try to identify it: They want to know what it is (e.g. see Gibson, 1962; Schifferstein and Cleiren, 2005). Empirical studies have shown that the tactual (haptic) system is quite rapid and accurate in recognizing three-dimensional, familiar objects. Klatzky, Lederman, and Metzger (1985) demonstrated that blindfolded subjects recognized 96% of common objects within 5 seconds, and 94% in 1–2 seconds. Tactual perception of an object and its properties is dependent on movement: It is through movement, in interaction, that people perceive tactually. Klatzky et al. (1985) studied the movements made by blindfolded people when physically exploring tactual properties of objects, and concluded that people use specific exploratory movements for the perception of specific tactual properties. In addition, these authors found that for a consistent tactual perception of the object, tactual scanning should be done in a systematic way. These tactual scanning strategies have to be learned: Experience in touching is needed to be able to explore efficiently and accurately (Davidson, 1985). Figure 2.2 shows an overview of the exploratory procedures distinguished by Klatzky et al. (1985) and indicates the corresponding tactual properties these procedures are specifically suited for. Turvey (1996) observed and researched ‘dynamic touch’ as an exploratory procedure. This procedure implies that people swing objects to ‘get a feel for them’. Dynamic touch is functional for the perception of geometrical properties, size, and weight, but especially appropriate for determining the moment of inertia of an object: Its reaction to rotation. Dynamic touch forms the basis for understanding how to use an object as a tool: How to hit a nail with a hammer and where to hit the ball with your tennis racket.

3.3. Motivations to move To understand how people interact with objects, it is important to understand why they interact in the way they do (Hodgson, 2001). In other words, we need to understand

47

The tactual experience of objects

Lateral Motion (Texture)

Pressure (Hardness)

Unsupported Holding (Weight)

Enclosure (Global Shape; Volume)

Static Contact (Temperature)

Contour Following (Global Shape; Exact Shape)

Moving Parts

FIGURE 2.2 Overview of the exploratory procedures that people use to determine specific tactual properties of objects (adapted from Klatzky et al., 1985).

their motivation for interaction. This motivation is evidently elicited by and reflected in the function of a product, but it is not necessarily limited to that function. In this Section, we give an overview of the different motivations we encountered in studies on tactual experience (see Figure 2.3): • Interaction for practical, functional use, as a tool: In many cases, objects are used with a functional goal, as a tool. The intention of the interaction is directed towards the outside world, for practical reasons. Evident examples are the use of scissors to cut paper, the use of a knife to slice bread, the use of a camera to take a picture, or the use of a car to get somewhere. However, the functional goals do not necessarily align with the goals the object was intended for. For example, scissors may be used to open a jar of paint. • Interaction to play: Another form of interaction occurs when the object is used for non-functional reasons, for playing in the broadest sense of the word, including playing sports or just messing around. Some objects are actually intended to play with; in this case the motivation is inherent to the object’s function. Examples are tennis rackets and yo-yos. But many people play with objects that were not initially meant to play with. This playing has a specific character: It involves physically moving and interacting with the object, just for the sake of the resulting sensation. It is sometimes referred to as ‘thoughtless’ playing with the object. However, in some cases the playing may involve a

48

Product Experience

to play? to explore?

to use?

how do I move … to take care? by accident?

to carry?

FIGURE 2.3 The map on Movements presents the different motivations people have to interact with an object: Why do you touch the object and how do you move?

challenge, like tearing up a coaster into as many parts as possible. In addition, playing may involve an element of cuddling, like when you stroke a pen against your skin, just because it feels so good. • Interaction to care for and to be taken care of: A specific theme in the interaction with objects is ‘to take care’. First, objects are used by people for personal care, that is, to brush one’s teeth, or to comb one’s hair. Furthermore, supporting someone can be regarded as a way of taking care; in that sense, chairs and beds take care of people. Again, this taking care can be the object’s primary function, like a coat that keeps someone warm, or a chair that supports, or it can be sought for independently from its function. For example, you can hold a coffee mug against your cheek to warm yourself. A second form of taking care is when people take care of the object: To wash it, repair it, store it, and so on. In other words, ‘taking care’ is a mutual aspect of the human–product interaction. • Interaction to explore: Regardless of its function, an object can be touched for the sake of exploring it, because it is unknown and a person wants to discover how it feels. This motivation is not necessarily restricted to unknown objects. A familiar object can also be touched just for the sake of touching it, to make contact with it. • Interaction to carry: Some interactions derive from the fact that objects are movable or portable, which involves a specific kind of interaction: Carrying the object. This carrying can be done in different ways: In a pocket, on your back, in your hands, and so on. • Interaction by accident, by coincidence: Furthermore, some interactions are not intentional or prompted by a specific motivation, but just happen by accident: People accidentally sit on something, or bump into something.

49

The tactual experience of objects

4. TACTUAL PROPERTIES OF OBJECTS Although people perceive an object as a whole, rather than as the sum of its different properties, this paragraph focuses on the perception of the different properties (Figure 2.4). For each property, the characteristic exploratory movements (see Section 3.2, Klatzky et al., 1985) are described, together with the possible dimensions on which people perceive these properties. These perceptual dimensions have been derived from experiments using sorting tasks, where participants sorted materials on the basis of their tactual similarities (e.g. see Giboreau, Navarro, Faye and Dumortier, 2001). Overall, tactual properties can be related to: The substance, i.e. the materials the object is made of: Its hardness, elasticity, plasticity, temperature and weight. The surface of the object: Its texture and patterns. The structure, i.e. the geometrical aspect of the object: Its global shape, exact shape, volume and weight distribution (balance). The moving parts of the object: The way the parts move in relation to one another.

elasticity?

movements?

force?

hardness?

material properties?

moving parts?

texture? how do I perceive its ...

geometry?

temperature?

shape? weight? size/ volume?

balance?

FIGURE 2.4 The map on Tactual Properties of an object presents the different properties perceived in tactual experience: How do you perceive the object’s tactual properties?

50

Product Experience

4.1. Hardness, elasticity and plasticity The hardness, stiffness and elasticity of an object’s materials are explored when people exert pressure on the object, for example when they squeeze the object. Other possible movements are pulling, pushing on it, and bending or wrenching it. These movements have in common that they try to transform the object. These properties evaluate the material’s resistance against, or compliance to, transformation: Hardness and softness are explored when exerting pressure (Klatzky, Lederman and Reed, 1987), and stiffness and flexibility, are explored when bending and wrenching (Ashby and Johnson, 2002). Once the material is transformed, the material’s elasticity and springiness depend on the way the material behaves when the pressure is released: Does it come back to its initial shape, or does it stay transformed? If it returns to its initial shape, the material is perceived as elastic. If it remains transformed (e.g. clay), this material property is referred to with the term ‘plasticity’.

4.2. Temperature People react differently to temperatures that are extremely high or low, compared with, temperatures that are close to their body temperature. Very high and very low temperatures are perceived instantaneously at contact, causing a strong withdrawal reflex. Temperatures that are not threatening need time to be perceived: People will leave their hands on a location for a while to perceive the difference between their body temperature and the temperature of the object. People perceive the warmness or coldness of objects, because objects with a temperature above or below body temperature cause a temperature flow. An object feels cold if it extracts warmth from the skin. To produce this sensation, the object does not only need to have a temperature below body temperature, it also has to extract warmth at a fast rate: The material needs to have low temperature resistance. Examples of such materials are glass and metal (Ashby and Johnson, 2002). Materials with high temperature resistance, such as wood or plastics, generally feel ‘warm’ even if their temperature is below body temperature. Due to the process of temperature flow, the temperature of objects changes over time when they are held, which eventually leads to neutral thermal perception (not warm, nor cold). The larger the difference between the object temperature and the skin temperature, the more accurate the temperature perception is (Tritsch, 1988).

4.3. Texture and patterns Texture is related to the properties of the material the object is made of and to the structure of the surface resulting from production techniques and surface treatment. Surface texture includes patterns, such as structured or randomly distributed details. Texture is explored through stroking the surface of an object. Stroking is necessary for the detection of fine surface textures (Hollins and Risner, 2000). Textures with coarser patterns may also be perceived through static touch (Lederman, 1981). In addition, surface texture is perceived when holding an object, thereby assessing the grip on that object (friction). Texture perception is probably one of the most studied tactual phenomena (Craig and Rollman, 1999), but researchers do not agree on an unequivocal set of perceptual dimensions for surface texture. Hollins et al. (1993) found that participants judged texture on three dimensions, the first two of which were the most important: Rough/smooth and soft/hard. The third dimension was related to the elasticity (‘springiness’) of the surface. In a follow-up study, the first two dimensions were again found, but the third

The tactual experience of objects

51

dimension was now defined as sticky/slippery (Hollins et al., 2000). Hollins et al. found bumpy/flat as an additional fourth dimension, but this dimension was not independent from the first three. In both experiments, a set of widely varying materials was used, such as sandpaper, velvet, and wood. Picard et al. (2003) investigated the perceptual dimensions of everyday tactual textures of textiles, and came up with the following set of four dimensions: Soft/harsh, thin/thick, relief/no relief, and hard/mellow. Some of the differences in outcomes between these studies may have depended on the presented materials. The perception of the roughness of a surface is not equal for all body parts: The lips and the fingers are most sensitive, while the heel, the back, and the thigh are least sensitive (Stevens, 1990). Furthermore, roughness perception is dependent on the way one moves. For example, roughness perception becomes more intense when the applied finger force increases (Lederman, 1974). For practical applications, the grip of objects is very important. The grip is mainly dependent on the degree of friction between hands and objects, which is influenced by the condition of the skin, such as dry or sweaty, and the amount of dirt on the skin. Slightly wet hands offer a greater friction force, which is why people spit on their hands before executing a task requiring firm grip. But too much water (or perspiration) forms a layer between hand and object, which causes slipping. The use of soap reduces the coefficient of friction of the hands (O’Meara and Smith, 2001). For soapy hands, textured surfaces offer the best grip, whereas for dry hands, smooth materials perform best (see Bobjer, Johansson and Piguet, 1993; Buchholz, Frederick and Armstrong, 1988). Friction is also dependent on the size of the contact area: The larger the contact area, the more friction (Highley, 1977).

4.4. Shape and size of the object Geometrical properties of objects are explored when grasping the object, holding it, manipulating it, and following the contours with the fingers. In addition, the size and shape of bigger objects are explored through dynamic touch, by swinging and wielding them. Shape discriminations can be based on the following characteristics (Lederman and Klatzky, 1987): (1) abrupt surface discontinuities, such as edges (no edge versus edge) and holes (hole versus no hole, shallow hole versus deep hole); (2) continuous 3D surface contours, such as curved versus flat; (3) orientation of surfaces (horizontal, vertical, slant). The importance of various shape properties seems to differ when they are perceived through touch compared to when they are perceived through vision. For example proportion, a typical aspect of visual perception, is not directly nor spontaneously perceived through tactual exploration (Appelle, Gravetter and Davidson, 1980). Changes in curvature are considered very important when perceiving visually, but changes in orientation of the object (its position in space) are not. This is quite the opposite when perceiving through the tactual system (Goodnow, 1969). Curvature perception depends on the direction of the scanning hand’s movement: Symmetrically curved edges often feel skewed in the direction of the moving hand (Goodnow, Baum and Davidson, 1971). In addition, shape perception is influenced by what has been perceived previously, a so called after-effect. For example, after prolonged perception of a concave surface, a flat surface is perceived as convex, and vice versa (Vogels, Kappers and Koenderink, 2001). Size in tactual perception is referred to as volume, length and width (Lederman and Klatzky, 1987). After-effects have also been described for the perception of size. After prolonged perception of an object with a certain length, longer objects are perceived as shorter and shorter objects as longer than their actual length. In general, compared to

52

Product Experience

the visual space, the tactual space seems smaller. Thus objects that were first touched and then seen look smaller than expected on the basis of touch (Teghtsoonian and Teghtsoonian, 1970).

4.5. Weight and balance An object’s weight is explored by holding the object, and by moving it up and down. Weight distribution is explored through dynamic touch, by swinging and wielding the object, or by trying to hold it still in a specific position (Turvey, 1996). Kreifeldt (2001) studied the object’s moment of inertia, perceived as an object’s resistance to rotational movement. This moment of inertia depends on where you hold the object. He illustrates this with the example of a baseball bat. The bat has a specific weight and a specific centre of gravity, but the way it is experienced depends on where you hold it: At the hitting hand or at the holding end. Weight is perceived as heavy or light, and weight distribution as balanced or unbalanced. Charpentier was the first to demonstrate, in 1891, that the perceived weight of an object depends not only on its physical mass but also on its size (Murray et al., 1999). When holding two objects of equal mass but of different size, subjects will consistently report the bigger one as lighter (Murray et al., 1999). This illusion also occurs in purely tactual situations, where the subjects are not able to see the objects (Amazeen, 1997). However, this effect emerges only when the grip on the objects is loose; it does not occur for firm grip (Ellis and Lederman, 1999). The perception of weight is influenced by what has been perceived previously. After prolonged holding of two objects of different weight in each hand, the weight of two objects of the same weight is estimated as different (de Mendoza, 1979). Weight perception is also influenced by the temperature of the object. For all body parts, cold objects rested on the skin feel heavier than thermally neutral ones. Warmth intensifies weight perception as well, but this effect is not present in all body parts. For example, it is present on the forearm, but it does not occur on the forehead (Stevens, 1980). Weight perception also seems to be influenced by other tactual properties of the object. For example, when lifting an object with the distal pads of the thumb and index finger at its sides (precision grip) the perceived weight depends on the object’s surface texture: The smoother the texture, the heavier the object is perceived. This is explained by the fact that to prevent the object from slipping, a greater normal force has to be applied when the object’s surface is smoother (Flanagan et al., 1995). Objects may consist of various constructions and mechanisms, and of moving parts. In the field of Human Factors, movement in human–product interaction has been studied extensively. This has resulted in an overview of the different types of grips used when holding an object (for example precision grip, force grip, antenna finger), different types of movements (for example translation, rotation), force exertion, and related types of control mechanisms (for example buttons, handles, wheels). For an overview of the resulting descriptions the reader is referred to MacKenzie and Iberall (1994).

5. TACTUAL SENSATIONS: BEING TOUCHED BY OBJECTS Like tactual perceptions, tactual sensations depend on the movements one makes when touching an object (Gibson, 1963). However, sensations and perceptions differ in the way they emerge from touching an object: The perception of an object may be considered as invariant throughout moving, whereas tactual sensations vary while moving. To illustrate

53

The tactual experience of objects

this, consider touching a wooden cube. You will perceive its shape through enclosure and contour following, moving the cube around in your hands. Throughout these different movements, the perception of the shape is invariant: It is a wooden cube. But this is not the case for the tactual sensations involved: The pressing of the edges of the corners, or of the flat surfaces of the cube on your skin varies with every movement of the hands and with every position of the cube. Thus the perception of the tactual invariant properties of an object coincides with varying sensations (Gibson, 1963). A physical property of an object may evoke many different sensations, depending on the way one interacts with it. This Section presents the skin and body senses and the related tactual sensations (Figure 2.5).

5.1. The skin and the skin senses Our skin is our largest organ: In adults it has a surface of 1.5 to 2 m2, is 0.5 to 4 mm thick (depending on the body part), and amounts to about 15% of total body weight (Saladin, 2001). Two types of skin cover the body: The glabrous (hairless) skin of the palms of the hands and the soles of the feet, and the hairy skin covering the rest of the body. The two types of skin seem to be equipped for different functions: The glabrous skin is suited for active touch in exploring and manipulating the world, and the hairy skin is suited for passive touch in signaling the locus of events, because of the following differences (MacKenzie and Iberall, 1994): • The glabrous skin is thicker (especially the epidermis), tougher, and more resistant to pressure. • The epidermis of the glabrous skin contains fat pads on the fingers and the bulges on the palm of the hand. These fat pads make the skin comply with the grasped object, thus facilitating a stable grip.

vibration?

itch?

body postures ? how do I sense ...

body temperature?

muscle force? pressure? pain?

FIGURE 2.5 The map on Bodily Sensations involved in tactual experience: What do you sense in your body when touching the object?

54

Product Experience

• The glabrous skin has a papillary structure: The epidermal ridges form the palmar print and the fingerprints. This structure has a sensory function: It allows the sensors to register lateral pressure. Hence, they contribute to the accuracy of the sense of touch. Furthermore, the ridges are important in grasping. They offer more grip on the grasped object, much like the profile of tires offers more grip on the road. • The distribution of the sweat glands is denser in the glabrous skin of hands and feet than in other parts of the body. The glands also differ in the way they respond to stimuli: The glands in the hand respond more to applied force (thus again facilitating grip), whereas the glands in the hairy skin respond more to temperature, thus facilitating temperature regulation. • The hairy skin lacks the Meissner’s corpuscles, responsible for the sensations of light touch and vibration. Therefore, people are unable to perceive subtle tactual details such as texture differences with body parts covered with hairy skin. Both skin types consist of three different layers, the epidermis, the dermis and the hypodermis containing a variety of sensors: Mechanoreceptors, sensitive to mechanical transformation of the skin; thermoreceptors, detecting cooling or warming of the skin; nociceptors, involved in the sensation of pain when the skin is (almost) damaged. Once the skin sensors are stimulated, neural fibers conduct this sensory information to the central nervous system. For each type of sensor, the corresponding neural fiber can either be slowly adapting (firing continuously) or rapidly adapting (only firing when the stimulus changes). These differences in adaptation rate strongly influence the duration of tactual sensations. Some sensations vanish rapidly because the sensor is adapted to the new situation (for example small temperature differences), whereas other sensations will last over time (for example deep pain) because the sensors do not adapt at all.

5.2. The skin sensations The skin sensations can be divided into three types, according to the type of sensors involved (Saladin, 2001; Vander, Sherman and Luciano, 2001): 1. Touch sensations and sensations derived from touch, such as superficial and deep pressure, and vibration (mainly mechanoreceptors involved). 2. Warm and cold sensations (mainly thermoreceptors involved). 3. Pain sensations (mainly nociceptors involved). Overall, sensations can be considered on the aspects of location (where), quality (what), intensity (how strong) and duration (how long does it last) (Gibson, 1963). We distinguish between the following types of skin sensations here (Figure 2.5): • Light touch: Light touch is what one senses when being touched without the skin being deformed. Light touch is mostly detected by rapidly adapting sensors (Saladin, 2001; Vander et al., 2001). This rapid adaptation allows people to forget about the clothes that touch them during the day. • Pressure: Pressure is maintained touch. It is experienced when an object is pressing on one’s skin and, thereby, deforming the skin. Pressure sensors are slowly adapting. That is why the sensations of deep and heavy pressure are usually difficult to ignore. • Vibration: Vibration in the skin is experienced when rapidly adapting touch sensors are stimulated rhythmically, for example when the hand strokes a texture, or when one sits on a chair and a truck is driving by, causing vibration of the floor. Receptor organs in the upper layer of the skin are sensitive to low frequency stimulation, the deeper receptors to high frequency stimulation (Sekuler and Blake, 1994).

The tactual experience of objects

55

• Cold and warmth: Although people intuitively consider warm and cold two opposites of one dimension, the sensations of warming and cooling of the skin are elicited by two different sensory systems. The skin easily adapts to temperatures between 20 and 40ºC, thus resulting in a thermal neutral perception of the object. Below 20 and above 40ºC, one perceives a cold or a warm object, but the sensation does not adapt over time. Above 45ºC, the tissue starts to be damaged, and the sensation becomes one of pain (Ganong, 2001; Sekuler and Blake, 1994). • Pain: When pain is induced by stimulation of the skin it is called superficial pain. Pain from muscles, bones and joints is called deep pain. Pain as a bodily sensation is referred to as somatic pain. The subjective experience of pain is a complex phenomenon; it differs from the other senses because it is intimately related to the affective meaning of the circumstances in which it is experienced, and the motivations of the subject. An extensive discussion on pain perception can be found in Kruger (1996). • Itch and tickle: Itch and tickle are skin sensations elicited by stimulation of nonmyelinated fibers in the skin that are highly similar to pain sensors. Itch and tickle are produced by mild stimulation when moving something across the skin (Saladin, 2001; Vander et al., 2001). Itch can also be produced by chemical stimulation of the skin. The difference between tickle and itch is not clearly defined. • Physical pleasure: Olausson et al. (2002) discovered a system of non-myelinated, slow conducting sensors in the hairy skin, that responds when a person is being touched lightly, producing a faint sensation of pleasant touch, without producing the sensation of being touched. Therefore, it seems plausible to consider the experience of physical pleasure as a physical sensation per se. The researchers suggested that people might possess a special system for limbic touch, underlying emotional responses to caress-like skin-toskin contact between individuals. Because the study used a soft brush as a stimulus, this system is not confined to interpersonal touch, but includes touching by objects.

5.3. The body senses and sensations In addition to the skin sensations, active touch involves the two body senses: Proprioception that perceives body posture, and kinaesthetics that perceives body movement (Figure 2.5). Body posture and body movement are sensed through sensors in muscles, tendons and joint tissues (Saladin, 2001; Vander et al., 2001).

5.4. Tactual sensitivity Tactual sensitivity relates to the capacity to sense if one is touched, where one is touched, for how long, and with what intensity (Lederman and Klatzky, 1998). Because the sensors in the skin are not equally distributed, tactual sensitivity depends on the locus on the body. The fingertips and lips contain most sensors per mm2 (Stevens, 1990). In addition, sensitivity depends on the spatial aspects of the afferent neurons. Receptive fields of fibers in the upper layer of the skin are relatively small (2–4 mm) and overlapping. Thus they form a sensitive system to locate a point on the skin. The deeper receptors have larger receptive fields, making the location of a point on the skin less accurate (Sekuler and Blake, 1994). Finally, sensitivity is related to the size of the reception areas in the somatosensory cortex. These areas are not proportional to the different body parts: The lips and hands cover the largest area, whereas the back and the calf cover relatively small parts. These differences in sensitivity are reflected in the homunculus defined by Penfield (Sekuler and Blake, 1994) that depicts the different body parts proportional to their reception areas in the cortex (Figure 2.6).

56

Product Experience

FIGURE 2.6 Penfield’s Homunculus.

The sensitivity due to the distribution of sensors and to the spatial characteristics of the corresponding neurons throughout the body cannot be altered by training. However, the size of the reception areas in the brain can be enlarged by training and experience. For example, blind people do not possess better discriminative abilities than sighted people (Hanninen, 1972), but they have become better in recognizing objects (Berla and Butterfield, 1977) and patterns (Craig, 1988) through touch. Tactual sensitivity is not a static aspect of an organism, but varies in time. Sensitivity decreases through aging. For example, the spatial acuity of the skin of the fingertip deteriorates with age (Stevens and Choo, 1996). Various diseases or pathological conditions can disturb the tactual senses, such as diabetes, leprosy, multiple sclerosis and Parkinson’s disease (Pratorius, Kimmeskamp and Milani, 2003). Damage of the central nervous system or peripheral nerve tissue through accident or tumors may cause a loss of sensitivity as well (Franzen and Lindblom, 1976).

6. THE BODY LANGUAGE OF OBJECTS The affective aspects of a tactual experience with an object can be characterized as the experienced body language of the object. In physical interaction objects are experienced as expressing affective behavior through their physical reactions to our actions. This affective behavior can be described along a number of themes that are discussed below (Figure 2.7).

6.1. Personality A person’s personality is sometimes characterized by words typically used to describe material properties that can be perceived tactually: People are experienced as weak, strong, hard, soft, flexible, rigid, warm, or cold, and so on. Conversely, objects may be regarded as entities expressing personality. In this case, people talk about objects as if they possess human characteristics. For example, they may say about an object that ‘it is obeying, but with dignity’, or that ‘it feels strong and playful’. According to Govers (2004), product personality can be defined as the set of human characteristics that people use to describe an object. What makes the tactual characteristics particularly interesting in this respect is that people seem to transfer the perceived tactual qualities

57

The tactual experience of objects

transparency?

physical skills? power match?

feedback?

perfect match?

how do I experience its ...

emotions?

personality?

familiarity? intentions?

FIGURE 2.7 The map on the Affective Behavior of an object involved in tactual experience: How do you experience the object’s affective behavior?

directly to the product personality: A cold object expresses a cold personality, and a flexible product may be experienced as a flexible personality. Because objects seem to have a personality, they seem to become social entities and, consequently, may evoke feelings and emotions that usually only apply to the interpersonal domain, such as feelings of sympathy. For example, people may report that they feel sorry for an object that seems sad, because it has a broken part.

6.2. Intentions People have specific motivations to interact with objects (Section 3.3), and people experience objects as having intentions as well. Sometimes an object seems to have a will of its own: It wants (or refuses) to be explored, to take care, to cooperate, to play, and so on. This perceived intentionality emerges because objects seem to display a specific physical behavior in interaction. People move in order to experience an object. In return, when the object moves this is experienced as meaningful, intentional behavior (Heider and Simmel, 1944; Michotte, 1963). Thus when an object does not react, it is experienced as refusing, rejecting. And when a chair collapses under somebody’s weight, that chair is experienced as letting that person down. Sonneveld (2007) found the following intentions in participants’ descriptions: • Wanting to be touched and explored (e.g. ‘the balloon had some kind of “I don’t want to be in your hands” reaction’). • Wanting to cooperate (e.g. ‘my walking shoes are much too heavy, they feel like they do not want to walk at all, they are too tired to do so’).

58

Product Experience

• Wanting to play (e.g. rubber bands, springs, and some on/off switches elicit repetitive movements, because they seem to ask you to go on playing with them). • Wanting to take care of and, related, wanting to love or hurt somebody (e.g. objects that literally support and take care of people due to their function, such as furniture and clothes).

6.3. Integrity: Tactual feedback In interaction, an object provides people with information. On the one hand, it provides information about itself, for example about its properties, about what it is and what it is doing. On the other hand, it provides information about the physical world around it, about what is going on. With the information the object supplies, it can guide people in what they are trying to achieve, but it may also mislead them. Therefore, the way in which objects give tactual feedback is experienced as the integrity of the object. To start with, integrity is related to whether the object gives any feedback at all: Products can be rich or poor in tactual information. For example, touch screens do not let the user feel what they are actually doing, whereas other interfaces, such as steering wheels of cars, let you know exactly what is going on. Next, if objects give feedback about what is happening, they may or may not seem honest. For example, coffee mugs of porcelain provide information about the temperature of the coffee inside, whereas polystyrene foam cups do not. In contrast to the porcelain cups that provide the correct information, people feel fooled when the coffee in the polystyrene cup is much hotter than expected. How tactual feedback is experienced is largely dependent on the expertise of the user: An experienced car mechanic will get a lot more information through manipulation of the different parts of an engine than a layperson. The appreciation of the integrity of objects is not unequivocal; being teased or fooled may be part of a pleasant experience, depending on the context of the interaction. For example, in the case of a toy crocodile the lack of integrity makes it exciting: The crocodile bites when you press one of his teeth, but you never know which tooth it is going to be.

6.4. The perfect match When touching an object, people evaluate the way it fits them. People seem to enjoy the feeling that something feels perfectly right: The perfect match. The degree of match is primarily determined by the object’s geometrical properties (Section 4.4), such as the fit of shoes and clothes, the shape and balance of tools, and the temperature of a shower. In addition, the degree of match can be related to dynamic properties, such as when the object moves with you as if it dances with you. A perfect match may be experienced immediately during the first encounter, or it may emerge from an intensive interaction over time. For example, a fountain pen adapts to the writer’s hand, and scissors adapt to the hairdresser’s hand. Over time, these objects become so well adapted to the owner’s hand that they seem impossible to work with for others.

6.5. Familiarity: Feeling ‘mine’ or alien When an object is touched frequently, it becomes familiar, it feels like one’s own, and one recognizes it as such. In contrast, an unfamiliar object may be experienced as ‘alien’ or ‘someone else’s’. Being new and unfamiliar can be a burden: People reported that as a child they had difficulties in accepting new clothes, because they did not feel as ‘own’.

The tactual experience of objects

59

This theme differs from the perfect match in the sense that familiarizing does not necessarily lead to a perfect fit, it just increases the familiarity of the object. Vice versa, if the size is right an unfamiliar object may already fit perfectly at the first encounter. Familiarity grows over time; the memories of past experiences contribute to the current experience. In addition, touch allows people to recognize objects in an affective, intimate way. One person said ‘I was wondering if it was my grandmother’s cupboard that I used to play with. When I let my hands glide across the surface of the woodcuttings, I suddenly knew: It’s hers. This tactual memory was very strong’. On the other hand, to feel somebody else ‘through’ an object can be a bewildering experience, because it evokes unwanted feelings of intimacy. For example, when you sit down on a toilet seat that is still warm from the person who sat there just before you, it feels like the seat was not yet ready to be used again; it almost feels like a strange, intimate contact with somebody else. Feelings of familiarity may be enhanced by elements that make the product unique (see Chapter 17). First, an object may be unique in shape, texture, or other details, which allows the user to recognize it as his ‘own’ object. Next, small changes that occur over time due to the interaction with the object, such as small dents, scratches or other imperfections, allow the user to recognize the object as the familiar object.

6.6. Power match and being in control A physical encounter may elicit an assessment of power: Who is the strongest? Some products literally challenge people to arm wrestling, like a marmalade jar with a tight lid. Who does not want to win, and stick the lid with triumph in the air once the jar has been defeated? In addition, the power match addresses the question about who is in control in the interaction: Who leads whom? Power seems to be mainly explored in the first encounter. It is not only about winning and being the strongest, but also about exploring limits: How far can I go? How much can I bend this stick? How far can I stretch this band? Furthermore, this theme assesses the requested physical effort one has to invest in the interaction. It can be overwhelming to sense the power one has, to feel that one is completely in control, for example in driving a car. But the opposite can be exciting as well: Luna park attractions are exciting because people experience that they are completely out of control.

6.7. Challenge of developing physical skills Physical interaction involves physical skills. Objects differ in the way they challenge people to develop their skills. Some objects require great skills, such as musical instruments, and some do not, such as objects with push buttons. People may have to invest a lot of effort to develop the required skills, for example in the case of learning to play the guitar. Once acquired, the practice of a physical skill can be a source of pleasure, of flow (Csikszentmihalyi, 1990). People are programmed biologically to like to develop their physical skills and to exercise them (Veenhoven, 2006). To develop skills, one has to develop ‘tactual knowledge’ through interacting with the objects. For example, a masseur needs to ‘know’ from experience the human body in its different qualities, to be able to work with it. Likewise, a sculptor needs to ‘know’ the material he is working on from experience. This aspect is closely related to the theme of tactual feedback. Objects differ in the way they allow people to develop a personal style when developing these skills, often referred to as doing things with a ‘personal touch’. Some objects

60

Product Experience

prescribe skills in a rather straightforward way, such as pressing the buttons on a microwave oven. Other objects allow more freedom to develop one’s own style, such as making cocktails with a cocktail shaker. In contrast, some objects seem to have their own, personal user manual that only the user knows. This is a particular form of physical skill and of ‘knowing’, because it has nothing to do with an established skill. This skill is specific for objects with stubborn personalities that can only be managed in a particular way. The theme of physical skills is closely related to the theme of power. When a person needs physical skills to interact with certain objects, the objects will have power over that person as long as (s)he has not developed these skills. For example, a car seems uncontrollable when you do not know how to drive it, but you gain control over it over the course of your driving lessons.

6.8. Attention: Tactual transparency and tactual noise When touching an object, people are in contact with that object, but their attention is not necessarily directed towards the object. The theme ‘transparency’ refers to the capacity of the object to allow people to feel through the object, to incorporate it, and to direct their attention to something else in their environment. For example, when riding a bicycle, you feel the road surface through the handle-bars and through the saddle. However, an annoying tactual sensation can create ‘tactual noise’ that reduces the degree of transparency, for example, when the handles get sticky they will attract the majority of attention and, thereby, divert the attention that was first directed towards the road. Condoms and surgical gloves with true touch are typical examples of products for which manufacturers have tried to decrease tactual noise, but nevertheless these objects may stay in one’s focus of attention. Condoms are a good example of the dual design possibilities of tactual transparency; either they are designed as tactually transparent as possible, or the designer accepts the fact that people perceive them anyway and tries to design a particular tactual experience.

6.9. Conclusion on the themes The previous paragraphs discussed the different themes involved in the body language of objects. Although some themes are more relevant than others to understand and describe specific experiences, they are all useful to describe the different aspects of a tactual experience as a whole. The themes should, therefore, be seen as different aspects of the same phenomenon. Also, they are not mutually exclusive, but they are related to each other. Thus, whatever the perspective taken, the other themes will emerge as context.

7. THE FEELINGS INVOLVED IN TACTUAL EXPERIENCE In human–product interaction, the experienced affective behavior of the object is reflected in the feelings elicited by this behavior. These different types of feelings are described below (Figure 2.8): • Physical pleasure: Lust and pain or disgust. Obviously, pleasantness in tactual experience is related to lust and physical pleasure. The pleasure is generally characterized by descriptions like ‘it feels good’, with superlatives such as ‘delicious’ and ‘delightful’. When people are asked to focus on tactual experiences, they are often amazed by the

61

The tactual experience of objects

bodily reaction? self experience?

physical pleasure? what is my ...

action tendency?

vulnerability?

energy?

affective reaction?

FIGURE 2.8 Map on Gut Feelings involved in tactual experience with an object: What feelings do you have when touching the object?

pleasantness of the experience: ‘It was even better than I thought it would be’. The physical pleasure does not seem to have one single counterpart, some descriptions of unpleasantness involve pain (e.g. ‘painful’, ‘it hurts’, ‘it stings’), some refer to light or strong discomfort (e.g. ‘uncomfortable’, ‘annoying’, ‘terrible’), while others contain elements of disgust (e.g. ‘repulsion’, ‘a dirty feeling’). People did not describe their feelings of physical pleasure or pain and disgust elaborately; these descriptions often missed nuances. In many cases, these feelings were exemplified by expressive sounds, such as ‘wow’, ‘mmmm’, ‘ugh’, or ‘yuk’ and the experience seemed to be grounded in a physical reaction accompanied by bodily responses, such as goose bumps, shivers, and feelings of nausea. Physical (un)pleasantness should not be confused with the overall assessment of the experience, because in some cases physical unpleasantness is nevertheless appreciated. As one person responded: ‘Its sliminess was disgusting, but at the same time that was also what made it attractive to touch’. And another one said: ‘It hurts so good’. • Affection: Love and hate. The feelings involved in interactions with products seem to be related to the experience of mutual affection, of feeling love for the object, as well as feeling loved by the object. This theme reflects the perspective described in Section 2.4, considering touch as a communication channel for affection. People refer to this theme in various ways. First of all, they indicate that they feel love or feel loved (e.g. ‘it has cuddle value for me’, ‘I feel tenderness’), or that they feel hatred or feel being hated (e.g. ‘I see it as a necessary evil’, ‘It drives me mad’). In some cases, the degree of affection seems to depend on the degree of intimacy between the person and the product (e.g. ‘I have my own way of cuddling it’).

62

Product Experience

A related set of feelings involve respect and contempt. As one person states: ‘When you push it harder it still feels soft, but at the same time it gives some resistance, which evokes respect’. Another set of feelings addresses acceptance and rejection. For example: ‘It accepts me the way I am’. Furthermore, affection may involve feelings of attachment. • Vulnerability: Trust and fear of getting hurt. Touch involves the body in physical contact, and confronts people with their physical vulnerability. This theme involves feelings of trust and distrust. It has to do with people’s fear of getting hurt, and also with feelings of being reassured, of trusting objects, and feeling safe with them. In addition, mixed experiences may be reported as well: ‘Under water, the muddy soil was an unknown world, but once you overcame your fear, it was a source of pleasure’. This theme is also related to feelings of freedom and oppression. For example, whereas some clothes give ‘the feeling of being free, like Peter Pan’, other clothes can make you ‘feel oppressed, suffocating’. • Energy: Tension and relaxation. The physical interaction with an object influences a person’s energy level. The energy may increase or decrease, and the energy may be experienced as positive or negative energy. This leads to reports of being physically excited, stressed, relaxed or washed-out. • Action tendency: Approach and avoidance. Action tendencies are components of emotions. An action tendency implies that the actual touching did not occur yet. Hence, the tactual experience may already start before you actually have physical contact with the object; you could call it ‘touching with the eyes’. Similarly, once a person knows how bad something feels, he may try to avoid it. Beside a tendency to approach or avoid, people may experience a tendency to hold on to or let go, and to take care of or neglect, or even to destroy. • Tactual characteristics reflected in self-experience: The tactual properties of objects can be transferred to the self-experience of the person who touches the object. For example, touching something cold can make one feel cold; or touching something dirty can make one feel dirty. The same phenomenon occurs for the experienced personality aspects of the object. For example, when a woman experiences an object as impressive, using the object may make her feel more impressive herself. An attempt to characterize these themes all together led to the umbrella concept of gut feelings as the basic concept for the emotions and feelings experienced in tactual interaction (Figure 2.8). Gut feelings are characterized as feelings that emerge from a non-reflective, direct interaction with the world, and may be related to the visceral level of interaction defined by Norman (2002). Gut feelings are related to our intuitive orientation on the world; they are grounded in our physical actions and physiological reactions, rather than in our cognitive orientation to the world, which is grounded in our thoughts. This explains why it is difficult for people to talk about touch and feelings: It is a non-verbal (or as some would say pre-verbal), intuitive mode of interaction.

8. EDUCATING THE TACTUAL SENSES In design education, students need to develop their awareness, insight and sensitivity for the different aspects of aesthetic experience in human–product interaction. This education cannot be achieved by a mere transfer of knowledge through lectures and readings. It is generally acknowledged that personal hands-on experiences should be part of the educational setting. To educate the designer’s senses, it seems therefore appropriate to develop tools and methods that offer a conceptual framework about the sensory experience,

The tactual experience of objects

63

that is embedded in practical, sensory experiences. Such an approach is based on the interplay between cognitive learning and perceptual learning; once people have a set of concepts to describe what they feel, they will be able to perceive more nuances, and thus describe their experiences more extensively. And vice versa, the more they have experienced physically, the more they will be able to give content to the specified concepts (Chollet, Valentin and Abdi, 2005). Product design education does not seem to offer such educational tools and methods yet. But other domains, as for example art education or wine tasting, offer valuable insights in possible starting points for such educational tools and methods. In art education, several methods were developed to learn to look. These methods offer a set of concepts people may use when looking at and experiencing art. For example, Visser (1986) developed a framework that guides people through the process of experiencing art, focusing on the formal aspects of art perception, such as materials, composition, size, space, point of view, abstraction, and so on. On the other hand, Armstrong (2000) proposed a set of different stages that people may go through when experiencing art, such as gathering information, dreaming, contemplation, investment (engagement), and so on. Both approaches are characterized by a conceptual framework illustrated with examples that the perceiver must apply in his own situation to assimilate their meaning. In the domain of the sensory evaluation of food, tools and methods were developed to enable people to train their sensory sensitivity. For example, the sensory user’s manual for wine tasting (LaMar, 1997) provides an overview of the different sensory aspects of wine and instructions on how to drink it in order to perceive its different aspects. People who attend wine-tasting workshops are thus presented with words that describe sensory aspects, but these words remain empty shells when they are not supported by wine tasting itself. The concepts become meaningful when they are experienced physically. It is only through experience that the concepts become embodied knowledge, and thus recognizable in experience. Therefore, to become wine-tasting experts, people need to build up a personal, embodied ‘database’ of sensory experiences, linked to the framework that is offered, to be able to work with that framework when actually tasting wine. Likewise, tools and methods in the education of the tactual senses in product design should provide designers with a frame of thought introducing the sensory, perceptual and experiential aspects of the specific sensory domain, a structure to guide the designers through these aspects, and insight in appropriate ways to explore these aspects. Moreover, these tools should support designers to build up a personal, embodied database linked to the conceptual framework that is offered on tactual experiences, and to customize the framework to fit the personal world of experience. It is from this starting point that the Tactual Experience Guide was developed. It offers a basis for the development of the aesthetic sensitivity of product design students, provided it is embedded in a personal hands-on exploration of the presented concepts. To ensure these hands-on experiences, product design education should include design courses addressing the aesthetic aspects of the senses.

9. FUTURE DEVELOPMENTS By presenting the content of the Tactual Experience Guide (Sonneveld, 2007) and some of the qualitative data on which it is based, the present chapter has provided a conceptual framework for the description of tactual experiences through words. During design

64

Product Experience

courses at Delft University of Technology, using and evaluating the guide has demonstrated the usefulness of words to get insight into the experience, but simultaneously it has made clear that words are limited in expressing nuances in tactual aesthetics. Therefore, we should explore the possibility to include the use of images, sounds, tangible samples, and so on, when using the Guide. In addition, the evaluation suggested that the Guide could be accompanied by a toolbox containing objects and materials that illustrate the different aspects of tactual experience. As it is only through hands-on experiences that one gains insight in tactual aesthetics, such hands-on experiences should be provided when being introduced to the Guide. Another limitation of the Tactual Experience Guide is that it does not provide insight into the underlying relations between the different aspects it describes. In visual aesthetics, relations between visual properties are explored to understand the underlying principles of aesthetic experience (Kreitler and Kreitler, 1972). These principles include concepts such as contrast, composition, proportion, (dis)continuities, ambiguities, and so on. In addition, the domain of visual aesthetics explores the relationship between visual properties of objects and exposure to these objects, leading to insights such as the MAYA-principle: Most Advanced Yet Acceptable (see Chapter 10). The field of tactual aesthetics lacks insight in such underlying principles. To develop the research field of tactual aesthetics, future research should focus on the underlying principles of aesthetic experience, thus possibly revealing some universal tactual Gestalt principles, and revealing insight in relations between aesthetic experience and interaction factors such as exposure frequencies (Hekkert, 2006). Finally, the insight that tactual experience in human–product interaction can be described as the body language of two social entities opens up new ways to explore the aesthetics of tactual experience, inspired by inter-human behavior and relationships. Possible sources of inspiration are insights in body language (Mehrabian, 1972); attachment (Mugge, Schifferstein and Schoormans, 2006); personality studies (Govers, 2004; Totton and Jacobs, 2001); haptonomics (Veldman, 1996); and social well-being (Ryan and Deci, 2001). For example, Ryan and Deci (2001) concluded that well-being in a relationship between two people is dependent on three factors: One should feel understood, feel competent, and have fun. It is intriguing to translate these findings into requirements for well-being in human–product relationships and to research their implications. Likewise, according to Veldman (1996) experiencing the integrity of the other, and trusting the other, form a necessary and solid base for an exchange of affection. Again, this may offer interesting hypotheses for human–product relationships that could lead to new insights in product design. To conclude this brief overview of possibilities, Totton and Jacobs (2001) observed that people all need to feel understood and accepted by other human beings. Likewise, we could hypothesize that people all need to feel understood and accepted by at least some of the objects that surround them. These directions for future developments in Tactual Aesthetics are not exhaustive; other directions may be explored as well. Nevertheless, the directions mentioned above illustrate that the presented conceptual framework forms a consistent basis to build future theory on; it can be the starting point for developing new hypotheses and research questions, exploring the full diversity of the field of Tactual Aesthetics. In addition, we think that the content and the structure of the conceptual framework will help to bridge the gap between research and design. By supporting designers in developing their tactual sensitivity and exploring the tactual aspects of the objects in their design projects, new insights in Tactual Aesthetics may be obtained with relevance to researchers in the field.

The tactual experience of objects

65

REFERENCES Amazeen, E. L. (1997). The effect of volume on perceived heaviness by dynamic touch: with and without vision. Ecological Psychology, 9, 245–263. Appelle, S., Gravetter, F. J. and Davidson, P. W. (1980). Proportion judgments in haptic and visual form perception. Canadian Journal of Psychology, 34, 161–174. Armstrong, J. (2000). The intimate philosophy of art. London: Penguin Books. Ashby, M. and Johnson, K. (2002). Materials and design: the art and science of material selection in product design. Oxford: Butterworth-Heinemann. Bergsma, J. (1987). Identiteit, lichamelijke verstoring en autonomie. Lochem: De Tijdstroom. Berla, E. P. and Butterfield, L. H. (1977). Tactual distinctive features analysis: training blind students in shape recognition and in locating shapes on a map. Journal of Special Education, 11, 335–346. Bermudez, J. L., Marcel, A. and Eilan, N. (1995). The body and the self. London: MIT Press. Bobjer, O., Johansson, S.-E. and Piguet, S. (1993). Friction between hand and handle. Effects of oil and lard on textured and non-textured surfaces; perception of discomfort. Applied Ergonomics, 24, 190–202. Bolanowski, S. J. (2004). Passive, active and intra-active (self) touch. Behavioral Brain Research, 148, 41–45. Buchholz, B., Frederick, L. J. and Armstrong, T. J. (1988). An investigation of human palmar skin friction and the effects of materials, pinch force and moisture. Ergonomics, 31, 317–325. Burton, G. (1993). Non-neural extensions of haptic sensitivity. Ecological Psychology, 5, 105–124. Chollet, S., Valentin, D. and Abdi, H. (2005). Do trained assessors generalize their knowledge to new stimuli? Food Quality and Preference, 16, 13–23. Craig, J. C. (1988). The role of experience in tactual pattern perception: a preliminary report. International Journal of Rehabilitation Research, 11, 167–183. Craig, J. C. and Rollman, G. B. (1999). Somesthetis. Annual Review of Psychology, 50, 305–331. Csikszentmihalyi, M. (1990). Flow: the psychology of optimal experience. New York: Harper Collins. Davidson, P. W. (1985). Functions of haptic perceptual activity in persons with visual and developmental disabilities. Applied Research in Mental Retardation, 6, 349–360. de Mendoza, J.-L. J. (1979). Demonstration of an aftereffect occurring in the tactile-kinesthetic domain. Psychological Research, 40, 415–422. Dewey, J. (1997 [1938]). Experience and education. New York: Touchstone. Ellis, R. R. and Lederman, S. J. (1999). The material-weight illusion revisited. Perception and Psychophysics, 61, 1564–1576. Fagan, J. (1998). Thoughts on using touch in psychotherapy. In: E. W. L. Smith, P. R. Clance and S. Imes (Eds.) Touch in psychotherapy. New York: Guilford. Fields, T. (2003). Touch. London: MIT Press. Finnegan, R. (2002). Communicating. The multiple modes of human interconnection. London: Routledge. Flanagan, J. R., Wing, A. M., Allison, S. and Spenceley, A. (1995). Effects of surface texture on weight perception when lifting objects with a precision grip. Perception and Psychophysics, 57, 282–290. Franzen, O. and Lindblom, U. (1976). Tactile intensity functions in patients with sutured peripheral nerve. In: Y. Zotterman (Ed.) Sensory functions of the skin with special reference to man. Oxford: Pergamon Press. Ganong, W. F. (2001). Review of medical physiology. New York: McGraw-Hill. Giboreau, A., Navarro, S., Faye, P. and Dumortier, J. (2001). Sensory evaluation of automotive fabrics: the contribution of categorization tasks and non-verbal information to set-up a descriptive method of tactile properties. Food Quality and Preference, 12, 311–322. Gibson, J. J. (1962). Observations on active touch. Psychological Review, 69, 477–491. Gibson, J. J. (1963). The useful dimensions of sensitivity. American Psychologist, 18, 1–15. Goodnow, J. J. (1969). Eye and hand: differential sampling of form and orientation properties. Neuropsychologia, 7, 365–373. Goodnow, J. J., Baum, B. and Davidson, P. (1971). A haptic error: skew in symmetrical curve. Perception and Psychophysics, 10, 253–256. Govers, P. (2004). Product personality. Unpublished PhD dissertation, Delft University of Technology, Delft, the Netherlands. Hanninen, K. A. (1972). Review of the educational potential of texture and tactually discriminable patterns. Journal of Special Education, 5, 133–141. Heider, F. and Simmel, M. A. (1944). An experimental study of apparent behavior. American Journal of Psychology, 57, 243–259. Hekkert, P. (2006). Design aesthetics: principles of pleasure in design. Psychology Science, 48, 157–172. Highley, D. R. (1977). Frictional properties of the skin. Journal of Investigative Dermatology, 69, 303–305. Hodgson, J. (2001). Mastering Movement. New York: Routledge.

66

Product Experience

Hollins, M., Bensmaia, S., Karlof, K. and Young, F. (2000). Individual differences in perceptual space for tactile textures: evidence from multidimensional scaling. Perception and Psychophysics, 62, 1534–1544. Hollins, M., Faldowski, R., Rao, S. and Young, F. (1993). Perceptual dimensions of tactile surface texture: a multidimensional scaling analysis. Perception and Psychophysics, 54, 697–705. Hollins, M. and Risner, S. R. (2000). Evidence for the duplex theory of tactile texture perception. Perception and Psychophysics, 62, 695–705. Klatzky, R. L., Lederman, S. J. and Metzger, V. A. (1985). Identifying objects by touch: an ‘expert system’. Perception and Psychophysics, 37, 299–302. Klatzky, R. L., Lederman, S. J. and Reed, C. (1987). There’s more to touch than meets the eye: the salience of object attributes for haptics with and without vision. Journal of Experimental Psychology: General, 116, 356–369. Kreifeldt, J. G. (2001, June 27–29). Designing ‘feel’ into a product. Paper presented at the International Conference on Affective Human Factors Design, Singapore. Kreitler, H. and Kreitler, S. (1972). Psychology of the arts. Durham, NC: Duke University Press. Kruger, L. (1996). Pain and touch. San Diego: Academic Press. Lakoff, G. and Johnson, M. (1999). Philosophy in the flesh: the embodied mind and its challenge to western thought. New York: Basic Books. LaMar, J. (1997). Tasting wine: a sensory user’s manual. Retrieved 23 March 2007, from http://avalonwine. com/Tasting-Wine.php Lederman, S. J. (1974). Tactile roughness of grooved surfaces: the touching process and effects of macro- and microsurface structure. Perception and Psychophysics, 16, 385–395. Lederman, S. J. (1981). The perception of surface roughness by active and passive touch. Bulletin of the Psychonomic Society, 18, 253–255. Lederman, S. J. and Klatzky, R. L. (1987). Hand movements: a window into haptic object recognition. Cognitive Psychology, 19, 342–368. Lederman, S. J. and Klatzky, R. L. (1998). The hand as a perceptual system. In: K. J. Connolly (Ed.) The psychobiology of the hand. London: MacKeith Press. Lederman, S. J. and Klatzky, R. L. (1999). Sensing and displaying spatially distributed fingertip forces in haptic interfaces for teleoperator and virtual environment systems. Presence, 8, 86–103. MacKenzie, C. L. and Iberall, T. (1994). The grasping hand. Amsterdam: North Holland. Mehrabian. (1972). Non-verbal communication. Chicago, Il: Aldine-Atherton. Michotte, A. (1963). The perception of causality. London: Methuen. Montagu, A. (1971). Touching. New York: Columbia University Press. Mugge, R., Schifferstein, H. N. J. and Schoormans, J. P. L. (2006). Product attachment and product lifetime: the role of personality congruity and fashion. In: K. M. Ekstrom and H. Brembeck (Eds.) European Advances in Consumer Research, 7, 460–466. Duluth, MN: Association for Consumer Research. Murray, D. J., Ellis, R. R., Bandomir, C. A. and Ross, H. E. (1999). Charpentier (1891) on the size–weight illusion. Perception and Psychophysics, 61, 1681–1685. Norman, D. A. (2002). Emotion and design: attractive things work better. Interactions, 9(4), 36–42. Olausson, H., Lamarre, Y., Backlund, H., et al. (2002). Unmyelinated tactile afferents signal touch and project to insular cortex. Nature Neuroscience, 5, 900–904. O’Meara, D. M. and Smith, R. M. (2001). Static friction between human palmar skin and five grabrail materials. Ergonomics, 44, 973–988. Picard, D., Dacremont, C., Valentin, D. and Giboreau, A. (2003). Perceptual dimensions of tactile textures. Acta Psychologica, 114, 165–184. Polanyi, M. (1967). The tacit dimension. Garden City, NY: Doubleday. Pratorius, B., Kimmeskamp, S. and Milani, T. L. (2003). The sensitivity of the sole of the foot in patients with Morbus Parkinson. Neuroscience letters, 346, 173–176. Ryan, R. M. and Deci, E. L. (2001). On happiness and human potentials: a review of research on hedonic and eudemonic well being. Annual Review of Psychology, 52, 141–166. Sacks, O. (1984). A leg to stand on. London: Picador. Sacks, O. (1987). The man who mistook his wife for a hat and other clinical tales. New York: Harper and Row. Saladin, K. S. (2001). Anatomy and physiology: the unity of form and function. New York: McGraw-Hill. Schifferstein, H. N. J. and Cleiren, M. P. H. D. (2005). Capturing product experiences: a split-modality approach. Acta Psychologica, 118, 293–318. Sekuler, R. and Blake, R. (1994). Perception (3rd Ed.) New York: McGraw-Hill. Sonneveld, M. H. (2007). Aesthetics of tactual experience. Unpublished PhD dissertation, Delft University of Technology, Delft, the Netherlands. Stevens, J. C. (1980). Thermal intensification of touch sensation: further extensions of the Weber phenomenon. Sensory Processes, 3, 240–248.

The tactual experience of objects

67

Stevens, J. C. (1990). Perceived roughness as a function of body locus. Perception and Psychophysics, 47, 298–304. Stevens, J. C. and Choo, K. K. (1996). Spatial acquity of the body surface over the life span. Somatosensory and Motor Research, 13, 153–166. Teghtsoonian, R. and Teghtsoonian, M. (1970). Two varieties of perceived length. Perception and Psychophysics, 8, 389–392. Thayer, S. (1982). Social touching. In: W. Schiff and E. Foulke (Eds.) Tactual perception: a sourcebook. Cambridge: Cambridge University Press. Totton, N. and Jacobs, M. (2001). Character and personality types. Buckingham: Open University Press. Tritsch, M. F. (1988). The veridical perception of object temperature with varying skin temperature. Perception and Psychophysics, 43, 531–540. Tsakiris, M. and Haggard, P. (2005). The rubber hand illusion revisited: visuotactile integration and selfattribution. Journal of Experimental Psychology, 31, 80–91. Turvey, M. T. (1996). Dynamic touch. American Psychologist, 51, 1134–1152. Vander, A. J., Sherman, J. and Luciano, D. (2001). Human Physiology. The mechanisms of body functions. New York: McGraw-Hill. Veenhoven, R. (2006). Healthy happiness. Rotterdam: Erasmus University. Veldman, F. (1996). Haptonomie: Wetenschap van de affectiviteit (Haptonomics: Science of affectivity). Utrecht: Erven J. Bijleveld. Visser, A. (1986). Hardop kijken (Looking aloud). Nijmegen: SUN. Vogels, I. M. L. C., Kappers, A. M. L. and Koenderink, J. J. (2001). Haptic after-effect of successively touched curved surfaces. Acta Psychologica, 106, 247–263. Winnicott, D. W. (1964). The child, the family and the outside world. Middlesex, UK: Penguin.

3

THE EXPERIENCE OF PRODUCT SOUNDS RENÉ VAN EGMOND Delft University of Technology, Delft, The Netherlands

1. WHETHER TO BE SILENT In our daily life we are confronted with the sounds of industrial products. However, people are often not aware that they ‘use’ these sounds in a functional or experiential way. Many people use auditory cues ‘unconsciously’. For example, the sound of a water boiler informs one in which phase the boiling process is. Furthermore, people derive not only functional information from the sound, but also information concerning an object’s physical properties. For example, you hear if a big truck is coming towards you even if the truck is far away. This sound is quite different from the sound of an approaching sports car. Moreover, even if the sports car was nearby instead of far away – making the sound louder – the truck still ‘sounds’ bigger. This indicates that people estimate the size of objects on cues different from loudness. Studies have shown that people use sound to estimate the size and the shape of objects (Carello, Anderson and Kunkler-Peck, 1998; Grassi, 2005; Kunkler-Peck and Turvey, 2000), to estimate the size and speed of rolling balls (Houben, Kohlrausch and Hermes, 2004; 2005), and to identify material properties (Giordano and McAdams, 2006; Hermes, 1998). In addition, sound has been used in estimating the quality of engines of vacuum cleaners (Benko et al., 2004; 2005). These examples show that sound is an important carrier of information that people use in interaction with the environment and objects. Research in the field of product sound design has often dealt with the issue of how to make products silent (e.g. Park, Lee and Lee, 2004; Rittmueller, Mann and Holger, 1997). In my view to silence products is not our main goal (although sometimes very important), but the goal is to enhance the experience of a product with proper sound design. Silence should be one of the means available to a sound designer, and will Product Experience Copyright © 2008 Elsevier Ltd.

69

70

Product Experience

be dependent on the type of task a product will be used for. For example, it is wellknown that background noise that exceeds 40 dB(a) limits the concentration of people (Lindeman and Stekelenburg, 1981, p. 114). Therefore, one should consider this aspect when designing a product for office use. The fans of personal computers that emit noise at such a loudness level that they have been compared with jet engines are notorious. Or the noise certain ink jet printers make, caused by the transport of the paper and the movement of the ink cartridge. The former example is, of course, worse because personal computers make the noise during the entire day, whereas in the latter example one chooses the time for printing. Silence can also – and probably should – be used in the design of alarm sounds for hospital situations. Personnel often switch off these sounds because they are too loud and too irritating (Chambrin, 2001; Edworthy and Hards, 1999; Friesdorf, Buss and Gobel, 1999; Solsona et al., 2001). Alarms go off when the value of a measurement device breaches a certain threshold (e.g. if the heart rate suddenly drops). However, often only the value of the threshold has been implemented, and not a duration that would indicate how long a certain threshold can be breached without becoming too dangerous. In other words, there is no time-dependency. Consequently, alarms continue to go off after the threshold has been breached for only a short time, even if the situation is no longer critical. The experience of alarm sounds would improve considerably if these sounds did not depend on these short temporal breaches of thresholds. The term sound quality has often been used in relation to reducing noise or in relation to psychoacoustic measures as, for example, sharpness and roughness (e.g. Brandl, Biermayer and Pfluger, 1999; Ih et al., 2003; Keiper, 1997; Lyon, 2000). However, recent discussions suggest that, dependent on the definition of quality, other aspects – e.g. the association of meaning – should be introduced (e.g. Blauert and Jekosch, 1997; Jekosch, 2004). In this chapter, the term product sound experience will be used, because it is a more general concept (related to the emotional, semantic, and sensorial experience), and because the term is more neutral. It also is a term that is related to the human aspect and not to how the sound can be described in terms of spectral composition or signalto-noise ratios. The latter sound features may evoke or be related to a certain experience and are therefore important to understand, but a one-to-one mapping of certain features to an experience will be too simplistic. In the next paragraph the domain of product sounds will be defined.

2. THE DOMAIN OF PRODUCT SOUNDS The domain of product sounds can be considered as a subdomain of environmental sounds (e.g. Ballas, 1993, 2002; Ballas and Howard, 1987; Björk, 1985; Gaver, 1993a, 1993b; Kidd and Watson, 2003; Marcell et al., 2000). A number of studies investigated memory, recognition, identification and the descriptors of environmental sounds. Environmental sounds comprise the sounds that we hear during our daily life, or – in other words – the sounds that surround us. This means that environmental sounds comprise a diversity of sounds, e.g. car sounds, birds whistling, water sloshing, thunder, horns, bells, dogs barking, etc. Of course, these sounds are often culture or country dependent. In the desert, one will perceive different sounds than in the streets of New York. For example, Ballas (1993) has asked people to keep a diary of the sounds they heard. The sound of highway traffic occurred quite often. There will be many places in the world where this sound is not heard, indicating the importance of contextual information.

The experience of product sounds

71

Although sounds of products are with us during our daily life, a perceptual domain of product sounds has not been established. We define the domain of product sounds as being the sounds that are produced by industrial products, cars, and user interfaces. Among product sounds, we distinguish between consequential and intentional sounds. Consequential sounds (mechanical sounds, Västfjäll et al., 2002) are a consequence of the moving parts of products and are affected by the interaction when a person uses a product. For example, a toothbrush sound will be generated by the engine (its rpm), the gear, the transmission, and the pressure and movement applied by the user. Intentional sounds are deliberately added to a product. These sounds are synthesized or recorded. They are radiated through a loudspeaker or by a vibrating surface. User interface and alarm sounds belong to this category. The sounds are often more musical or speech-like than the consequential sounds. Consequently, the design and the experience of intentional sounds will differ from consequential sounds. Before describing the specific properties of the consequential and intentional sounds, a short introduction of the spectral and temporal structure of sound will follow in the next paragraph.

3. SPECTRAL AND TEMPORAL STRUCTURE OF SOUNDS Three perceptual domains of sound can be distinguished: Speech sounds, musical sounds, and environmental sounds (sometimes called everyday sounds). Certain sounds may be classified in more than one domain. In these cases, the context in which the sound is heard will be the determinant factor to relate it to a specific domain. For example, singing produces speech sounds (especially if one considers the Sprechstimme (‘speech-voice’)), but this does not warrant to state that every speech sound is a musical sound. Each domain has its own characteristic features. Sounds are described by an amplitude fluctuation as a function of time in the temporal domain and as a sum of sinusoids (overtones) in the spectral domain. Each event in the temporal domain can be mathematically transformed (i.e. Fourier transformation) into the spectral domain and vice versa. A Fourier-transform of a time-signal results in a series of sinusoids (with their relative amplitude and phases). This sum of sinusoids determines the spectral composition of the sounds. Natural sounds – and most synthetic sounds – are composed from several sinusoids called the overtone series. The number of overtones, their frequencies and amplitude fluctuations determine the sound color or timbre. Vowels and consonants can be considered as the basic constituents of speech. Vowels are recognized by their timbre characteristics. Formants are specific areas in the speech spectrum. The relationship between formants is an important aspect in the recognition of vowels. Consonants are noise or click-like sounds. Although speech has been applied in user interfaces, it has mostly been used to inform a user. Some interfaces have been developed to recognize a user’s speech commands, but they often fail. One of the reasons is that the systems have to be trained to recognize specific voices. Human speech recognition not only analyzes the sound, but in addition applies top-down knowledge on grammar, lexicon, and uses the meaning of the produced sentences. One fills in gaps that have not been perceived in the speech sound. In this chapter we will not consider the perception or design of speech in user interfaces. Two categories of musical instruments can be distinguished: (a) pitch-producing (e.g. violin, guitar, piano, etc.); and (b) ‘noise’ producing (e.g. drums, wood-blocks). These categories are not that strict. For example, timpani are often tuned in fourths, thus pitch can be attributed to these ‘noise-like’ sounds. Most sounds of pitch-producing instruments consist of overtones that are proportionally related by the harmonic series (1:2:3:4:5:6:7:8, etc.), or have small deviations from the harmonic series (1:2:3.01:4.02,

72

Product Experience

etc.). The deviations from the harmonic series introduce inharmonicity. This inharmonicity has a physical cause, e.g. the stiffness in metal strings. A sound with a fundamental of 100 Hz would consist of a series of harmonics of: 100 Hz, 200 Hz, 300 Hz, 400 Hz, 500 Hz, 600 Hz, etc. This is the ‘ideal’ sound that hardly exists. It is interesting to note that the harmonic series contains the intervals of the Western tonal system. The proportion 1:2 is the octave, 2:3, the fifth, 3:4, the fourth, 4:5 the major third, and 5:6 the minor third. In terms of pitches, if the fundamental tone is C then the series of overtones (until the 6th harmonic) will be the following: C, C, G, C, E, G. This series is often considered as the underlying principle of Western tonal harmony (see, e.g. Parncutt, 1989). The number of overtones between the harmonics determines how noiselike a sound is (the amount of tonalness decreases). This can be captured by the harmonicsto-noise ratio (see, e.g. Boersema and Weenink, 2006). It is also worthwhile to note that the onset (start) of a musical tone often introduces noise-like elements that are important in people’s judgment on the naturalness and quality of a sound (e.g. Fletcher, 1962). Musical sounds – and even melodies – are considered to be product sounds if they have been implemented in industrial products. For example, sounds in user interfaces, alarm sounds, and start-up sounds in computers are considered product sounds.

4. PRODUCT SOUNDS As mentioned before, product sounds are a perceptual subdomain of environmental sounds and also comprise the use of musical sounds (musical instruments or synthesized sounds). In most instances, synthetic sounds are implemented in the way they have been designed; they will not change due to product–user interaction. As a result the sound will be the same every time it is triggered. Although this simplifies the design of these sounds, the experience of the sound will be affected. The reason for this is that a perfectly designed sound will evoke a desired experience. However, in this case the experience of the sound will also depend on the repetition by which it is heard. A repeated sound that remains perceptually equal will come to be experienced more negatively. This appears to be in conflict with the mere exposure effect reported by Zajonc (1968). However, if one considers the nature of user interface sounds then the following explanation may be given. User interface sounds are often short, of a simple structure, and are repeated many times. In a meta-analysis of the relevant literature, Bornstein (1989) found that the mere exposure effect is stronger for complex stimuli, for multiple stimuli, and for brief stimuli that are not repeated that often. Most user interface sounds have just the opposite features and may therefore be disliked with repetition. Consequently, a sound designer should consider introducing small deviations in the spectral-temporal structure of the sounds. These fluctuations will result in minor differences between repetitions, but nonetheless all sounds will be perceived as similar to the original sound. The similarity will evoke the familiar experiential properties, whereas the deviations will avoid boredom or irritation. In the next paragraph, the spectral and temporal structure of product sounds will first be discussed without the influence user actions have on the sound. Then the effect of the user on the sound production will be discussed, and its consequences on the sound properties.

4.1. Spectral and temporal structure of product sounds The amount of structure in both the temporal and spectral domain of a product sound determines how well a sound can be recognized and remembered (Özcan and

The experience of product sounds

73

Van Egmond, 2007). Sounds with no-structure in the spectral domain will sound very noise-like. In this case, the sound will be hard to manipulate to achieve a proper experience. The lack of structure in the temporal domain (e.g. absence of a strong perceived periodicity in sounds with an engine or a hierarchical rhythmic structure in alarm sounds) will make it harder to encode a sound. If a sound is highly structured in both domains recognition and memory are high, whereas for sounds that are unstructured in both domains recognition memory will be low. Özcan and van Egmond (2007) distinguished three levels of structure in the spectral domain and three levels of structure in the temporal domain. In the spectral domain, a highly structured sound consists of an (harmonic) overtone series, and therefore will sound tonal. These sounds almost do not contain noise (or noise that is relatively low in intensity compared to the overtones). Medium structured sounds in the spectral domain stem from products that contain an engine with a certain rpm (revolutions per minute). The rpm will result in a perceivable periodicity in the sound that generates an (harmonic) overtone series. However, these sounds consist of a considerably large amount of noise caused by the moving of, for example, gears. In other words, the intensity of the noise is relatively high compared to the intensity of the overtones generated by the periodicity. Sounds that only consist of noise or (rapidly) changing spectral structure over time are the least structured. Not only will the memory and recognition decrease with the amount of structure, but the possibility of designing a sound experience for a product will be much more difficult if structure is lacking. We have also distinguished three levels of temporal structure for product sounds. Product sounds that are highly structured in the temporal domain will have a distinctive (hierarchical) rhythm-like structure, and will therefore be more easily remembered (see Povel and Essens, 1985). An example is the sound of an alarm clock with a distinctive rhythm. Products with engines that run at a certain rpm will have a cyclical temporal structure. These sounds are considered medium structured. The rpm causes a periodicity – one experience the cyclic temporal structure – but this periodicity produces continuous events, not discrete events in the temporal domain. In contrast to discrete events, continuous (cyclic) events in the temporal domain will not evoke a strong rhythmic pattern. Products that operate with airflow or water-flow will produce sounds with the least recognizable temporal structure. The aforementioned descriptions of the spectral and temporal structure of product sounds are without the influence of user actions. In other words, these are products that run stationary. The user actions will influence the spectral–temporal structure of a product sound and will be determined by the type of control system. Three control systems can be distinguished: Autonomous; partly autonomous; and user-dependent control systems. The comparability of the sounds produced by similar events decrease in the following way: Autonomous; partly autonomous; and user-dependent. The comparability of the sounds of similar events will affect the possibility of making proper predictions concerning experiential aspects. Products that have autonomous control systems (e.g. central heating systems) will produce sounds that are very similar (equal) if events are repeated. In fact, minor fluctuations in the sound will be more dependent on the external factors of the environmental context than that a user action will influence the sound. Consequently, the experience of the sound will be highly predictable. For example, a central heating system will be dependent on the temperature of the surrounding context that influences the time of heating and the temperature of the water that needs to be heated. The spectral–temporal composition of the sound of a product with a partly autonomous control system will be dependent on user actions. However, the deviations in the

74

Product Experience

spectral–temporal composition will be small. Consequently, the experience of the sound will still be predictable within small perceptual boundaries. For example, the sound event of a drip filter coffeemaker will be dependent on the amount of coffee it has to brew, how the filter is placed, etc. However, when the coffeemaker is switched on it will run autonomously. The spectral–temporal composition of products with user-dependent control systems can vary considerably. Two manifestations of sound can be discerned: (i) The stationary sound (i.e. the sound a product generates when it is switched on but not used); (ii) The sound when the product is in use. The experience of the sound of these products in use and when they are stationary differ. For a company this may mean that if they want to design the shelf-presence of a product sound then they should consider the experience of the sound when the product is switched on without any user action. For example, a toothbrush can often only be switched on and cannot actually be used in a store. The design of the stationary sound is easier than the design of a sound that pertubates perceptually under the influence of usage. There are two main reasons for this. First, the spectral composition of the sound will change when the product is used (e.g. by increasing the amount of pressure on a toothbrush). Differences in spectral compositions may evoke different experiences. Consequently, it will be more difficult to design the preferred sound experience. Secondly, user actions are not systematic and will generate random fluctuations that are harder to predict, and therefore more difficult to incorporate in the sound design. Three examples of product sounds will be given. In Figure 3.1 the spectrogram of an alarm clock sound is presented. In the spectrogram frequency as a function of time is represented with grey shading to indicate the intensity of the frequencies. It can be seen that a tone with duration of approximately 350 ms is repeated for four times (i.e. discrete events). The pitch of this sound is approximately 890 Hz, it can be seen that there is a strong emphasis for the sixth overtone (around 5300 Hz, darkest shading). In addition,

Figure 3.1 Spectrogram of an alarm clock sound. Time has been represented on the x-axis. Frequency has been represented on the y-axis. Intensity of overtones has been indicated with shades of gray.

The experience of product sounds

75

the ninth through the fourteenth overtones also have a relatively strong intensity. This indicates that the sound will be perceived as somewhat sharp, because the sensation of sharpness depends on the amount of energy in the higher frequencies. Consequently, the sound may be perceived as relatively unpleasant. This is, of course, a desirable experiential property for an alarm sound if one needs to wake up. In Figure 3.2 the spectrogram of the sound of an electric toothbrush is presented. It can be seen that there are no discrete events; the sound events are continuous. It can also be seen that the sound contains noise, because hardly any frequency bands can be distinguished. Nevertheless, in Figure 3.2 some darker frequency bands can still be distinguished, caused by the periodicity of the engine. The user involvement is reflected in this figure, especially if one considers the dark band in the spectrogram around 2500 Hz. This band fluctuates over time due to the pressure the user applies to the brush. With an increase of pressure the frequency decreases, whereas if the pressure decreases the frequency increases. One can hear this in the perceived pitch of the sound. This demonstrates that there is a certain bandwidth in which the spectral content of the sound fluctuates. Consequently, the experience of the sound may be dependent on the amount and the strength of these fluctuations. In a pilot study, we investigated the effect of pressure on the preference and experience of a toothbrush sound. In this study, the pressure was systematically varied by putting weights on the head of the toothbrush. Thirty participants were asked to indicate which sound they preferred the most using a paired comparison paradigm. Preference was negatively associated with the amount of pressure (r
.92). In other words, if the pressure increases then the preference decreases. Zampini, Guest and Spence (2003) have investigated the effect of a toothbrush sound on the experience of the vibrotactile stimulation on teeth. They have found that if the loudness level or the intensity of the higher frequencies decreases the ‘tactile’ feeling of roughness and unpleasantness of the toothbrush also decreases.

Figure 3.2 Spectrogram of the sound of an electric toothbrush. Time has been represented on the x-axis. Frequency has been represented on the y-axis. Intensity of overtones is indicated with shades of gray.

76

Product Experience

Figure 3.3 Spectrogram of the sound of a vacuum cleaner. Time has been represented on the x-axis. Frequency has been represented on the y-axis. Intensity of overtones has been indicated with shades of gray.

In Figure 3.3 the spectrogram of the sound of a vacuum cleaner is shown. It can be seen that there is even less structure in the frequency domain. In addition, the energy of the sound appears to be more concentrated in the lower frequency range, whereas for the toothbrush the lower frequencies do not contain that much energy and the higher frequencies are relatively high in energy (note that the sounds have all been normalized for intensity to the same sound pressure level). In Figure 3.3 vertical lines have been drawn to identify the areas in the spectrogram in which changes occur. In these areas, the dark bands in the spectrogram are interrupted and the amount of energy for the higher frequencies increases (this area becomes darker). This change results from the blocking of the nozzle of the vacuum cleaner when the hose is moved back and forth. In conclusion, consequential product sounds are often noisy. This noise (random structure) complicates the design and predictability of these sounds. The spectral– temporal composition of the sound is not only dependent on how the product has been constructed, but in addition is dependent on the usage actions (e.g. pressure). The structural aspects of intentional sounds (e.g. feedback or alarm sounds) can be designed, and thus the experience of these sounds is largely under control (not withstanding that contextual aspects may also influence these sounds). In addition, the improved technology nowadays allows the generation of high quality (CD standard) sounds. However, in many cases the design of the sounds has not been explicitly based on the type of interaction the designer wanted. In comparison with consequential sounds there is so much design freedom (e.g. timbre, pitch, inter-onset intervals) that it becomes difficult to make choices. From ecological acoustics (e.g. Gaver, 1993a,b) it is known that sounds often convey a meaning. That is, one can hear the type of source, the location of the sound, the context in which the sound is present, and whether the source of the sound is dangerous. Therefore, it is important that sounds in user interfaces are designed to convey a specific meaning. Alarm sounds in intensive care units are a ‘good’ example of improper – or not well-thought-through – sound designs.

The experience of product sounds

77

Figure 3.4 Two spectrograms of alarm sounds of a device that controls a patient’s breathing from the same manufacturer.

The use of alarm sounds in intensive care units generates many problems (Edworthy, Loxley and Dennis, 1991; Edworthy and Meredith, 1994; Welch, 1999; Yan Xiao and Seagull, 2003). Manufacturers often implement alarm sounds because they want to reduce the probability of liability law-suits (Edworthy and Adams, 1996). Because alarms go off unnecessarily and too often, staff in intensive care units often turn the alarms off. Because they are occupied with coping with their annoyance, they forget to attend to the actual problem (e.g. Block, Nuutinen and Ballast, 1999; Edworthy and Adams, 1996; Momtahan, Hetu and Tanssley, 1993). The large number of sounds, the lack of conveying a well-defined meaning (e.g. type of device, the level of urgency), and the masking effects by other alarms and noise-producing sources make alarm sounds very difficult to remember and to recognize (Edworthy and Adams, 1996; Momtahan, Hetu and Tanssley, 1993; Welch, 1999). Often alarms that need to convey a certain meaning (e.g. cardiac arrest) may differ between devices. This finding is not only true for devices of different manufacturers, but also for devices from the same manufacturer. For example, we recorded the sound of a device that controls the breathing of a patient. If the pump stops then this alarm sounds. In Figure 3.4 spectrograms of sounds are shown that stem from two versions of this device from the same manufacturer. It can be readily seen that the sounds differ in their spectral composition. In addition, the spectrogram depicted on the left consists of the same tones that are repeated. The tones have a duration of 250 ms followed by a pause of 250 ms. The spectrogram depicted on the right consists of five tones. The arrows indicate at what point the new tone starts. It can be seen that there is a small overlap of the tones (the next tone starts while the previous tone is still sounding). Furthermore, the temporal structure of the second sound is more rhythmic than of the first sound. This rhythm can be characterized as: short-short-longshort-long. After the last tone has sounded, a pause starts in which the decay of the last sound can still be heard. After 3 seconds this rhythm is repeated again. It can be concluded that the left and the right sound do not contain any similarity. Consequently, staff of the intensive care unit will have to remember two different sounds that correspond to the same cause. This example demonstrates that a lot can still be improved in the consistent design of alarm sounds.

5. PROCESS OF AUDITORY PERCEPTION Several theories and models exist that describe the process of auditory perception or specific aspects in this process (e.g. Bregman, 1994; Handel, 1991; McAdams, 1993; Yost, 2007;

78

Product Experience

Fastl and Zwicker, 2007). I have been developing a conceptual framework for the process of auditory perception that guided my thinking and the set up of research projects. In this conceptual framework, I make a distinction between experiential aspects of sounds that stem from a bottom-up process and experiential aspects of sounds that stem from top-down processing (or ‘knowledge-rule based aspects’). This distinction is coarse, and I am aware of the many subtleties that have been reported in the literature concerning the interaction of bottom-up and top-down processing of sound. However, the distinction is of importance in the experience of product sounds because it allows pinpointing certain effects as a result of bottom-up processing which are salient and unaffected by prior knowledge. Figure 3.5 represents a flowchart of the conceptual framework. On top a sound has been symbolically represented in the temporal domain. The sound will be modified by

FIGURE 3.5 Conceptual framework of auditory perception.

The experience of product sounds

79

the outer and middle ear, but the main processing occurs in the inner ear. The middle ear functions as a half-open organ pipe with a resonance frequency between 2 kHz and 5 kHz. This enhances our sensitivity for this frequency range. The sound is processed in the inner ear by the cochlear mechanics and neural activity (see, e.g. von Békésy and Rosenblith, 1951; Yost, 2006). The inner ear (with the basilar membrane as an important constituent) can be considered as a low pass filter. The higher frequencies are mapped at the start of the cochlea, whereas the lower frequencies are mapped at the end of the cochlea. Thus, each region in the inner ear is sensitive for a certain frequency. The inner ear decomposes a sound into its constituent frequencies. In a way, this resembles the mathematical technique of the Fourier transform in which a signal in the temporal domain is transformed into the frequency domain (as a result the constituent frequencies can be determined). The spatial mapping of frequency is projected on the brain through the auditory nerve. This is called a tonotopic representation of the sound. Because of the cochlear composition (fluids and organ of Corti cause integration of single resonances) there are areas of resonance. Critical bands are areas in which the summation of loudness and masking effects take place (see Fletcher, 1940; Fastl and Zwicker, 2007). For example, frequencies within a critical band contribute less to the perceived loudness than frequencies outside a critical band. Furthermore, the understanding of the masking of tones (a tone with a certain intensity can be made inaudible by another tone, noise-band, or noise) made it possible to develop coding techniques like mp3. In addition to loudness, sharpness, harmonicity, and roughness can be explained by our understanding of the functioning of the inner ear. These measures are sensations that contribute to the sensory pleasantness of a sound. Many of the psychoacoustical measures (see for an overview, Allen, 1999; Fastl and Zwicker, 2007; Hirsh and Watson, 1996; Moore, 2003; Yost, 2007) have been implemented in algorithms for sharpness, harmonicity, loudness, roughness, and tonalness. Aures (1985) and Fastl and Zwicker (2007, p. 245) proposed models of sensory pleasantness on the basis of these values. The models differ in the weighing of the factors. An increase in the value of loudness, sharpness, and roughness all result in a decrease of sensory pleasantness. An increase of tonalness (named tonality by Fastl and Zwicker, 2007, but this is to my opinion too much related to the musical concept of tonality and causes confusion) results in an increase of sensory pleasantness. Tonalness can best be described as the amount of harmonic structure in a sound in relation to the amount of noise. Because psychoacoustic experiments demand a large amount of experimental control, the sounds employed to determine these effects are often synthesized (e.g. frequency modulation, amplitude modulation, noise bands). As mentioned before, psychoacoustical measures are often used because these measures can be implemented in algorithms (see Keiper, 1999). These implemented psychoacoustics measures will be referred to as psychoacoustic metrics. Lim (2001) has used psychoacoustic metrics to predict the quality of the sound of power windows. Västfjäll et al. (2002) have found that these metrics do not predict whether a sound is related to the pleasantness and activation dimensions of emotional experience. In a graduation project (Bouter, 2006) we have investigated if the psychoacoustic metrics of roughness, loudness, sharpness, and harmonicity (an implementation for tonalness) are associated with participant responses on self-report scales of roughness, loudness, sharpness, and harmonicity for product sounds. The metrics of loudness, sharpness and harmonicity were highly correlated with the constructs (r
.82, r
.97, r
 .89, respectively), whereas the correlation between the roughness metric and the corresponding construct was not that high (r
.47).

80

Product Experience

These auditory sensations are important in our understanding of the experience of product sounds because they are very difficult to suppress. In other words, if these factors are perceptually salient their effect is hard to change by an attribution of meaning. Top-down influences (the next step in the model) may influence but will not completely override these primary sensations. Thus, a proper sound design should always control for these primary sensations. Consequently, if these primary sensations are under control then top-down influences may contribute more to the experience of the sound. This will eventually lead to a richer sound experience. In the last stage of the conceptual framework, information from memory influences the processing of a sound. This information may consist of the attribution of meaning, but it can also consist of more abstract structural aspects that describe a sound (e.g. the increase and decrease pattern in pitch). For example, if one of the alarm sounds of Figure 3.4 is coded in memory, then several aspects are represented. First, the timbre of the sound (audition-specific image) should be coded. Secondly, the temporal structure (the rhythmic pattern) will be represented. Thirdly, the name of the medical device that produces the sound is represented. Fourthly, the type of medical failure and the level of urgency associated with the sound will be represented (attribution of meaning). Fifthly, the context in which the sound is heard (intensive care unit) is important. When the sound is heard and encoded on a subsequent occasion, these aspects (if there is recognition) will be activated and will be combined with the sensory sense of urgency (caused by the tempo, the repetition) that will evoke the necessity for a staff member of the intensive care unit to act. Note that the combination of top-down and bottom-up sensations that the sound makes evokes a need to act. If somebody who is not a staff member hears the sound but is not present in the intensive care unit, a sense of urgency may be induced by the rhythmic nature of the sound, but no knowledge is available concerning the actual failure. Therefore, the need to act will not be induced. Note that there is a difference between evaluating a sound and experiencing an emotion while listening to the sound. Nevertheless, Västfjäll et al. (2002) have demonstrated that affective reactions and affective evaluations are associated. Thus, although this distinction is important from a theoretical point of view, it may be difficult to measure experimentally. Possibly the context and the attribution of meaning are important to induce a difference in experience and in evaluation. However, these aspects have not been controlled in the study of Västfjäll et al. (2002). Thus far, the sensation-driven and more cognitively induced experiences have been discussed. However, is it possible to enhance our understanding of how these aspects interact and what type of emotions they evoke? In addition, which aspects in product sounds contribute to the sensations and which contribute to the cognitive aspects? In other words, can we gain more insight into where in the process of auditory perception certain perceptions or emotions occur? Several studies have shed light on these issues. For example, for the recognition of the rhythmic structure a mental frame of reference needs to be induced (see e.g. Povel and Essens, 1982), suggesting that it involves topdown knowledge. On the other hand, the perception of the tempo will be more bottomup driven. Many studies have investigated affective reactions to acoustic stimuli using rating scales (Bradley and Lang, 2000; Edworthy, Hellier and Hards, 1995; Meunier, Habault and Canevet, 2001; Solomon, 1959; Von Bismarck, 1974a,b). Rating scales have been successfully employed in many studies to obtain emotional responses and convey the intensity of the evoked emotion. However, they do not convey where the evaluation took place in the perceptual process. Thus, these scales cannot be used to differentiate between basic and more cognitive emotions. Basic emotions are the outcome of a simple appraisal.

The experience of product sounds

81

They are needed for survival purposes and are often directly related to an action (e.g. Izard, 1977; Plutchik, 1980). For example, the emotion ‘fear’ that is the outcome of an appraised ‘physical threat’ will result in the action to flee. More complex, or evolved, appraisals evoke ‘cognitive emotions’ (see Ortony, Clore and Collins, 1988). An example of a cognitive emotion is inspiration, which requires an appraised ‘mental illumination’ (see Desmet, 2002). This knowledge is essential if one wants to design a product sound, because it will determine which features in the sound to manipulate. For example, to evoke basic emotions manipulating aspects like, for example, roughness or sharpness will be important, whereas for the more cognitive emotions attribution of meaning will be more important. It is hypothesized that the distinction between basic and cognitive emotions results from a difference in processing. The more cognitive emotions will need more processing time because more information (e.g. structural aspects, association of meaning) from memory has to be accessed. It is hypothesized that the basic emotions will rely more on the sensations discussed before. If the two emotion types are measured using verbal selfreports, the meaning of these emotion words needs to be accessed in memory. In addition, the bottom-up processing of the sounds (i.e. the sensations) will be independent of the type of emotion. Thus, if a sound is evaluated on a basic emotion, then long-term memory is only accessed once. However, in order to evaluate the more cognitive emotions, longterm memory in the conceptual framework needs to be accessed twice: Once to access the meaning of the words, and once to access the structural aspects, associations of meaning of the sounds. Consequently, the processing time will take longer. Van Egmond, Desmet and Van der Helm (2004) have tested this hypothesis by using different sounds and letting listeners evaluate these on ‘basic’ and more ‘cognitive’ emotions. Rating values and response times have been collected in this study. Responses to the ‘basic’ emotions were significantly faster than responses to the ‘cognitive’ emotions. This study shows that response times can be successfully used to differentiate between emotion types.

6. DESIGNING THE EXPERIENCE OF CONSEQUENTIAL PRODUCT SOUNDS Just as one designs visual concepts for products and tests the experience of these concepts, there is a need to design auditory concepts that a ‘future’ product will produce. Several studies have investigated perceptual attributes for sound design and have demonstrated the usefulness of sound design for product sounds (e.g. Bowen and Lyon, 2003; Collins, 2001; Jordan and Engelen, 1999; Louwers, 1993; Lyon, 2000, 2003). Consequential sounds are a result of how an appliance has been constructed. Thus, the sound is dependent on, for example, the type of engine, the type of material, and the gears. Therefore, if a designer wants to create a consequential sound in the conceptual phase it will be a difficult task, because the sound needs to be realistic without the product being there. For example, if one wants to design the sound of an electric toothbrush one needs to model or to mimic the new product. This means that knowledge of the construction, the engine, the gears and the material has to be implemented in order to create a new sound. Gaver (1986) has suggested that sounds in user interfaces (i.e. intentional sounds) should afford certain actions. Therefore, these sounds should be caricatures of natural sounds (named auditory icons) because this will facilitate conveying the meaning of a sound. Gaver (1993a,b) elaborated on these ideas more extensively. He proposes not only involving the perceptual and spectral aspects in our understanding of environmental sounds, but also understanding the physical processes and mechanisms that cause the

82

Product Experience

sounds to sound as they sound. Furthermore, he has identified that the type of interaction of an object with the environment is important in our understanding of the produced sound. Therefore, several interaction types have been discerned (scratching, scraping, bouncing, etc.). Lyon (2000) has modeled several vibrating systems and has related them to perceptual judgment in order to obtain predictions on the sound quality of products. The question still remains: How realistic are these synthesized sounds? In a number of studies and projects for industry we have proposed a design method for producing sounds on the basis of recordings and manipulations of recorded parts of product sounds, and we proposed to make this part of the design process (Özcan and Van Egmond, 2006; Van Balken, 2001; Van Egmond, 2006). This methodology consists of four stages. In the first stage the sounds are recorded. Secondly, the sounds are analyzed (spectral-temporal, psychoacoustical, and perceptually). Thirdly, a realistic sound design concept is proposed based on preferred experience, and the sounds are designed. In the fourth stage, the sounds will be tested with respect to how successful the design is in relation to the required sound experience. These stages can be recursively applied if the sound experience is not as desired. In the next paragraphs the different stages will be discussed.

6.1. Recording One of the most important aspects in the recording of sound is the choice and the placement of a microphone, or the use of even more than one microphone. In music recordings, knowledge is available on how to record different instruments. This has resulted in standards of which microphones to use and where to place them (see Dickreiter, 1984). However, this knowledge is not easily applicable to the recording of product sounds. Although one can learn from the way musical instruments are recorded (e.g. where to place a microphone for a wind instrument or where to put the microphone for a relatively large instrument), the placement and the choice of microphones for the recording of product sounds is often a combination of common sense and trial-and-error. In addition, the sound level needs to be measured in order to know the loudness of sound. A standard calibrated sound level meter is appropriate to measure the sound before recording it (e.g. Bruel and Kjaer Type 2260). As mentioned before, usage actions may influence the sound of an appliance. Therefore, the sound should be recorded with, and without, user involvement. In addition, the separate sound-producing parts of the appliance will be recorded separately. This allows an analysis of the separate sound sources that will enable one to determine which aspect (or part) contributes to a certain wanted or unwanted experience. Other candidates for sources (replacement) are also recorded. These sources should offer the same functionality but achieve this functionality in another way. For example, one could use a boiler to warm water, but also a heating element. These will all be used in the design phase to generate sound concepts.

6.2. Analysis In the second stage, several methods of analysis will be conducted on the sounds using implemented algorithms, in addition to an analysis by the investigator’s or designer’s ear. The spectral–temporal composition of the sound will be determined and will be related to the sound-producing parts. This process will be exemplified with the analysis and design of a coffeemaker sound (Van Balken, 2001). In Figure 3.6 the amplitude of the signal in the time domain (upper graph) and the spectrogram (lower graph) of the recording of a coffeemaker have been depicted. The amplitude of the time signal has been normalized. It can be seen that the entire process after the coffeemaker has been switched on takes

The experience of product sounds

83

Figure 3.6 The sound of a coffeemaker represented in a time–amplitude and in a time–frequency/intensity domain (Van Balken, 2001).

approximately 100 seconds. In the graph specific events and sound producing parts have been indicated. Because the valve ticking is a very characteristic sound, this part has been magnified in the amplitude-time graph (the small click can thus be distinguished). After approximately 75 seconds the water has reached the correct temperature (approximately 95ºC) and a decay of the amplitude can be seen. Five seconds later, the pump starts and forces the water through the valve and the coffee pad. The pump sound is relatively high in intensity and will therefore evoke a sense of unpleasantness. From 85 seconds through 100 seconds the coffee flows into the cup, after which the pump stops. An event that is noticeable is the squeak that occurs approximately 2 seconds after the pump starts. It can be seen in the spectrogram (bottom graph) that this squeak contains many high frequencies of a relatively high intensity. This means that the sound will be experienced as very sharp and unpleasant. In Table 3.1 the sound events are represented as a function of the sound producing parts and use stages. It can be seen that there are nine distinctive sound events, and that most events are independent of user actions (only the opening of the lid is user dependent). In addition, most of the events occur during the brewing process. This brewing process is also the period in which multiple sources contribute to the sound (see also Figure 3.6). The next step in the analysis phase will be to determine the psychoacoustical metrics (e.g. roughness, sharpness, loudness) previously described for each of the identified events. Because sound varies over time and many different events occur, it is important

84

Product Experience

TABLE 3.1 Characteristic cofeemaker sounds with related product parts and use stages Sound

Sound producing product parts

Use stage

Squeaking Sucking Rustling Ticking Whirring Squeaking Rustling Coffee flow Rustling

Metal opening system Air escaping valve for the reservoir Boiler Valves between boiler and spout Pump Air driven through the hole in pad holder Brew chamber Spout Boiler

Opening lid Placing the reservoir Heating Heating Brewing Brewing Brewing Brewing Post-brewing

Figure 3.7 Top figure: Aggregated pleasantness ratings as a function of time (in seconds). Bottom figure: The sound scenario of Figure 3.6 (Van Balken, 2001).

to know which of the events results in a – for example – unpleasant experience. In order to relate sound events to a judgment of pleasantness, an online measurement task is used. In this task a listener is asked to indicate the experience of the sound (e.g. how pleasant) while listening to it. A slider device consisting of a linear potentiometer is employed to determine the rating value and the time on which the rating occurred. A computer samples the responses on the slider with a certain frequency that depends on the needed temporal resolution. In this way the rating data can be mapped onto the sound. With this device fluctuations in people’s judgment can be tracked, and the boundaries can be determined at which a judgment changes. In Figure 3.7 an example of this type of response has been represented. In the bottom part of this figure an abstract representation of the sound events as they can be seen in Figure 3.6 is given (note that we shortened the duration of the first part of the heating process in the recording). In the top figure the aggregated number of times (over listeners) people scored in one of the five categories (pleasant, fairly pleasant, neutral, fairly unpleasant, unpleasant) has been presented as a function of time. It can be readily observed by comparing the top and bottom graph that the heating process has been judged overall fairly pleasant, whereas as soon as the pump starts the judgment becomes unpleasant. Note that because the squeak is very short in relation to people’s behavioral reaction of moving

The experience of product sounds

85

Figure 3.8 The sound events of a redesigned sound of a coffeemaker (Van Balken, 2001).

the slider, this event cannot be found in the pleasantness ratings. However, additional interviews have indicated that this has been perceived as a very unpleasant event. In the next stage the results of the first two stages will be used to come up with a different sound scenario and a concept for a sound design.

6.3. Concept and sound design phase After the problems with the original sound have been determined, the sound needs to be improved. In this phase several scenarios and sound design concepts are created that should evoke the desired experience. The experience will be determined by the usecues one wants to convey (e.g. when does something start or stop) and the experiential aspects (sometimes based on a company’s brand values). In Figure 3.8 a new sound scenario for the coffeemaker is presented based on the experience of quality, warmness, coziness, and hominess. The sound events have been presented as a function of time. It can be seen that the duration is shorter than the duration of the scenario presented at the bottom of Figure 3.7 of the original sound. It is envisioned that this should evoke a sense of quality because devices are considered to be of higher quality if they are faster than similar devices (comparable to the acceleration rate of cars). In addition, the heating starts and then stops and starts again. This introduces a kind of wave-like feeling in the sound that can be considered cozier because it resembles the breaking waves on a sea-shore. The heating also continues when the pump starts, removing the small silent period in the original sound just before the pump starts. This masks the start of the pump and creates a more natural flow. In the original sound the silence before the brewing and the pump start possibly evokes a suspenseful experience that enhances the negative effect induced by the pump. Because the pump and the squeak were the main contributors to the most unpleasant reactions to the sound (see Figure 3.7), the squeak has been removed and the pump runs more regularly and starts up more gently. The new sound is synthesized using recordings of sound-producing parts, other parts than those present in the current appliance, sound filtering techniques to simulate different damping or resonant aspects, and some physical modeling algorithms (for example, to simulate air flow). It is essential at this stage to synthesize a realistic sound concept of which the constituent parts can be used in the construction of a new appliance.

6.4. Evaluation In the evaluation phase, listeners will judge the sounds in two ways. First, the sounds are evaluated using the aforementioned online measuring technique. Because this technique takes a long time, only a small selection of attributes will be used. Secondly, after

86

Product Experience

a listener has completely listened to a sound, it is evaluated by rating a larger list of attributes. In this way, the holistic experience of the sounds is measured. The attributes will reflect the desired experience. Together these evaluations will determine if the new sound design concepts fulfill the required experience. If not, then new concepts will be developed and the cycle starts all over again.

7. CONCLUSION This chapter presents an overview of the existing literature that is relevant for the understanding of the experience of product sounds and their design. It also contains a positioning to clarify what ‘good’ sound design can mean in the experience of industrial products. However, several topics have not been tackled. For example, the design of intentional sounds has not been discussed at full length, although the design of these sounds offers designers a lot of freedom. In fact, it should be possible to design these sounds in such a way that they would evoke exactly the preferred experience. In this way, the visual and auditory design could complement or strengthen the experience. Furthermore, in Section 6 the redesign of a sound has been discussed. In other words, the sound of an existing product has been improved to evoke a preferred experience. However, in the conceptual phase of the design process for entirely new products it is much harder to include sound design, because there is a lack of tools and methods. Just like sketching tools for visual design, a tool for sketching a sound would help designers in this conceptual phase. This type of tool would allow the design of a sound on the basis of the desired product experience. In this way, sound design would be an integral part in the development process for new products. Although most people are confronted with the sound of products throughout their daily life, the research into the auditory processing of these sounds, the experience they evoke and how they can be changed is limited. The challenge lies in that one has to try to integrate knowledge from several disciplines (psychology, acoustics, psychophysics, music) and to disclose this in such a way that designers can use it. The current chapter started with the question ‘Whether to be silent’ is the best to develop sounds for products. I hope it has been demonstrated in that there is more to the experience of product sounds than silence.

ACKNOWLEDGMENTS Spectrograms have been made with Praat (Boersma and Weenink, 2006). I would like to thank my colleagues at the Faculty of Industrial Design Engineering at the Delft University of Technology and those in the field of auditory perception for their discussions. Since 2003 I have worked with Elif Özcan and Rolf den Otter, this chapter would not have been possible without their contributions and discussions on this topic.

REFERENCES Allen, J. B. (1999). Psychoacoustics. In: J. G. Webster (Ed.) Wiley encyclopedia of electrical and electronics engineering, pp. 422–437. New York: John Wiley. Aures, W. (1985). Berechnungsverfahren für den sensorischen Wohlklang. Acta Acustica, 59, 130–141. Ballas, J. A. (1993). Common factors in the identification of an assortment of brief everyday sounds. Journal of Experimental Psychology – Human Perception and Performance, 19, 250–267. Ballas, J. A. (2002). What is that sound? Some implications for sound design. Proceedings of Journées Design Sonore. www.sfa.asso.fr/DS2002/procs/ballas.pdf.

The experience of product sounds

87

Ballas, J. A. and Howard, J. H. (1987). Interpreting the language of environmental sounds. Environment And Behavior, 19, 91–114. Benko, U., Petrovcic, J., Juricic, D., Tavcar, J. and Rejec, J. (2005). An approach to fault diagnosis of vacuum cleaner motors based on sound analysis. Mechanical Systems And Signal Processing, 19, 427–445. Benko, U., Petrovcic, J., Juricic, D., Tavcar, J., Rejec, J. and Stefanovska, A. (2004). Fault diagnosis of a vacuum cleaner motor by means of sound analysis. Journal Of Sound And Vibration, 276, 781–806. Björk, E. A. (1985). The perceived quality of natural sounds. Acustica, 57, 185–188. Blauert, J. and Jekosch, U. (1997). Sound-quality evaluation – a multi-layered problem. Acustica, 83, 747–753. Block, F. E., Nuutinen, L. and Ballast, B. (1999). Optimization of alarms: a study on alarm limits, alarm sounds, and false alarms, intended to reduce annoyance. Journal of Clinical Monitoring and Computing, 15, 75–83. Boersma, P. and Weenink, D. (2006). Praat: doing phonetics by computer (Version 4.5.08) [Computer program]. Retrieved December 20, 2006, from http://www.praat.org/ Bornstein, R. F. (1989). Exposure and effect: Overview and meta-analysis of research, 1968–1987. Psychological Bulletin, 106, 265–289. Bouter, P. (2006). The association between the psychoacoustic measurement and subjective measurement of loudness, roughness, sharpness, and noisiness of product sounds. Internal Report, Delft University of Technology. Bowen, D. L. and Lyon, R. H. (2003). Mapping perceptual attributes of sound to product design choices. Noise Control Engineering Journal, 51, 271–279. Bradley, M. M. and Lang, P. J. (2000). Affective reactions to acoustic stimuli. Psychophysiology, 37, 204–215. Brandl, F. K., Biermayer, W. and Pfluger, M. (1999). Objective assessment of vehicle interior noise quality. Sound and Vibration, 33, 20–28. Bregman, A. S. (1994). Auditory scene analysis: the perceptual organization of sound. Boston: The MIT Press. Carello, C., Anderson, K. L. and Kunkler-Peck, A. J. (1998). Perception of object length by sound. Psychological Science, 9, 211–214. Chambrin, M. C. (2001). Alarms in the intensive care unit: how can the number of false alarms be reduced? Critical Care, 5, 184–188. Collins, M. (2001). The sound and the future. The engine design of the Harley-Davidson. I.D. magazine, 52–57. Desmet, P. M. A. (2002). Designing emotions. Doctoral Dissertation, Delft University of Technology, Delft, The Netherlands. De Wit, M. G. M., Verbunt, J. P. M. and Vael, J. E. M. (1993). From noise and vibration to functional sound. Philips Research Bulletin on Sands, 1–4. Dickreiter, M. (1984). Mikrofon-Aufnahmetechnik. Stuttgart: Hirzel Verlag. Edworthy, J. and Meredith, C. S. (1994). Cognitive-psychology and the design of alarm sounds. Medical Engineering and Physics, 16(6), 445–449. Edworthy, J. and Adams, A. (1996). Warning design: a research prospective. London: Taylor and Francis. Edworthy, J. and Hards, R. (1999). Learning auditory warnings: the effects of sound type verbal labeling and imagery on the identification of alarm sounds. International Journal of Industrial Ergonomics, 24, 603–618. Edworthy, J., Hellier, E. and Hards, R. (1995). The semantic associations of acoustic parameters commonly used in the design of auditory information and warning signals. Ergonomics, 38, 2341–2361. Edworthy, J., Loxley, S. and Dennis, I. (1991). Improving auditory warning design – relationship between warning sound parameters and perceived urgency. Human Factors, 33, 205–231. Edworthy, J. and Stanton, N. (1995). A user-centered approach to the design and evaluation of auditory warning signals.1. Methodology. Ergonomics, 38, 2262–2280. Fastl, H. and Zwicker, E. (2007). Psychoacoustics. Facts and models (3rd Ed.) Berlin: Springer Verlag. Fletcher, H. (1940). Auditory patterns. Review of Modern Physics, 12, 47–66. Fletcher, H. (1962). Quality of piano tones. Journal of the Acoustical Society of America, 34, 749–761. Friesdorf, W., Buss, B. and Gobel, M. (1999). Monitoring alarms – the key to patients’ safety in the ICU? Intensive Care Medicine, 25, 1350–1352. Gaver, W. W. (1986). Auditory Icons: Using sound in computer interfaces. Human–Computer Interaction, 2, 167–177. Gaver, W. W. (1993a). How do we hear in the world – Explorations in ecological acoustics. Ecological Psychology, 5, 285–313. Gaver, W. W. (1993b). What in the world do we hear. 1. An ecological approach to auditory event perception. Ecological Psychology, 5, 1–29. Giordano, B. L. and McAdams, S. (2006). Material identification of real impact sounds: Effects of size variation in steel, glass, wood, and Plexiglas plates. Journal of the Acoustical Society of America, 119, 1171–1181. Grassi, M. (2005). Do we hear size or sound? Balls dropped on plates. Perception and Psychophysics, 67, 274–284.

88

Product Experience

Handel, S. (1991). Listening (2nd Ed.) Boston: The MIT Press. Hermes D. J. (1998) Auditory material perception. IPO Annual Progress Report 33, 95–102. Hirsh, I. J. and Watson, C. S. (1996). Auditory psychophysics and perception. Annual Review of Psychology, 47, 461–484. Houben, M. M. J., Kohlrausch, A. and Hermes, D. J. (2004). Perception of the size and speed of rolling balls by sound. Speech Communication, 43, 331–345. Houben, M. M. J., Kohlrausch, A. and Hermes, D. J. (2005). The contribution of spectral and temporal information to the auditory perception of the size and speed of rolling balls. Acta Acustica United with Acustica, 91, 1007–1015. Ih, J. G., Lim, D. H., Shin, S. H. and Park, Y. (2003). Experimental design and assessment of product sound quality: application to a vacuum cleaner. Noise Control Engineering Journal, 51, 244–252. Izard, C. E. (1977). Human emotions. New York: Plenum. Jekosch, U. (2004). Basic concepts and terms of ‘quality’, reconsidered in the context of product-sound quality. Acta Acustica United with Acustica, 90, 999–1006. Jordan, P. W. and Engelen, H. (1999). Sound design for consumer products. Paper presented at the Proceedings of Stockholm, Hey Listen! Stockholm. Keiper, W. (1997). Sound quality evaluation in the product cycle. Acustica, 83, 784–788. Keiper, W. (1999). Psychoacoustics in industry: Needs and benefit. Acustica, 85, 665–666. Kidd, G. R. and Watson, C. S. (2003). The perceptual dimensionality of environmental sounds. Noise Control Engineering Journal, 51, 216–231. Kunkler-Peck, A. J. and Turvey, M. T. (2000). Hearing shape. Journal of Experimental Psychology – Human Perception and Performance, 26, 279–294. Lim, T. C. (2001). Correlations between deficiencies in power window systems influencing sound quality and some psychoacoustic metrics. Applied Acoustics, 62, 1025–1047. Lindeman, H. E. and Stekelenburg, M. (1981). Lawaai (Noise). Stafleu’s Wetenschapelijke Uitgeversmaatschappij. Louwers, M. T. (1993). Introductory research on the effects of operational sounds on the quality perception of consumer products. IPO Report 932, Eindhoven. Lyon, R. H. (2000). Designing for product sound quality. New York: Dekker. Lyon, R. H. (2003). Product sound quality – from perception to design. Sound and Vibration, 37, 18–23. Marcell, M. E., Borella, D., Greene, M., Kerr, E. and Rogers, S. (2000). Confrontation naming of environmental sounds. Journal of Clinical and Experimental Neuropsychology, 22, 830–864. McAdams, S. (1993). Recognition of sound sources and events. In: S. McAdams and E. Bigand (Eds.) Thinking in sound: the cognitive psychology of human audition. Oxford: Oxford University Press. Meunier, S., Habault, D. and Canevet, G. (2001). Auditory evaluation of sound signals radiated by a vibrating surface. Journal of Sound and Vibration, 247, 897–915. Momtahan, K., Hetu, R. and Tanssley, B. (1993). Audibility and identification of auditory alarms in the operating-room and intensive-care unit. Ergonomics, 36, 1159–1176. Moore, B. C. J. (2003). An introduction into the psychology of hearing (5th Ed.) Amsterdam: Academic Press. Ortony, A., Clore, G. L. and Collins A. (1988). The cognitive structure of emotions. Cambridge: Cambridge University Press. Özcan, E. and Van Egmond, R. (2006). Product sound design and application: overview. In: M. Karlsson (Ed.) Design and Emotion 2006 Conference Proceedings, pp. 1–19. Göteborg: Chalmers University. Özcan, E. and Van Egmond, R. (2007). Memory for product sounds: The effect of label type. Acta Psychologica, 126, 196–215. Park, C. W., Lee, S. I. and Lee, S. J. (2004). Effect of suction nozzle modification on the performance and aeroacoustic noise of a vacuum cleaner. Ksme International Journal, 18, 1648–1660. Parncutt, R. H. (1989). Harmony: A Psychoacoustical Approach. Berlin: Springer Verlag, Plutchik, R. (1980). Emotion: A psychoevolutionary synthesis. New York: Harper and Row. Povel, D. J. and Essens, P. (1985). Perception of temporal patterns. Music Perception, 2, 411–440. Rittmueller, S. P., Mann, J. A. and Holger, D. K. (1997). Iterative design method to reduce the exhaust and venting noise of a vacuum cleaner motor with minimum use of absorptive foam. Noise Control Engineering Journal, 45, 63–68. Solomon, L. N. (1959). Search for physical correlates to psychological dimensions of sounds. Journal of the Acoustical Society of America, 31, 492–497. Solsona, J. F., Altaba, C., Maull, E., Rodriguez, L., Bosque, C. and Mulero, A. (2001). Are auditory warnings in the intensive care unit properly adjusted? Journal of Advanced Nursing, 35, 402–406. Van Balken, J. (2001). Designing coffeemaker sounds. Exploring the experience of Senseo’s sound. Unpublished MSc Thesis, Delft University of Technology.

The experience of product sounds

89

Van Egmond, R. (2004). Emotional experience of frequency modulated sounds: Implications for the design of alarm sounds. In: D. de Waard, K. A. Brookhuis and C. M. Weikert (Eds.) Human factors in design, pp. 345–356. Maastricht, The Netherlands: Shaker Publishing. Van Egmond, R. (2006). Designing an emotional experience for product sounds. In: M. Karlsson (Ed.) Design and Emotion 2006 Conference Proceedings, pp. 1–19. Göteborg: Chalmers University. Van Egmond, R., Desmet, P., Van Der Helm, A. (2004). Basic and cognitive emotions in sound perception. In: A. Kurtgözü (Ed.) Proceedings of the Fourth International Conference on Design and Emotion, pp. 1–11. Ankara: METU. Västfjäll, D., Gulbol, M. A., Kleiner, M. and Garling, T. (2002). Affective evaluations of and reactions to exterior and interior vehicle auditory quality. Journal of Sound and Vibration, 255, 501–518. Von Békésy, G. and Rosenblith, W. A. (1951). The mechanical properties of the inner ear. In: S. S. Steven (Ed.) The handbook of experimental psychology. New York: John Wiley. Von Bismarck, G. (1974a). Sharpness as an attribute of the timbre of steady sounds. Acustica, 30, 159–172. Von Bismarck, G. (1974b). Timbre of steady sounds: A factorial investigation of its verbal attributes. Acustica, 30, 146–159. Welch, J. (1999). Auditory alarms in intensive care. In: N. A. Stanton and J. Edworthy (Eds.) Human factors in auditory warnings. Aldershot: Ashgate, 319–342. Yan Xiao and Seagull, F. J. (2003). An analysis of problems with auditory alarms: Defining the roles of alarms in process monitoring tasks. In: E. Haas and J. Edworthy (Eds.) The ergonomics of sound: Selections from Human Factors and Ergonomics Society Annual Meetings, 1985–2000. Santa Monica: Human Factors and Ergonomics Society, 43–47. Yost, W. A. (2007). Fundamentals of Hearing: An Introduction (5th Ed.). New York: Academic Press. Zajonc, R. B. (1968). Attitudinal effects of mere exposure. Journal of Personality and Social Psychology Monographs, 9, 1–27. Zampini, M., Guest, S. and Spence, C. (2003). The role of auditory cues in modulating the perception of electric toothbrushes. Journal of Dental Research, 82(11), 929–932.

4

TASTE, SMELL AND CHEMESTHESIS IN PRODUCT EXPERIENCE ARMAND V. CARDELLO Natick Soldier Center, MA, USA

PAUL M. WISE Monell Chemical Senses Center, PA, USA

1. INTRODUCTION 1.1. Importance of taste, smell and chemesthesis to product experience The oldest human senses are those that respond to chemicals in the environment. A primitive amoeba responds to a drop of vinegar placed in a Petri dish by moving away from it. Anthropomorphically, we could say that the amoeba ‘experiences’ this unpleasant ‘product’ and behaves in accordance with its negative consequences. Among mammals, the chemical senses play no less of an important role. In humans, the senses of taste, smell, and chemesthesis have become secondary to those of vision and audition, but they still play a pre-eminent role in how we experience a wide range of everyday objects, including all foods and beverages, as well as many important products in the perfume, healthcare and personal product markets. In spite of the importance of the chemical senses to human experience, they have not received the scientific attention that the ‘higher’ senses have received. However, exciting discoveries during the past decade have produced greater interest in the chemical senses. Among these discoveries have been the identification of receptor proteins that encode odorant qualities, and the discovery of basic tastes beyond those we were taught as children. These and other scientific findings have opened the doors to further discoveries that will enable a better understanding of how we experience products using the chemical senses and how we might design products that better meet the utilitarian and aesthetic needs of today’s user. Product Experience Copyright © 2008 Elsevier Ltd.

91

92

Product Experience

1.2. Chemical senses as integrated perceptual systems Scientists working in the areas of physiology, biochemistry, and sensory psychology have traditionally treated sensory systems as discrete, passive, linear systems. Although this view is still predominant in the area, from the perspective of ‘product experience,’ J. J. Gibson’s analysis of human sensory systems is worth considering. Gibson (1966) viewed the human senses as ‘perceptual systems’, i.e. active interrelated systems designed to extract information from the environment for a purpose. In the case of smell, he described this purpose as ‘detecting things at a distance by means of their odors’. In the case of taste, the purpose was ‘the control of ingestion … standing guard over eating, selecting certain things and rejecting others.’ Taken together, the primary purpose ascribed by Gibson to the chemical senses was appetitive or ‘food getting’ (Gibson, 1966, p. 138). Gibson’s notion that the human senses are information gathering systems has relevance to the study of product experience. When we encounter products, whether they are food products on a supermarket shelf or in a restaurant, perfumes in a department store, automobiles in a showroom, or apparel in a clothing store, we are engaged in the process of information gathering for a purpose. The purpose is decision-making related to purchase, consumption or use of the product. However, obtaining sensory information about a product through the senses is only part of the story. Sensory information almost always occurs within a context of other information. Food items in a restaurant occur within a context that includes the price of the item, the appearance and cleanliness of the restaurant, and the perceived nutritive value of the food. Perfumes and automobiles are perceived within a context of marketing images and sales offers, while clothing and certain personal care items are perceived within gender-based, social, and perhaps moral or religious contexts. Thus, the experience of a product must take into account not only its sensory attributes, but the situational, cognitive, social, and cultural lenses through which that information must pass before a judgment can be made regarding its goodness, pleasantness, desirability for purchase, consumption or use. In this chapter we examine the role of three sensory systems that signal the presence of chemicals in the environment, and that influence our product experiences. Taste (gustation) is the sensory system that detects chemicals dissolved in the liquids we drink and the foods we eat. Smell (olfaction) detects air-born chemicals emitted by natural, living, and synthetic products. Chemesthesis detects chemical irritants via the same skin-sense nerves that give us sensitivity to touch, temperature, and pain. These three sensory systems have different specialties, but work together to contribute to our overall experience of the products we use. We will begin by outlining the anatomical, physiological, and biochemical mechanisms that underlie chemosensory experience. This will be followed by a discussion of fundamental sensory and perceptual phenomena that operate when products are experienced through these senses. From there, we will examine how chemosensory product experiences are quantified for practical applications in industry. Lastly, we will examine the roles of non-sensory information, beliefs, expectations and product context on experience, concluding with a discussion of how age, gender, social, and cultural variables can influence chemosensory product experience.

2. TASTE 2.1. The experience of taste Taste is the sensory experience that results from stimulation of chemoreceptors located on the tongue, palate, pharynx, larynx, and other areas of the oral cavity. The human

Taste, smell and chemesthesis

93

experience of taste includes at least five distinct qualities: Bitter, sour, sweet, salty and umami taste, all of which play a significant role in product experience and in helping us to decide what potential foods and/or beverages we should ingest or reject. In addition to these qualities of taste experience, there are two other dimensions of sensory experience that apply to all sensory systems. These are the intensity or magnitude of the taste, smell, etc. and its affective (like/dislike) dimension. These three elements of sensory experience will recur throughout our discussions in this chapter. As noted in the Introduction, the chemical senses evolved as important survival mechanisms, signaling stimuli in the environment that should be avoided and those that are safe to approach and/or consume. The sensory experience of ‘bitterness’ signals the potential presence of toxic substances and triggers rejection. Sourness signals the presence of acids and can be an indicator of spoiled food or unripe fruit, which may also lead to product rejection. Sweetness, on the other hand, indicates the presence of calories needed for survival and triggers ingestion. Saltiness indicates the presence of sodium and other ions needed for nervous function and fluid homeostasis and can also trigger ingestive responses. Lastly, umami taste (savoriness), which is exemplified by the taste of monosodium glutamate (MSG), chicken broth or aged cheddar cheese, suggests the presence of amino acids, the building blocks of life, and leads to ingestion. All of the taste qualities that lead to ingestion also trigger physiological or ‘cephalic phase’ responses that prepare the body to process food, including production of stomach acid and the release of insulin (see Mattes, 1987).

2.2. The mechanisms of taste experience Oral anatomy Although taste receptors are located throughout the oral cavity, many are clustered on taste papillae located on the dorsal surface of the tongue. The most plentiful of these are fungiform papillae, which are mushroom-shaped structures distributed over the anterior tongue. Foliate papillae consist of folds located on the margins of the tongue, while circumvallate papillae are large, circular structures forming a chevron near the back of the tongue. A fourth type of papilla, filiform, is widely distributed over the dorsal surface, but houses no taste buds. Taste buds are rosebud shaped groups of cells (Smith and Margolskee, 2001). Each bud contains about 100 cells, some of which are taste receptor cells. Tastants (molecules of products in solution) pass through a small pore at the top of the taste bud and contact finger-like projections from the taste receptor cells. These projections (microvilli) contain receptor-proteins. Receptors for sweet, savory (umami), and bitter (see below) are probably expressed in different subsets of taste receptor cells, although a given cell may express multiple types of bitter receptors (Zhao et al., 2003; Mueller et al., 2005; Breslin and Huang, 2006). Accordingly, a given taste receptor cell may respond to a number of different compounds, but those compounds will tend to signal the same taste quality (though this picture may change when sour and salty receptors are identified). Receptor mechanisms for experiencing taste Over the last decade, researchers have discovered receptor-proteins for sweet, umami (savory), and bitter tastes (Lindemann, 2001; Mombaerts, 2004; Breslin and Huang, 2006). When these proteins ‘recognize’ particular chemicals, they initiate a cascade of chemical reactions in the taste receptor cell that eventually causes the cell to respond. Receptors for salty and sour tastes have been harder to pinpoint (Lindemann, 2001;

94

Product Experience

Breslin and Huang, 2006). Candidate receptors include ion channels that allow ions to pass through cell membranes. The search for sour and salty receptors continues, and may lead to greater understanding soon (see Liu and Simon, 2001; Huang et al., 2006). Individual differences in the genes that encode receptor-proteins can lead to differences in taste experience (see Breslin and Huang, 2006). Many readers will remember classroom demonstrations using the bitter substance propylthiouracil (PROP). Those who express one form of a particular bitter receptor-protein can detect PROP at much lower concentrations than those who express other forms (see Bufe et al., 2005). The former individuals prove more sensitive to certain other bitter chemicals, including compounds present in a number of healthful vegetables, and are less likely to consume such vegetables (Bell and Tepper, 2006; Dinehart et al., 2006; but also see Yackinous and Guinard, 2002). Other studies suggest that PROP-sensitive people experience greater bitterness and burn from alcoholic beverages and that this sensory response, in turn, reduces consumption (Duffy et al., 2004; Lanier et al., 2005). Still other studies suggest that sensitive children prefer higher levels of sugar in their foods, perhaps to mask the more intense bitter sensations that they experience (Mennella et al., 2005). Differences in sweet preference are also related to genetic variation in the receptor-proteins for sweet taste (McDaniel and Reed, 2004; Li et al., 2005). In short, some aspects of everyday product experience can be predicted by variation in the genes that encode taste receptor-proteins. Knowledge of variations in genes may suggest specific product formulations targeted toward particular phenotypes. For example, as technology progresses, a consumer might someday scan a card with information about his/her genes for taste and smell receptors and receive a cup of coffee with just the right balance of taste and aroma compounds to optimize his/her personal flavor experience. Processing taste receptor information into taste experience The first relay station for taste signals coming from receptors in the oral cavity lies in the brain stem (Smith-Swintosky et al., 1991; Smith and Shepherd, 1999). Connections to other areas in the hindbrain mediate very basic motor and visceral responses connected to acceptance or rejection, including characteristic facial expressions in response to bitter and sweet tastes (see Mattes, 1987) and certain cephalic phase responses, like salivation and increased production of stomach acid. In primate models, and thus perhaps in humans, projections from the brainstem go to primary taste areas in the cortex via the thalamus (Small, 2006). Research suggests that the primary taste cortex plays a role in coding taste intensity (Smith-Swintosky et al., 1991). Next, connections pass to other nearby areas (Baylis et al., 1995; Carmichael and Price, 1995). Responses in the amygdala seem to be driven by interactions among intensity, pleasantness, and familiarity (Small et al., 1997, 2003). Thus, the amygdala might also play an important role in representing stimulus salience (Small, 2006). Responses in the orbitofrontal cortex (OFC) do not seem closely tied to intensity, but different patterns of activation emerge for pleasant versus unpleasant tastes (Small et al., 2003). As will be discussed elsewhere in this chapter, the OFC seems to represent reward value, and probably plays a key role in decisions about chemosensory stimuli. The neural basis of perceived taste quality remains unclear, but the experience of quality probably begins in higher (cortical) taste areas. In these areas, quality is represented in patterns of activity across neurons, and perhaps in timing of responses (Hallock and Di Lorenzo, 2006; Lemon and Smith, 2006). The primary taste cortex and the OFC also receive input from other sensory modalities, including somatosensation and smell (Rolls and Baylis, 1994; Cerf-Ducastel et al., 2001; Savic et al., 2002). In a brain imaging study, Small and colleagues (1997) found

Taste, smell and chemesthesis

95

that both odors and tastes (presented separately) caused activation in primary taste cortex and the OFC. However, when subjects received some taste-odor pairs simultaneously, total activation exceeded the sum of the activations from either stimulus alone. This suggests a special representation in these areas for flavor (the combined effect of taste and smell). Interestingly, both physiological and behavioral studies suggest that effective combinations between tastes and smells only occur for congruent pairs, e.g., vanilla odor plus sweet taste, but not vanilla odor plus salty taste (Small et al., 1997, 2004; Dalton et al., 2000). Learning plays an important role in what sensations yield a combined effect (Diamond et al., 2005).

3. TASTE: BASIC PHENOMENA OF TASTE EXPERIENCE 3.1. Taste adaptation To understand product experience, one must go beyond genetics and physiology to look at the fundamental perceptual phenomena that underlie taste experience. One such phenomenon that occurs in all sensory systems is adaptation, i.e. the sensory experience grows weaker during sustained stimulation. Figure 4.1 shows a typical adaptation curve for taste. When a taste stimulus comes into contact with receptors, the minimum concentration of that stimulus that can be detected (its absolute threshold) is elevated. With continued stimulation this threshold concentration continues to increase until it asymptotes after about one minute. When the stimulus is removed, the threshold concentration begins to fall dramatically, returning to its baseline level after about 30 seconds. This phenomenon is the reason why your first taste of a sour ball candy can be unbearably sour, but as you continue to suck on it, it becomes less sour. Taste ‘adaptation’ also occurs when eating one product reduces the perceived taste intensity of a different product that has the same or similar taste, such as tasting a bite of a Granny Smith apple after having the sour ball candy in your mouth. This latter

FIGURE 4.1 A typical taste adaptation curve (from Matlin and Foley, 1992).

96

Product Experience

phenomenon is known as cross-adaptation. When products that have different tastes are eaten one after another, an increase in the perceived intensity of the second product may occur. One of the most well studied examples of this cross-enhancement of taste experience reveals itself in the bitter taste of distilled water. The bitterness that results from drinking distilled water is due to the fact that, normally, the salt in saliva adapts taste receptors in the oral cavity. When distilled water is consumed, the adapting saliva is removed, producing a characteristic bitter ‘water taste’ (Bartoshuk et al., 1964). In fact, distilled water can take on a variety of different tastes depending upon the nature of the pre-adapting taste (McBurney and Schick, 1971). Thus, our taste product experiences depend not only on our genetic make-up, physiology, and the product being consumed, but also on other products that the user has recently consumed.

3.2. Repetitive taste experiences and liking A similar, but different, phenomenon to taste adaptation occurs with our affective (liking) experiences following repeated exposure to a food or beverage. Cabanac (1971) showed that the pleasantness of a stimulus increases as a function of how much the stimulus satisfies a biological need. Thus, liking for a food will increase the hungrier is the individual. He termed this effect ‘allesthesia’. However, when a food is consumed repeatedly over time, its pleasantness declines relative to foods that have not been eaten (Rolls et al., 1981, 1988). This change in the affective experience of a food or beverage after repeated exposure is known as ‘sensory specific satiety’. In simple terms, it is the reason why, if you eat the same meal for several days in a row, you start to desire other foods. It is the basis for our need for variety in taste experiences. Sensory specific satiety was first viewed to be related to physiological satiation. However, since the phenomenon can occur in response to the taste of foods and beverages that are not consumed (Murphy, 1982; Drewnowski et al., 1982) it seems clear that this is a perceptual effect. Although there is evidence that some sensory attributes and, therefore, some foods/beverages are less susceptible to sensory specific satiety (Drewnowski et al., 1982; Johnson and Vickers, 1991), the phenomenon is a powerful one that shapes our daily experience of foods, how much we like them, and whether and how much we eat of them (Swithers and Hall, 1994; Hetherington and Rolls, 1996).

3.3. Combined taste experiences Most foods and beverages are comprised of a complex combination of ingredients. Each of these contributes their own unique taste to the product. So what happens to our taste experiences when these separate ingredients are combined? Most commonly, the perceived intensity of the mixed tastes will be less than the total of the perceived intensities of the unmixed tastes (Lawless, 1987; Kroeze, 1990). This phenomenon is known as ‘taste suppression’. A number of studies, including those that have shown that the suppression occurs even when different tastes are presented to different sides of the tongue, support the fact that suppression occurs in the brain and is not due to adaptation of taste receptors in the mouth (Lawless, 1979; Kroeze and Bartoshuk, 1985). However, the suppression that occurs when different tastes are combined does not always affect each taste equally. For example, if you mix certain bitter tastants with salty ones, the bitterness is suppressed, but the salty experience typically remains unchanged (Breslin and Beauchamp, 1995). Similarly, if you mix certain bitter compounds with sour ones, you can get enhancement of the taste. Breslin (1996), in a review of the interactions that occur between bitter, salty and sour tastes, concluded that: (1) salty and sour tastes

Taste, smell and chemesthesis

97

suppress each other at high concentrations, but enhance each other at lower concentrations; (2) bitter and sour tastes either suppress or enhance each other, depending on concentration and other factors; and (3) salty and bitter tastes usually suppress the bitterness, but leave the salty taste unaffected. In many food formulations, ingredients are combined that have the same taste quality. Even here interactions can occur to alter the experienced taste intensity of the product. Thus, combining MSG and 5⬘ ribonucleotides can produce enhancement of the umami taste (Rifkin and Bartoshuk, 1980). Similarly, combining high intensity sweeteners will produce enhancement of their sweetness (Ayya and Lawless, 1992; Schiffman et al., 1995). Combining ingredients with either the same or different tastes can produce a variety of interactions, depending on the taste compounds that are mixed, their absolute and relative concentrations, and other factors. The art of the food product developer is to be aware of these mutually interactive effects and to compensate for their impact in order to optimize the sensory experience of the product for the consumer. But can the quality (rather than the intensity) of taste experience change when two or more tastes are combined? On this point most research supports the notion that the components of any taste mixture are still perceived separately and that taste is, therefore, an ‘analytic’ sense. However, the data are not singular in this regard. Not all taste experience comes from foods. One common ‘taste interaction’ that many people experience is how toothpaste ‘changes’ the taste of foods after brushing. At first glance, one might think that the ‘taste’ of the toothpaste is somehow responsible. However, brushing changes a variety of factors that control the general milieu of the mouth (e.g. amount and composition of saliva, presence of minute food particles, micro flora, etc.), so the change in the taste of foods may be due to multiple factors. Further, it must be remembered that much of what we experience as ‘taste’ is really smell. On this point, a recent study of the effects of toothpaste on the flavor of foods (Allison and Chambers, 2005) showed that toothpaste did not affect the perceived taste experience of food, rather it changed the perceived olfactory attributes of the food. For the flavor of orange juice, this effect was seen for up to an hour after brushing.

3.4. Modifying taste experiences Can other sensory properties of a product influence our experience of its taste, and is it possible that our taste experiences can alter perception of other sensory properties? The answer to both of these questions is ‘yes’. Perhaps the most ubiquitous interactions with taste experiences are those that occur with odors. Research by Murphy, e.g. Murphy and Cain (1980), has shown that the overall intensity of taste-odor mixtures presented orally are perceived to be a simple additive function of the components separately. However, when samples of the odorants in solution are presented alone to the mouth, subjects perceive the experience to be one of taste, not odor. The effect, which they termed ‘taste-referred olfaction’, is illusory, because pinching the nose to block retronasal passage of the volatile compounds eliminates the effect. Similar effects have been demonstrated by others with some evidence showing that the more qualitatively similar the tastes and odors, the more likely is the effect (Frank and Byram, 1988; Dalton et al., 2000; Delwiche and Heffelfinger, 2005). Figure 4.2 is taken from research by Labbe et al. (2006), in which the investigators added either cacao or vanilla flavorants (odorants) to a cocoa beverage. As can be seen, the addition of a cocoa odor (top panel) increased both the odor intensity and the bitter taste of the beverage, while depressing its sweetness. Adding vanilla odor (bottom panel) increased aroma and also elevated sweetness at the highest concentration. Thus, the odor

98

Product Experience

FIGURE 4.2 The impact of the addition of three different concentrations of cocoa (top panel) or vanilla flavorings (bottom panel) on taste and olfactory attributes (and body) of cocoa beverage. The stars indicate significant differences at the levels specified above them (from Labbe et al., 2006).

imparted by flavorants in food can differentially affect our experiences of the product’s taste qualities. In commercial practice, the phenomenon of taste-referred olfaction is now being used to ‘flavor’ bottled water by simply embedding odorants into the bottle caps (ScentSational Technologies, LLC, Jenkintown, PA), and quite surprisingly, even imagined odors can produce illusory taste experiences (Algom et al., 1993; Djordjevic et al., 2004). In the latter study, the investigators measured the accuracy of detecting sucrose in water versus plain water during simultaneous presentation of real or imagined ham and strawberry odors. Figure 4.3 shows the results of this study. The accuracy (% correct) of detecting sweetness in the forced choice trials was enhanced in the presence of actual strawberry odor and depressed in the presence of ham odor. Nearly identical effects were found when subjects merely imagined these odors. Chemesthesis, the tactile feel of chemical irritants, can also alter our taste experiences of food products. Several researchers have shown that a rinse of either piperine or capsaicin (black or red pepper solutions) reduces the perceived intensity of sweet, sour, salty and bitter tastes (Lawless and Stevens, 1984; Lawless et al., 1985) Similarly, Gilmore and Green (1993) examined the effect of a capsaicin rinse on the perceived saltiness, sourness, and irritation of high concentration NaCl and citric acid. Their results showed a significant desensitization effect of the capsaicin on both the experienced taste intensity and irritation of these compounds. Thus, it seems that chemesthesis has broad

99

Taste, smell and chemesthesis

100

Detection of sucrose (percentage)

90

80 Strawberry Ham No odor 70

60

50 Perception

Imagery

FIGURE 4.3 Accuracy of detecting sucrose as a function of real (perception) and imagined (imagery) strawberry odor, ham odor, or no odor (redrawn from Djordjevic et al., 2004).

interactive effects with basic tastes and that cross-adaptation can occur between the irritant effects of capsaicin and other taste-related oral irritants. As far as taste experience affecting other perceived aspects of a product, research by Christensen (1980) showed that increasing the sweet taste of solutions resulted in an increase in the experience of their thickness or viscosity. One explanation often given for this ‘cross-modal’ interaction is that, through learning, people come to associate high sugar levels with greater thickness in beverages, e.g. tea sweetened with honey. However, in today’s marketplace, the prevalence of high intensity sweeteners that impart little viscosity to a solution diminishes the likelihood of such a learned association. The taste of certain foods can also alter our experience of somesthetic sensations. A common example of this is how the sweetness of sugar can reduce the experience of burning that results from eating black or red pepper spices (Stevens and Lawless, 1986; Nasrawi and Pangborn, 1989). (See Section 6 for a broader discussion of chemesthetic effects.)

3.5. Innate and early taste experiences and preference How much of our experiences and product preferences is learned and how much is innate? This is a complex question because functioning taste receptors appear in the human fetus at around 14 weeks. Thus, by birth, infants have been exposed to a wide variety of taste compounds present in the amniotic fluid. However, the work cited earlier on the characteristic facial expressions of infants in response to bitter and sweet compounds provides support to the notion that some aspects of our adult taste experiences may be hard-wired. Other research also supports this view. Desor et al. (1973, 1975) measured the amount of solution that young infants consumed and found a strong ‘innate’ preference for sweetness, but little difference in their intake of sour, salty, and bitter solutions. Desor et al. (1975) also demonstrated that adding citric acid to a sweet

100

Product Experience

solution suppressed the infants intake of the mixture, suggesting that the sour experience produced by the citric acid was innately unpleasant. In a review of the area, Cowart and Beauchamp (1990) concluded that there is good evidence of an innate preference for sweet taste and reasonable evidence supporting an innate rejection of sour tastes. While it seems that many everyday products like sour milk, bitter coffee, and sweet pastry may produce pleasurable or unpleasurable experiences as a result of factors present at birth, many other product experiences are modifiable at an early age. For example, Beauchamp et al. (1986) found that infants, who are relatively insensitive to salty tastes at birth, develop a preference for it within the first four months of life. Cowart and Beauchamp (1990) have argued that this is the result of developmental changes, not learning. Other researchers have shown that exposure to certain tastes early in life can alter our taste experiences and preferences later in life (Birch and Marlin, 1982; Harris and Booth, 1987; Beauchamp and Cowart, 1993). The reader is referred to Ganchrow (1995) for a more detailed analysis of the development of human taste experience and preference.

3.6. The effects of learning on taste experience While the perceptual and hedonic natures of many taste experiences are innate, others are learned. An example of this is the development of taste aversions. Researchers have shown that a single pairing of food with the experience of illness is enough to create a lasting aversion to the food (Garb and Stunkard, 1974; Bernstein and Webster, 1980; Pelchat and Rozin, 1982). In many cases this aversion develops within minutes, although the delay between eating the suspect food and illness can be as long as 12 hours. In one survey of college students, 57% reported to have at least one learned food aversion (Midkiff and Bernstein, 1985). Not unexpectedly for that sample population, many of these aversions were to the taste of an alcoholic beverage that had been over-consumed, resulting in alcohol-induced illness. Bartoshuk and Wolfe (1990) have suggested that the aversions that form to foods result from an association with the food’s odor, not its taste. This argument makes sense from a survival perspective, because to be of future advantage to the organism, the aversion to illness-producing foods should occur as early as possible upon encountering the food. By all accounts, taste aversion learning is a primitive form of learning (Schafe and Bernstein, 1996) that can powerfully shape our product experiences. An opposite ‘medicine effect’ in which taste experiences associated with recuperation from illness become preferred has been shown in animals, but it has been difficult to demonstrate in humans. The reader is referred to Chambers and Bernstein (1995) and Schafe and Bernstein (1996) for reviews of this area. In the case of learned taste preferences, mere exposure to novel or disliked tastes (and odors) can increase one’s liking and intake of them (Cain and Johnson, 1978; Davis and Porter, 1991; Stein et al., 2003). This form of simple repetitive exposure to tastes and smells can serve as an important strategy for overcoming neophobia (the reluctance to try new foods or other products). This is why parents, usually unwittingly, continue to offer their children foods that they dislike, e.g. vegetables, in an attempt to ‘overcome’ their disliking with time and repeated exposure. The influence that repeated exposure can have on our product experiences has been powerfully demonstrated in research by Rozin and others on the development of liking for chili pepper. In one such study, Rozin and Schiller (1980) demonstrated that simple repeated exposure to increasing levels of chili in food is all that is needed to reduce the aversion to its burning sensation and to create a preference for it. Other studies have shown that a neutral or disliked taste paired with a pleasant taste experience will increase the preference for the neutral flavor (Zellner et al., 1983; Breslin

Taste, smell and chemesthesis

101

et al., 1990; Capaldi, 1996). This flavor–flavor conditioning occurs unknowingly among coffee and tea drinkers who often initiate their coffee or tea drinking by adding liberal amounts of sugar or non-nutritive sweeteners to offset the bitterness of the coffee/tea. With time, the pairing of the pleasantness of the sweet taste with that of the taste of the coffee/ tea produces a greater liking for the taste of them alone, resulting in the gradual reduction of the added sweetener. The opposite effect of associating a non-preferred taste, e.g. bitter, with a preferred taste, e.g. sweet, has been demonstrated to reduce the liking for the preferred taste (Capaldi and Hunter, 1994). Still other research has shown that flavor preferences can be increased by associating a neutral flavor or food with nutritional consequences (calories) or with feelings of satiety (fullness) (Booth, 1981; Capaldi et al., 1987). Although some taste and hedonic experiences in the real world may be ‘innately’ determined on the basis of our genetic make-up, our needs to avoid toxic substances and to obtain calories have created the ability for our taste experiences to be modified to achieve survival goals. In Section 8 we will discuss how taste and other product experiences can also be modified by such factors as context, information and expectations. The important point to take away from the data discussed here is that our taste experiences of products result from a combination of innate, genetic, cognitive and learned factors. Product experience is only partly to be found in the product. An equally important part is to be found in the consumer and his/her past experiences.

4. SMELL 4.1. The experience of smell Smell is the sensory experience that results from stimulation of receptors in the olfactory epithelium of the nose by airborne molecules. When these molecules interact with olfactory receptors, experiences such as ‘grassy’, ‘woody’ or ‘musky’ result. Odors in the environment warn us of dangers both near, like spoiled food, and remote, like the smell of smoke from a distant fire. Odors also alert us to more pleasant prospects, such as breakfast cooking downstairs, and they play an important role in social communication.

4.2. The mechanisms of olfactory experience Nasal anatomy Information gathering in the olfactory system begins with the passive (normal breathing) or active (sniffing) arrival of airborne molecules. Upon passage into the nares, odorant molecules interact with specialized olfactory receptor-neurons located in the olfactory epithelium, a small patch of receptor tissue located high in the nasal cavity (Rawson and Yee, 2006). Sniffing is more effective for stimulating smell than natural breathing because sniffs create eddy-currents that drive odorized air up to this epithelium. In fact, some authors believe that the act of sniffing is an important unit of olfactory processing and experience, with each sniff constituting a ‘snapshot’ of the olfactory information in the environment (Kepecs et al., 2006; Mainland and Sobel, 2006). While sniffing causes the passage of inhaled air through the nares (orthonasal), chewing of food causes air from the back of the mouth to be passed upward into the nasal chamber in the reverse direction, through the nasopharynx (retronasal). When one perceives the ‘flavor’ of a food in the mouth, it is its retronasal smell that is the major contributing factor to the experience. When a cold produces nasal congestion, it interferes with the flow of air both through the nares and

102

Product Experience

retronasally, which is why our flavor experience of foods eaten when we have a head cold is so diminished. Receptors for olfactory experience Olfactory receptor neurons extend hair-like cilia into the mucous layer that covers the olfactory epithelium (Lledo et al., 2005; Rawson and Yee, 2006). Like taste receptor cells, these cilia contain specialized receptor-proteins that recognize odor molecules. Recent research has identified many of the genes that encode receptor-proteins (Glusman et al., 2001). Humans have approximately 350 potentially functional genes, which implies that we may have as many functional receptor-proteins (Rawson and Yee, 2006). Research suggests that each type of receptor-protein is expressed by many olfactory receptor neurons, but that each individual receptor neuron expresses only a single type of receptor-protein (Buck, 1996). A given receptor-protein, and therefore the olfactory receptor neuron that expresses that protein, will respond to multiple odorants, and a given odorant will stimulate multiple receptor neurons (Hildebrand and Shepherd, 1997). The overall pattern, or mosaic, of stimulated receptor neurons signals the identity of the molecule at this level of the olfactory system. Recent research on odor receptors provides new insights into consumer olfactory experience. It is clear that odor perception differs substantially among people with a ‘normal’ sense of smell. For example, the compound androstenone, which is found in products like celery, pork, and truffles, is experienced quite differently among individuals. It is described as being either urinous, musky, or floral by different people. Yet, upwards of 50% of the population cannot even smell it (Labows and Wysocki, 1984). Wysocki and Beauchamp (1984) have shown a strong genetic basis for this difference and have demonstrated the involvement of genetic, developmental, and experiential factors (Wysocki and Beauchamp, 1991). Researchers are now attempting to link differences in olfactory experiences to specific mutations in genes that encode receptors. Those who develop consumer olfactory products, like perfume, know that a single formulation will rarely please all consumers. That we differ in our mosaic of smell receptors helps explain why we each seem to live in our own world of scents. As with taste experience, it may soon prove possible to infer much about the olfactory experiences and preferences of an individual by knowing the genes that he or she expresses. Receptors for pheromones Pheromones are chemical signals released by one individual that influence the physiology or behavior of another individual of the same species (see Wysocki and Preti, 2004). Many animals can sense pheromones both with the olfactory system and with a specialized chemosensory organ called the VNO, or vomeronasal organ (see Buck, 2000). Humans lack a functional VNO, but we may express some receptors in our olfactory epithelium very similar to receptors found in the VNO of other animals (Rodriguez et al., 2000). Further, members of a newly-discovered family of receptors respond to volatile amines, including stress-related amines in urine and amines that differ in concentration between males and females (Liberles and Buck, 2006). Genes for some of these receptors can be found in human olfactory epithelium. In short, between regular olfactory receptors and other receptors that may exist, humans have adequate capabilities for sensing pheromones. It is clear that humans respond to pheromones (Wysocki and Preti, 2004), although not with the automatic, stereotyped behavior of moths that fly miles upwind to mate. A magic love potion that irresistibly draws the opposite sex seems unlikely. However, pheromones may influence mood and physiology. Male underarm extract (odor masked) placed on the upper lip makes women ‘less tense’ and ‘more relaxed’ relative to the

Taste, smell and chemesthesis

103

masking vehicle alone (Preti et al., 2003). Underarm extract from a female donor can influence release of hormones to either accelerate or delay ovulation, depending on the stage of the donor (Stern and McClintock, 1998; Shinohara et al., 2001). This finding helps explain menstrual synchrony, i.e., the tendency for women who live together to fall into register over time. In addition, underarm extract from male donors can make women more regular and accelerates release of hormones associated with ovulation (Cutler et al., 1986; Preti et al., 2003). More research is needed, but data suggest that a properly formulated olfactory product could alter both our mood and our physiology. Processing olfactory receptor information into smell experiences Signals from the olfactory receptor neurons pass to the brain for further processing (Rolls, 2001; Lledo et al., 2005; Gottfried, 2006). First, signals are sharpened in specialized structures called the olfactory bulbs. Then, signals diverge into several pathways (Price, 1973). The direct cortical targets of the signals are situated in the primary olfactory cortex. Activity in parts of the primary olfactory cortex is modulated by sniffing, again reflecting its importance in olfactory processing (Sobel et al., 1998). Other work suggests that primary olfactory cortex plays a role in olfactory learning and memory, and may be involved in separating odors of interest from steady (background) odors (Gottfried et al., 2003; Kadohisa and Wilson, 2006). Interestingly, there are direct or nearly direct connections to other evolutionarily old parts of the brain, including the hypothalamus. The hypothalamus regulates hormonal release via the pituitary gland and plays a key role in the regulation of hunger, thirst, sexual behavior, and other basic functions, The hypothalamus may play a roll in the olfactory circuits that help regulate these functions, and may modulate sensory response based on both internal and external signals (Rolls, 2001). There are also direct connections to the amygdala, which responds to interactions between intensity and pleasantness (Anderson et al., 2003). Thus, similar to conclusions regarding taste processing (see p. 94), olfactory activity in the amygdala seems to play a role in representing stimulus salience. Interestingly, personally meaningful odors are described with strong emotional language and are associated with relatively strong activation in the amygdala (Herz et al., 2004a). Contrary to popular perception, studies suggest that odor evoked memories are no more accurate than those evoked by auditory, visual, or tactile cues, but they do entail stronger emotional reactions (Herz, 1997). Thus, people may confuse emotional vividness with accuracy. From the primary cortical areas, odor signals pass to higher brain areas, either directly or through the thalamus. The orbitofrontal cortex (OFC) is one important target (see Rolls, 2001; Gottfried, 2006). As with taste, the OFC combines input from multiple sensory modalities to make evaluative ‘judgments’. Also like taste, odors that people find pleasant evoke different patterns of activity in the OFC than do odors they find unpleasant (Anderson et al., 2003). Other studies suggest that some cells in the OFC respond according to the reward-value of a stimulus (reviewed in Rolls, 2001). In other words, a given cell might respond differently to a particular odor, depending on whether or not the odor was associated with a pleasant reward. This result suggests that evaluative judgments about odors are subject to learning. Other work has shown that, relative to a vanilla odor control, OFC activation to banana odor declines after subjects have eaten bananas to satiety (O’Doherty et al., 2000). This effect is not simple adaptation, since both neural response at lower levels of processing and rated intensity might change very little. Rather, this is a case of sensory specific satiety and reflects a change in the rewardvalue of the stimulus.

104

Product Experience

5. SMELL: BASIC PHENOMENA OF EXPERIENCE 5.1. Olfactory adaptation Continued exposure to the odor of a product decreases our ability to detect it, reduces its perceived intensity, and increases our reaction time to its presence (see Cometto-Muñiz and Cain, 1995; Lawless, 1997). A typical pattern consists of rapid loss of odor intensity within 5–10 seconds, and a more gradual decline thereafter, reaching a new steady state within a minute or two. However, the rate and degree to which odors adapt varies. Intense odors are more resistant to adaptation than weak ones (Cain, 1974). Further, unpleasant odors adapt more rapidly than pleasant ones (Jacob et al., 2003). The time course of adaptation could also depend on physical properties of the molecules, related to diffusion into the mucus and eventual clearance (Dalton, 2000). Adaptation is strongest for odors identical to the adapting odor (self-adaptation), but exposure to one odor may reduce the perceived experience of another odor to some lesser degree (cross-adaptation). One might expect similar smelling odors to stimulate overlapping sets of physiological mechanisms and thus cross-adapt effectively, and dissimilar odors to cross-adapt less effectively. Some data are consistent with this notion (e.g. Todrank et al., 1991; Cain and Polak, 1992). However, molecular structure, apart from the perceived quality, seems to play a roll. For example, 3-methyl-2-hexenoic acid (3M2H), a sweaty-smelling compound important in human underarm odor, can be crossadapted by the fruity-smelling ethyl ester of 3M2H (Pierce et al., 1995). Other research suggests it may even be possible to cross-adapt an odor using an odorless molecule with similar structure (Pierce et al., 1996). Wysocki et al. (2003) identified common fragrance ingredients that cross-adapt human underarm odor. This work showed that for males, many odors cross-adapt underarm odor, but far fewer cross-adapt underarm odor for females. Thus, properties of the observer, as well as properties of the molecule, can play a roll in efficacy of cross-adaptation. Adaptation probably occurs at various levels of the olfactory system (Cometto-Muñiz and Cain, 1995; Dalton, 2000). Receptor cells adapt, but neither the magnitude nor the duration of peripheral adaptation fully accounts for human olfactory responses (Zufall and Leinders-Zufall, 1997). Further, long-term reductions in sensitivity observed with repeated exposure on the job (Dalton et al., 2003) or in the home (Dalton and Wysocki, 1996) may have quite different underlying causes than short-term adaptation seen while sampling multiple perfumes at the department store. At some level, cognitive influences play a role. We may cease to pay attention to an odor that is always present, a general phenomenon many researchers call habituation (Lawless, 1997). Interestingly, if subjects believe an odor is safe and beneficial, rated intensity will wane more over time than if subjects believe the same odor is potentially harmful (Dalton, 1996). Thus, cognitive and emotional reactions to odors may modulate their experienced intensity (see Distel and Hudson, 2001, for a discussion of these influences on perceived odor intensity).

5.2. Combined olfactory experiences The odors of most consumer products come from mixtures of odorants. Products that counteract malodors often seek to mask offending smells by creating a mixture of the masker and the offending odor. Understanding the relationship between the experience of unmixed components and the experience of their mixtures will aid in product design, optimization, and quality control.

Taste, smell and chemesthesis

105

Most research on odor mixtures has been limited to two-component mixtures. The following general rules have emerged with respect to odor strength: (1) a mixture of two odorants will generally smell less intense than the sum of the intensities of the unmixed components, but stronger than the weaker of the two components; (2) the components, when they can be individually perceived, will smell weaker than when unmixed; and (3) the stronger of the two mixture-components will reduce the intensity of the weaker component more than the weaker will reduce the stronger (Laing et al., 1984; Cain et al., 1995; Lawless, 1997). Less work has been done on more complex mixtures, but some research suggests that beyond three or four components, adding additional odors has little impact on overall intensity (Moskowitz and Barbe, 1977). In addition, interactions depend to some extent on the overall intensity of the odors. Studies have found more complete summation when the unmixed components are relatively weak (Laing et al., 1984; Cain et al., 1995). With weak odors, even synergy has been observed (Engen, 1982; Laska et al., 1990). This general trend of concentration-dependence in mixture interactions also holds true at threshold levels (Cometto-Muñiz et al., 2005). Like taste mixtures, simple mixtures of odorants can be analyzed into their components. However, while taste qualities can be discerned in five to six component mixtures (Marshall et al., 2006), complex odor mixtures often form holistic sensations, and even expert perfumers fail to correctly identify all the constituents of a mixture that consists of more than about three to five chemicals (Livermore and Laing, 1996). This phenomenon is commonly assumed to support the fact that olfaction is a synthetic sense (Burgard and Kuznicki, 1990). According to most sensory work, the perceived qualities of mixtures generally fall between those of their unmixed components, with no radically new notes emerging, though some research leaves the possibility of emergent qualities open (Moskowitz and Barbe, 1977; Jinks and Laing, 2001; Zou and Buck, 2006). In addition, some odors tend to have an impact on the quality of mixtures disproportionate to their individual perceived intensities. For example, woody odors tend to dominate fruity ones, consistent with the laments of wine makers who find that tannins have a large impact on flavor (Atanasova et al., 2005). Unfortunately, science cannot precisely predict how a mixture will smell based on the properties of the molecules that comprise it. There is no substitute for the flavorist and perfumer, who create flavors and scents using a combination of personal experience and experimentation. However, systematic studies of how molecular properties influence mixture-interactions could eventually lead to perceptual models that greatly streamline the process of formulating novel flavors and aromas.

5.3. Modification of olfactory experiences Olfactory experiences can often be modified by taste. Von Sydow et al. (1974) showed that the experience of the intensity of the pleasant odors of fruit juice increases, while the experience of unpleasant odors decreases, in response to added sugar. McBride and Johnson (1987) found enhancement of lemon flavor experience with increasing levels of sucrose and citric acid in lemon juices, while Perng and McDaniel (1989) showed that sucrose enhanced the fruit flavor intensity of blackberry juice. However, in contrast to McBride and Johnson, the latter authors found that increasing the sourness of the juice lowered the experience of its fruit flavor. Other investigators have shown that aspartame enhances and prolongs the fruitiness of fruit-flavored solutions more than sucrose (e.g. Bonnans and Noble, 1993). A number of additional studies have also shown the strong interactions of taste and smell experience in foods (Frank et al., 1993; Stevenson et al., 1995, 1999). When odorant compounds are mixed with irritant compounds, mutual suppression occurs, much like with mixtures of odors. Cain and Murphy (1980) demonstrated this

106

Product Experience

effect by mixing the nasal irritant, carbon dioxide, with concentrations of amyl butyrate. The result was a mutual suppressive interaction, i.e. increasing concentrations of carbon dioxide resulted in decreasing judgments of the odor of amyl butyrate and vice versa. Moreover, by presenting the odorant compound to one nostril and the irritant compound to the other, it was possible to show that the locus of interaction occurred centrally in the nervous system (Cain and Murphy, 1980).

5.4. Innate versus learned odor experiences and preferences Some research has found that infants and young children have odor preferences that differ from those of adults, including a relative indifference to some odors that adults find offensive (Engen, 1982, 1988). These data suggest that odor preferences are learned over time. However, other research has found support for innate odor preferences. Among these latter studies are those showing that infants, like adults, display more negative facial responses to fishy and rotten egg odors than to the odors of vanilla, banana or butter (Steiner, 1977; see also Schmidt and Beauchamp, 1988) and that very young infants display preferences for certain biologically relevant odors, like the odor of breast milk (Porter et al., 1992). However, the fact that we begin learning odor preferences in the womb (Schaal et al., 1998; Winberg and Porter, 1998), e.g. infants of mothers who consume garlic are born with a preference for garlic odor (Mennella et al., 1995), makes it difficult to dissociate innate odor preferences from early learning. Regardless of whether there are innately preferred or rejected odors, learning and context certainly play important roles later in life. One line of evidence comes from intercultural differences in odor preferences (Schleidt et al., 1981; Wysocki and Gilbert, 1989; AyabeKanamuura et al., 1998). An odor that appeals to one culture does not necessarily appeal to another (see Section 9.3). Also consider a study sponsored by the US military that sought a universally offensive odor for non-lethal crowd control (Dilks et al., 1999). No single odor in the study was considered equally unpleasant across all ethnic groups. Another line of evidence comes from studies of odors paired with either positive or negative emotional experiences (Baeyens and Wrzesniewski, 1996; Herz et al., 2004b). The results suggest that, through associative learning, the experience of an odor takes on the emotional tone of the context in which the odor was originally experienced. As suggested elsewhere in this chapter, beliefs and expectations play a critical role. In one study, panelists rated a given odor as less pleasant if they believed it consisted of synthetic rather than natural materials (Herz, 2003), while in another, people expressed more liking for a mixture of fatty acids when the stimulus was labeled ‘parmesan cheese’ rather than ‘vomit’ (Herz and von Clef, 2001). Perfumers and flavorists know the benefits of selecting odors that have positive associations in the culture in which the product will be marketed. The above research suggests that the consumer product industry might actively influence the acceptability of new scents through proper advertising, packaging, and labeling that creates associations with positive images and concepts. Retailers can further help by ensuring that consumers experience new odors in an emotionally positive context. Such techniques might prove particularly effective in influencing the pleasantness of newly developed odors and scents.

6. CHEMESTHESIS 6.1. The experience of chemesthesis We feel chemicals when they stimulate nerve endings in our skin (see Doty and ComettoMuñiz, 2003; Wysocki and Wise, 2003). Chemical feel is called ‘chemesthesis,’ a

Taste, smell and chemesthesis

107

contraction of ‘chemical somesthesis’. When chemicals reach nerve endings in sufficient concentration, we experience sensations such as the ‘cooling’ of menthol, the ‘bite’ of carbonation, the ‘burn’ of distilled spirits, the ‘heat’ of chili peppers, and the ‘sting’ of smelling salts. Chemesthesis warns us against high chemical concentrations, and helps protect tissue through reactions such as tearing, increased mucus production, sneezing, and coughing. However, mild to moderate chemesthesis plays an important role in our everyday experience of foods, beverages, and many personal products.

6.2. Peripheral anatomy and receptor mechanisms for experiencing chemesthesis Mucous membranes, like those that cover the eyes, nose, and mouth, are more sensitive to chemicals because they offer easier access to nerve endings than the keratinized skin on the rest of the body. As with taste and smell, some irritants stimulate specific receptor proteins in nerve endings (Caterina and Julius, 2001; Peier et al., 2002; McKemy, 2005). For example, one protein responds to capsaicin, the active principle in hot peppers, whereas another responds to cooling compounds like menthol. Interestingly, these and other chemically sensitive receptors also respond to temperature changes, so the terms ‘hot pepper’ and ‘cool mint’ make perfect sense. In addition, some compounds may act through non-specific effects on cell membranes rather than on specific receptor proteins. The action of carbon dioxide (CO2), the gas that gives bite to carbonated beverages, depends on enzyme activity (see Tarun et al., 2003). CO2 diffuses into tissue, where enzymes facilitate formation of carbonic acid (the effective stimulus). Recent work has found individual differences in the enzyme genes expressed in the nose (Tarun et al., 2003) and sensory receptors for nicotine may also vary among individuals (Keiger et al. 2003). Furthermore, PROP tasters and PROP non-tasters (p. 94) differ in sensitivity to oral irritation, probably due to differences in nerve density in the tongue (Prescott and SwainCampbell, 2000; Duffy et al., 2004). As with smell and taste, we may eventually predict some chemesthetic product experiences and preferences based on gene make-up.

6.3. Processing peripheral nerve signals into chemesthetic experiences Nerve fibers that carry somatosensory information project first to the brain stem (for signals from the eyes, nose, and mouth) (Hendry et al., 1999). From there, signals pass to the thalamus, then to the primary somatosensory cortex, then to higher cortical areas. Data on central processing of chemesthesis are sparse. An imaging study using an irritating concentration of acetone found activation in the brainstem and somatosensory cortex (Savic et al., 2002). In addition, research measuring electrical potentials has found differences in patterns of activity across the scalp between pure odors and irritants (reviewed in Shusterman, 2002). However, little research bears on quality coding past the periphery.

6.4. Chemesthesis: Basic phenomena At both supra-threshold (Cometto-Muñiz and Hernádez, 1990) and threshold levels (Cometto-Muñiz et al., 2004), there is greater cooperation (better additivity) among mixture-components for nasal chemesthesis than for smell. Better cooperation might indicate that detection-mechanisms for chemesthesis are more broadly tuned (Cometto-Muñiz et al., 2004). Practically speaking, good cooperation means a mixture might irritate, even if none of the individual compounds irritate, having implications for the experience of multi-odorous indoor air and odor-containing personal products used in indoor spaces.

108

Product Experience

In terms of compounds that have both chemesthetic and olfactory components, Figure 4.4 shows data from an experiment in which ammonia was presented to subjects in two concentrations and for two durations. As can be seen, the perceived irritation (and thus, the total intensity) increased more dramatically as a function of both concentration and stimulus duration than did the perceived odor. Thus, when one inhales ammonia, its odor reaches a peak rapidly while irritation continues to grow over time, perhaps even suppressing its odor. Although strong chemesthesis is innately aversive, many individuals develop a preference for mild to moderate irritation with repeated exposure (see Rozin and Schiller, 1980; Stevenson and Yeomans, 1993). Also, people in warmer climates tend to consume more hot spices. Since compounds in many spice plants have anti-microbial activity, societies in which food spoilage poses greater problems might develop a taste for spices, in part, for practical reasons. Further, as with odor, emotional reactions will depend on beliefs about the stimulus. A pleasantly pungent fragrance will be received quite differently if subjects believe the aroma comes from a potentially dangerous chemical (see Dalton and Wysocki, 1996). Relative to taste and smell, the dynamics of chemesthesis are slow (Cain et al., 1986; Green, 1990; Prescott and Swain-Campbell, 2000). Depending on the chemical, concentration, method of presentation, and location of stimulation (mouth, nose, or eyes), irritation may build over many minutes and fade slowly once a peak is reached. Build-up, seen with steady presentation or with a sufficiently brief inter-stimulus interval (ISI), is often referred to as sensitization (Green, 1990; Prescott and Swain-Campbell, 2000). The waning of irritation, seen after some time with steady presentation or with a sufficiently long ISI, is often called desensitization (Green, 1990; Prescott and Swain-Campbell, 2000; Jacquot et al., 2005). However, the mechanisms underlying these phenomena are not completely understood.

FIGURE 4.4 Perceived odor, pungency (irritation) and total intensity for two concentrations of ammonia inhaled for two different durations (from Cometto-Muniz and Cain, 1984).

Taste, smell and chemesthesis

109

7. MEASURING CHEMOSENSORY PRODUCT EXPERIENCE 7.1. Chemosensory experiences: Consumers versus experts Quantifying the taste, smell and chemesthetic experiences produced by products is critical to many product development efforts. In many commodity areas, e.g. wine, tea, coffee, dairy products, and perfumes, expert ‘palates’ or ‘noses’ are used to characterize sensory properties, to create new formulations, and to delineate what constitutes a ‘good’ or ‘bad’ product. However, by virtue of their extensive experience and their acquired abilities to distinguish subtle product differences, the product experiences of experts are not the same as those of consumers. A reason commonly cited by product experts for not using consumers to judge product quality is that consumers differ from one another in their experiences and preferences. Yet, while this is true, it has been shown that the expert’s experience of wine and beer quality differs significantly from one expert to another (Langron et al., 1983; Guinard et al., 1999), and that even the vocabulary they use to describe these products differs dramatically among them (Sauvageot et al., 2006). In the perfume industry, ‘experts’ also disagree. Witness a leading perfumist’s review of Dark Chocolate perfume by Coty: ‘The name suggests a beguilingly sinful, high calorie confection. Instead, the fragrance has all the charm and mystery of a neon-lit shopping mall, and smells of cheap chocolates with a fruit heart. A case of pastel clad obesity’ (Turin, 2006). The notion that experts have special sensory abilities that enable them to detect product differences that consumers cannot is also often overstated. For example, it has been shown that while expert tea tasters can discriminate the experience produced from tasting tea in which the milk has been poured into the tea versus the tea having been poured into the milk, it has been shown that consumers have the same ability (Powers, 1988). Of the many purported abilities of experts, the one that most differentiates expert product experiences from those of consumers is their ability to identify and characterize differences in taste and olfactory product experiences using a well-defined vocabulary. This ability to provide a detailed description of product attributes, while beyond the capacity of most consumers, can often be taught using standard descriptive sensory methods, as discussed below.

7.2. Methods for quantifying chemosensory product experiences: Trained panels Although the use of experts is still common in some specialized product areas, most commercial products are now characterized by trained ‘descriptive’ panels. Descriptive panels are groups of individuals, screened to ensure lack of defects in taste or smell. These individuals, usually 10–20 in number, are trained in one of a variety of descriptive sensory methods that enable them to articulate and describe their product experiences using standard terminology and operational definitions. This process produces a product profile that identifies all of the taste, smell, and chemesthetic attributes of the product and the intensities of each. There are a wide variety of methods for conducting descriptive analysis. The oldest is the flavor profile method developed by Arthur D. Little Co. (Cairncross and Sjostrom, 1950). This method utilized a panel of six to eight individuals who received training in taste and smell physiology, sensory testing, and flavor vocabulary. Physical reference standards for specific taste and smell attributes, e.g. ‘salty’, ‘rancid’ or ‘piney’ were employed to be used during training and product evaluation. A panel leader was chosen

110

Product Experience

to lead panel discussions and to arrive at a consensus on the experienced aroma, taste, flavor, mouthfeel, and aftertaste attributes of the product, the order in which they were perceived, their intensity, and the ‘amplitude’ or overall quality of the product. Although the A. D. Little Flavor Profile method was the first popular descriptive method, it has largely been replaced by more contemporary methods that have improved upon various aspects of the methodology. Among these more contemporary approaches is Quantitative Descriptive Analysis (QDA) (Stone et al., 1974), a method that replaced consensus agreement with the use of quantitative, statistical analysis of panel data. The perceived intensity ratings were improved in QDA by using a linear graphic rating scale, resulting in a method with reportedly good quantitative reliability (Stone and Sidel, 2004). This technique also spurred the use of ‘spider web’ charts for displaying product attributes. Figure 4.5 shows a spider web chart of descriptive data for two personal care products (bar soap). The lengths of the spines reflect the maximum intensity for each attribute and the mean product ratings for that attribute are plotted along the spine. By overlaying the data for two or more products on a single spider web (as in Figure 4.5), the differences in the sensory profiles of the products are easily visualized. A third technique, known as Spectrum Descriptive Analysis, has also been popularized (Meilgaard et al., 1999). This latter technique adapts approaches from earlier methods, while utilizing ‘universal’ rating scales with physical reference anchors in order to quantify the product experiences of panelists. While each of these methods differ to some degree, comparisons of the methods for the same products typically produce similar results.

7.3. Methods for quantifying chemosensory product experiences: Consumers Quantifying sensory experience After training and extensive panel work, it can be argued that the product experiences of descriptive panelists are also unlike those of the average user. In an effort to avoid the use of either experts or trained panelists to describe taste and olfactory product

FIGURE 4.5 A typical spider web plot of descriptive data for two bar soaps (from Lawless and Heymann, 1998).

Taste, smell and chemesthesis

111

experience, the method known as ‘free-choice profiling’ was developed (Williams and Langron, 1984; Steenkamp and van Trijp, 1988). This approach relies on naive consumers who use their own vocabulary to describe their experience of the product or who select terminology from a pre-prepared list. Terminology may also be generated using the repertory grid method (e.g. Jaeger et al., 2005). The data are then analyzed using multivariate techniques in order to develop a user-based perceptual space of products, the dimensions of which can be defined in terms of the actual experiential vocabulary that consumers use. Other consumer-friendly techniques have also been developed. Free-sorting procedures, in which products are sorted into groups and a common vocabulary is used to describe the groups, have been used to analyze the perceptual product space for tastants (Yang and Lawless, 2005), odorants (Rawcliffe et al., 1992), and a variety of foods and beverages (Lawless et al., 1995; Popper and Heymann, 1996; Tang and Heymann, 2002). In a recent study comparing sorting by consumers and QDA trained panelists, the perceptual product space for the two groups was found to be quite similar. Moreover, the sorting procedure for trained panelists provided similar results to traditional QDA methods (Cartier et al., 2006). The authors concluded that ‘sorting combined with verbalization led to meaningful and consistent product sensory mapping, whatever the panelists’ level of training’ (Cartier et al., 2006, p. 562). Similar results also have been reported by Faye et al. (2006) for non-food products. Quantifying affective experience As noted in other sections, most chemosensory products produce a strong affective or ‘hedonic’ experience in the user. When an individual smells a new cologne or tastes a new chewing gum, there is an emotional experience of its pleasantness or unpleasantness. These hedonic experiences of products are the primary drivers of the consumer’s choice, purchase, consumption, and/or utilization of the product. The first studies of affective experience were undertaken by Beebe-Center in the 1930s, but the systematic study of food hedonics (from the Greek word for ‘pleasure’) and its measurement began after World War II when the US Army Quartermaster Food and Container Institute in Chicago began to study why soldiers liked and/or disliked certain foods. This research program ultimately produced the now well known, ‘ninepoint hedonic scale’ (Peryam and Pilgrim, 1957) (see left side of Figure 4.6). This simple method for quantifying the intensity of a person’s affective experience is now the most commonly used affective measure in the food, cosmetics, perfumery, and personal care products industries. In spite of its simplicity, the nine-point hedonic scale does have problems. Since it is a form of ‘category’ scale there is a tendency for people to avoid the extreme categories (Stevens and Galanter, 1957). In addition, the intervals between scale points are not perceptually equal (Jones and Thurstone, 1955; Moskowitz and Sidel, 1971), so the level of measurement it provides is, at best, ordered metric. Other methods that have been used over the years include magnitude estimation, linear graphic rating scales and, most recently, a Labeled Affective Magnitude (LAM) scale (Schutz and Cardello, 2001; Cardello and Schutz, 2004). The latter scale was developed to provide a simple-to-use, ratio-like scale of product liking experience that is modeled after other contemporary labeled magnitude scales (Green et al., 1993; Bartoshuk, 2000). As shown on the right side of Figure 4.6, this scale uses a vertical line with the end points labeled ‘greatest imaginable liking’ and ‘greatest imaginable disliking’. Between these end-point anchors the nine labels used in the nine-point hedonic scale are distributed along the liking/disliking dimension according to the perceived ratios in their semantic meaning, as determined

112

Product Experience

Scales for rating product liking experience Nine-point Hedonic Scalea

Labeled Affective Magnitude Scaleb,c Greatest imaginable liking

Like extremely Like very much Like moderately Like slightly Neither like nor dislike Dislike slightly Dislike moderately Dislike very much Dislike extremely

Like extremely Like very much Like moderately Like slightly Neither like nor dislike Dislike slightly Dislike moderately Dislike very much Dislike extremely Greatest imaginable disliking

a

Peryam and Pilgrim (1957)

bSchutz

and Cardello (2001) and Schutz (2004)

cCardello

FIGURE 4.6 Scales for rating product liking/disliking (affective) experiences.

using a ratio scaling procedure. Using this scale it is possible to estimate whether the liking experience for a particular product is twice, one-third, etc. as large as the liking experience for a different product (Schutz and Cardello, 2001). In addition, the scale has been shown to produce better sensitivity, equal reliability, and equal ease of use to the nine-point hedonic scale by consumers (Schutz and Cardello, 2001).

8. CONTEXT, INFORMATION AND EXPECTATIONS IN CHEMOSENSORY AND PRODUCT EXPERIENCE 8.1. Non-sensory influences on product experience The taste and olfactory experiences that arise from the intrinsic properties of a product do not constitute the sole determinants of product experience. There are a host of other ‘extrinsic’ factors, e.g. product names, packaging, labeling, pricing, context and promotional information that influence product experience through cognitive and psychological processes. In this section we will explore the effect of some of these factors.

8.2. The effects of context Both the sequential and situational contexts in which a product is experienced can significantly alter our experiences of it. Sequential effects are commonplace. When you are shopping for fragrances in a department store, you might first smell a perfume on one wrist and then its eau de toilette on your other wrist. You would likely conclude that the eau de toilette was quite weak. However, if you tried them in the reverse order, you would

Taste, smell and chemesthesis

113

likely feel differently about the strength of the eau de toilette, even if you waited a relatively long time between trials to ensure that no adaptation to the scent was occurring. In fact, in a recent study of sequential context effects, the perceived intensity of the odor of perfumery grade California Orange Oil was affected by presentation of either a weaker or stronger sample as much as 25 minutes before the target odorant was presented (Pol et al., 1998). A number of other studies using tastants and odorants have shown that the intensity of a target product presented within a context of products of either higher or lower taste/smell intensity or liking will be ‘contrasted’ with the intensity of the contextual products (Lawless, 1983; McBride, 1985; Schifferstein, 1995). These results are commonly explained in terms of a ‘perceptual’ adaptation level (Helson, 1964) or in terms of range-frequency theory (Parducci, 1974). Recently, Schifferstein has addressed this phenomenon in terms of the nature and characteristics of the contextual stimuli and their impact on the magnitude of these contrast effects (Schifferstein, 1995; Schifferstein and Frijters, 1992; Schifferstein and Oudejans, 1996). His results support the notion that the more similar the contextual and target stimuli, the larger will be the contrast effect. In addition to sequential context effects, the situational and environmental context in which products are encountered can alter our experiences of them. For example, the product experience that results from eating ice cream on a hot summer day is quite different from eating that same ice cream on a cold, winter day. Likewise, the experience of encountering a fishy odor in the hallway of your home is quite different from the experience of encountering that same odor in a seafood market. In the world of taste and smell, context can make all the difference. The number of different environmental, situational, and other contextual factors that can alter product experiences is quite large. The reader is referred to articles and book chapters by Rozin and Tuorila, 1993; Cardello, 1994, 2007; Deliza and MacFie, 1996; Schifferstein, 2001; Wansink et al., 2005; Jaeger, 2006 and Chapter 24 by Meiselman in this volume for discussions of the contextual influences on product experience. Perhaps the most common contextual or ‘framing’ influences in the commercial products industry are those involving product information. Whether written, verbal, or non-verbal, information influences consumer experience and behavior by altering the attitudes and beliefs that the consumer holds toward the product. In turn, these attitudes and beliefs can give rise to product expectations, i.e. psychological anticipations that the product will possess certain attributes or have certain anticipated effects. Expectations can arise from previous experiences with the product, with other products in the category, or from any one of a variety of information sources about the product. Two major types of expectations are those that relate to expected sensory experiences, e.g. a belief that a skin cream will have a floral scent, and those that relate to expected affective experiences, e.g. a belief that you will like the smell of a new automobile’s leather interior.

8.3. The effects of information and expectations Effects on olfactory product experiences Olfactory experiences seem to be especially prone to the influence of expectations. As early as 1899, Slosson performed a classroom demonstration in which he told students that an odorant was to be dispersed into the back of the classroom and that he wanted to see how long it took for the odor to reach the front of the room. He asked his students to raise their hand when they smelled the odor. Although no odor was actually dispersed, a majority of students reported smelling the odor, some even becoming upset by its unpleasant quality. Much more recently, O’Mahony (1978) performed a similar but more dramatic experiment. Using a British television broadcast of a documentary on

114

Product Experience

the chemical senses, viewers were informed that smells could be produced by transmitting sound waves of the same vibrational frequency as the frequency of vibration of the molecules of the odorant. The documentary then went on to ‘demonstrate’ the phenomenon by ‘broadcasting’ a specific frequency of sound alleged to correspond to a pleasant country odor. Viewers were asked to phone or write the station if and when they smelled the odor. Of 130 responses received, 114 viewers said that they experienced a smell when the sound was emitted. The most common smell experiences reported were ‘hay’ or ‘grass’. Using more traditional paradigms, Engen (1972) performed experiments in which an odorant was presented to subjects at regular intervals prior to the presentation of a ‘blank odorant’ in order to establish an expectation for the odor. The mere expectation that the odorant was to be presented resulted in an odor experience. In another experiment, he flowed odorants through a tube that was visible to the subjects. The odorants were either colored yellow or left transparent. A much higher rate of odor experience resulted on trials in which the odorant was colored yellow than when it was transparent. The color appeared to have induced an expectation that an odor was being presented and this expectation led to a greater likelihood of the olfactory experience. Dalton (1996) exposed subjects to the odorant ‘isobornyl acetate’ under three different information conditions. One information group was told that they would be exposed to a natural extract of balsam trees. A second group was told they would be exposed to an industrial solvent that could produce health and cognitive problems after longterm use. A control group was simply told that they would be exposed to a standard odorant approved for smell research. The subjects were exposed to the odorant over the course of 20 minutes and were asked to judge its intensity every minute. The results of this study are depicted in Figure 4.7. Although the positive information group (balsam trees) showed a typical adaptation curve to the odorant over time, the negative bias group (industrial solvent) first began to adapt, but then their experience of the odor began to increase in intensity, reaching strong levels by the end of the test. The neutral

50

Very strong Positive bias Negative bias Neutral bias

Perceived intensity

40

Strong 30

20 Moderate 10 Weak Barely detectable 0

5

10

15

20

Exposure duration (minutes)

FIGURE 4.7 Perceived odor intensity of isobornyl acetate for subjects in the positive, negative, and neutral bias conditions during a 20-minute exposure (from Dalton, 1996).

Taste, smell and chemesthesis

115

(control) group produced an intermediate level of adaptation. This study underscores the relevance of survival mechanisms and how our olfactory experiences can be shaped by perceived dangers. Another study showing the importance of information about the source of experienced odors was conducted by Distel and Hudson (2001). In this study odors of everyday products were experienced as more pleasant and more intense when subjects were told the name of the product than when the source of the odor was left unstated. Cognitive elements, perhaps related to the reduction in uncertainty about the odors, appear to have induced changes in the olfactory experiences of them. In a study on perfumes, Scharf and Volkmer (2000) examined the olfactory expectations produced by brand information. They tested two different perfumes at two different concentrations. Consumers first evaluated the fragrances in a blind test for their olfactory experiences of ‘intensity’, ‘sweetness’, ‘freshness’ and ‘floweriness’. In a second session, consumers evaluated images of four different package designs for the perfumes that differed in color (dark red vs. pastel green) and brand (‘Christian Dior elegance’ vs. ‘TIP woman’). Each design was evaluated for the olfactory experiences that would be expected of the product using the same attributes used in the blind test. Lastly, consumers rated combinations of the four fragrances and the four package images on the attributes. Results showed that the olfactory expectations created by the packages influenced the actual olfactory experiences of the perfumes. In some cases, the olfactory experience was altered to be more consistent with the expectation that was created by the packaging. In other cases, the experience was altered to become less like that which was expected. The former finding is commonly referred to as ‘assimilation’, i.e. the experience assimilates (incorporates) the expectation of it. The latter finding is referred to as ‘contrast’. As we shall see in our discussion of taste and liking expectations, ‘assimilation’ and ‘contrast’ are but two of several possible outcomes, yet they seem to form the basis of what appears to be an evolving cognitive model to explain how expectations can influence our product experiences. Effects on taste product experiences One early study of the effects of information on taste experience was conducted by Carlsmith and Aronson (1963). These researchers presented a series of sweet or bitter solutions to consumers who evaluated them for their intensity. Before presenting each, they gave a signal to indicate that either a bitter or sweet stimulus would be presented. In some cases the signal accurately cued the taste solution being presented, but in other cases the cue was misleading, e.g. a signal for a bitter solution was given, but a sweet solution was presented. Their results showed that sweet solutions that disconfirmed a taste expectation were rated less intense than sweet solutions that confirmed a taste expectancy. However, bitter solutions that disconfirmed an expectation were rated more intense. In both cases, disconfirmed expectations resulted in a more negative experience (less sweet or more bitter). They concluded that the data supported Cognitive Dissonance Theory (Festinger, 1957), a psychological theory that proposes that when cognitions (beliefs, values, attitudes, perceptions) or behaviors are inconsistent with one another, a negative psychological state of ‘dissonance’ occurs that makes the individual evaluate the stimulus more negatively. Olson and Dover (1976) performed a similar study on the taste of coffee. These researchers assigned consumers to an information or control condition. In the information condition, consumers were given written information that the coffee had ‘no bitterness’. After reading the information, consumers rated the strength of their belief that the coffee would be either ‘very bitter’, ‘fairly bitter’, ‘somewhat bitter’, or ‘not at all bitter’. Several days later, they returned to taste and evaluate the actual coffee using the same procedure to assess their perceptions of its bitterness. The consumers in the

116

Product Experience

Effect of expectations on product experience (coffee bitterness) 8 7

Control group E

Expectation group

6

Expectation level (E)

5 4 3 2

E

1 E

E

0 Not at all bitter Somewhat bitter Fairly bitter

Very bitter

Bitterness cateogries

FIGURE 4.8 The effects of product expectations on product experience (plot of the data from Olson and Dover, 1976). The bars indicate the mean ratings of the reported ‘belief strength’ of the bitterness of coffee. The arrows indicate the level of pre-trial expectation in the information condition.

control condition merely came to the taste test session and evaluated the coffee. Figure 4.8 shows the results of the study. The arrows indicate the expected belief strength of the information group before tasting the coffee. As can be seen, when compared to the control group, the information group’s experience of the bitterness of the tasted coffee moved in the direction of their expectations of it, supporting an ‘assimilation’ of the expectation into the product experience. Cardello and Sawyer (1992) conducted a study in which consumers were given one of four different types of information about pomegranate juice, i.e. that the juice was ‘very bitter’, had ‘average bitterness’, was ‘not bitter at all’, or no information. The consumers rated their expected experience of the bitterness and other sensory and hedonic attributes of the juice prior to tasting it. The results showed that the information established different expectations for the bitterness of the juice that were consistent with the information. However, the ratings of the bitterness experienced in the tasted product showed both assimilation and contrast effects, similar to the findings of Scharf and Volkmer (2000). Lastly, in a recent study examining the role of information about the area of origin of olive oils (Carporale et al., 2006), users of Italian olive oils were led to believe that a particular oil had been produced in one of three different olive oil producing regions of Italy. Expectations of the bitterness and pungency of the oil were found to differ depending upon the regional information provided about the oil. Moreover, these expectations were assimilated into the ratings of the actual experience of the bitterness and pungency of the oil. Effects on affective product experiences Over the years, a number of studies have also examined the effects of product information on expectations of the liking for foods. Among these have been studies that have examined the effects of nutritional or health content labeling of foods (Tuorila et al., 1994, 1998; Kahkonen and Tuorila, 1998; Kahkonen, et al., 1999; Wansink and Park, 2002). In these studies, product labels such as ‘low-fat’ or ‘contains soy’ lowered user expectations for both the taste and other desirable sensory attributes of the product and, in many cases, resulted in an assimilation of the lowered expectation into the actual liking

Taste, smell and chemesthesis

117

experience (Tuorila et al., 1994; Kahkonen et al., 1996, 1999). Other studies have examined the effects of information about the geographic origins of the product (Stefani et al., 2006; Caporale et al., 2006) or the methods of processing and preserving them (Siret and Issanchou, 2000; Cardello, 2003; Caporale and Monteleone, 2004; Iaccarino et al., 2006). In the majority of these studies product liking has been shown to assimilate the expectation. However, as with sensory expectations, contrast effects have also been observed. Researchers in the area of consumer expectations are now shifting to a hybrid ‘assimilation-contrast model’ to account for the disparate data, with factors related to the magnitude of the discrepancy between expectations and actual product features and the individual’s confidence in his/her sensory abilities determining whether assimilation or contrast effects will occur. The reader is referred to Cardello (2007) for a recent review of this general topic area and for a more detailed discussion of predictive models of the effects of expectations on product liking experiences. Brain mechanisms, expectations and product experience In recent years progress has been made in understanding the neural bases of cognitive effects on taste, smell and other sensory experiences. Much of this progress has been made through the use of fMRI technology that measures blood flow in various regions of the brain. McClure et al. (2004) measured fMRI brain activity in response to the taste of cola beverages (Coke and Pepsi). When a simple visual stimulus of a Coke can (versus a circle of light) was presented before the test stimulus, increased brain activity was observed in several brain areas. The authors concluded that these latter brain areas are involved in the process of ‘biasing perception based on prior affective behavior’. In a more recent study, remarkably similar to the early expectation studies on bitter taste, fMRI was used to investigate the neural mechanisms by which taste expectations can attenuate actual taste perception (Sarinopolous et al., 2006). These investigators provided misleading information to subjects that a taste solution would be mildly bitter. This false information produced neural activity in specific areas of the cortex, including the orbitofrontal cortex (OFC) that predicted decreases in other brain activity in response to the presentation of a very bitter taste solution. The authors concluded that the (neural) network processing aversive taste was suppressed by ‘expectancy-related brain activity’ in specific regions of the cortex, including the OFC. Since the OFC encodes the ‘reward value’ of olfactory and taste stimuli (Rolls, 2000; Hornack et al., 2004), it appears that mere expectations about products activate reward areas in the brain, enabling cognitive-based stimuli to alter, at a neural level, the hedonic experience of the products we taste, eat, smell and drink.

9. AGE, GENDER, CULTURAL AND SOCIAL FACTORS IN CHEMOSENSORY AND PRODUCT EXPERIENCE 9.1. The effects of age As life expectancy rises, the elderly comprise an increasingly larger group of consumers. With advanced age comes declines in sensitivity to taste, smell and chemesthesis (Murphy 1993; Hummel et al., 2003; Boyce and Shone, 2006) and concomitant changes in our experiences of products. Aging takes the largest toll on olfactory experience, with substantial losses in sensitivity to a wide range of odorants. Figure 4.9 shows data from a study by Murphy and Cain (in Murphy, 1986) showing the percent correct identification for a battery of 80 odorants as a function of age. As age increases, there is a steady and progressive decline in olfactory identification. In terms of sensory thresholds, some

118

Product Experience

90

80

Percent correct

70

60

50

40

30

20

10

r ⫽ ⫺0.75

20

30

40

50

60

70

Age (yrs)

FIGURE 4.9 Percent correct identification for a battery of 80 odorants as a function of age (from Murphy and Cain [unpublished] in Murphy, 1986).

evidence suggests that the minimum concentration that subjects can smell increases by two orders of magnitude between the 2nd and 8th decades of life (Wysocki et al., 2003). Declines in taste sensitivity are more modest than those for smell, with age-associated increases in thresholds ranging from 1.3- to 9-fold, depending upon the taste compound studied (Stevens and Cain, 1993; Mojet et al., 2001). Results vary, but in several studies sensitivity to bitter and salty stimuli tends to decrease more than sensitivity to other tastes (reviewed in Mojet et al., 2001). Figure 4.10 from Mojet et al. (2003) shows data on age and gender differences for the five basic tastes. Each taste quality is represented by two taste compounds. Age effects are largest for umami taste, with other differences varying by both taste compound and gender. What is clear, though, is that in all cases, taste responsiveness declines with age. For chemesthesis, declines in sensitivity are also modest, with an age-associated increase in threshold of less than 10-fold for one study (Wysocki et al., 2003), and no significant effect of age in another study (Stuck et al., 2006). Declining chemosensory function poses risks. Declining taste sensitivity could increase the risk of consuming bitter food contaminants and of chronically over-consuming sodium (Stevens and Cain, 1993). Declining olfactory function also might render product warning odors less effective. In one study, 45% of elderly subjects failed to detect the warning odor in natural gas at standard levels (Murphy et al., 2002). Another study found that, relative to normal controls, those with olfactory dysfunction were more likely to miss fires and gas leaks, and more likely to ingest spoiled foods or toxic substances (Santos et al., 2004). These results suggest that concentrations of warning odors in products could require adjustment for elderly consumers. In addition, reduced flavor perception may lead to reduced food intake, reduced dietary variety, and less healthful food choices (Murphy, 1993; Duffy et al., 1995; Rolls 1999). Research suggests that appropriate flavor enhancement can improve consumption and nutritional status among the elderly (Schiffman, 2000; Mathey et al., 2001), but not all manipulations of this kind are effective

Taste, smell and chemesthesis

119

FIGURE 4.10 Perceived intensity of ten taste compounds in water for young (solid lines) and elderly (dotted lines) males (triangles) and females (circles) (from Mojet et al., 2003).

(Koskinen et al., 2003; Essed et al., 2006). Thus, formulation of optimal taste and smell product experiences to improve consumption, nutrition, and safety for the elderly represents both an opportunity and a challenge for the consumer products industry.

9.2. The effects of gender When gender differences occur in simple laboratory tests of sensitivity to taste, smell, and chemesthesis, the differences tend to favor women (Dalton, 2000; Shusterman, 2002; Mojet et al., 2001, 2003; Stuck et al., 2006). However, differences in taste and smell experiences are usually small and often fail to reach statistical significance (Dalton, 2000; Mojet et al., 2001, 2003) (see Figure 4.10 as an example). Differences are somewhat more consistent for chemesthetic experience, but are still moderate and still often fail to

120

Product Experience

reach significance (Wise et al., 2003; Wysocki et al., 2003; Hummel et al., 2003; Stuck et al., 2006). Some evidence suggests that gender differences may increase with age (e.g. Wysocki and Gilbert, 1989; Mojet et al., 2001), but on the whole, individual differences within the sexes are much larger than average differences between them. Although simple tests of sensitivity reveal relatively little difference between men and women, some interesting higher-order effects occur, particularly in olfaction. For example, with repeated testing, women can become progressively more sensitive to some odors (Dalton et al., 2002). Over the course of 30 sessions, sensitivity can increase by several orders of magnitude. However, beyond an initial practice period, men fail to show this sensitization. Interestingly, neither pre-pubescent girls nor post-menopausal women show this sensitization, which suggests that the effect may depend on hormonal function. Fluctuations across the menstrual cycle also suggest that hormonal function might influence olfactory sensitivity in women, though the exact role that hormones play remains unclear (Pause et al., 1996; Navarrete-Palacios et al., 2003). Other studies suggest that women are more likely than men to develop intolerance to certain environmental odors (Kreutzer, 1999; Caress and Steinman, 2004; Johansson et al., 2005). Yet another study found that a pleasant odor could reduce rated pain intensity in women, but not in men (Marchand and Arsenault, 2002). Sex differences in emotional responses to odors appear to play a role here. Regardless, it is clear that chemosensory experiences do differ between men and women. Although it may not always make sense to create gender-specific product formulations, gender differences are worth considering in product development and evaluation.

9.3. The effects of cultural and social factors Cultural factors are, perhaps, the most powerful of factors that can control product experience. In fact, one set of authors has concluded ‘If one were interested in determining as much as possible about an adult’s food preferences and could only ask one question, the question should undoubtedly be: “What is your culture or ethnic group?” ’ (Rozin and Vollmecke, 1986). Insofar as odors are concerned, there are large intercultural differences in odor preferences (Schleidt et al., 1981; Wysocki and Gilbert, 1989; Ayabe-Kanamuura et al., 1998). For example, wintergreen is experienced as a ‘candy’ odor in the US (Cain and Johnson, 1978), but as a ‘medicine’ odor in the UK (Moncrieff, 1966). Also the study cited in Section 5.4 that sought a universally offensive odor for non-lethal crowd control (Dilks et al., 1999) found no single odor to be unpleasant across all ethnic groups. In a recent study on the pleasantness of ‘Japanese’, ‘international’, and ‘European’ odors as experienced by Japanese and German consumers, ‘Japanese’ odors were perceived as much less pleasant by the German consumers than by the Japanese consumers. The ‘international’ odors were generally perceived as pleasant by both groups, and the ‘European’ odors elicited mixed responses (Ayabe-Kanamura et al., 1998). For taste, ethnic, and cultural differences abound. Kimchi is the national food of Korea and is eaten ubiquitously there. However, its strong salty/sour taste is not well accepted in most European and North American countries. Nor are the super-tart umeboshi pickled plums that are eaten in Japan. These and other exotic taste and olfactory experiences that one encounters in Asian, Indian and other ethnic markets are product experiences that few Americans or Europeans fully appreciate. Thus, it appears that the food habits and olfactory experiences of different ethnic groups can influence future sensory and affective preferences for chemosensory stimuli and products. Even within the same country, differences in product experiences are commonplace. Take, for example, the preference for brown eggs over white eggs in the Northeastern

Taste, smell and chemesthesis

121

United States and the preference for ‘grits’ (coarsely ground, hulled grain) in the Southern US, preferences not seen elsewhere in the U.S. Rozin (1996) has argued that some product avoidances, e.g. those for bean curd or pork kidneys among Midwesterners in the US, are due to people not viewing them as food. Thus, cultural factors can even define what items are considered to be foods and which are not. Religious and moral taboos provide the most dramatic examples of cultural effects on product experience. The dietary avoidance of pork by Jews and Muslims, beef in India, and eggs among certain African tribes (Simoons, 1961) are powerful examples of religious and cultural beliefs influencing taste, smell, and product preferences. The moral and cultural differences that make horse, cat, and dog consumption unheard of in North America, but a common practice in certain other countries is another example. Interestingly, the peoples who disdain these and other food products for cultural or religious reasons often consider them ‘unclean’ or ‘disgusting’; yet those who are not part of that culture view them to be quite normal for consumption. Disgust has also been found to be a major reason for morally motivated vegetarians to avoid all meat products (Rozin et al., 1996). Socioeconomic factors, such as status in society and income level can also affect our product experiences. In an early study of smell preferences, Brill (1932) reported that among 43 common odors the most disliked was that of perspiration, which he said was due to its ‘association with people of the lower class’. Perfumes, on the other hand, are often associated with royalty and wealth, while incense is associated with religious piety and sacredness (Largey and Watson, 2006). In the case of taste, witness the expensive, gourmet and health food tastes of today’s urban socialites that makes drinking a non-specialty shop coffee an experience to be avoided. And compare that to the oftenreported consumption of back alley restaurant waste by the homeless in our society. Clearly, the different socioeconomic situations in which these individuals find themselves drive their product experiences, defining for them what is a good or acceptable product experience and what is a bad or unacceptable product experience. Lastly, when talking about product experiences in today’s marketplace, one must, of course, mention the important social influences of role models. Whether it is a famous Hollywood actress pitching her own brand of perfume, a handsome actor drinking the latest introduction into the beer market, or the soccer superstar eating his favorite candy bar, role models are the essence of product advertising. A variety of role model effects on taste and smell have been reported in children, where elders, teachers, and peers have been used to increase liking for specific foods (see Rozin, 1988, for a review). It has also been shown that father–son and mother–daughter food preferences are more similar than father–daughter or mother–son preferences (Pliner, 1983). However while taste preferences begin early in life, systematic data on the role of parental influences in forming taste preferences in children is weak (Rozin, 1996). In addition to experimental studies, there is a large case study literature in sociology and anthropology describing the variety of social and cultural differences in taste and smell. The reader is referred to recent anthologies and books by Korsmeyer, (1999, 2005), Watson and Caldwell (2005) and Drobnick (2006) for fascinating discussions of the role of culture in chemosensory experience.

10. CONCLUSION In this chapter, we have discussed the physiological, psychological, cognitive and sociocultural factors that affect the taste, smell, chemesthetic, and liking experiences of products in today’s market. Although our treatment of topics has relied heavily on the

122

Product Experience

understanding of the fundamental mechanisms of human taste and smell experience, we feel that only through a thorough understanding of these underlying principles can we begin to understand the complex problem of product experience. It is our opinion that the future of product development in the food, perfume, and personal care industries lies in a combination of breakthroughs in the genetics of taste, smell, and chemesthesis and in a better understanding of the psychological, cultural, and societal factors that form the bases of our everyday product experiences.

REFERENCES Algom, D., Marks, L. E. and Cain, W. (1993). Memory psychophysics for chemosensation: perceptual and mental mixtures of odor and taste. Chemical Senses, 18, 151–160. Allison, A. and Chambers, D. H. (2005). Effects of residual toothpaste flavor on flavor profiles of common foods and beverages. Journal of Sensory Studies, 20, 167–186. Anderson, A. K., Christoff, K., Stappen, I., et al. (2003). Dissociated neural representations of intensity and valence in human olfaction. Nature Neuroscience, 6(2), 196–202. Atanasova, B., Thomas-Danguin T., Chabanet, C., Langlois, D., Nicklaus, S. and Etiévant, P. (2005). Perceptual interactions in odour mixtures: Odour quality in binary mixtures of woody and fruity wine odorants. Chemical Senses, 30, 209–217. Ayabe-Kanamuura, S., Schicker, I., Laska, M., et al. (1998). Differences in perception of everyday odors: A Japanese–German cross-cultural study. Chemical Senses, 23, 31–38. Ayya, N. and Lawless, H. T. (1992). Quantitative and qualitative evaluation of high-intensity sweeteners and sweetener mixtures. Chemical Senses, 17, 245–259. Baeyens, F. and Wrzesniewksi, A. (1996). Toilet rooms, body massages, and smells: Two field studies on human evaluative odor conditioning. Current Psychology: Developmental, Learning, Personality, Social, 15, 77–96. Bartoshuk, L. M. (2000). Comparing sensory experiences across individuals: Recent psychophysical advances illuminate genetic variation in taste perception. Chemical Senses, 25, 447–460. Bartoshuk, L. M., McBurney, D. H. and Pfaffmann, C. (1964). Taste of sodium chloride solutions after adaptation to sodium chloride: Implications for the ‘water taste’. Science, 143(3609), 967–968. Bartoshuk, L. M. and Wolfe, J. M. (1990). Conditioned taste aversions in humans: Are they olfactory aversions? Chemical Senses, 15, 551. Baylis, L. L., Rolls, E. T. and Baylis, G. C. (1995). Afferent connections of the caudolateral orbitofrontal cortex taste area of the primate. Neuroscience, 64, 801–812. Beauchamp, G. K. and Cowart, B. J. (1993). Development of salt taste responses in human newborns, infants and children. In: NIH Publication No. 93-3483, 3. Development, Growth and Senescence in the Chemical Senses, pp. 61–68. Washington, DC: U.S. Department of Health and Human Services. Beauchamp, G. K., Cowart, B. J. and Moran, M. (1986). Developmental changes in salt acceptability in human infants. Developmental Psychology, 19, 75–83. Bell, K. I. and Tepper, B. J. (2006). Short-term vegetable intake by young children classified by 6-n-propylthiouracil bitter-taste phenotype. American Journal of Clinical Nutrition, 84(1), 245–251. Bernstein, I. L. and Webster, M. M. (1980). Learned taste aversions in humans. Physiology and Behavior, 25, 363–366. Birch, L. L. and Marlin, D. W. (1982). I don’t like it; I never tried it: Effects of exposure on two-year old children’s food preferences. Appetite, 3, 353–360. Bonnans, S. and Noble, A. C. (1993). Effect of sweetener type and of sweetener and acid levels on temporal perception of sweetness, sourness and fruitiness. Chemical Senses, 18(3), 273–283. Booth, D. A. (1981). The physiology of appetite. British Medical Bulletin, 37, 135–140. Boyce, J. M. and Shone, G. R. (2006). Effects of ageing on smell and taste. Postgraduate Medical Journal, 82(966), 239–241. Breslin, P. A. S. (1996). Interactions among salty, sour and bitter compounds. Trends in Food Science and Technology, 7, 390–399. Breslin, P. A. S., Davidson, T. L. and Grill, H. J. (1990). Conditioned reversal of reactions to normally avoided tastes. Physiology and Behavior, 47, 535–538. Breslin, P. A. S. and Beauchamp, G. K. (1995). Suppression of bitterness by sodium: variation among bitter taste stimuli. Chemical Senses, 20, 609–623.

Taste, smell and chemesthesis

123

Breslin, P. A. and Huang, L. Q. (2006). Human taste: Peripheral anatomy, taste transduction, and coding. In: T. Hummel and A. Welge-Lüssen (Eds.) Taste and smell: An update (Advances in Oto-rhino-laryngology) Vol. 63, pp. 152–190. Basel, Germany: Karger. Brill, A. A. (1932). The sense of smell in the neuroses and psychoses. Psychoanalytic Quarterly, 1 (Spring), 7–42. Buck, L. B. (1996). Information coding in the vertebrate olfactory system. Annual Review of Neuroscience, 19, 517–544. Buck, L. B. (2000). The molecular architecture of odor and pheromone sensing in mammals. Cell, 100, 611–618. Bufe, B., Breslin, P. A., Kuhn, C., et al. (2005). The molecular basis of individual differences in phenylthiocarbamide and propylthiouracil bitterness perception. Current Biology, 15, 322–327. Burgard, D. R. and Kuznicki, J. T. (1990). Chemometrics: chemical and sensory data. Boca Raton, FL: CRC Press. Cabanac, M. (1971). Physiological role of pleasure. Science, 173, 1103–1107. Cain, W. S. (1974). Perception of odor intensity and the time-course of olfactory adaptation. ASHRAE Transactions, 80, 53–75. Cain, W. S. and Johnson, F. (1978). Lability of odor pleasantness: influence of mere exposure. Perception, 7, 459–465. Cain, W. S. and Murphy, C. (1980). Interaction between chemoreceptive modalities of odour and irritation. Nature, 284, 255–257. Cain, W. S. and Polak, E. H. (1992). Olfactory adaptation as an aspect of odor similarity. Chemical Senses, 17, 481–491. Cain, W. S., Schiet, F. T., Olsson, M. J. and De Wijk, R. A. (1995). Comparison of models of odor interaction. Chemical Senses, 20, 625–637. Cain, W. S., See, L. C. and Tosun, T. (1986). Irritation and odor from formaldehyde: Chamber studies. In: IAQ ’86: Managing the Indoor Air for Health and Energy Conservation, pp. 126–137. The American Society of Heating and Refrigerating and Air-Conditioning Engineers, Atlanta. Cairncross, S. E. and Sjostrom, L. B. (1950). Flavor profiles – a new approach to flavor problems. Food Technology, 4, 308–311. Capaldi, E. D. (1996). Conditioned food preferences. In: E. D. Capaldi (Ed.) Why we eat what we eat: the psychology of eating, pp. 53–80. Washington, DC: American Psychological Association. Capaldi, E. D., Campbell, D. H., Sheffer, J. D. and Bradford, J. P. (1987). Conditioned flavor preferences based on delayed caloric consequences. Journal of Experimental Psychology: Animal Behavior Processes, 13, 150–155. Capaldi, E. D. and Hunter, M. J. (1994). Taste and odor in conditioned flavor preference learning. Animal Learning and Behavior, 22, 355–365. Caporale, G. and Monteleone, E. (2004). Influence of information about manufacturing process on beer acceptability. Food Quality and Preference, 15, 271–278. Caporale, G., Policastro, S., Carlucci, A. and Monteleone, E. (2006). Consumer expectations for sensory properties in virgin olive oils. Food Quality and Preference, 17, 116–12. Cardello, A. V. (1994). Consumer expectations and their role in food acceptance. In: H. J. MacFie and D. M. H. Thomson (Eds.) Measurement of food preferences, pp. 253–297. London: Blackie Academic. Cardello, A. V. (2003). Consumer concerns and expectations about novel food processing technologies: effects on product liking. Appetite, 40, 217–233. Cardello, A. V. (2007). Measuring consumer expectations to improve product development. In: H. MacFie (Ed.) Consumer-led Food Product Development. Cambridge: Woodhead, 223–261. Cardello, A. V. and Sawyer, F. M. (1992). Effects of disconfirmed consumer expectations on food acceptability. Journal of Sensory Studies, 7, 253–277. Cardello, A. V. and Schutz, H. G. (2004). Numerical scale-point locations for constructing the LAM (labeled affective magnitude) scale. Journal of Sensory Studies 19, 341–346. Caress, S. M. and Steinemann, A. C. (2004). Prevalence of multiple chemical sensitivities: A population-based study in the southeastern United States. American Journal of Public Health, 94(5), 746–747. Carlsmith, J. M. and Aronson E. (1963). Some hedonic consequences of the confirmation and disconfirmation of expectancies. Journal of Abnormal and Social Psychology, 66, 151–156. Carmichael, S. T. and Price, J. L. (1995). Limbic connections of the orbital and medial prefrontal cortex in macaque monkeys. Journal of Comparative Neurology, 363, 615–641. Caterina, M. J. and Julius, D. (2001). The vanilloid receptor: A molecular gateway to the pain pathway. Annual Reviews in Neuroscience, 24, 487–517. Cartier, R., Rytz, A., Lecomte, A., Poblete, F., Krystlik, J., Belin, E. and Martin, N. (2006). Sorting procedure as an alternative to quantitative descriptive analysis to obtain a product sensory map. Food Quality and Preference, 17, 562–571. Cerf-Ducastel, B., Van de Moortele, P. F., MacLeod, P., Le Bihan, D. and Faurion, A. (2001). Interaction of gustatory and lingual somatosensory perceptions at the cortical level in the human: a functional magnetic resonance imaging study. Chemical Senses, 26, 371–383.

124

Product Experience

Chambers, K. C. and Bernstein, I. L. (1995). Conditioned flavor aversions. In: R. L. Doty (Ed.) Handbook of olfaction and gestation, pp. 745–773. New York: Dekker. Christensen, C. M. (1980). Effects of taste quality and intensity on oral perception of viscosity. Perception and Psychophysics, 28(4), 315–320. Cometto-Muñiz, J. E. and Cain, W. S. (1984). Temporal integration of pungency. Chemical Senses, 8, 315–327. Cometto-Muñiz, J. E. and Cain, W. S. (1995). Olfactory adaptation. In: R. L. Doty (Ed.) Handbook of olfaction and gestation, pp. 257–282. New York: Marcel Dekker. Cometto-Muñiz, J. E., Cain, W. S. and Abraham, M. H. (2004). Chemosensory additivity in trigeminal chemoreception as reflected by detection of mixtures. Experimental Brain Research, 158, 196–206. Cometto-Muñiz, J. E., Cain, W. S. and Abraham, M. H. (2005). Odor detection of single chemicals and binary mixtures. Behavioural Brain Research, 156, 115–123. Cometto-Muñiz, J. E. and Hernandez, S. M. (1990). Odorous and pungent attributes of mixed and unmixed odorants. Perception and Psychophysics, 47, 391–399. Cowart, B. J. and Beauchamp, G. K. (1990). Early development of taste perception. In: R. L. McBride and H. J. H. MacFie (Eds.) Psychological Basis of Sensory Evaluation, pp. 1–17. London: Elsevier Applied Science. Cutler, W. B., Preti, G., Krieger, A. M., et al. (1986). Human axillary secretions influence women’s menstrual cycles: The role of donor extract from men. Hormones and Behavior, 20, 463–473. Dalton, P. (1996). Odor perception and beliefs about risk. Chemical Senses, 21, 447–458. Dalton, P. (2000). Psychophysical and behavioral characteristics of olfactory adaptation. Chemical Senses, 25, 487–492. Dalton, P., Cowart, B., Dilks, D., et al. (2003). Olfactory function in workers exposed to styrene in the reinforced-plastics industry. American Journal of Industrial Medicine, 44(1), 1–11. Dalton, P., Doolittle, N. and Breslin, P. A. (2002). Gender-specific induction of enhanced sensitivity to odors. Nature Neuroscience, 5(3), 199–200. Dalton, P., Doolittle, N., Nagata, H. and Breslin, P. A. (2000). The merging of the senses: Integration of subthreshold taste and smell. Nature Neuroscience, 3, 431–432. Dalton, P. and Wysocki, C. J. (1996). The nature and duration of adaptation following long-term odor exposure. Perception and Psychophysics, 58(5), 781–792. Davis, L. B. and Porter, R. H. (1991). Persistent effects of early odor exposure on human neonates. Chemical Senses, 16(2), 169–174. Deliza R. and MacFie, H. J. H. (1996). The generation of sensory expectations by external cues and its effect on sensory perception. Journal of Sensory Studies, 11, 103–128. Delwiche, J. F. and Heffelfinger, A.L. (2005). Cross-modal additivity of taste and smell. Journal of Sensory Studies, 20, 512–525. Desor, J. A., Maller, O. and Andrews, K. (1975). Ingestive responses of human newborns to salts, sour and bitter stimuli. Journal of Comparative and Physiological Psychology, 89, 966–970. Desor, J. A., Maller, O. and Turner, R. E. (1973). Taste in acceptance of sugars by human infants. Journal of Comparative and Physiological Psychology, 84, 496–501. Diamond, J., Breslin, P. A. S., Doolittle, N., Nagata, H. and Dalton, P. (2005). Flavor processing: Perceptual and cognitive factors in multi-modal integration. Chemical Senses, 30 (Suppl. 1), i232–i233. Dilks, D. D., Dalton, P. and Beauchamp, G. K. (1999). Cross-cultural variation in responses to malodors. Chemical Senses, 24, 599. Dinehart, M. E., Hayes, J. E., Bartoshuk, L. M., Lanier, S. L. and Duffy, V. B. (2006). Bitter taste markers explain variability in vegetable sweetness, bitterness, and intake. Physiology and Behavior, 87(2), 304–313. Distel, H. and Hudson, R. (2001). Judgement of odor intensity is influenced by subjects’ knowledge of the odor source. Chemical Senses, 26, 247–251. Djordjevic, J., Zatorre, R. J. and Jones-Gotman, M. (2004). Effects of perceived and imagined odors on taste detection. Chemical Senses, 29, 199–208. Doty, R. L. and Cometto-Muñiz, J. E. (2003). Trigeminal chemosensation. In: R. L. Doty (Ed.) Handbook of olfaction and gustation (2nd Ed.), pp. 981–999. New York: Marcel Decker. Drewnoski, A., Grinker, J. A. and Hirsh, J. (1982). Obesity and flavor perception: Multidimensional scaling of soft drinks. Appetite, 3, 361–368. Drobnick, J. (2006). The smell culture reader. New York: Oxford. Duffy, V. B., Backstrand, J. R. and Ferris, A. M. (1995). Olfactory dysfunction and related nutritional risk in free-living, elderly women. Journal of the American Dietetic Association, 95(8), 879–884. Duffy, V. B., Peterson, J. M. and Bartoshuk, L. M. (2004). Associations between taste genetics, oral sensation and alcohol intake. Physiology and Behavior, 82(2–3), 435–445. Engen, T. (1972). The effect of expectation on judgments of odor. Acta Psychologica, 35, 450–458.

Taste, smell and chemesthesis

125

Engen, T. (1982). The perception of odors. New York: Academic. Engen, T. (1988) The acquisition of odour hedonics. In: S. Van Toller and G. H. Dodd (Eds.) Perfumery: The psychology and biology of fragrance, pp. 79–90. London: Chapman and Hall. Essed, N. H., Van Staveren, W. A., Kok, F. J. and De Graaf, C. (2007). No effect of 16 weeks flavor enhancement on dietary intake and nutritional status of nursing home elderly. Appetite, 48, 29–36. Faye, P., Bremaud, D., Teillet, E., Courcoux, P., Giboreau, A. and Nicod, H. (2006). An alternative to external preference mapping based on consumer perceptive mapping. Food Quality and Preference, 17, 604–614. Festinger, L. (1957). A theory of cognitive dissonance. Evanston, IL: Row and Peterson. Frank, R. A. (2002). Response context affects judgments of flavor components in foods and beverages. Food Quality and Preference, 14, 139–145. Frank, R. A. and Byram, J. (1988). Taste–smell interactions are tastant and odorant dependent. Chemical Senses, 13, 445–455. Frank, R. A., Van der Klaauw, N. J. and Schifferstein, H. N. J. (1993). Both perceptual and conceptual factors influence taste–odor and taste–taste interactions. Perception and Psychophysics, 54, 343–350. Ganchrow, J. R. (1995). Ontogeny of human taste perception. In: R. L. Doty (Ed.) Handbook of olfaction and gustation, pp. 715–772. New York: Dekker. Garb, J. L. and Stunkard, A. (1974). Taste aversions in man. American Journal of Psychiatry, 131, 1204–1207. Gibson, J. J. (1966). The senses considered as perceptual systems. Boston: Houghton Mifflin. Gilmore, M. M. and Green, B. G. (1993). Sensory irritation and taste produced by NaCl and citric acid: effects of capsaicin desensitization. Chemical Senses, 18(3), 257–272. Glusman, G., Yanai, I., Rubin, I. and Lancet, D. (2001). The complete human olfactory subgenome. Genome Research, 11, 685–702. Gottfried, J. A. (2006). Smell: Central nervous processing. In: T. Hummel and A. Welge-Lüssen (Eds.) Taste and smell: An update (Advances in Oto-rhino-laryngology), pp. 44–69. Basel, Germany: Karger. Gottfried, J. A., O’Doherty, J. and Dolan, R. J. (2003). Encoding predictive reward value in human amygdala and orbitofrontal cortex. Science, 301, 1104–1107. Green, B. G. (1990). Effects of thermal, mechanical, and chemical stimulation on the perception of oral irritation. In: Green, B. G., Mason, J. R. and Kare, M. R. (Eds.) Chemical senses, Vol. 2, Irritation, pp. 171–175. New York: Marcel Dekker. Green, B. G., Shaffer, G. S. and Gilmore, M. M. (1993). Derivation and evaluation of a semantic scale of oral sensation magnitude with apparent ratio properties. Chemical Senses, 18, 683–702. Guinard, J., Yip, D., Cubero, E. and Mazzucchelli, R. (1999). Quality ratings by experts, and relation with descriptive analysis ratings: a case study with beer. Food Quality and Preference, 10, 59–67. Hallock, R. M. and Di Lorenzo, P. M. (2006). Temporal coding in the gustatory system. Neuroscience and Biobehavioral Reviews, 30(8), 1145–1160. Harris, G. and Booth, D. A. (1987). Infants’ preference for salt in food: Its dependence upon recent dietary experience. Journal of Reproductive and Infant Psychology, 5, 97–104. Helson, H. (1964). Adaptation-level theory: An experimental and systematic approach to behavior. New York: Harper. Hendry, S. H. C., Hsiao, S. S. and Bushnell, M. C. (1999). Somatic sensation. In: M. J. Zigmond, F. E. Bloom, S.C., et al. (Eds.) Fundamental neuroscience, pp. 761–789. San Diego: Academic Press. Herz, R. S. (1997). Emotion experienced during encoding enhances odor retrieval cue effectiveness. American Journal of Psychology, 110, 489–505. Herz, R. S. (2003). The effect of verbal context in olfactory perception. Journal of Experimental Psychology: General, 132, 595–606. Herz, R. S., Eliassen J., Beland S. and Souza T. (2004a). Neuroimaging evidence for the emotional potency of odor-evoked memory. Neuropsychologia, 42, 371–378. Herz, R. S., Beland, S. and Hellerstein, M. (2004b). Changing odor hedonic perception through emotional associations in humans. International Journal of Comparative Psychology, 17, 315–339. Herz, R. S. and Von Clef, J. (2001). The influence of verbal labeling on the perception of odors: Evidence for olfactory illusions? Perception, 30, 381–391. Hetherington, M. M. and Rolls, B. J. (1996). Sensory-specific satiety: theoretical frameworks and central characteristics. In: E. Capaldi (Ed.) Why we eat what we eat: The psychology of eating, pp. 267–290. Washington, DC: American Psychological Association. Hildebrand, J. G. and Shepherd, G. M. (1997). Mechanisms of olfactory discrimination: Converging evidence for common principles across phyla. Annual Review of Neuroscience, 20, 595–631. Hornak J., O’Doherty J., Bramham J., Rolls E.T., et al. (2004). Reward-related reversal learning after surgical excisions in orbito-frontal or dorsolateral prefrontal cortex in humans. Journal of Cognitive Neuroscience, 16(3), 463–478.

126

Product Experience

Huang, A. L., Chen, X., Hoon, M. A., et al. (2006). The cells and logic for mammalian sour taste detection. Nature, 442, 934–938. Hummel, T., Futschik, T., Frasnelli, J. and Huttenbrink, K. B. (2003). Effects of olfactory function, age, and gender on trigeminally mediated sensations: A study based on the lateralization of chemosensory stimuli. Toxicology Letters, 140–141, 273–280. Iaccarino T., DiMonaco R., Mincione A., Cavella S. and Masi P. (2006). Influence of information on origin and technology on the consumer response: The case of soppressata salami. Food Quality and Preference, 17, 76–84. Jacob, T. J., Fraser, C., Wang, L., Walker, V. and O’Connor, S. (2003). Psychophysical evaluation of responses to pleasant and mal-odour stimulation in human subjects; adaptation, dose response and gender differences. International Journal of Psychophysiology, 48, 67–80. Jacquot, L., Monnin, J., Lucarz A. and Brand, G. (2005). Trigeminal sensitization and desensitization in the nasal cavity: a study of cross interactions. Rhinology, 43, 93–98. Jaeger, S. R. (2006). Non-sensory factors in sensory science research. Food Quality and Preference, 17, 132–144. Jaeger, S. R., Rossiter, K. L. and Lau, K. (2005). Consumer perceptions of novel fruit and familiar fruit: A repertory of grid application. Journal of the Science of Food and Agriculture, 85(3), 480–488. Jinks, A. and Laing, D. G. (2001). The analysis of odor mixtures by humans: evidence for a configurational process. Physiology Behavior, 72, 51–63. Johansson, A., Bramerson, A., Millqvist, E., Nordin, S. and Bende, M. (2005). Prevalence and risk factors for self-reported odour intolerance: the Skovde population-based study. International Archives of Occupational and Environmental Health, 78(7), 559–564. Johnson, J. L. and Vickers, Z. (1991). Sensory-specific satiety for selected bread products. Journal of Sensory Studies, 6(2), 65–79. Jones, L. V. and Thurstone, L. L. (1955). The psychophysics of semantics: an experimental investigation. Journal of Applied Psychology, 39(1), 31–36. Kadohisa, M. and Wilson, D. A. (2006). Olfactory cortical adaptation facilitates detection of odors against background. Journal of Neurophysiology, 95(3), 1888–1896. Kahkonen P., Hakanpaa P. and Tourila H. (1999). The effects of information related to fat content and taste on consumer responses to a reduced-fat frankfurter and a reduced-fat chocolate bar. Journal of Sensory Studies, 14, 35–46. Kahkonen P. and Tuorila H. (1998). Effect of reduced-fat information on expected and actual hedonic and sensory ratings of sausage. Appetite, 30, 13–23. Kahkonen P., Tuorila H. and Rita, H. (1996). How information enhances the acceptability of low-fat spread. Food Quality Preference, 7, 87–94. Keiger, C. J., Case, L. D., Kendal-Reed, M., et al. (2003). Nicotinic cholinergic receptor expression in the human nasal mucosa. Annals of Otology, Rhinology and Laryngology, 112(1), 77–84. Kepecs, A., Uchida, N. and Mainen, Z. F. (2006). The sniff as a unit of olfactory processing. Chemical Senses, 31, 167–179. Korsmeyer, C. (1999). Making sense of taste, food and philosophy. Ithaca: Cornell University Press. Korsmeyer, C. (2005). The taste culture reader, experiencing food and drink. Oxford: Berg. Koskinen, S., Kalviainen, N. and Tuorila, H. (2003). Flavor enhancement as a tool for increasing pleasantness and intake of a snack product among the elderly. Appetite, 41(1), 87–96. Kreutzer, R., Neutra, R. R., and Lashuay, N. (1999). Prevalence of people reporting sensitivities to chemicals in a population-based survey. American Journal of Epidemiology, 150, 1–12. Kroeze, J. H. A. (1990). The perception of complex taste stimuli. In: R. L. McBride and H. J. H. MacFie (Eds.) Psychological Basis of Sensory Evaluation, pp. 41–68. London: Elsevier Applied Science. Kroeze, J. H. A. and Bartoshuk, L. M. (1985). Bitterness suppression as revealed by split-tongue taste stimulation in humans. Physiology and Behavior, 35, 779–783. Labbe, D., Damevin, L., Vaccher, C., Morgenegg, C. and Martin, N. (2006). Modulation of perceived taste by olfaction in familiar and unfamiliar beverages. Food Quality and Preference, 17, 582–589. Labows, J. N. and Wysocki, C. J. (1984). Individual differences in odor perception. Perfumer and Flavorist, 9, 21–26. Laing, D. G., Panhuber, H., Wilcox, M. E. and Pittman, E. A. (1984). Quality and intensity of binary odor mixtures. Physiology and Behavior, 33, 309–319. Langron, S. P., Noble, A. C. and Williams, A. A. (1983). Measurement and Control. In: R. K. Atkin and A. A. Williams (Eds.) Sensory quality in foods and beverages: Definition, p. 325. Chichester: Ellis Horwood. Lanier, S. A., Hayes, J. E. and Duffy, V. B. (2005). Sweet and bitter tastes of alcoholic beverages mediate alcohol intake in of-age undergraduates. Physiology and Behavior, 83(5), 821–831. Largey, G. and Watson, R. (2006). The sociology of odors. In: J. Drobnick (Ed.) The smell culture reader, pp. 29–40. Oxford: Berg.

Taste, smell and chemesthesis

127

Laska, M., Hudson, R. and Distel, H. (1990). Olfactory sensitivity to biologically relevant odors may exceed the sum of component thresholds. Chemoecology, 1, 139–141. Lawless, H. T. (1979). Evidence for neural inhibition in bittersweet taste mixtures. Journal of Comparative and Physiological Psychology, 93, 538–547. Lawless, H. T. (1983). Contextual effects in category ratings. Journal of Testing and Evaluation, 11, 346–349. Lawless, H. T. (1987). Gustatory psychophysics. In: T. E. Finger and W. L. Silver (Eds.) Neurobiology of taste and smell, pp. 401–420. New York: John Wiley. Lawless, H. T. (1997). Olfactory psychophysics. In: G. Beauchamp and L. Bartoshuk (Eds.) Tasting and smelling, pp. 125–168. San Diego, CA: Academic Press. Lawless, H. T. and Heymann, H. (1998). Sensory evaluation of food: Principles and practices. New York: Chapman and Hall. Lawless, H. T., Rozin, P. and Shenker, J. (1985). Effects of oral capsaicin on gustatory, olfactory and irritant sensations and flavor identification in humans who regularly or rarely consume chilli pepper. Chemical Senses, 10, 579–589. Lawless, H. T. and Stevens, D. A. (1984). Effects of oral chemical irritation on taste. Physiology and Behavior, 32, 995–998. Lawless, H. T., Sheng, N. and Knoops, S. S. C. P. (1995). Multidimensional scaling of sorting data applied to cheese perception. Food Quality and Preference, 6, 91–98. Lemon, C. H. and Smith, D. V. (2006). Influence of response variability on the coding performance of central gustatory neurons. Journal of Neuroscience, 26(28), 7433–7443. Li, X., Li, W., Wang, H., et al. (2005). Pseudogenization of a sweet-receptor gene accounts for cats’ indifference toward sugar. PLoS Genetics, 1, 27–35. Liberles, S. D. and Buck, L. B. (2006). A second class of chemosensory receptors in the olfactory epithelium. Nature, 442(7103), 645–650. Lindemann, B. (2001). Receptors and transduction in taste. Nature, 413, 219–225. Liu, L. and Simon, S. A. (2001). Acidic stimuli activate two distinct pathways in taste receptor cells from rat fungiform papillae. Brain Research, 923, 58–70. Livermore, A. and Laing, D. G. (1996). Influence of training and experience on the perception of multicomponent odor mixtures. Journal of Experimental Psychology: Human Perception and Performance, 22, 267–277. Lledo, P. M., Gheusi, G. and Vincent, J. D. (2005). Information processing in the mammalian olfactory system. Physiological Reviews, 85, 281–317. Mainland, J. and Sobel, N. (2006). The sniff is part of the olfactory percept. Chemical Senses, 31, 181–196. Marchand, S. and Arsenault, P. (2002). Odors modulate pain perception: A gender-specific effect. Physiology and Behavior, 76(2), 251–256. Marshall, K., Laing, D.G., Jinks, A. L. and Hutchinson, I. (2006). The capacity of humans to identify components in complex odor-taste mixtures. Chemical Senses, 31, 539–545. Mathey, M. F., Siebelink, E., De Graaf, C. and Van Staveren, W. A. (2001). Flavor enhancement of food improves dietary intake and nutritional status of elderly nursing home residents. Journals of Gerontology Series A: Biological Sciences and Medical Sciences, 56(4), M200–M205. Matlin, M. W. and Foley, H. J. (1992). Sensation and perception. Boston: Allyn and Bacon. Mattes, R. D. (1987). Sensory influences on food intake and utilization in humans. Human Nutrition: Applied Nutrition, 41A, 77–95. McBride, R. L. (1985). Stimulus range influences intensity and hedonic ratings of flavour. Appetite, 6, 125–131. McBride, R. L. and Johnson, R. L. (1987). Perception of sugar-acid mixtures in lemon juice drink. International Journal of Food Science and Technology, 22, 299–408. McBurney, D. H. and Shick, T. R. (1971). Taste and water taste of twenty-six compounds for man. Perception and Psychophysics, 10, 249–252. McClure, S. M., Li, J., Tomli, D., Cypert, K. S., Montagu, L. M. and Montague, P. R. (2004). Neural correlates of behavioral preference for culturally familiar drinks. Neuron, 44, 379–387. McDaniel, A. H. and Reed, D. R. (2004). The human sweet tooth and its relationship to obesity. In: N. Moustaid-Moussa and C. Berdanier (Eds.) Genomics and proteomics in nutrition, pp. 49–67. New York: Marcel Dekker. McKemy, D. D. (2005). How cold is it? TRPM8 and TRPA1 in the molecular logic of cold sensation. Molecular Pain, 1, 1–7. Meilgaard, M., Civille, G. V. and Carr, B. T. (1999). Sensory evaluation techniques. Boca Raton, Fl: CRC Press. Mennella, J. A., Johnson, A. and Beauchamp, G. K. (1995). Garlic ingestion by pregnant women alters the odor of amniotic fluid. Chemical Senses, 20, 207–209. Mennella, J. A., Pepino, M. Y. and Reed, D. R. (2005). Genetic and environmental determinants of bitter perception and sweet preferences. Pediatrics, 115(2), e216–e222.

128

Product Experience

Midkiff, E. E. and Bernstein, I. L. (1985). Targets of learned food aversions in humans. Physiology and Behavior, 34, 839–841. Mojet, J., Christ-Hazelhof, E. and Heidema, J. (2001). Taste perception with age: Generic or specific losses in threshold sensitivity to the five basic tastes? Chemical Senses, 26, 845–860. Mojet, J., Heidema, J. and Christ-Hazelhof, E. (2003). Taste perception with age: Generic or specific losses in supra-threshold intensities to the five taste qualities? Chemical Senses, 28, 397–413. Mombaerts, P. (2004). Genes and ligands for odorant, vomeronasal, and taste receptors. Nature Reviews: Neuroscience, 5, 263–278. Moncreiff, R. W. (1966). Odour preferences. New York: Wiley. Moskowitz, H. R. and Barbe, C. D. (1977). Profiling of odor components and their mixtures. Sensory Processes, 1, 212–226. Moskowitz, H. R. and Sidel, J. L. (1971). Magnitude and hedonic scales of food acceptability. Journal of Food Science, 36, 677–680. Mueller, K. L., Hoon, M. A., Erlenbach, I., et al. (2005). The receptors and coding logic for bitter taste. Nature, 434, 225–229. Murphy, C. (1982). Effects of exposure and context on hedonics of olfactory–taste mixtures. Sensory Processes, 1, 212–226. Murphy, C. (1986). Taste and smell in the elderly. In: H. L. Meiselman and R. S. Rivlin (Eds.) Clinical measurement of taste and smell, pp. 343–375. New York: Macmillan. Murphy, C. (1993). Nutrition and chemosensory perception in the elderly. Critical Reviews in Food Science and Nutrition, 33(1), 3–15. Murphy, C. and Cain, W. S. (1980). Taste and olfaction: Independence versus interaction. Physiology Behavior, 24, 601–605. Murphy, C., Schubert, C. R., Cruickshanks, K. J., et al. (2002). Prevalence of olfactory impairment in older adults. Journal of the American Medical Association, 288, 2307–2312. Nasrawi, C.W. and Pangborn, R.M. (1989). The influence of tastants on oral irritation by capsaicin. Journal of Sensory Studies, 3, 287–294. Navarrete-Palacios, E., Hudson, R., Reyes-Guerrero, G. and Guevara-Guzman, R. (2003). Lower olfactory threshold during the ovulatory phase of the menstrual cycle. Biological Psychology, 63(3), 269–279. O’Doherty, J., Rolls, E. T., Francis, S., et al. (2000). Sensory-specific satiety-related olfactory activation of the human orbitofrontal cortex. Neuroreport, 11, 399–403. Olson J.C. and Dover P. (1976). Effects of expectation creation and disconfirmation on belief elements of cognitive structure. In: B. B. Anderson (Ed.) Advances in Consumer Research, 3, 168–175. Association for Consumer Research. O’Mahoney, M. (1978). Smell illusions and suggestion: reports of smells contingent on tones played on radio and television. Chemical Senses and Flavor, 3, 183–189. Parducci, A. (1974). Contextual effects: a range-frequency analysis. In: E. C. Carterette and M. P. Friedman (Eds.) Handbook of perception, psychophysical judgment and measurement, pp. 127–141. New York: Academic. Pause, B. M., Sojka, B., Krauel, K., Fehm-Wolfsdorf, G. and Ferstl, R. (1996). Olfactory information processing during the course of the menstrual cycle. Biological Psychology, 44(1), 31–54. Peier, A. M., Moqrich, A., Hergarden, A. C., Reeve, A. J., Andersson, D. A., Story, G. M., et al. (2002). A TRP channel that senses cold stimuli and menthol. Cell, 108, 705–715. Pelchat, M. L. and Rozin, P. (1982). The special role of nausea in the acquisition of food dislikes by humans. Appetite, 3, 341–351. Perng, C. M. and McDaniel, M. R. (1989). Optimization of a blackberry juice drink using response surface methodology. Presented at the 49th Annual Meeting of the Institute of Food Technologists. Chicago, IL. Peryam, D. R. and Pilgrim, F. J. (1957). Hedonic scale method of measuring food preferences. Food Technology, 11, 9–14. Pierce, J. D., Jr., Wysocki, C. J., Aronov, E. V., Webb, J. B. and Boden, R. M. (1996). The role of perceptual and structural similarity in cross-adaptation. Chemical Senses, 21, 223–237. Pierce, J. D., Jr., Zeng, X. N., Aronov, E. V., Preti, G. and Wysocki, C. J. (1995). Cross-adaptation of sweatysmelling 3-methyl-2-hexenoic acid by a structurally-similar, pleasant-smelling odorant. Chemical Senses, 20, 401–411. Pliner, P. (1983). Family resemblance in food preferences. Journal of Nutrition Education, 15, 137–140. Pol, H. E. H., Hijman, R., Baare, W. F. C. and van Ree, J. M. (1998). Effects of context on judgements of odor intensities in humans. Chemical Senses 23, 131–135. Popper, R. and Heymann, H. (1996). Analyzing differences among products and panelists by multidimensional scaling. In: T. Naes and E. Risvik (Eds.) Multivariate analysis of data in sensory science, pp. 159–184. Amsterdam: Elsevier.

Taste, smell and chemesthesis

129

Porter, R. H., Makin, J. W., Davis, L. B. and Christensen, K. M. (1992). Breast-fed infants respond to olfactory cues from their own mother and unfamiliar lactating females. Infant Behavior and Development, 15, 85–93. Powers, J. J. (1988). Mathematics of a lady tasting tea revisited. Journal of Sensory Studies, 3, 151–158. Prescott, J. and Swain-Campbell, N. (2000). Responses to repeated oral irritation by capsaicin, cinnamaldehyde and ethanol in PROP tasters and non-tasters. Chemical Senses, 25, 239–246. Preti, G., Wysocki, C. J., Barnhart, K. T., Sondheimer, S. J. and Leyden, J. J. (2003). Male axillary extracts contain pheromones that affect pulsatile secretion of luteinizing hormone and mood in women recipients. Biology of Reproduction, 68, 2107–2113. Price, J. L. (1973). An autoradiographic study of complementary laminar patterns of termination of afferent fibers to the olfactory cortex. Journal of Comparative Neurology, 150, 87–108. Rawcliffe, T., Howgate, P. and Geelhoed, E. N. (1992). Patterns of odours similarity among carbonyls and their mixture. Chemical Senses, 17, 119–125. Rawson, N. E. and Yee, K. K. (2006). Transduction and coding. Advances in Oto-Rhino-Laryngology, 63, 23–43. Rifkin, B. and Bartoshuk, L. M. (1980). Taste synergism between monosodium glutamate and disodium 5⬘-guaylate’. Physiology and Behavior, 24, 1169–1172. Rodriguez, I., Greer, C. A., Mok, M. Y. and Mombaerts, P. (2000). A putative pheromone receptor gene expressed in human olfactory mucosa. Nature Genetics, 26, 18–19. Rolls, B. J. (1999). Do chemosensory changes influence food intake in the elderly? Physiology and Behavior, 66(2), 193–197. Rolls, E. T. (2000). The orbitofrontal cortex and reward. Cerebral Cortex, 10, 284–294. Rolls, E. T. (2001). The rules of formation of the olfactory representations found in the orbitofrontal cortex olfactory areas in primates. Chemical Senses, 26, 595–604. Rolls, E. T. and Baylis, L. L. (1994). Gustatory, olfactory, and visual convergence within the primate orbitofrontal cortex. Journal of Neuroscience, 14, 5437–5452. Rolls, B. J., Hetherington, M. and Burley, V. J. (1988). The specificity of satiety: the influence of foods of different macronutrient content on the development of satiety. Physiology and Behavior, 43, 145–153. Rolls, B. J., Rolls, E. T., Rowe, E. A. and Sweeney, K. (1981). Sensory specific satiety in man. Physiology and Behavior, 27, 137–142. Rozin, P. (1988). Social learning about foods by humans. In: T. Zental and B. G. Galef, Jr. (Eds.) Social learning: A comparative approach, pp. 165–187. Hillsdale, NJ: Erlbaum. Rozin, P. (1996). Sociocultural influences on human food selection. In: E. Capaldi (Ed.) Why we eat what we eat: The psychology of eating, pp. 233–263. Washington, DC: American Psychological Association. Rozin, P., Markwith, M. and Stoess, C. (1996). Moralization and becoming a vegetarian: the transformation of preferences into values and the recruitment of disgust. Psychological Science, 8, 67–73. Rozin, P. and Schiller, D. (1980). The nature and acquisition of a preference for chili pepper by humans. Motivation and Emotion, 4, 77–100. Rozin, P. and Tuorila, H. (1993). Simultaneous and temporal contextual influences on food acceptance. Food Quality and Preference, 4, 11–20. Rozin, P. and Vollmecke, T. A. (1986). Food likes and dislikes. Annual Review of Nutrition, 6, 433–456. Santos, D. V., Reiter, E. R., DiNardo, L. J. and Costanzo, R. M. (2004). Hazardous events associated with impaired olfactory function. Archives of Otolaryngology – Head and Neck Surgery, 130, 317–319. Sarinopoulos, I., Dixon, G. E., Short, S. J., Davidson, R. J. and Nitschke, J. B. (2006). Brain mechanisms of expectation associated with insula and amygdala response to aversive taste: implications for placebo. Brain, Behavior, and Immunity, 20, 120–132. Sauvageot, F., Urdapilleta, I. and Peyron, D. (2006). Within and between variations of texts elicited from nine experts in wines. Food Quality and Preference, 17(6), 429–444. Savic, I., Gulyas, B. and Berglund, H. (2002). Odorant differentiated pattern of cerebral activation: comparison of acetone and vanillin. Human Brain Mapping, 17, 17–27. Schaal, B., Marlier, L. and Soussignan, R. (1998). Olfactory function in the human fetus: Evidence from selective neonatal responsiveness to the odor of amniotic fluid. Behavioral Neuroscience, 112, 1438–1449. Schafe, G. E. and Bernstein, I. L. (1996). Taste aversion learning. In: E. Capaldi (Ed.) Why we eat what we eat: the psychology of eating, pp. 31–52. Washington, D.C.: American Psychological Association. Scharf, A. and Volkmer, H. P. (2000). The impact of olfactory product expectations on the olfactory product experience. Food Quality and Preference, 11, 497–504. Schifferstein, H. N. J. (1995). Contextual shifts in hedonic judgments. Journal of Sensory Studies, 10, 381–392. Schifferstein, H. N. J. (2001). Effects of product beliefs on product perception and liking. In: L. Frewer, E. Risvik. and H. Schifferstein. Food, people and society: A European perspective of consumers’ food choices, London: Springer Verlag. Schifferstein, H. N. J. and Frijters, J. E. R. (1992). Contextual and sequential effects on judgments of sweetness intensity. Perception and Psychophysics, 52(3), 243–255.

130

Product Experience

Schifferstein, H. N. J. and Oudejans, I. M. (1996). Determinants of cumulative successive contrast in saltiness intensity judgments. Perception and Psychophysics, 58(5), 713–724. Schiffman, S.S. et al. (1995). Investigation of synergism in binary mixtures of sweeteners. Brain Resource Bulletin, 38, 105–120. Schiffman, S. S. (2000). Intensification of sensory properties of foods for the elderly. Journal of Nutrition, 130(Suppl. 4S), 927S–930S. Schleidt, M., Hold, B. and Attila, G. (1981). A cross-cultural study on the attitude towards personal odors. Journal of Chemical Ecology, 7, 19–31. Schmidt, H. J. and Beauchamp, G. K. (1988). Adult-like preferences and aversions in three-year-old children. Child Development, 59, 1136–1143. Schutz, H. G. and Cardello, A. V. (2001). A labeled affective magnitude (LAM) scale for assessing food liking/ disliking. Journal of Sensory Studies, 16, 117–159. Shinohara, K., Morofushi, M., Funabashi, T. and Kimura, F. (2001). Axillary pheromones modulate pulsatile LH secretion in humans. Neuroreport, 12, 893–895. Shusterman, D. (2002). Individual factors in nasal chemesthesis. Chemical Senses, 27, 551–564. Simoons, F. J. (1961). Eat not this flesh. Madison: University of Wisconsin Press. Siret, F. and Issanchou, S. (2000). Traditional process: influence on sensory properties and on consumers’ expectation and liking. Application to pate’ de campagne. Food Quality and Preference, 11, 217–228. Slosson, E. E. (1899). A lecture experiment in hallucinations. Psychology Review, 6, 407–408. Small, D. M. (2006). Central gustatory processing in humans. In: T. Hummel and A. Welge-Lüssen (Eds.) Taste and smell: An update (Advances in Oto-rhino-laryngology), pp. 191–220. Basel, Germany: Karger. Small, D. M., Gregory, M. D., Mak, Y. E., et al. (2003). Dissociation of neural representation of intensity and affective valuation in human gustation. Neuron, 39, 701–711. Small, D. M., Jones-Gotman, M., Zatorre, R. J., Petrides, M. and Evans, A. C. (1997). Flavor processing: More than the sum of its parts. Neuroreport, 8, 3913–3917. Small, D. M., Voss, J., Mak, Y. E., Simmons, K., Parrish, T. and Gitelman, D. (2004). Experience-dependent neural integration of taste and smell in the human brain. Journal of Neurophysiology, 92, 1892–1903. Smith, D. V. and Margolskee, R. F. (2001). Making sense of taste. Scientific American, 284, 32–39. Smith, D. V. and Shepherd, G. M. (1999). Chemical senses: Taste and olfaction. In: M. J. Zigmond, F. E. Bloom, S. C. Landis, et al. (Eds.) Fundamentals of neuroscience, pp. 719–760. San Diego, CA: Academic Press. Smith-Swintosky, V. L., Plata-Salaman, C. R. and Scott, T. R. (1991). Gustatory neural coding in the monkey cortex: Stimulus quality. Journal of Neurophysiology, 66, 1156–1165. Sobel, N., Prabhakaran, V., Desmond, J. E., et al. (1998). Sniffing and smelling: Separate subsystems in the human olfactory cortex. Nature, 392, 282–286. Steenkamp, J. B. E. M. and van Trijp, H. C. M. (1988). Free-choice profiling in cognitive food acceptance research. In: D. M. H. Thomson (Ed.) Food acceptability, pp. 363–376. London: Elsevier Applied Science. Stefani G., Romano D. and Cavicchi A. (2006). Consumer expectations, liking and willingness to pay for specialty foods: Do sensory characteristics tell the whole story? Food Quality and Preference, 17, 53–62. Stein, L. J., Nagai, H., Nakagawa, M. and Beauchamp, G. K. (2003). ‘Effects of repeated exposure and healthrelated information on hedonic evaluation and acceptance of a bitter beverage.’ Appetite 40(2),119–129. Steiner, J. E. (1977). Facial expressions of the neonate infant indicating the hedonics of food-related chemical stimuli. In: J. M. Weiffenbach (Ed.) Taste and development: The genesis of sweet preference, pp. 173–189. Bethesda, MD: DHEW Publication No. (NIH) 77-1068, US Department of Health, Education, and Welfare. Stern, K. and McClintock, M. K. (1998). Regulation of ovulation by human pheromones. Nature, 392, 177–179. Stevens, D. A. and Lawless, H. T. (1986). Putting out the fire: Effects of tastants on oral chemical irritation. Perception and Psycholophysics, 39, 346–350. Stevens, J. C. and Cain, W. S. (1993). Changes in taste and flavor in aging. Critical Reviews in Food Science and Nutrition, 33, 27–37. Stevens, S. S. and Galanter, E. H. (1957). Ratio scales and category scales for a dozen perceptual continua. Journal of Experimental Psychology, 54(6), 377–411. Stevenson, R. J., Prescott, J. and Boakes, R. A. (1995). The acquisition of taste properties by odors. Learning and Motivation, 26, 433–455. Stevenson, R. J., Prescott, J. and Boakes, R. A. (1999). Confusing tastes and smells: how odours can influence the perception of sweet and sour tastes. Chemical Senses, 24, 627–635. Stevenson, R. J. and Yeomans, M. R. (1993). Differences in ratings of intensity and pleasantness for the capsaicin burn between chili likers and non-likers; implications for liking development. Chemical Senses, 18, 471–482. Stone, H. and Sidel, J. L. (2004). Sensory evaluation practices (3rd Ed.) New York: Academic Press. Stone, H., Sidel, J. L., Oliver, S. Woolsey, A. and Singleton, R. C. (1974). Sensory evaluation by qualitative descriptive analysis. Food Technology, 28, 24–34.

Taste, smell and chemesthesis

131

Stuck, B. A., Frey, S., Freiburg, C., et al. (2006). Chemosensory event-related potentials in relation to side of stimulation, age, sex, and stimulus concentration. Clinical Neurophysiology, 117(6), 1367–1375. Swithers, S. E. and Hall, W. G. (1994). Does oral experience terminate ingestion? Appetite, 23(2), 113–138. Tang, C. and Heymann, H. (2002). Multidimensional sorting, similarity scaling and free-choice profiling of grape jellies. Journal of Sensory Studies, 17(6), 493–510. Tarun, A., Bryant, B., Zhai, W., Soloman, C. and Shusterman, D. (2003). Gene expression for carbonic anhydrase isoenzymes in human nasal mucosa. Chemical Senses, 28, 621–629. Todrank, J., Wysocki, C. J. and Beauchamp, G. K. (1991). The effects of adaptation on the perception of similar and dissimilar odors. Chemical Senses, 16, 467–482. Tuorila H., Andersson A., Martikainen A. and Solovaara H. (1998). Effect of product formula, information and consumer characteristics on the acceptance of a new snack food. Food Quality and Preference, 9, 313–320. Tuorila, H., Cardello, A. V. and Leshe, L. L. (1994). Antecedents and consequences of expectations related to fat-free and regular-fat foods. Appetite, 23, 247–263. Turin, L. (2006). Accords and discords: perfume reviews. In: J. Drobnick (Ed.) The Smell Culture Reader, pp. 216–227. New York; Oxford: Berg. Von Sydow, E., Moskowitz, H., Jacobs, H. and Meiselman, H. (1974). Odor–taste interaction in fruit juices. Lebensmittel Wissenschaft und Technologie, 7, 9–16. Wansink, B. and Park, S. (2002). Sensory suggestiveness and labeling: Do soy labels bias taste? Journal of Sensory Studies, 17, 483–491. Wansink, B., van Ittersum, K. and Painter, J. E. (2005). How descriptive food names bias sensory perceptions in restaurants. Food Quality and Preference, 16, 393–400. Watson, J. L. and Caldwell, M. L. (2005). The cultural politics of food and eating: A reader. Oxford: Blackwell Publishing. Williams, A. A. and Langron, S. P. (1984). The use of free-choice profiling for the evaluation of commercial ports. Journal of the Science of Food and Agriculture, 35, 558–568. Winberg, J. and Porter, R. H. (1998). Olfaction and human neonatal behaviour: Clinical implications. Acta Paediatrica, 87, 6–10. Wise, P. M., Radil, T. and Wysocki, C. J. (2003). Time-intensity ratings of nasal irritation from carbon dioxide. Chemical Senses, 28, 751–760. Wysocki, C. J. and Beauchamp, G. K. (1984). Ability to smell androstenone is genetically determined. Proceedings of the National Academy of Sciences of the United States of America, 81, 4899–4902. Wysocki, C. J. and Beauchamp, G. K. (1991). Individual differences in human olfaction. In: C. J. Wysocki and M. R. Kare (Eds.) Chemical senses: genetics of perception and communication, pp. 353–373. New York: Marcel Dekker. Wysocki, C. J., Cowart, B. J. and Radil, T. (2003) Nasal-trigeminal chemosensitivity across the adult lifespan. Perception and Psychophysics, 65, 115–122. Wysocki, C. J. and Gilbert, A. N. (1989). The National Geographic Smell Survey: The effects of age are heterogeneous. In: C. L. Murphy, W. S. Cain, and D. M. Hegsted (Eds.) Chemical senses and nutrition in aging, pp. 12–28. New York: New York Academy of Sciences. Wysocki, C. J. and Preti, G. (2004). Facts, fallacies, fears, and frustrations with human pheromones. The Anatomical Record Part A: Discoveries in Molecular, Cellular, and Evolutionary Biology, 281(1), 1201–1211. Wysocki, C. J. and Wise, P. (2003). Methods, approaches, and caveats for functionally evaluating olfaction and chemesthesis. In: K. Deibler and J. F. Delwiche (Eds.) Handbook of flavor characterization: Sensory, chemical and physiological, pp. 1–40. New York: Marcel Dekker. Yackinous, C. A. and Guinard, J. X. (2002). Relation between PROP (6-n-propylthiouracil) taster status, taste anatomy and dietary intake measures for young men and women. Appetite, 38(3), 201–209. Yang, H. H. and Lawless, H. T. (2005). Descriptive analysis of divalent salts. Journal of Sensory Studies, 20, 97–113. Zellner, D. A., Rozin, P., Aron, M. and Kulish, C. (1983). Conditioned enhancement of humans’ liking for flavor by pairing with sweetness. Learning and Motivation, 14, 338–350. Zhao, G. Q., Zhang, Y., Hoon, M. A., et al. (2003). The receptors for mammalian sweet and umami taste. Cell, 115, 255–266. Zou, Z. and Buck, L. B. (2006). Combinatorial effects of odorant mixes in olfactory cortex. Science, 311, 1477–1481. Zufall, F. and Leinders-Zufall, T. (1997). Identification of long-lasting form of odor adaptation that depends on the carbon monoxide/cGMP second-messenger system. Journal of Neuroscience, 8, 2703–2712.

5

MULTISENSORY PRODUCT EXPERIENCE HENDRIK N.J. SCHIFFERSTEIN Delft University of Technology, Delft, The Netherlands

CHARLES SPENCE University of Oxford, Oxford, UK

1. INTRODUCTION People use all of their senses in order to explore the world around them. Under normal everyday conditions the senses all work together to create the overall product experiences that fill a typical person’s life. For example, a man who drives a car will determine its position on the road by means of the objects that he sees through the windshield. He can ascertain the car’s speed from the numbers he reads on the dashboard, the speed at which he sees the trees passing him by on the side of the road, the sounds made by the car’s engine, and/or by means of the vibrations he feels through his body. He can judge the stability of the car from the way in which the vehicle responds to his actions: The changes in velocity he feels when he changes gear, the forces he experiences when changing lane. In general, whenever we use a product, we perform actions on (or with) that product and our senses provide us with feedback regarding how the product, or the environment, reacts to those actions. By now, many different studies have suggested that the greater the number of sensory modalities that are stimulated at any one time, the richer our experiences will be (e.g. Bahrick and Lickliter, 2000; Spence, 2002; Stein and Meredith, 1993). As a consequence, increasing the number of modalities of sensory input presented in a virtual environment can help to increase people’s sense of presence and also increase their memory for objects placed within the virtual environment (e.g. Dinh et al., 1999; Hoffman et al., 1998; Washburn et al., 2003). An interesting application of this was recently described by Vlahos (2006), who reported that the addition of a scent collar to standard virtual-reality equipment (e.g. goggles offering a stereoscopic view, headphones providing binaural sounds, and movement sensors) can help to create an immersive multisensory Product Experience Copyright © 2008 Elsevier Ltd.

133

134

Product Experience

environment in which soldiers can be prepared for the kinds of situations that they may subsequently encounter in a war zone. Soldiers on the ground generally find themselves in an unknown area under circumstances that are unfamiliar and often highly stressful. On the one hand, the unusual smells produced by exploding bombs and mines, burning trucks and buildings, and decaying corpses may overwhelm the newly-deployed soldiers and may interfere with the completion of their mission. On the other hand, the olfactory cues may provide information that is vital for the soldier’s survival: Burning wires may, for example, indicate an electrical problem in an aircraft, while the smell of cigarettes may signal the presence of the enemy. Despite the important role that the ensemble of sensory modalities plays in a consumer’s interactions with the environment, very few industrial companies have attempted to make full use of the multisensory potential of the products that they market/produce (see Hine, 1995; Lindstrom, 2005). For a product designer, it is important to know what kind of actions people will perform with a product, how they will perceive it during these interactions, and how the consumer’s senses work together to deliver rich and varied multisensory experiences. This knowledge can then be applied to the design of new products that will more effectively stimulate the senses of the consumer, and hence lead to more pleasurable and memorable multisensory product interactions. In this chapter, we highlight the roles of the various senses, and their interplay when people interact with different products. Besides highlighting the key theoretical debates in this area, our discussion will center on empirical data gathered in well-controlled experimental studies, as well as on survey data. This chapter also highlights a number of examples where the theoretical principles of multisensory perception have actually been incorporated and tested in the design of everyday products. In each of the seven sections that follow, we start out by describing a particular topic and then critically discuss a number of scientific studies that have investigated this topic. Typically, each topic can be characterized not only by its content, but also by the specific approaches and research methods that are used. Additionally, where possible we outline how this knowledge has been used or can be used in the development of new products. In the first three sections we consider the modalities as if they were separate systems. First, we evaluate the roles of the various sensory modalities in people’s interactions with products during everyday life (Section 2) and in mental imagery (Section 3). Next, we review what happens when people switch their attention between different sensory systems (Section 4). Given that the senses typically do not work in isolation, but rather operate as an integrated whole, the next sections discuss the links that people experience intuitively between phenomena occurring in different sensory modalities (Section 5), and the ways in which sensory information from the different modalities is integrated into a holistic product experience. While Section 6 discusses a number of interactions that have been observed between various forms of sensory stimulation, Section 7 focuses specifically on the role of sensory (in)congruity in these interactions. Section 8 highlights what is currently known about the factors that determine the relative importance people attach to each of their senses in various situations. Finally, at the end of the chapter, we highlight some of the important emerging challenges currently facing researchers and designers in the area of multisensory product design.

2. COMPARING THE DIFFERENT SENSORY MODALITIES Each of our senses is most sensitive to a different type of stimulation (energy). Because each sensory modality may be considered as a separate information channel, not all of

135

Multisensory product experience

the incoming sensory information will necessarily communicate the same message to a person. For instance, whereas some information is only used to perform a task with a product, other information may also evoke memories or associations to other objects, or may trigger feelings and emotions. It is important to note that product experiences are based on all of the incoming sensory information, no matter whether it is perceived consciously by a person or not. The discussion outlined here therefore raises the question of what roles the different senses play in people’s everyday lives. In order to obtain a more detailed insight into the similarities and differences between the various senses in modulating our multisensory product experiences, Schifferstein and Cleiren (2005) developed a split-modality approach. The participants in their study were presented with six simple products (a marker pen, a spray deodorant, a tennis ball, a bag of crisps, a boiled egg, and a can of soft drink) via only a single sensory modality at a time. The active interaction between user and product was choreographed in equivalent ways for the four modalities: A prerecorded video (vision) or sound (audition); a product presented in a sound-isolated box (touch); or a smell contained in a glass jar (olfaction). For the particular range of products studied by Schifferstein and Cleiren (2005), vision and touch turned out to be approximately equally successful in providing participants with detailed information concerning a product; audition proved somewhat less useful, and olfaction provided the least detailed information. Furthermore, products perceived by vision and touch were found to be the easiest to identify, and yielded the clearest memories of previous events and associations to persons and other products (see Table 5.1). In a complementary experimental study, Schifferstein and Desmet (2007) assessed the roles of the various senses on people’s perception of different everyday products by comparing the effects of blocking one sense. They found that preventing people from seeing the products had the most detrimental effect on the amount of functional product information that they perceived. Because fewer product features are perceived simultaneously, and a considerable proportion of the product-related information is lost when one cannot see a product, task difficulty and task duration typically increase, up to the point at which simple tasks can no longer be completed without help from another person. Interestingly, when products cannot be seen, people report that their experiences become more intense and that they start to use their other senses more. When tactual perception was blocked to some degree (in this case by wearing very thick gloves), a substantial amount of product information was lost as well. Similar to vision, perceived task difficulty and task duration increased significantly. Tasks requiring subtle coordinated movements (such as composing an SMS message on a mobile phone) became almost impossible to perform. In addition, an emotional dimension of tactual product experiences was revealed: Familiar products felt strange; they did not feel familiar anymore. It seems as if through blocking tactual perception one becomes somewhat alienated from one’s own surroundings. TABLE 5.1 Mean evaluation responses for the products experienced by participants in Schifferstein and Cleiren’s (2005) study. Higher values indicate that participants found a particular aspect more applicable

Detailed information Difficulty of identification Clarity of memory associated with the event

Scale type

Vision

Touch

Sound

Smell

7-point 7-point 4-point

6.2 1.3 3.2

6.0 1.7 3.3

4.7 3.7 2.9

3.0 4.6 2.8

136

Product Experience

Furthermore, Schifferstein and Desmet (2007) found that blocking auditory perception primarily resulted in communication problems: People felt cut-off from the outside world (see also Ramsdall, 1978). Blocking olfaction only led to a reported decrease in appetite for foods (see also Blomqvist et al., 2004). Hence, blocking the ears or the nostrils did not interfere much with functional product usage. However, it did decrease how good, how stimulating, and how intense the products were experienced as being. Therefore, consumers’ emotional product experiences nevertheless seem to suffer when audition or olfaction is blocked. When taken together with the existing literature, the two studies discussed above indicate that visual information is primarily important in user-product interactions, because it seems to be used mainly for the functional interaction, that is for executing tasks. In general, vision provides the largest amount of information on a product within the shortest time frame. Furthermore, visual input seems to be linked most directly to stored knowledge, such as information regarding the method of production, region of origin, and product safety (e.g. see Burns et al., 1995; Hinton and Henley, 1993). This large quantity of information most likely attracts the majority of a consumer’s attentional resources, which leaves fewer resources available for the processing of any other sensory experiences. This may explain why people claim that they use their other senses more after vision has been blocked. Given that olfaction and touch are often regarded as being our more ‘emotional senses’ (see Spence, 2002), this may help to explain the somewhat counterintuitive finding that products are experienced ‘more intensely’ when vision is denied. Similar to visual exploration, through touching people gather a lot of information about a product. On the one hand, this makes it relatively easy for people to identify many common objects by touch alone (e.g. Klatzky, Lederman and Metzger, 1985). On the other hand, this information is very helpful during functional product usage. Furthermore, tactual experiences are likely to possess a substantial emotional component, given the key role that touch plays in mediating interpersonal intimacy (e.g. Bolanowski, Verrillo and McGlone, 1999; Field, 2001). Given that audition is critical in verbal communication by means of which factual information may be effectively distributed to others, one might perhaps expect that sounds play an analogous role in communicating factual information about products. However, the studies discussed above indicate that product sounds play only a very limited role in our functional interaction with products. Because audition plays an important part in the expression of emotion, for example in the non-verbal aspects in human speech (e.g. Scherer, 2003) and in music (e.g. Herz, 1998; Krumhansl, 2002), product sounds may nevertheless affect the emotional product expression. The functional role of taste and smell is evident in people’s everyday interactions with food products, such as those occurring when people prepare and eat a meal. In addition, olfactory cues play a functional role in judging whether or not a person, object, or space is clean and safe. This is illustrated by the importance people attach to the smell of personal care and cleaning products (Schifferstein, 2006). As far as consumers’ interactions with non-food products are concerned, however, Schifferstein and Desmet’s (2007) study suggests that blocking olfaction does not have a deleterious effect on the functional execution of many everyday tasks. Olfactory cues may nevertheless have a major effect on a consumer’s emotional reaction to these products. In general, olfactory cognition seems to be dominated by the affective dimension (e.g. Engen, 1982). What’s more, memories elicited by odors tend to be more emotional in character than memories elicited by other types of stimuli (e.g. Herz and Schooler, 2002). Odor pleasantness even plays an important role in the way people categorize odors. It is, however, important to note that many of our associations with smells are

Multisensory product experience

137

idiosyncratic in nature (Ayabe-Kanamura et al., 1998; Dubois, 2000). The consequent variation in a consumer’s associations with a particular olfactory experience/attribute may, therefore, be expected to show a much more profound variation across individuals/ cultures than that found for visual, auditory, or tactual stimuli. In conclusion, all of the human senses contribute to how a product is experienced and are, therefore, important to consider in the design process. Insight into the roles that the different senses play in people’s interactions with products can help designers to choose the optimum sensory channel by which to communicate a certain message to a consumer (Lindstrom, 2005). For example, because the olfactory modality seems to be most closely linked to emotional experiences, designers who want to influence the emotions elicited by their products may want to consider altering the product’s smell. Although changes in appearance may be linked more explicitly and more consistently to shifts in specific emotional associations (Desmet, 2002) than smells, changes in the smell of a product may have a more direct impact on the emotional experience evoked. As a result, a specially designed smell could prove to be a powerful reinforcer of the emotions activated by the product’s appearance. Furthermore, because touch plays such an important role in perceiving a product as one’s own versus ‘foreign’, the use of distinctive tactual details may provide an effective way to generate differential advantage that operates on a more personal level than that obtained with visual features. In this way, manufacturers’ consistent use of particular tactual features could generate distinctive and appealing tactual brand images for products such as hand tools, mobile phones, or shavers.

3. SENSORY IMAGERY The way in which people experience multisensory products is not determined solely by their physical interactions with the products themselves. Part of the experience is often formed in a consumer’s mind prior to their actually buying or using a given product. That is, they may already imagine what it would feel like to use the product or to show it off to friends. Consumers may fantasize about how soft a new mattress will feel, how dazzling the smell of a new bath oil would be, how great they would look in that new pair of jeans, how delicious a piece of cheesecake will taste, or how magnificent that new stereo will sound. While for most people mental imagery may be most vivid for the case of visual images (Kosslyn, 1994), it is important to note that people can generate rich mental images for the other senses as well (e.g. Klatzky, Lederman and Matula, 1991; Reisberg, 1992; Stevenson and Case, 2005). That said, it nevertheless seems legitimate to ask whether all sensory images can be formed with the same ease and, if so, whether they are all equally vivid? Any test of mental imagery that prespecifies a number of products or events (e.g. Betts, 1909) is unlikely to provide an unbiased test of whether, for example, olfactory images are any less clear and vivid than visual and tactual images, because the clarity of a mental image may be at least partly product-dependent. With this in mind, Schifferstein (2007) instructed the participants in his study to imagine a product with a conspicuous (or specific) sensory characteristic (i.e., appearance, sound, feel, smell, or taste), without specifying a particular event or product category. By using this instruction, the products chosen were likely to vary between sensory modalities (see Table 5.2), but the roles of the sensory stimuli in everyday life would, as far as possible, be matched. The vividness ratings obtained under such conditions were only slightly, but significantly, lower for smell and taste than for the other sensory modalities: Whereas mean vividness ratings for

138

Product Experience

TABLE 5.2 Frequency with which objects in various categories were imagined for five sensory modalities (N ⴝ 50 participants) (adapted from Schifferstein, 2007)

Vehicles Electrical apparatuses Interior product Toys and sports gear Tools Musical instruments Care products Foods and beverages Surroundings (industry, nature) Art, culture, music Person Other

Vision

Touch

Audition

Smell

Taste

13 9 4 5 0 1 0 2 10 5 1 0

4 9 4 6 6 2 6 3 3 1 5 1

12 8 7 0 8 0 0 0 2 9 4 0

1 0 1 0 2 1 7 15 20 0 2 1

0 0 0 0 1 0 3 46 0 0 0 0

vision, touch, and audition ranged from 5.4 to 5.7, the ratings for taste and smell were both 4.9 on a seven-point scale. In a follow-up experiment, Schifferstein (2007) instructed his participants to imagine interacting with eight different products. He asked them to imagine what the product would smell like, what it would sound like, what it would look like, and finally how it would feel to interact with the particular product. When the mean imagery vividness ratings were correlated with the mean importance ratings for these sensory modalities during product usage (see Schifferstein, 2006), significant positive correlations were observed for touch, audition, and olfaction. This suggests that for three out of the four sensory modalities under investigation in Schifferstein’s study, the vividness of a characteristic part of a mental image was related to how important people found that aspect of the product when they interacted with it. The lack of any significant correlation in the case of vision may be due to the fact that visual attributes were important for the perception of nearly all products. The role of sensory imagery in product experience is particularly important in situations in which products cannot be experienced physically, such as when products are advertised or when they are sold through catalogs or through virtual stores. In some cases, providing a particular kind of sensory input in an available modality (typically vision or audition) may be used to create a cross-modal illusion that can compensate for the absent sensory stimulation. For example, Li, Daugherty and Biocca (2002, 2003) suggested that interactive visual images can compensate for a lack of tactual input in product advertising in a virtual environment. Enhancing the sense of presence can facilitate the occurrence of cross-modal illusions, such as feeling physical resistance from virtual (visual) objects that are moved (Biocca, Kim and Choi, 2001). Furthermore, auditory cues can be used to give an impression of the tactile attributes of a product or surface (Kitagawa and Spence, 2006; Spence and Zampini, 2006). For instance, Lederman (1979) describes a television commercial for a shaving cream in which the edge of a credit card was drawn across both sides of a model’s face shown in close-up. One side of his face had been shaved using a proprietary shaving cream, the other with a competitor product. Although both sides of the model’s face looked identical after shaving in the

Multisensory product experience

139

advert, the difference in the closeness of the shave was effectively illustrated by the difference in the sound made by the credit card as it was drawn across the skin on the two sides of the model’s face.

4. ATTENTION SWITCHING BETWEEN THE SENSES Mechanisms of selective attention allow people to concentrate on particular events of interest in their environments, and to avoid sensory overload. Researchers have shown that people can covertly (i.e. in the absence of any change in posture) direct their selective attention to a particular sense (e.g. Spence, Nicholls and Driver, 2001). In fact, attending to one sensory modality can facilitate the perception of stimuli in that modality relative to when attention is diverted to another sensory modality (or else divided simultaneously between several senses). For example, the participants in a study reported by Spence and Driver (1997b) had to make speeded spatial discrimination responses to an unpredictable sequence of auditory and visual target stimuli presented to the left or right of a central fixation point. At the beginning of each trial, a symbolic visual cue indicated the likely modality of the upcoming target. This cue was valid on the majority (80%) of the trials and invalid on the remaining 20% of the trials. The participants responded significantly faster (and somewhat more accurately) when the target occurred in the expected modality than when the very same target stimulus was presented in the unexpected modality. Using similar experimental protocols, it has subsequently been shown that people can also direct their attention selectively toward the sense of touch (Spence et al., 2001), to the sense of smell (Spence et al., 2001), to the gustatory modality (Marks, 2002; Marks and Wheeler, 1998), and even to thermal pain (Miron, Duncan and Bushnell, 1989; Spence et al., 2002). Switching one’s attention from one sensory modality to another takes time; that is, it comes at a cost. Spence, Nicholls and Driver (2001) determined the reaction time costs associated with people shifting their attention between their eyes, their ears, and their skin. They found that the reaction time costs associated with shifting attention away from the sense of touch were larger than the costs associated with shifting attention away from either vision or audition. The costs associated with shifting attention toward touch from either the eyes or ears were larger than the costs associated with shifting attention to the eyes or ears. These results suggest that touch is, in a sense, ‘sticky’: Once our attention has been directed to what we are feeling, we find it harder to shift our attention toward what we are seeing or hearing instead. Spence et al. argued that this may be because when attending to touch people can either adopt an internal or an external perspective (see Chapter 2), whereas sounds and sights are always experienced as distal properties of the external world (Martin, 1995). It may then simply take people longer to shift between internal and external perspectives than to shift their attention between sensory modalities that both represent a more external perspective. Interestingly, differences in the reaction time costs associated with switching attention between the various senses have not only been reported in perceptual tasks, but also in the cognitive processing of words and concepts. For instance, Pecher, Zeelenberg and Barsalou (2003) used a property verification task in which their participants had to indicate whether a particular concept (e.g. blender) usually had a certain sensory property (e.g. loud) or not. Pecher et al. found that the average reaction times of participants increased when the sensory modality of the property switched between consecutive trials of the experiment: For example, responses to ‘blender-loud’ would be faster after ‘leavesrustling’ (both auditory), than after ‘toast-warm’ (tactual).

140

Product Experience

Perceived location is an extremely important factor governing multisensory integration and perception. This is because stimuli that appear to originate from the same spatial location are more likely to be attributed to a single multisensory source rather than to separate sources (Spence, 2007a). Consequently, it is far more likely that sensory stimuli will be integrated if they appear to come from the same spatial location than if they appear to come from different locations. The ventriloquist illusion (e.g. Bertelson, 1999), however, represents an important exception to this rule. Typically, the sounds that emanate from a ventriloquist’s mouth appear to originate from the mouth of his dummy. In order for the illusion to work well, the voice and lip-movements should co-vary in time; otherwise people have no particular reason to believe that the two streams of stimuli belong together. Similarly, in a movie theatre, the words uttered by an actor seem to originate from his mouth on the screen, whereas the sound is actually reproduced by the loudspeakers located elsewhere in the auditorium. In this case, given the temporal correspondence and the overall congruence between the auditory and visual streams, our brain assumes that the two sources of information ought to go together and we ventriloquize the actor’s speech sounds into spatial alignment with the seen lip movements. It should be noted that spatial and temporal correspondence are, in fact, just two of the many rules used by the human brain to group (bind, or unify) multisensory inputs. The interested reader is referred to Spence (2007a) for a recent discussion of the various other higher order variables (such as common number and semantic congruency) that have been shown to modulate the degree to which multisensory integration takes place. One of the key application areas for these research findings on attentional capture is in the design of warning and alarm signals. Many modern electronic devices, such as microwave ovens and telephones, make use of synthesized sounds to capture a consumer’s attention or to provide audible warning signals. In situations where multiple devices are used simultaneously, such as in kitchens, aircraft cockpits, or hospital operating rooms, several different alarms may go off at more or less the same time. Under such conditions, the source of a particular alarm can be hard to identify. For example, Momtahan, Hétu and Tansley (1993) reported that medical staff in operating rooms and intensive care units recognized only around 40% of the warning sounds correctly. An additional complication in many practical settings is that many interface operators have been known to disable or silence alarms if they consider them to be too irritating (e.g. Meredith, Edworthy and Rose, 1999; Sorkin, 1988). This problem is particularly pronounced under conditions where the false alarm rate is high. One possible strategy to overcome these limitations involves the inclusion of more sensory modalities in the design of alarm signals. In general, the chance of detecting an important warning signal and the speed with which it can be detected increases with the number of sensory modalities that redundantly communicate the same message. For instance, Selcon, Taylor and McKenna (1995) have shown that the verbal warning signal ‘left’ can elicit a left response more effectively if it is also presented on an interface operator’s left. Similarly, introducing a visual signal on the left can also help. Hence, gains in performance are most likely to occur if all of the relevant sensory signals originate from the same apparent spatial location, and if all the signals are congruent in terms of their perceived meaning (see Ho and Spence, 2005a; Spence and Driver, 1999). It is also important that the multiple sensory signals constituting a multisensory warning signal appear to be synchronized in time. One potential problem for interface designers here relates to the fact that auditory, visual, and tactile stimuli are all processed at different speeds (Spence and Squire, 2003). Even though light travels through air far more rapidly than sound, it takes much longer to process visual stimuli (chemically) in the eye than it does to mechanically transduce sound waves in the inner ear. It has been

Multisensory product experience

141

estimated that these two factors cancel each other out at a distance of approximately 10 m from an observer (known as the ‘horizon of simultaneity’, Pöppel, 1988). However, given that the majority of interfaces are located much closer to the typical interface operator (Spence and Driver, 1999), the visual component of an audiovisual warning signal should be presented slightly earlier in time than the auditory component in order to ensure that both stimuli reach the relevant parts of the brain at the same time (see also Chan and Chan, 2006). Ho and Spence (2005a) showed that warning signals presented from the direction in which a potentially dangerous driving event was about to occur (‘front’ or ‘back’), yielded faster and more accurate acceleration or braking responses than when they were presented from below the car driver’s seat. Auditory icons (the car horn sound) and verbal warning signals were found to be significantly more effective as warning signals than the pure tone cues typically favored by many designers. For application in real-world driving situations, Ho and Spence favored the implementation of auditory icons, because non-verbal signals are less likely to be susceptible to the influence of other concurrent linguistic elements in the driving environment, such as human speech from other passengers or from the car radio. In addition, their meaning is language-independent and understood intuitively (i.e. without the need for explicit training, Edworthy and Hellier, 2006). In subsequent studies, Ho and her colleagues (Ho, Reed and Spence, 2006; Ho, Tan and Spence, 2005) have shown that directional vibrotactile cues can also be used to effectively capture a driver’s attention. Vibrotactile warning signals have the additional advantage that they are private to the driver and that they remain effective, no matter how loud the radio may be turned up. However, depending on the body location where the vibrotactile warning signal is to be perceived, its utility may be limited by the thickness of the driver’s clothing. Although odors may also function as alarms, e.g. in indicating spoilage of foods or in indicating a fire burning, the deliberate use of odors in designed warning signals has, until now, mainly been confined to adding odorants to gas, so that a gas leakage will be noticed. Nonetheless, several other possible applications of using odors to attract attention or to alarm people have been suggested. For example, designers have considered the use of odorants (e.g. the smell of brewing coffee) as the wake-up signal in alarm clocks. However, by themselves olfactory stimuli are not capable of reliably awakening people who are asleep (Carskadon and Herz, 2004) and, consequently, an odor is unlikely to yield an effective wake-up signal. Others have attempted to isolate smells that are repulsive to as many people as possible to be used by the police in crowd control. However, no single odor has as yet been found to be suitably repulsive to all ethnic groups (Dilks, Dalton and Beauchamp, 1999). Elsewhere, it has been suggested that odors might be used to alert or relax people (e.g. Spence, 2002; Warm, Dember and Parasuraman, 1991). Such smells might find an application in office buildings, to energize employees during their post-lunch dip, or to help them to relax after a stressful meeting. Analogously, odorants might potentially be used in car interface design, either to alert drowsy drivers or to relax stressed drivers (see Baron and Kalsher, 1998; Ho and Spence, 2005b). However, more research is needed in order to find out which odorants can be used effectively to achieve the desired effects, and to specify the exact conditions under which they will be most effective. One other area where the spatial co-location of sensory signals has been shown to be important relates to the use of mobile phones in vehicles. Mobile phone conversations tend to be more demanding than conversations with other car passengers (see Kames, 1978), possibly because car passengers have less demanding/emotional conversations with the driver. In addition, car passengers tend to be aware of road conditions and

142

Product Experience

TABLE 5.3 Performance (% of words correctly recalled in the target speech stream) on a verbal shadowing task in a driving simulator, as a function of the location of the relevant auditory stream (adapted from Spence and Read, 2003)

Single task (Shadowing) Dual task (Shadowing while driving)

Front

Side

Difference

90.3 79.2

63.9 33.3

26.4 45.9

hence normally fall silent when the road conditions become more taxing. Using a mobile phone typically results in a four-fold increase in the risk of having an accident and, contrary to intuition, drivers are just as likely to have an accident using a hand-held device as when using a hands-free kit (Redelmeier and Tibshirani, 1997). This suggests that the key problem is related with attentional distraction (Horrey and Wickens, 2006; Spence and Read, 2003; Strayer and Johnston, 2001). Although certain influential ergonomics researchers (e.g. Wickens, 1980; 1992) have argued that people have separate pools of attentional resources for the processing of auditory and visual information, contemporary research has shown that there are in fact extensive crossmodal links in our spatial attention (Spence and Driver, 1997a, 2004). Spence and Read (2003) reported a driving simulator study showing that one of the important difficulties faced by drivers is in terms of trying to direct their visual attention spatially onto the road ahead, while at the same time trying to direct their auditory attention to the location from which the person’s voice is coming. Car drivers found it significantly easier to drive and to listen to a speech stream when both sources of information were presented from the same direction (i.e. from the front), rather than different directions (see Table 5.3). Spence and Read suggested that the introduction of ‘talking windscreens’ (where the sound appeared to come from directly in front of the driver) might make it easier to drive while simultaneously maintaining a mobile phone conversation.

5. CROSS-MODAL CORRESPONDENCES Certain product properties may be perceived by multiple sensory modalities. In particular, vision and touch show considerable overlap in terms of the information that they can provide: For example, people can both see and feel the shape of an object, its size, and the roughness of its surfaces (Lederman and Klatzky, 2004). Although the other senses are less well equipped to perceive these product properties, the shape and size of an object also affect the sound it will make when you tap it, and the surface properties affect the sound it makes when you run your fingers over its surface (see Spence and Zampini, 2006). Through their experiences with real-life objects, people will have some intuitive idea about which types of stimuli tend to co-occur and which do not. These patterns of covariation may form the basis of the correspondences that people perceive across different sensory modalities. If certain pairs of sensory stimuli are more likely to occur together (or seem to fit together better than others), presenting one stimulus of the pair may increase a consumer’s expectation that the other stimulus should also be present (see Garber, Hyatt and Starr, 2001; Schifferstein, 2001).

Multisensory product experience

143

Besides these associations that seem to have formed through prior experience, people might perceive some resemblances among colors, sounds, tastes, smells, and other sensory stimuli directly, because certain dimensions of sensory experience are shared across the different sensory modalities. For example, perception in all of the senses seems to share the dimensions of intensity (weak-strong), duration, and spatial location (Boring, 1942), which suggests that ‘the several senses display a fundamental unity in part because a class of suprasensory attributes pertains to sensations on all modalities’ (Marks, 1978, p. 51). Consistent with this line of thought, von Hornbostel (1931) collected evidence that brightness was a universal sensory dimension. The fact that people perceive correspondences across sensory modalities bears some resemblance to the phenomenon of synesthesia (Cytowic, 1989). However, we prefer to reserve the term ‘synesthesia’ for the phenomenon experienced by some special individuals, who experience a definite and reliable sensation in one sensory modality after being stimulated in a different sensory modality (e.g. Grossenbacher and Lovelace, 2001), and to use the term ‘cross-modal correspondence’ to refer to the connections that most people make between various sensory attributes in different modalities. One of the meaningful distinctions between these two groups is that individual reports of correspondences from synesthetic individuals have been shown to be more consistent over time than those from non-synesthetes (e.g. Baron-Cohen et al., 1993). However, given that these reports differ considerably between individual synesthetes, it is difficult to draw any generalizations from single reports that will be pertinent to design practice. Nevertheless, aggregate data from groups of synesthetes seem to show certain patterns of correspondence that are highly similar to those observed in non-synesthetes (Ward, Huckstep and Tsakanikos, 2006). We now present an example of how designers can make use of the associations that people perceive between different sensory domains. Even though people may find it hard to predict exactly how a product will smell if they have only had a chance to look at it, consistent crossmodal correspondences nevertheless do seem to exist between olfaction and vision. For example, Kemp and Gilbert (1997) have shown that colors that are matched to various odors generally show greater variation in hue across different odor than between various intensive levels of a single odor, suggesting that odor quality is primarily related to hue. Kemp and Gilbert also reported that perceived odor intensity, but not stimulus concentration, was related to visually-perceived brightness. People typically find it difficult to describe their olfactory experiences verbally (see Engen, 1982). Therefore, fragrance companies who want to inform potential consumers of the properties of their fragrances often try to communicate this information nonverbally through the color of the fragrance’s liquid itself, the bottle, the packaging, the advertisements, and other promotional materials. Packaging color has been shown to significantly affect people’s expectations with regard to perfume intensity (dark red is more intense than pastel green), sweetness (dark red is sweeter than pastel green), and freshness (pastel green is fresher than dark red) (Scharf and Volkmer, 2000). To determine which colors will match with a specific fragrance, Schifferstein and Tanudjaja (2004) asked 69 female participants to judge the degree to which each of 17 different colors fitted the smells of each of 14 fine fragrances. Figure 5.1 presents the color profiles for 5 of the fragrances that differ widely in their character. DKNY rated particularly high on saturated orange and yellow, whereas Wish rated high on all red colors, and Kouros fitted best with saturated blue. In addition, Kouros and Miss Dior gave rise to relatively high ratings for gray, black, and brown, whereas most other fragrances gave rise to very low ratings for these colors. Although individual degree-of-fit ratings for odor-color associations showed considerable variation over time in Schifferstein and Tanudjaja’s (2004) study, the aggregated

Pastel

9 8 7 6 5 4 3 2 1

Neutral DKNY Kouros Indecence Miss Dior Wish

i te gre y bla ck bro wn

9 8 7 6 5 4 3 2 1

wh

Saturated

pu rpl e red pin ora k n ye ge llo gre w en blu e

9 8 7 6 5 4 3 2 1

Product Experience

pu rpl e red ora ng ye e llo gre w en blu e

Degree-of-Fit

144

FIGURE 5.1 Color profiles for five fragrances (adapted from Schifferstein and Tanudjaja, 2004; reprinted with permission from Pion Limited, London).

data exhibited a relatively consistent pattern. In addition, it has recently been shown that the associations people report can be replicated using non-verbal psychophysical techniques, thus supporting the claim that they are a consistent, crossmodal phenomenon. In fact, Demattè, Sanabria, and Spence (2006) conducted the first crossmodal study of the Implicit Association Test (Greenwald, McGhee and Schwartz, 1998). The participants in their study had to make speeded discrimination responses to a random sequence of odors (strawberry vs. spearmint) and color patches (pink vs. turquoise) presented via computer. A color patch or an odor was presented as a target stimulus on each trial. The assignment of these targets onto the two response keys was manipulated in order to generate compatible (e.g. responding to the pink color and to the strawberry odor with the same response key) and incompatible (e.g. responding to the pink color and to the spearmint odor with the same response key) blocks of trials. Participants responded significantly more rapidly and more accurately to odor-color pairings having a stronger crossmodal association than to those having a weaker (or no) crossmodal association (see Figure 5.2). Designers can make use of these specific, individually matching colors in order to design packages that communicate a product’s olfactory properties to a consumer (Smets and Overbeeke, 1989). Given a limited set of matching colors, the designer can then pick those colors to create a package that will stand out against the competitors on the store shelves. In Schifferstein and Tanudjaja’s (2004) fragrance study, degree-of-fit ratings were, on average, highest for pastel colors, somewhat lower for saturated colors, and lowest for the neutral colors (see Table 5.4). Some colors showed many significant differences between odors, whereas others showed almost none. Saturated purple, red, and orange, and pastel purple, blue, and green obtained high degree-of-fit ratings and showed virtually no significant differences between odors. Consequently, these colors can be used in almost any fragrance package or advertisement. On the other hand, colors that show more differences between fragrances such as pastel red, pink and pastel yellow might better be reserved for those fragrances to which they best match. Although neutral colors (white, black, gray, brown) do not seem to match particularly well with fragrances in general, they do provide opportunities for some fragrances to create very distinctive packages that will stand out on the perfume shelves, and that still provide a reasonable fit (e.g. Miss Dior – brown; see Figure 5.1). The design strategy to adapt a package to match a fragrance’s olfactory characteristics assumes that consumers do not perceive the fragrance and the package separately, but may perceive them as a unitary whole. Indeed, a number of anecdotal reports suggest

145

Multisensory product experience

FIGURE 5.2 Percentage of correct responses to smells (olfactory) and colors (visual) when responses were coupled in a compatible (strawberry-pink and spearmint-turquoise) or incompatible (strawberry-turquoise and spearmint-pink) manner (adapted from Demattè et al., 2006; reprinted with permission from Oxford University Press).

TABLE 5.4 Mean ratings of degree-of-fit and the number of significant differences between odors for each color (from Schifferstein and Tanudjaja, 2004; reprinted with permission from Pion Limited, London)

Mean degree-of-fit

Number of different odor pairs

Saturated colors Purple Red Orange Yellow Green Blue

4.8 4.5 4.6 4.6 4.0 3.8

2 1 6 15 12 13

Pastel colors Purple Red Pink Orange Yellow Green Blue

5.1 5.1 4.9 5.3 4.9 4.6 4.8

9 22 17 12 19 7 2

Neutral colors White Gray Black Brown

3.3 2.8 2.2 2.6

0 7 19 10

that consumers experience the smell of spray-on deodorants somewhat differently if the coloring of their canisters is changed. Similarly, Thomas Hine (1995) cites the example of when 15% more yellow was added to the green on 7-Up cans, consumers found the taste to be a lot more limey/lemony. In fact, some consumers were so upset by the

146

Product Experience

FIGURE 5.3 MDS similarity spaces for graphic designs (visual – left) and sound characteristics of musical fragments (auditory – right), in both cases represented by CD covers (adapted from van Egmond, 2004; reprinted with permission from Taylor and Francis).

apparent change in taste as to say ‘You’re changing my Seven-Up! Don’t do a “New Coke” on me.’ Because the product formulation in these cases had not changed, these examples show that a product’s packaging can dramatically impact on a consumer’s perception of the product within. Another example of how designers can make use of the correspondences that people perceive across different sensory modalities was provided by van Egmond (2004). He asked participants to group existing CD covers (visual) and representative samples of music taken from those CDs (auditory) on the basis of either their visual or auditory similarity, respectively. Multi-Dimensional Scaling analyses of the similarity data revealed highly similar clusterings for the two types of stimuli (see Figure 5.3). This suggests that graphic designers are, to some extent, able to communicate the characteristics of the music through the designs they create for CD covers.

6. INTERACTIONS BETWEEN VARIOUS SENSORY DOMAINS Much of the research on multisensory integration has focused on the question of how perception in one sensory modality affects perception in a different sensory modality. In some cases, multisensory integration results in multisensory illusions, some of which may find an application in product design. Below, we give a number of examples. In the food realm, multiple interactions have been described between taste, smell, vision, and touch. For example, adding a colorant to an odorous solution generally increases the perceived intensity of its smell; darker solutions are often judged as smelling more intense than lighter solutions. The appropriateness of the color has little or no effect on the perceived intensity of the odor, but it can nevertheless facilitate odor identification and increases people’s liking for a particular odor (Zellner, Bartoli and Eckard, 1991; Zellner and Whitten, 1999). Research on odor-texture interactions has shown that increasing the hardness of a gel leads to a decrease in perceived intensity of its flavor,

Multisensory product experience

147

while flavor release from the gel is hardly affected (Bult, de Wijk and Hummel, 2007; Weel et al., 2002). Furthermore, essentially tasteless odorants may enhance the perceived intensity of a specific taste (Frank, van der Klaauw and Schifferstein, 1993). Examples of odor-texture interactions for non-food products are also available. In a classic study performed more than 70 years ago, Laird (1932) found that women evaluated the quality of silk stockings more positively when they were scented with a narcissus scent compared to their natural (slightly rancid) scent. When asked for the reasons behind their preference, the majority of the housewives pointed to differences in durability, sheen, or weave (i.e., to the tactile and/or visual properties of the stockings), rather than to differences in their olfactory properties. Demattè et al. (2006) recently followed up on Laird’s (1932) classic study. In their first experiment, fabric swatches were presented from a carousel under computer-control, and olfactory stimuli were delivered through a custom-built olfactometer. The participants were asked to rate the perceived softness of each fabric sample, while an odor of clean air was delivered directly to their nostrils. Demattè et al. observed that the fabric swatches were rated as significantly softer when presented with a lemon or lavender odor than when presented with an animal-like odor. While their experimental procedure allowed the authors to test a large number of possible combinations of fragrance and tactile feel within a short time frame, the ecological validity of the testing situation was obviously somewhat limited. Therefore, in a subsequent experiment, Demattè et al. went on to show that when participants evaluated fabric swatches that had actually been impregnated with one or the other fragrance, the fragrance affected the fabric’s softness in much the same way. In fact, if anything, the increase in fabric softness induced by the fragrance when emanating from the fabric itself was somewhat larger than when it was presented via the olfactometer (presumably due to the increased unity of the two signals in the former case). In another study looking at the nature of crossmodal interactions between hearing and touch, Zampini, Guest, and Spence (2003) asked their participants to evaluate the pleasantness and roughness of an electric toothbrush that they had to brush repeatedly across their front teeth. The toothbrush was judged to be rougher and less pleasant when the overall sound level increased and when the high-frequency sounds (above 2 kHz) made by the toothbrush were amplified. The same type of auditory manipulation has also been found to affect the perceived forcefulness of aerosol spray sounds (Spence and Zampini, 2007); the level of perceived carbonation in sparkling water drinks (Zampini and Spence, 2005); and even the perceived crispness of potato chips (Zampini and Spence, 2004). These examples show that people’s perception of the attributes of a product in a given sensory modality is frequently affected by the sensations that are simultaneously being perceived by another modality. It is crucial to note that participants/consumers are typically completely unaware of the occurrence (or nature) of these crossmodal effects. Thus, introspective consumer reports often fail to provide any useful insight into such crossmodal interactions. Therefore, experimental research on individual cases of multisensory perception is necessary in order to unravel the mechanisms behind each of these interactions. The belief among many researchers in the field of cognitive neuroscience is that many of the same rules will govern multisensory integration, no matter what combination of stimulus modalities (and sensory attributes) one looks at. However, it is becoming increasingly clear that some of the rules governing multisensory integration are more important for certain combinations of sensory modalities than for others. Thus, while common spatial origin and temporal alignment play a particularly important role in the integration of auditory, visual, and tactual stimuli (e.g. Congedo, Lécuyer and Gentaz,

148

Product Experience

2006; Kallinen and Ravaja, 2007; Spence, 2007a; Stein and Meredith, 1993), precise temporal synchronization would seem to be less important for the integration of olfactory and gustatory cues with information from the other sensory modalities. Because stimulus processing occurs more slowly for these modalities, the time frame for integration is much wider (Spence and Squire, 2003). Furthermore, taste–smell interactions occur in large part because people tend to confuse smell and taste sensations, probably because they cannot correctly localize the stimuli (see Rozin, 1982). Due to the fact that taste and smell seem to originate from the same location when eating, people may integrate some of the odorant intensity with the intensity of the taste (Frank et al., 1993). On the other hand, the robust crossmodal interactions that have been documented between vision and both taste and smell seem to rely mainly on learned associations (see Demattè, Sanabria and Spence, 2006; Maga, 1974). In such cases, the cognitive associations evoked by sensing something may generate expectations that affect people’s subsequent perception (Cardello and Sawyer, 1992; Schifferstein, 2001).

7. SENSORY (IN)CONGRUITY Knowledge of the roles of the different senses is important for designers who want to communicate a coherent message to the potential customers for their products. In particular, designers should consider which messages an existing product currently communicates through each of the senses (Lindstrom, 2005). The contents of certain of these messages may conflict, and thus the designer may want to check the possibility of changing certain product features in order to improve the coherence of the consumer’s overall multisensory product experience. The congruence of sensory messages in product design is also desirable from an ergonomic perspective, where coherence helps to clarify what a product is about and what it can do. In addition, perceived unity in visual stimuli has been shown to correlate with ratings of both aesthetic appeal and liking (Bell, Holbrook and Solomon, 1991; Veryzer and Hutchinson, 1998). Therefore, multisensory coherence is likely to be positively related to consumer preference. In striving for multisensory coherence, it becomes obvious that a product’s package should be regarded as an integral part of the overall product design. Unfortunately, many packages seem to be designed primarily to protect their content and, thereby, create an almost invincible barrier to experience the product inside. One possible strategy to counteract this is to introduce holes in the package to let potential consumers experience how the product inside feels, smells and sounds. Another strategy is to convey sensory product information through the sensory characteristics of the packaging. We have already seen how visual components (e.g. color) can influence the perception of fragrances and foods. Analogously, the tactual, olfactory, and auditory aspects of package design are also potentially important, even if currently underutilized by many designers. For example, a brand of toilet tissue that claims to be velvety-smooth recently started treating the plastic packaging in which rolls of their toilet paper were sold, so that the plastic itself actually felt velvety-smooth (see Spence, 2007b). It is also possible to impregnate packaging materials with fragrances that are released upon touching the package. Furthermore, through the selection of packaging materials the sounds perceived when opening the package can be modified to enhance, for example, freshness and crispiness of foods (crispy paper or metal foil, see Brown, 1958) or softness of personal care products (thick plastic with rounded shapes producing only soft sounds).

Multisensory product experience

149

FIGURE 5.4 ‘Mary Biscuit’ biscuit box, designed by Stefano Giovannoni (1995) (courtesy Alessi spa).

A nice example of a product in which all the senses are stimulated to bring about a unitary multisensory experience is Alessi’s ‘Mary Biscuit’ designed by Stefano Giovannoni (see Figure 5.4). This biscuit-box seems to enhance the coziness of social visits. The majority of biscuit boxes are made from metal: They feel cold and may contain sharp edges; the sounds they make when they are opened are also sharp, and when the cover accidentally falls on the floor it can make a loud and unpleasant noise. Hence, through the senses of touch and audition, the majority of biscuit tins communicate hostility and coldness rather than friendship and warmth. The ‘Mary Biscuit’ container, however, is different in that it is made of plastic, and contains only rounded edges, and its shape resembles that of a pillow. The box feels soft and warm to touch, it makes only soft noises when you open it or put it down, and the box itself seems to invite the user to cuddle it. In contrast to the metal box, which usually does not have a distinctive smell of its own, the cover of the ‘Mary Biscuit’ is impregnated with a vanilla-like odor that becomes apparent when the container is opened. Because many cookies contain the flavor of vanilla anyway, the additional smell may enhance the experience of tasting a cookie. In addition, the smell might also evoke nostalgic memories of family visits to one’s grandmother and may, thereby, enhance the feeling of sharing an experience with intimate friends or relatives. Upon closer inspection, the ‘Mary Biscuit’ container can therefore be seen to use the majority of a consumer’s senses in order to communicate similar, but also seemingly redundant, information to produce a pleasing multisensory impression. In the years to come, the increasing awareness of the roles of the different sensory modalities and the interactions between them will shift the focus for many designers away from the physical product to the specific experience that a product evokes. For example, designers can develop a scenario of the sensory events that occur when a person encounters a product (MacDonald, 2002). They may then use this scenario as the starting point for the design of a new product. Such an approach is likely to result in more interesting, involving, and ultimately more engaging products and product experiences, because such products exploit the full potential of people’s sensory connections with the surrounding world (Howes, 2005). However, designers may not always want to design products for which all of the senses communicate the same message. On the contrary, in some cases designers may want to evoke surprise by introducing sensory discrepancies or uncertainties. Examples of such incongruities include a chair made from rope, which makes a person wonder

150

Product Experience

whether it is stable enough to sit on, or a cup that seems to be made from metal but is actually made from plastic, and is thus much lighter than one would expect. The challenge in these cases seems to be to combine familiarity and originality within the same design (see Hekkert, Snelders and van Wieringen, 2003). Even designers who want to surprise people will generally make sure that the majority of the product-related information communicates the same message, while only one particular aspect is responsible for the element of surprise. An analysis of designs presented in the International Design Yearbooks (1999– 2004) suggests that up to 6% of these designs incorporate some form of visual-tactual incongruity (Ludden, Schifferstein and Hekkert, 2007a), implying that the product feels different from what would be expected on the basis of visual information alone. The surprising products found mainly consisted of furniture and other interior products, such as lamps, vases, tiles, and dinner-sets. This might imply that surprise as a design strategy is typically limited to products in well-known categories that consumers can easily identify. On the basis of an analysis of 101 products exhibiting visual-tactual incongruity, Ludden, Schifferstein, and Hekkert (2007a,b) made an important distinction between two different types of surprise. The ‘Visible Novelty’ (VN) type of surprise consists of products that seem unfamiliar to the perceiver. Consequently, only uncertain expectations can be formed on the basis of associations with the product category, overall shape, or material use. Although the consumer’s surprise reaction is likely to be limited, because they are unsure of the product’s exact properties and purpose, the product may be interesting and intriguing because of the ambiguity it creates in their mind. An example is provided by the bench ‘Shrunken furniture’ designed by Bertjan Pot (Figure 5.5 (left)). This bench is made out of polystyrene, which is covered in knitted cloth and then vacuumed and hardened with wax. The combination of materials with the new shape results in a bench that looks like it is made out of a familiar soft material, like foam rubber. In reality, the bench feels hard. In addition,

FIGURE 5.5 Products evoking the Visible Novelty (left) and the Hidden Novelty (right) surprise types.

Multisensory product experience

151

because this material is lighter than the materials that are commonly used in the construction of benches (such as wood), this bench is much lighter than one would expect. On the other hand, products belonging to the ‘Hidden Novelty’ (HN) surprise type seem very familiar to the perceiver, and thus lead to stronger reactions of surprise, because the tactual properties are very different from those that are expected. For example, the vase in Figure 5.5 (right) looks like a familiar, traditional vase made from glass or crystal. In actual fact, it is made from plastic and, consequently, a consumer will experience it as being surprisingly light when he/she picks it up. Ludden et al. (2007b) found that people react differently to products evoking these two types of surprise. Surprise ratings were higher and facial expressions typically associated with surprise occurred more often for HN than for VN products. People tended to use more vocal expressions when interacting with HN products, and more exploratory behavior while interacting with VN products. It is possible that people enjoyed exploring VN products more than HN products, or else that they were more curious about the exact material properties of VN products. Alternatively, however, they may have needed more time in order to interact with the VN products, because they wanted to try and figure out what exactly it was that had triggered their surprise reaction. On the other hand, it seems that the surprise experienced by consumers upon touching the HN product was understood immediately, and that further exploration or cognitive effort was unnecessary. This may partly explain why people typically experience VN products as more interesting than HN products.

8. SENSORY DOMINANCE When a person interacts with a product, the inputs from the various senses should be integrated in order to give rise to a unified multisensory product experience. How does this multisensory integration take place? Laboratory studies of multisensory integration have typically tried to determine the relative contribution of each input variable to (an aspect of) the overall product experience. But what are the factors that affect this integration process? The results of many empirical studies now show that the relative importance of the various senses does not only depend on the particular type of product being investigated, but also on the specific task that the user has to perform with the product (e.g. Schifferstein, 2006), as well as on the personal characteristics of the user (e.g. Peck and Childers, 2003; Wrzesniewski, McCauley and Rozin, 1999). One general rule-of-thumb is that the interpretation of sensory information that is more ambiguous usually receives less weight in the formation of multisensory product judgments (Ernst and Bülthoff, 2004; Hoch and Ha, 1986; van Beers, Sittig and van der Gon, 1999). Therefore, on the basis of an evaluation of the roles of the individual sensory modalities (see Section 2), one might conclude that vision and touch are likely to dominate product perception and experience in many real-life situations (Schifferstein and Cleiren, 2005). Moreover, vision is likely to have an even larger impact on product experiences than touch, because visual information is processed more quickly and is typically available sooner (i.e. we see an object on the shelf before we decide to pick it up; we inspect the food on our plates before we decide what to eat next). Furthermore, while vision can be used to explore large objects with a minimum of delay, this would be impossible using only touch. As a consequence, vision has been found to direct exploratory behavior involving the other senses (e.g. Heller, 1982; Klatzky, Lederman and Matula, 1993). However, the dominant role of vision and, to a lesser extent, touch is likely to be mainly limited to the functional user-product interaction and to the conscious

152

Product Experience

TABLE 5.5 Mean importance ratings of the five modalities on a five-point scale during usage of several products (adapted from Schifferstein, 2006)

Car Vacuum cleaner TV Couch Vase Hammer Paper Wristwatch Shoes Underwear Deodorant Flowers Cookies

Vision

Touch

Audition

Smell

Taste

4.3 3.5 4.4 4.6 4.4 2.7 3.6 4.6 4.7 4.2 2.5 4.8 4.1

3.8 3.3 2.3 4.4 2.9 3.8 3.5 4.1 4.5 4.5 3.8 3.1 3.3

4.3 4.2 4.5 2.6 1.7 2.2 2.1 3.5 3.0 1.9 1.7 1.4 1.9

3.5 2.7 1.9 2.9 2.0 1.3 2.8 2.0 3.1 3.4 4.6 4.4 4.3

1.1 1.1 1.1 1.1 1.1 1.1 1.2 1.3 1.1 1.4 1.2 1.6 4.9

experience of that interaction. The other sensory modalities may nevertheless still play important roles in terms of modulating the emotional experiences that are evoked by products (Schifferstein and Desmet, 2007). In one recent study, Schifferstein (2006) compared the conscious evaluations of the relative importance of the different senses during actual product usage. Eighty people reported on a five-point scale how important they found the information provided by each of the five main senses during the usage of 45 different products (see Table 5.5). The results showed that the usage value of a vase, for instance, was determined primarily by its visual appearance. For a TV, the sound was judged as being just as important as its appearance. For simple tools and utensils, such as a hammer, the tactile characteristics were of primary importance, followed by their visual appearance. For products associated with cleaning and personal care, olfactory cues generally played a key role in combination with their tactual and, to a lesser extent, visual properties. For food products, taste was judged as being the most important sense, generally followed by smell and then vision. Going through the list of products, examples were found for which only a single sensory modality was important during usage (vase), but also examples for which four or more sensory modalities were found important (car, vacuum cleaner, shoes, cookies). In controlled experimental studies, the relative importance of the different senses has primarily been investigated by providing participants with conflicting information on (an aspect of) a single object through two or more sensory modalities at the same time. Although incongruent information is presented in these studies, the research focus has not been on the participant’s surprise reaction to that incongruity, but rather on if (and how) the available discrepant sensory information is integrated (weighted). By comparing the overall responses to the objects, the impact of the various modalities can be quantified. Note that the degree of conflict in these studies has often been kept below the threshold where such discrepancy becomes noticeable (e.g. Ernst and Banks, 2002; Rock and Victor, 1964). These controlled experimental studies have shown that the importance of a sensory modality depends both on the stimuli being evaluated, as well as on the specific task instructions given to participants (Guest and Spence, 2003; Lederman, Thorne and Jones,

Multisensory product experience

153

1986; Welch and Warren, 1980). Several explanations for these findings have been put forward. For example, according to the ‘modality appropriateness’ hypothesis (Freides, 1974; Welch and Warren, 1980), people weigh sensory inputs according to their relative unimodal performance capabilities for a specific task. According to this hypothesis, the more suitable a sensory modality to perceive a product attribute, the larger its impact on the perception of that particular product aspect will be. According to another hypothesis (Posner, Nissen and Klein, 1976), however, the modality that receives the observers’ directed attention will dominate perception. Note that these two accounts yield similar predictions in many cases. Strictly speaking, perception depends mainly on the nature of the information and the physical laws that govern the situation, and not so much on the specific sensory modality the information happens to be presented in. For instance, by means of a camera that transforms visual information into electronic pulses that can be felt on the tongue, blind people can (in some sense) learn to ‘see’, and people who have lost sensitivity in the vestibular system can regain their sense of balance (Bach-y-Rita, 2004; O’Regan and Noë, 2001). In addition, devices can be developed to convey important information through a different sensory modality under extreme conditions where the original sensory system is unable to function properly. For example, van Erp and van Veen (2006) recently reported on the development of a vest containing 56 vibrators that can help an astronaut to obtain a sense of ‘up’ and ‘down’ in a microgravitational environment. Nonetheless, under natural conditions the type of information perceived is always linked to a particular sensory modality. Consequently, the perceived importance of a particular sensory modality is likely to be determined by whether it can transduce important information or not. Accordingly, Schifferstein (2006) concluded that ‘the often referred to dominance of vision is likely to reflect people’s overall tendency to find visual input relatively important when its role is evaluated for the ensemble of activities performed. As a consequence, the role of the senses is likely to depend on the specific products used, the frequency with which they are used, and the importance attached to the activities performed.’ (Schifferstein, 2006, p. 60). Schifferstein suggested that the importance of vision in Western societies may have increased over time due to the range of products that have been created. For example, many products that facilitate human communication (such as television, newspapers, the Internet) require major visual input. When such products become part of people’s daily lives, the importance of vision is, if anything, likely to increase (see also Spence, 2002). Interestingly, the importance of a sensory modality does not only depend on the nature of a particular product, but also on its stage of ownership. People’s subjective reports suggest that the role of vision is primarily dominant during the acquisition process. After purchase, the role of the other modalities often increases significantly, at the expense of vision. However, during ownership, the importance of vision increases once again, because the owner’s appreciation of the product’s visual appearance decreases due to fashion changes and to the accumulation of signs of usage, such as scratches (Fenko, Schifferstein and Hekkert, 2007). Design is an integrative discipline that requires the ability to materialize predefined intentions into new design solutions. Designers can manipulate many different variables, such as product shape, weight, smell, and finish, to fulfill these intentions. In the end it is the interplay between all of these design variables that will determine the experiential value of the design, and in most cases it is impossible to determine the contribution of individual variables to the end result. Nevertheless, knowledge regarding the relative importance of the different senses can be valuable because it helps to prevent a fixation

154

Product Experience

on the visual modality. Understanding the relative importance of the different senses can be useful for balancing the time and resources invested in new product development projects (Lindstrom, 2005). A broad focus on all the senses can stimulate designers to explore the value of hitherto neglected sensory modalities for creating new market opportunities. For example, for a mobile phone the smell and feel of the cover are generally not regarded as important product attributes. Nevertheless, a mobile phone can be regarded as a very personal, almost intimate, object that people carry around all day long and that they keep close to their face during calls. In this case, introducing phone covers made from different materials, varying in surface texture and smell characteristics, might make phones feel and smell more personal, thereby clearly discriminating between ‘own’ and ‘foreign’, and increasing the intimacy of the calling experience. As a consequence, people may become more attached to their phone over time (see Chapter 17).

9. CONCLUSIONS AND DIRECTIONS FOR FUTURE RESEARCH The rapid growth of research in recent years on the topic of multisensory perception has produced many fascinating insights into how our brain integrates the information received through the different senses. In the previous sections we have attempted to provide an overview of the state-of-the-art knowledge regarding the ways in which people use their senses in their interactions with products. Through the accelerated application of this growing source of knowledge, a future generation of consumer products can be developed that offer experiences that involve more sensory modalities, and that make the stimulation of the senses more unified and coherent or, possibly, more interesting and surprising, but in all cases, more pleasing. However, a number of additional topics will need to be investigated more systematically in the years to come. They have not been treated here in full because of the dearth of available literature on these topics at present. One particularly important topic for future research relates to individual differences. Given that there are robust and striking individual differences in sensory perception, manufacturers may increasingly start to develop products that have been sensorially tailored for a particular section of the population. Differences between individuals in sensory capabilities may, for example, be due to sex (Velle, 1987) or age (Corso, 1971; Doty et al., 1984; Robinson, 1988). It is important to note that the elderly do not form a homogeneous population: Their capabilities, health, functional status, and nutritional needs vary widely. What’s more, sensory capabilities do not necessarily decline to the same degree for each individual (e.g. see Stevens, Cain and Burke, 1988) and, therefore, the heterogeneity of sensory capabilities in the population is likely to increase. It is therefore possible that demographic changes will not only lead to products that have been specifically designed to appeal to the advanced-age segment of the population (e.g. see Pirkl, 1994; Schiffman, 2000), but also to products that are designed to fit the individual needs of the end user (e.g. Mugge, Schifferstein and Schoormans, 2006). Apart from these variations in sensory capabilities, individuals may also differ in the ways in which they prefer to respond to sensory stimulation. Schifferstein and Smeets (2006) named the latter construct a person’s perceptual style. Differences in perceptual style may, for example, influence whether consumers tend to rely on visual or tactual input when they buy a product (e.g. Peck and Childers, 2003). In an educational context, it has been argued that differences in perceptual style partly determine whether a teacher should present visual images, verbal messages, or physical objects that can be explored tactually, in order to communicate information efficiently to a particular student

Multisensory product experience

155

(e.g. Dunn and Dunn, 1992). Interestingly, the sensory expertise of the product user may also be identified as a determining factor of perceptual style (e.g. Binns, 1937). Another area that has, as yet, received relatively little attention from researchers relates to possible cultural differences in how people use their senses in their interaction with products, and the effect this has on their evaluation of these products. A culture specifies the rituals that are connected to product usage (Howes, 1991; MacDonald, 2002). Through exposure to nature, surroundings, and products in everyday life, people develop a sensory frame of reference. On the one hand, this frame of reference contains a person’s sensory knowledge that has accumulated over time, which may affect what a person can perceive through a particular sensory modality (e.g. Ayabe-Kanamura et al., 1998). On the other hand, it may determine which sensory impressions go together and which do not (Scanlon, 1985; Walker, 1987). For example, learned odor-color associations are likely to differ cross-culturally: Lemons are yellow in Europe, but green in parts of South America; cinnamon is associated with red in the USA (sweets), but with brown in the UK (spice); meanwhile the odor of cucumber is associated with green in the UK, but is often associated with red in Spain due to its ubiquitous presence in gazpacho soup (Spence, 2007b). These differences clearly need to be taken into account when developing products for different countries or for multi-cultural societies. It is worth highlighting the fact that technology can also change the way in which people interact with everyday products. For example, when Unilever introduced their newly-designed canister for the Axe/Lynx deodorant brand (where the spray mechanism was built into the cap), it was noted that certain consumers (particularly young males) often used their thumb to operate the aerosols, while consumers have traditionally used their index finger instead. One of Unilever’s packaging directors attributed this change to the enhanced usage of the thumbs when text messaging or playing video games (Carter, 2006). Product designers need to take such changes in haptic behaviour into account when engaging in the design process. Another topic that is of great interest to designers and that is likely to increase in popularity is the use of sensory icons. Traditionally, companies have invested a lot of money in trying to define and communicate a desirable identity for their brands. Next to formulating a verbal message, they have typically developed visual stimuli in order to support this process, such as a brand logo, and a selection of colors and fonts to be used in packaging, advertising, and other types of graphic design (see Hine, 1995). More recently, however, sensory impressions created in the other sensory modalities have increasingly been used to create and modify brand identity (see Lindstrom, 2005). For example, in the automotive industry, the contribution of auditory cues to product perception has been known about for many years (e.g. Packard, 1957, p. 111). On occasion the unique, or characteristic, sounds made by a specific vehicle have been linked to a specific manufacturer or brand name. For example, Harley-Davidson went to great lengths to try and trademark the distinctive sound of their motorcycle engines, which they considered to be an essential part of their product experience (Lyon, 2003; Sapherstein, 1998). In fact, car designers have invested a great deal of research into the sound of everything from the engine to the sound made when the door is closed (see Spence and Zampini, 2006). Distinctive product sounds, sometimes known as ‘signature sounds’, can denote ‘character’ and can become strongly associated with a particular product and its functionality. For food products, think of the snap of the ‘Kitkat’, the distinctive crack of the chocolate breaking as one bites into a ‘Magnum’ ice cream, or the sound of the release of carbonation as a bottle of ‘Schweppes’ is opened. It also seems increasingly likely that companies will develop their own unique ‘signature scents’ that will help to create a distinctive, unifying, and memorable image that

156

Product Experience

will help to distinguish their products from those of their competitors (see Miller, 1993; Spangenberg, Crowley and Henderson, 1996). In fact, according to one report in the Wall Street Journal (Ellison and White, 2000), British Airways has already trialled the use of a signature scent called ‘Meadow Grass’ in their executive airport lounges. After all, what could be nicer for the world-weary traveler than to be greeted by a familiar scent when they walk into the lounge, signaling that they are almost home? It is probably only a matter of time before companies start copyrighting their signature scents in order to protect their own unique olfactory images (Classen, Howes and Synnott, 1994). Recently in the Netherlands, Lancôme successfully defended the copyright on the smell of their Trésor fragrance against a copyist (Frijters and van Houte, 2004). Furthermore, several products are already on the market that have registered a specific scent as (part of) their trademark. UK consumers can now buy darts arrows that have been impregnated with the smell of beer, while consumers in the Netherlands can buy tennis balls impregnated with the smell of green grass as part of a registered brand (Goosen, 1998). By analogy to the previous examples, it should also be possible to register a certain, characteristic feel as part of a certain brand. This may then open up possibilities for the creation of multisensory brands, consisting of the definition of multiple sensory impressions underlying brand identities, with possibilities to register characteristic sensations or combinations of sensations as part of the brand (Howes, 2005; Lindstrom, 2005).

ACKNOWLEDGMENTS This research was supported by MAGW VIDI grant 452-02-028 of the Netherlands Organization for Scientific Research (N.W.O.) awarded to H. N. J. Schifferstein. The authors are indebted to all of their co-workers for their contributions to the research presented.

REFERENCES Ayabe-Kanamura, S., Schicker, I., Laska, M., et al. (1998). Differences in perception of everyday odors: a Japanese–German cross-cultural study. Chemical Senses, 23, 31–38. Bach-y-Rita, P. (2004). Tactile sensory substitution studies. Annals of the New York Academy of Sciences, 1013, 83–91. Bahrick, L. E. and Lickliter, R. (2000). Intersensory redundancy guides attentional selectivity and perceptual learning in infancy. Developmental Psychology, 36, 190–201. Baron, R. A. and Kalsher, M. J. (1998). Effects of a pleasant ambient fragrance on simulated driving performance: the sweet smell of ... safety? Environment and Behavior, 30, 535–552. Baron-Cohen, S., Harrison, J., Goldstein, L. H. and Wyke, M. (1993). Coloured speech perception: is synesthesia what happens when modularity breaks down? Perception, 22, 419–426. Bell, S. S., Holbrook, M. B. and Solomon, M. R. (1991). Combining esthetic and social value to explain preferences for product styles with the incorporation of personality and ensemble effects. Journal of Social Behavior and Personality, 6, 243–274. Bertelson, P. (1999). Ventriloquism: a case of cross-modal perceptual grouping. In: G. Aschersleben, T. Bachmann and J. Müsseler (Eds.) Cognitive contributions to the perception of spatial and temporal events, pp. 347–362. Amsterdam: Elsevier. Betts, G. H. (1909). The distribution and functions of mental imagery. New York: Teachers College, Columbia University. Binns, H. (1937). Visual and tactual ‘judgement’ as illustrated in a practical experiment. British Journal of Psychology, 27, 404–410. Biocca, F., Kim, J. and Choi, Y. (2001). Visual touch in virtual environments: an exploratory study of presence, multimodal interfaces, and cross-modal sensory illusions. Presence: Teleoperators and Virtual Environments, 10, 247–265.

Multisensory product experience

157

Blomqvist, E. H., Bramerson, A., Stjarne, P. and Nordin, S. (2004). Consequences of olfactory loss and adopted coping strategies. Rhinology, 42, 189–194. Bolanowski, S. J., Verrillo, R. T. and McGlone, F. (1999). Passive, active and intra-active (self) touch. Somatosensory and Motor Research, 16, 304–311. Boring, E. G. (1942). Sensation and perception in the history of experimental psychology. New York: Appleton-Century-Crofts. Brown, R. L. (1958). Wrapper influence on the perception of freshness in bread. Journal of Applied Psychology, 42, 257–260. Bult, J. H. F., de Wijk, R. A. and Hummel, T. (2007). Investigations on multimodal sensory integration: texture, taste, and ortho- and retronasal olfactory stimuli in concert. Neuroscience Letters, 411, 6–10. Burns, L. D., Brown, D. M., Cameron, B., Chandler, J. and Kaiser, M. J. (1995). Sensory interaction and descriptions of fabric hand. Perceptual and Motor Skills, 81, 120–122. Cardello, A. V. and Sawyer, F. M. (1992). Effects of disconfirmed consumer expectations on food acceptability. Journal of Sensory Studies, 7, 253–277. Carskadon, M. A. and Herz, R. S. (2004). Minimal olfactory perception during sleep: why odor alarms will not work for humans. Sleep, 27, 402–405. Carter, S. (2006, 5–6 December). Brand owner case study: Axe ‘Millenium’ body spray. Paper presented at the 3rd Packaging Impact and Innovation conference, London, UK. Chan, A. H. S. and Chan, K. W. L. (2006). Synchronous and asynchronous presentations of auditory and visual signals: implications for control console design. Applied Ergonomics, 37, 131–140. Classen, C., Howes, D. and Synnott, A. (1994). Aroma: the cultural history of smell. London: Routledge. Congedo, M., Lécuyer, A. and Gentaz, E. (2006). The influence of spatial delocation on perceptual integration of vision and touch. Presence: Teleoperators and Virtual Environments, 15, 353–357. Corso, J. F. (1971). Sensory processes and age effects in normal adults. Journal of Gerontology, 26, 90–105. Cytowic, R. E. (1989). Synesthesia: a union of the senses. New York: Springer Verlag. Demattè, M. L., Sanabria, D. and Spence, C. (2006). Cross-modal associations between odors and colors. Chemical Senses, 31, 531–538. Demattè, M. L., Sanabria, D., Sugarman, R. and Spence, C. (2006). Crossmodal interactions between olfaction and touch. Chemical Senses, 31, 291–300. Desmet, P. M. A. (2002). Designing emotions. Unpublished PhD dissertation, Delft University of Technology, Delft. Dilks, D. D., Dalton, P. and Beauchamp, G. K. (1999). Cross-cultural variation in responses to malodors. Chemical Senses, 24, 599. Dinh, H. Q., Walker, N., Song, C., Kobayashi, A. and Hodges, L. F. (1999, 13–17 March). Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments. Paper presented at the IEEE Virtual Reality Conference, Houston, TX. Doty, R. L., Shaman, P., Applebaum, S. L., Giberson, R., Siksorski, L. and Rosenberg, L. (1984). Smell identification ability: changes with age. Science, 226, 1441–1443. Dubois, D. (2000). Categories as acts of meaning: the case of categories in olfaction and audition. Cognitive Science Quarterly, 1, 35–68. Dunn, R. and Dunn, K. (1992). Teaching secondary students through their individual learning styles. Needham Heights, MA: Allyn and Bacon. Edworthy, J. and Hellier, E. (2006). Complex nonverbal auditory signals and speech warnings. In: M. S. Wogalter (Ed.) Handbook of warnings, pp. 199–220. Mahwah, NJ: Erlbaum. Ellison, S. and White, E. (2000, November 27). ‘Sensory’ marketers say the way to reach shoppers is the nose. The Wall Street Journal. Engen, T. (1982). The perception of odors. New York: Academic Press. Ernst, M. O. and Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433. Ernst, M. O. and Bülthoff, H. H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8, 162–169. Fenko, A., Schifferstein, H. N. J. and Hekkert, P. (2007). Shifts in sensory dominance between various stages of user–product interactions (submitted). Field, T. (2001). Touch. Cambridge, MA: MIT Press. Frank, R. A., van der Klaauw, N. J. and Schifferstein, H. N. J. (1993). Both perceptual and conceptual factors influence taste–odor and taste–taste interactions. Perception and Psychophysics, 54, 343–354. Freides, D. (1974). Human information processing and sensory modality: cross-modal functions, information complexity, memory, and deficit. Psychological Bulletin, 81, 284–310. Frijters, J. E. R. and van Houte, C. P. M. (2004). De geur van parfum zit niet in de fles (The smell of perfume does not reside in the bottle). Nederlands Juristen Blad, 79, 1987–1989.

158

Product Experience

Garber, L. L., Hyatt, E. M. and Starr, R. G. (2001). Placing food color experimentation into a valid consumer context. Journal of Food Products Marketing, 7(3), 3–24. Goosen, W. F. P. (1998, 5 March 1998). Geuren als merk (Smells as brands). Paper presented at the Geurenmarketing congress, Amsterdam. Greenwald, A. G., McGhee, D. E. and Schwartz, J. L. K. (1998). Measuring individual differences in implicit cognition: the implicit association test. Journal of Personality and Social Psychology, 74, 1464–1480. Grossenbacher, P. G. and Lovelace, C. T. (2001). Mechanisms of synesthesia: cognitive and physiological constraints. Trends in Cognitive Sciences, 5, 36–41. Guest, S. and Spence, C. (2003). Tactile dominance in speeded discrimination of textures. Experimental Brain Research, 150, 201–207. Hekkert, P., Snelders, D. and van Wieringen, P. C. W. (2003). ‘Most advanced, yet acceptable’: typicality and novelty as joint predictors of aesthetic preference in industrial design. British Journal of Psychology, 94, 111–124. Heller, M. A. (1982). Visual and tactual texture perception: intersensory cooperation. Perception and Psychophysics, 31, 339–344. Herz, R. S. (1998). An examination of objective and subjective measures of experience associated to odors, music, and paintings. Empirical Studies of the Arts, 16, 137–152. Herz, R. S. and Schooler, J. W. (2002). A naturalistic study of autobiographical memories evoked by olfactory and visual cues: testing the Proustian hypothesis. American Journal of Psychology, 115, 21–32. Hine, T. (1995). The total package: The secret history and hidden meanings of boxes, bottles, cans, and other persuasive containers. New York: Little Brown. Hinton, P. B. and Henley, T. B. (1993). Cognitive and affective components of stimuli presented in three modes. Bulletin of the Psychonomic Society, 31, 595–598. Ho, C., Reed, N. and Spence, C. (2006). Assessing the effectiveness of ‘intuitive’ vibrotactile warning signals in preventing front-to-rear-end collisions in a driving simulator. Accident Analysis and Prevention, 38, 988–996. Ho, C. and Spence, C. (2005a). Assessing the effectiveness of various auditory cues in capturing a driver’s visual attention. Journal of Experimental Psychology: Applied, 11, 157–174. Ho, C. and Spence, C. (2005b). Olfactory facilitation of dual-task performance. Neuroscience Letters, 389, 35–40. Ho, C., Tan, H. Z. and Spence, C. (2005). Using spatial vibrotactile cues to direct visual attention in driving scenes. Transportantion Research Part F, 8, 397–412. Hoch, S. J. and Ha, Y. W. (1986). Consumer learning: advertising and the ambiguity of product experience. Journal of Consumer Research, 13, 221–233. Hoffman, H. G., Hollander, A., Schroder, K., Rousseau, S. and Furness, T. I. (1998). Physically touching and tasting virtual objects enhances the realism of virtual experiences. Journal of Virtual Reality, 3, 226–234. Horrey, W. J. and Wickens, C. D. (2006). Examining the impact of cell phone conversations on driving using meta-analytic techniques. Human Factors, 48, 196–205. Howes, D. (1991). The varieties of sensory experience: a sourcebook in the anthropology of the senses. Toronto: University of Toronto Press. Howes, D. (2005). Hyperesthesia, or, the sensual logic of late capitalism. In: D. Howes (Ed.) Empire of the senses: the sensual culture reader, pp. 281–303. Oxford: Berg. Kallinen, K. and Ravaja, N. (2007). Comparing speakers versus headphones in listening to news from a computer – individual differences and psychophysiological responses. Computers in Human Behavior, 23, 303–317. Kames, A. J. (1978). A study of the effects of mobile telephone use and control unit design on driving performance. IEEE Transactions on Vehicular Technology, VT-27, 282–287. Kemp, S. E. and Gilbert, A. N. (1997). Odor intensity and color lightness are correlated dimensions. American Journal of Psychology, 110, 35–46. Kitagawa, N. and Spence, C. (2006). Audiotactile multisensory interactions in information processing. Japanese Psychological Research, 48, 158–173. Klatzky, R. L., Lederman, S. J. and Matula, D. E. (1991). Imagined haptic exploration in judgments of object properties. Journal of Experimental Psychology: Learning Memory and Cognition, 17, 314–322. Klatzky, R. L., Lederman, S. J. and Matula, D. E. (1993). Haptic exploration in the presence of vision. Journal of Experimental Psychology: Human Perception and Performance, 19, 726–743. Klatzky, R. L., Lederman, S. J. and Metzger, V. A. (1985). Identifying objects by touch: an ‘expert system’. Perception and Psychophysics, 37, 299–302. Kosslyn, S. M. (1994). Image and brain: the resolution of the imagery debate. Cambridge, MA: MIT Press.

Multisensory product experience

159

Krumhansl, C. L. (2002). Music: a link between cognition and emotion. Current Directions in Psychological Science, 11, 45–50. Laird, D. A. (1932). How the consumer estimates quality by subconscious sensory impressions. Journal of Applied Psychology, 16, 241–246. Lederman, S. J. (1979). Auditory texture perception. Perception, 8, 93–103. Lederman, S. J. and Klatzky, R. L. (2004). Multisensory texture perception. In: G. A. Calvert, C. Spence and B. E. Stein (Eds.) The handbook of multisensory processes, pp. 107–122. Cambridge, MA: MIT Press. Lederman, S. J., Thorne, G. and Jones, B. (1986). Perception of texture by vision and touch: multidimensionality and intersensory integration. Journal of Experimental Psychology: Human Perception and Performance, 12, 169–180. Li, H., Daugherty, T. and Biocca, F. (2002). Impact of 3-D advertising on product knowledge, brand attitude, and purchase intention: the mediating role of presence. Journal of Advertising, 31(3), 43–57. Li, H., Daugherty, T. and Biocca, F. (2003). The role of virtual experience in consumer learning. Journal of Consumer Psychology, 13, 395–407. Lindstrom, M. (2005). Brand sense: build powerful brands through touch, taste, smell, sight, and sound. New York: Free Press. Ludden, G. D. S., Schifferstein, H. N. J. and Hekkert, P. (2007a). Surprise as a design strategy. Design Issues (in press). Ludden, G. D. S., Schifferstein, H. N. J. and Hekkert, P. (2007b). Visual – tactual incongruities in products as sources of surprise. Empirical Studies of the Arts (submitted). Lyon, R. H. (2003). Product sound quality – from perception to design. Sound and Vibration, 37(3), 18–22. MacDonald, A. (2002). The scenario of sensory encounter: cultural factors in sensory–aesthetic experience. In: W. S. Green and P. W. Jordan (Eds.) Pleasure with products: beyond usability, pp. 113–123. London: Taylor and Francis. Maga, J. A. (1974). Influence of color on taste thresholds. Chemical Senses and Flavour, 1, 115–119. Marks, L. E. (1978). The unity of the senses: interrelations among the modalities. New York: Academic Press. Marks, L. E. (2002). The role of attention in chemosensation. Food Quality and Preference, 14, 147–155. Marks, L. E. and Wheeler, M. E. (1998). Attention and the detectability of weak taste stimuli. Chemical Senses, 23, 19–29. Martin, M. G. F. (1995). Bodily awareness: a sense of ownership. In: J. L. Bermudez, A. Marcel and N. Eilan (Eds.) The body and the self, pp. 267–289. Cambridge, MA: MIT Press. Meredith, C., Edworthy, J. and Rose, D. (1999). Observational studies of auditory warnings on the intensive care unit. In: N. A. Stanton and J. Edworthy (Eds.) Human factors in auditory warnings, pp. 305–317. Aldershot: Ashgate. Miller, C. (1993). Scent as a marketing tool: retailers – and even a casino – seek sweet smell of success. Marketing News, 27 (January 18), 271–272. Miron, D., Duncan, G. H. and Bushnell, M. C. (1989). Effects of attention on the intensity and unpleasantness of thermal pain. Pain, 39, 345–352. Momtahan, K., Hétu, R. and Tansley, B. (1993). Audibility and identification of auditory alarms in the operating room and the intensive care unit. Ergonomics, 36, 1159–1176. Mugge, R., Schifferstein, H. N. J. and Schoormans, J. P. L. (2006). The different ways to personalize products: a categorization study. Working paper: Delft University of Technology, the Netherlands. O’Regan, J. K. and Noë, A. (2001). A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences, 24, 939–1031. Packard, V. (1957). Hidden persuaders. Harmondsworth, Middlesex: Penguin Books. Pecher, D., Zeelenberg, R. and Barsalou, L. W. (2003). Verifying different-modality properties for concepts produces switching costs. Psychological Science, 14, 119–124. Peck, J. and Childers, T. L. (2003). Individual differences in haptic information processing: the ‘need for touch’ scale. Journal of Consumer Research, 30, 430–442. Pirkl, J. J. (1994). Transgenerational design: products for an aging population. New York: Van Nostrand Reinhold. Pöppel, E. (1988). Mindworks: time and conscious experience. New York: Harcourt Brace Jovanovich. Posner, M. I., Nissen, M. J. and Klein, R. M. (1976). Visual dominance: an information-processing account of its origins and significance. Psychological Review, 83, 157–171. Ramsdall, D. A. (1978). The psychology of the hard-of-hearing and the deafened adult. In: H. Davis and S. R. Silverman (Eds.) Hearing and deafness (4th Ed.), pp. 499–510. New York: Holt, Rinehart and Winston. Redelmeier, D. A. and Tibshirani, R. J. (1997). Association between cellular-telephone calls and motor vehicle collisions. New England Journal of Medicine, 336, 453–458.

160

Product Experience

Reisberg, D. (1992). Auditory imagery. Hillsdale, NJ: Erlbaum. Robinson, D. W. (1988). Threshold of hearing as a function of age and sex for the typical unscreened population. British Journal of Audiology, 22, 5–20. Rock, I. and Victor, J. (1964). Vision and touch: an experimentally created conflict between the two senses. Science, 143, 594–596. Rozin, P. (1982). ‘Taste–smell confusions’ and the duality of the olfactory sense. Perception and Psychophysics, 31, 397–401. Sapherstein, M. B. (1998). The trademark registrability of the Harley-Davidson roar: A multimedia analysis. Retrieved 12/03/2007, from http://www.bc.edu/bc_org/avp/law/st_org/iptf/articles/content/1998101101. html#fnB24 Scanlon, B. A. (1985). Race differences in selection of cheese color. Perceptual and Motor Skills, 61, 314. Scharf, A. and Volkmer, H. P. (2000). The impact of olfactory product expectations on the olfactory product experience. Food Quality and Preference, 11, 497–503. Scherer, K. R. (2003). Vocal communication of emotion: A review of research paradigms. Speech Communication, 40, 227–256. Schifferstein, H. N. J. (2001). Effects of product beliefs on product perception and liking. In: L. Frewer, E. Risvik and H. Schifferstein (Eds.) Food, people and society: A European perspective of consumers’ food choices, pp. 73–96. Berlin: Springer Verlag. Schifferstein, H. N. J. (2006). The relative importance of sensory modalities in product usage: a study of selfreports. Acta Psychologica, 121, 41–64. Schifferstein, H. N. J. (2007). Comparing mental imagery between the sensory modalities (submitted). Schifferstein, H. N. J. and Cleiren, M. P. H. D. (2005). Capturing product experiences: a split-modality approach. Acta Psychologica, 118, 293–318. Schifferstein, H. N. J. and Desmet, P. M. A. (2007). The effect of sensory impairments on product experience and personal well-being. Ergonomics, (in press). Schifferstein, H. N. J. and Smeets, M. A. M. (2006, 27–29 September). Towards the assessment of perceptual style. Paper presented at the 5th Design and Emotion Conference, Göteborg, Sweden. Schifferstein, H. N. J. and Tanudjaja, I. (2004). Visualising fragrances through colours: the mediating role of emotions. Perception, 33, 1249–1266. Schiffman, S. S. (2000). Intensification of sensory properties of foods for the elderly. Journal of Nutrition, 130, 927S–930S. Selcon, S. J., Taylor, R. M. and McKenna, F. P. (1995). Integrating multiple information sources: using redundancy in the design of warnings. Ergonomics, 38, 2362–2370. Smets, G. J. F. and Overbeeke, C. J. (1989). Scent and sound of vision: expressing scent or sound as visual forms. Perceptual and Motor Skills, 69, 227–233. Sorkin, R. D. (1988). Why are people turning off our alarms? Journal of the Acoustical Society of America, 84, 1107–1108. Spangenberg, E. R., Crowley, A. E. and Henderson, P. W. (1996). Improving the store environment: do olfactory cues affect evaluations and behaviors? Journal of Marketing, 60(April), 67–80. Spence, C. (2002). The ICI report on the secret of the senses. London: The Communication Group. Spence, C. (2007a). Audiovisual multisensory integration. Acoustical Science and Technology, 28, 61–70. Spence, C. (2007b, 8–11 February). Creating innovative packaging that appeals to all the senses. Paper presented at the PACE: Packaging and Converting Executive Forum meeting, Paris. Spence, C., Bentley, D. E., Phillips, N., McGlone, F. P. and Jones, A. K. P. (2002). Selective attention to pain: A psychophysical investigation. Experimental Brain Research, 145, 395–402. Spence, C. and Driver, J. (1997a). Cross-modal links in attention between audition, vision, and touch: implications for interface design. International Journal of Cognitive Ergonomics, 1, 351–373. Spence, C. and Driver, J. (1997b). On measuring selective attention to an expected sensory modality. Perception and Psychophysics, 59, 389–403. Spence, C. and Driver, J. (1999). A new approach to the design of multimodal warning signals. In: D. Harris (Ed.) Engineering psychology and cognitive ergonomics Vol. 4: Job design, product design, and human– computer interaction, pp. 455–461. Hampshire: Ashgate Publishing. Spence, C. and Driver, J. (Eds.) (2004). Crossmodal space and crossmodal attention. Oxford: Oxford University Press. Spence, C., McGlone, F. P., Kettenmann, B. and Kobal, G. (2001). Attention to olfaction: a psychophysical investigation. Experimental Brain Research, 138, 432–437. Spence, C., Nicholls, M. E. R. and Driver, J. (2001). The cost of expecting events in the wrong sensory modality. Perception and Psychophysics, 63, 330–336. Spence, C. and Read, L. (2003). Speech shadowing while driving: on the difficulty of splitting attention between eye and ear. Psychological Science, 14, 251–256.

Multisensory product experience

161

Spence, C., Shore, D. I. and Klein, R. M. (2001). Multisensory prior entry. Journal of Experimental Psychology: General, 130, 799–832. Spence, C. and Squire, S. (2003). Multisensory integration: maintaining the perception of synchrony. Current Biology, 12, R519–R521. Spence, C. and Zampini, M. (2006). Auditory contributions to multisensory product perception. Acta Acustica united with Acustica, 92, 1009–1025. Spence, C. and Zampini, M. (2007). Affective design: modulating the pleasantness and forcefulness of aerosol spray: by manipulating aerosol spraying sounds. Co Design, 3 (Supplment 1), 107–121. Stein, B. E. and Meredith, M. A. (1993). The merging of the senses. Cambridge MA: MIT press. Stevens, J. C., Cain, W. S. and Burke, R. J. (1988). Variability of olfactory thresholds. Chemical Senses, 13, 643–653. Stevenson, R. J. and Case, T. I. (2005). Olfactory imagery: a review. Psychonomic Bulletin and Review, 12, 244–264. Strayer, D. L. and Johnston, W. A. (2001). Driven to distraction: dual–task studies of simulated driving and conversing on a cellular telephone. Psychological Science, 12, 462–466. van Beers, R. J., Sittig, A. C. and van der Gon, J. J. D. (1999). Integration of proprioceptive and visual position-information: an experimentally supported model. Journal of Neurophysiology, 81, 1355–1364. van Egmond, R. (2004). Compact disc cover design: transfer of musical style and emotional contents into graphical design. In: D. McDonagh, P. Hekkert, J. van Erp and D. Gyi (Eds.) Design and emotion: The experience of everyday things, pp. 238–242. London: Taylor and Francis. van Erp, J. B. F. and van Veen, H. A. H. C. (2006). Touch down: the effect of artificial touch cues on orientation in microgravity. Neuroscience Letters, 404, 78–82. Velle, W. (1987). Sex differences in sensory functions. Perspectives in Biology and Medicine, 30, 490–522. Veryzer, R. W. and Hutchinson, J. W. (1998). The influence of unity and prototypicality on aesthetic responses to new product designs. Journal of Consumer Research, 24, 374–394. Vlahos, J. (2006). The smell of war. Popular Science, 8, 73–95. von Hornbostel, E. (1931). Über Geruchshelligkeit (About olfactory brightness). Pflügers Archiv für die Gesamte Physiologie des Menschen und der Tiere, 227, 517–538. Walker, R. (1987). The effects of culture, environment, age, and musical training on choices of visual metaphors for sound. Perception and Psychophysics, 42, 491–502. Ward, J., Huckstep, B. and Tsakanikos, E. (2006). Sound–colour synaesthesia: to what extent does it use crossmodal mechanisms common to us all? Cortex, 42, 264–280. Warm, J. S., Dember, W. N. and Parasuraman, R. (1991). Effects of olfactory stimulation on performance and stress in a visual sustained attention task. Journal of the Society of Cosmetic Chemists, 42, 199–210. Washburn, D. A., Jones, L. M., Vijaya Satya, R., Bowers, C. A. and Cortes, A. (2003). Olfactory use in virtual environment training. Modelling and Simulation, 2(3), 19–25. Weel, K. G. C., Boelrijk, A. E. M., Alting, A. C., et al. (2002). Flavor release and perception of flavored whey protein gels: perception is determined by texture rather than by release. Journal of Agricultural and Food Chemistry, 50, 5149–5155. Welch, R. B. and Warren, D. H. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 88, 638–667. Wickens, C. D. (1980). The structure of attentional resources. In: R. Nickerson (Ed.) Attention and Performance VIII, pp. 239–257. Hillsdale, NJ: Erlbaum. Wickens, C. D. (1992). Engineering psychology and human performance (2nd Ed.) New York: HarperCollins. Wrzesniewski, A., McCauley, C. and Rozin, P. (1999). Odor and affect: individual differences in the impact of odor on liking for places, things and people. Chemical Senses, 24, 713–721. Zampini, M., Guest, S. and Spence, C. (2003). The role of auditory cues in modulating the perception of electric tooth brushes. Journal of Dental Research, 82, 929–932. Zampini, M. and Spence, C. (2004). The role of auditory cues in modulating the perceived crispness and staleness of potato chips. Journal of Sensory Studies, 19, 347–363. Zampini, M. and Spence, C. (2005). Modifying the multisensory perception of a carbonated beverage using auditory cues. Food Quality and Preference, 16, 632–641. Zellner, D. A., Bartoli, A. M. and Eckard, R. (1991). Influence of color on odor identification and liking ratings. American Journal of Psychology, 104, 547–561. Zellner, D. A. and Whitten, L. A. (1999). The effect of color intensity and appropriateness on color-induced odor enhancement. American Journal of Psychology, 112, 585–604.

6

HUMAN CAPABILITY AND PRODUCT DESIGN JOHN CLARKSON University of Cambridge, Cambridge, UK

1. INTRODUCTION A US-wide survey of 15,477 working-age adults and computer users, commissioned by Microsoft (2003), asked questions about levels of difficulty users experienced with ordinary daily tasks. The results showed that ‘In the US, 60% (101.4 million) of working age adults who range from 18 to 64 years old are likely or very likely to benefit from accessible technology.’ Another US-wide web-based survey of 1501 internet users, aged 18–75⫹, undertaken by Philips (2004), found that: • Only 13% of the American public believes that in general ‘technology products are easy to use’. • Close to 65% of Americans say ‘they have lost interest in purchasing a technology product because it seemed too complex to setup or operate’. • Only 23% of Americans say ‘they use the full range of features on most new technology products’. • Consumers are beginning to demand that products be accessible for easy operation and set-up … ‘ease-of-use’ is so important to the public (76%) that it is now equal in importance to the dimension of ‘high quality’ (76%). As a direct response to the 2004 survey, Philips have focused on simplifying their products, as evidenced by their new corporate strapline, ‘sense and simplicity’. These surveys show that the proliferation of technology products has not always made life easier for prospective users, rather in some cases it has made it more difficult. Some are excluded from using particular products, some experience undue difficulty and some are merely frustrated (Figure 6.1). Thankfully, some find the same products easy to Product Experience Copyright © 2008 Elsevier Ltd.

165

Population

166

Product Experience

Finds easy

Frustrated

Has difficulty

Most able

Excluded

Least able

FIGURE 6.1 An illustration of capability variation within the user population and its relationship to product exclusion.

use. The mark of a ‘good’ product is then one that minimizes the levels of exclusion, difficulty, and frustration, while maximizing the sense of user satisfaction. Designing a product to minimize exclusion requires knowledge of the demands made by a product on its users’ sensory, cognitive and motor capabilities, and knowledge of the range of these capabilities within the user population. Any users whose capability does not meet that demanded by the product is likely to be excluded from using that product, or at the very least experience difficulty. Hence ‘good’ design requires an understanding of the characteristics and capabilities of the target users of a new product or service, so that it can be designed to operate within their capabilities and meet their needs and desires. The remainder of this chapter describes various forms of information that help to create a better understanding of these characteristics, along with the users’ sensory, cognitive and motor capabilities.

2. USER CHARACTERISTICS Users may be described in many ways. Businesses talk about market segments and preferences, medical professionals talk about symptoms and conditions, government talks about socioeconomic factors, and designers talk about anthropometry and capability. All represent different ways in which to describe the diversity of human form, performance and behaviour. All have some relevance to the design of products and services, however, the focus here will be on anthropometry and, in particular, capability.

2.1. Anthropometry Anthropometric data is used for the study of human body measurement for anthropological classification and comparison. It includes body measurements, such as height, weight and hand size, and functional measurements, principally concerning how far people can reach in different directions. The distribution of these measurements in a population tends to follow a typical ‘bell-shaped’ curve, as illustrated in Figure 6.2. It has been common practice to design for the middle 90% of this variation. However, this approach can exclude the smallest 5% and the largest 5%, who are likely to find the product hard or impossible to use. In practice, the numbers excluded are likely to be even higher, as those who are excluded by height may not be the same as those excluded by arm length, etc. It is also important to take into account the variation in data distribution by gender, age and geographical location. For example, a product which includes 90% of UK men may include only a small proportion of UK women, a small proportion of men over 75 or fewer than 90% of men from another country.

167

Human capability and product design

Average (mean)

Middle 90%

Smallest 5%

Largest 5%

FIGURE 6.2 Typical variation of anthropometric measurement.

Anthropometric data is relatively easy to find, with basic data available for many countries. However, consistency across different data sets, and often within the same data set, is less easy to find. Data for seemingly related measurements is often gathered using different population samples, making more detailed comparative analysis difficult. More recently, some efforts have been made to recognize the variation of measurement by age. For example, the UK Department of Trade and Industry (DTI) has published separate data for children (Norris and Wilson, 1995), adults (Peebles and Norris, 1998) and older adults (Smith et al., 2000). This provides a useful introduction to the anthropometric variation that may be associated with an aging population.

2.2. Capability People have a range of sensory, cognitive and motor capabilities which they use when interacting with products. Losses in any of these capabilities can make it difficult or even impossible to use a product or service. Since the incidence of capability loss increases with age (Figure 6.3), the number of those who have difficulty using products and services will be greater for older users. This is compounded by the fact that the population of the developed world is aging (Figure 6.4), with increases in the number of those in older age bands, and corresponding

50

85⫹

80–84

75–79

70–74

65–69

60–64

55–59

50–54

45–49

40–44

35–39

30–34

25–29

0

20–24

25

16–19

GB population (%)

75

Age band FIGURE 6.3 Prevalence of capability loss in Great Britain as a function of age.

168

Product Experience

% Increase 1990–2050

Italy

UK

Germany

Spain

0

USA

60

France

120

Japan

Increase in number of people over 65 (%)

180

FIGURE 6.4 Population aging.

decreases in younger groups. In many countries 50⫹ is the fastest growing segment of the population. In the UK since 1900, life expectancy has increased by an average of 2.5 years per decade and by 2020 nearly 50% of the UK population will be over 50. Similarly, 20% of the USA population and 25% in Japan will be over 65. Interacting with most products involves three main stages: Perceiving, thinking and acting. For example, opening a carton of juice involves: • Perceiving: Sensing where the opening is. • Thinking: Determining how to open it. • Acting: Carrying out the movements required to actually open the carton. Each of these primarily uses a different kind of capability, for example: • Sensory: Which covers areas like vision and hearing. • Cognitive: Areas considered with thought, intelligence and communication. • Motor: Relating to movement, such as dexterity, reach and stretch, and locomotion. As a result, performing an action as a whole involves a combination of sensory, cognitive, and motor capabilities. Even simple functions like seeing are, in reality, a result of sensory, motor, and cognitive capabilities working together. What our separate senses detect is complex and unorganized, and needs to be filtered and processed by the brain to enable decisions and actions. Higher level functions like attention; recognizing objects visually; understanding speech and planning movement; also involve cognition. This is best described as an interaction loop, such as the one illustrated in Figure 6.5. Sensory input is processed by attention and higher cognition, which generate a motor response, causing a visible effect in the world. A person’s overall capability may be described by the combination of their sensory, cognitive and motor capabilities. Conversely their overall capability loss may be ascribed to one or more individual losses, since it is often the case that people suffer multiple minor capability losses in later life.

2.3. Prevalence According to the UK Office of National Statistics (ONS), the total number of people in Great Britain identified as having some degree of reduced capability is over 8.5 million, approximately 18% of the GB population. This figure is in line with those reported by many other countries across the world (UN, 1990). The prevalence and extent of capability loss is not the same for all capability types. Figure 6.6 shows the amount of loss for some of the capabilities most relevant to the use

169

Human capability and product design

Memory

Cognitive processor

Perceptual processor

Product

Motor processor

Actions

Body / limbs

FIGURE 6.5 The interaction between sensory, cognitive and motor capabilities.

GB population (thousands)

3000 High

Medium

Low

2000

1000

ea om m

un

H

ic at io n In fu tel nc lec tio tu ni al ng

g rin

on si Vi

y rit te ex D

C

Lo

co

m

ot io n R ea c st h re a tc nd h

0

Areas of disability

FIGURE 6.6 Prevalence of capability loss in Great Britain by type for those aged 16 or over.

of products and services for those aged 16 or over. Closer examination of the underlying data reveals that those aged 75 or over are ten times more likely to exhibit some capability loss than those aged 16–49. With reference to Figure 6.6, a person’s sensory capability describes the combination of capabilities in vision (recognition and reading) and hearing (communication and signal discernment). According to the ONS, the total number of people in Great Britain identified as having reduced sensory capability (vision and hearing only) is over 3.9 million (8.3% of the population). A person’s cognitive capability describes the combination of capabilities in intellectual functioning (reading, writing, counting and memory) and communication (interpersonal

170

GB population (%)

6

Product Experience

Male

Female

4

2

0

High ability

Medium ability

Low ability

FIGURE 6.7 Overall capability – variation by ability and gender for those in Great Britain aged 16 or over.

understanding). Again, according to the ONS, the total number of people in Great Britain identified as having reduced cognitive capability is over 2.6 million (5.5% of the population). A person’s motor capability describes the combination of capabilities in locomotion (walking, stair climbing, bending, and balance), reach and stretch (upper-body strength and control) and dexterity (picking up, carrying, holding and twisting using the hands). According to the ONS, the total number of people in Great Britain identified as having reduced motor capability is over 6.7 million (14.3% of the population). Note that the sum of these figures exceeds the total prevalence of reduced capability (3.9 ⫹ 2.6 ⫹ 6.7 ⬎ 8.5) since many people, especially those in older age groups, experience more than one capability loss. Designs that only address one of these losses will still exclude these people from using the product. It is therefore important to consider combinations of capability loss when evaluating and improving designs. It is also important to recognize the hierarchy of sensory, cognitive and motor capabilities: A person’s lack of sensory capability may exclude them from using a product regardless of their cognitive or motor capabilities. From the product perspective, an excessive sensory demand will not be counterbalanced by a low cognitive or motor demand. For example, unreadable labels on well-designed controls for a simple product will result in unnecessary exclusion. Finally, for the purposes of examining products, it can be useful to measure each sensory, cognitive, and motor capability on a four-point scale: 1. Fully able – has adequate use of the capability for everyday activities. 2. Moderately able – experiences some difficulties with the capability in several everyday situations. 3. Partially able – has significant problems with the capability in most everyday situations. 4. Minimally able – cannot use that capability for practical everyday purposes. The original ONS data, illustrated in Figure 6.7 (Grundy et al., 1999), may be integrated to create a cumulative curve, which in turn may be mapped against the four-point scale as shown in Figure 6.8. The latter graph shows how many people in Great Britain (GB) have less than the specified level of ability, indicating how many people would have particular difficulty using a product with that level of demand. It is difficult to identify comparative data for other countries, and what exists is rather superficial in terms of detail (UN, 1990). However, while it is accepted that ‘local’ variation of capability loss might be expected, the discussions that follow in this chapter remain pertinent and indicative of the challenges that designers would face in designing for their local, or indeed an international, user community.

171

Human capability and product design

GB population (%)

20

15

10

5

0 Full ability

Moderate ability

Partial ability

Minimal ability

FIGURE 6.8 Overall capability – cumulative frequency graph showing the proportion of Great Britain population aged 16 or over that does not meet the specified ability level.

3. PRODUCT DESIGN The previous section presented an outline description of users’ capabilities. The following sections will provide detailed descriptions of each capability and their impact on product design. However, we first need to clarify the link between product accessibility and user capability. All products make demands of their users, in the context of a particular task being undertaken in a particular environment. When such demands exceed the capabilities of a particular user, exclusion or difficulty may arise. A product that makes minimal demands is likely to exclude only those users with minimal ability, while one that makes high demands might also exclude those with full ability. The environment will also have an impact on the level of exclusion. For example, bright light cast on a text display can increase exclusion, not only by making the characters more difficult to see, but also by compromising the vision capability of the user. Products that are designed with these challenges in mind are more likely to moderate their demands on the user, increasing accessibility and ultimately the utility of the product. Figure 6.1 illustrated the proposition that for every user that might be excluded from using a product, there are likely to be more that find difficulty or frustration in using the same product. Equally, it is proposed that effort expended in reducing exclusion is likely to have a beneficial effect on levels of difficulty and frustration. In practice, there is a continuum between these modes of interaction since, for example, severe difficulty may quickly lead to exclusion for a tired user, or the development of a coping strategy may enable a previously excluded user to operate a product. Consequently, the exact shape of the curve in Figure 6.1, and the boundaries between the different modes of interaction, will vary from product to product. A thorough understanding of user capability enables informed design decisions to be made regarding user interaction with the product. For example, an appreciation of how dexterity capability might be reduced due to increased age or presence of disease, enables a designer to design interactions involving the hands for a wider range of capabilities, thus reducing exclusion. Hence, the following sections describe each of the sensory, cognitive and motor capabilities in turn, and their likely variation within the user population. In addition, design guidance is provided to aid inclusion, along with descriptions as to how the four-point scale presented in the previous section may be applied to each capability.

172

Product Experience

4. VISION Our sense of vision allows us to perceive the world in images, motion and color. We use information from the visual sense in order to move around and interact with objects and environments. The effective design of any product or environment should take into account the range of human visual abilities. The human eye works by allowing light to enter through the pupil (Figure 6.9). The light rays then pass through the lens where they are refracted. These rays are then focused on the retina, the surface at the back of the eye that is sensitive to light. The retina consists of two types of receptor cells known as rods and cones. Rods allow us to differentiate between shades of black and white, while cones allow us to see color in adequate levels of illumination. Most cones are concentrated in a region of the retina called the fovea. This region allows us to see the greatest detail. The presence of stimulation of the retinal cells is transmitted to the brain via the optic nerve at the back of the eye. The brain utilizes and interprets the signals from both eyes to construct the visual image that we see. Because images from both eyes are used, we are able to see in three dimensions and perceive depth (see Chapter 1). The visual system serves various functions, four of which are discussed in the context of designing products: • • • •

Visual acuity. Contrast sensitivity. Color perception. Usable visual field.

Because of aging and various eye conditions, the structure and function of the eye can change. These changes can cause various levels of reduction in visual acuity and contrast sensitivity.

4.1. Visual acuity Visual acuity is the ability of the eye to see fine detail. It is dependent on two factors: The viewing distance from the product; and the smallest feature or space that the eye can detect

Optic nerve

Pupil Lens

Retina

FIGURE 6.9 The shape and structure of the human eye.

173

Human capability and product design

(for example, text size or stroke thickness). Reduced visual acuity, due to a reduced ability of the lens in the eye to focus light on the retina, results in images that appear blurry. Designing for reduced visual acuity involves considering the type size, weight, and font style of text (Arditi, 2006a). The ability to perceive details also depends on the ambient illumination and the contrast between the foreground and background. Visual acuity is important for tasks such as: Seeing and reading text in print material and on products (for example, product manuals and product controls); seeing and identifying graphical symbols in print material and on products; reading signage on the roads and in public spaces; and recognizing faces at various distances. Particular problems can arise when there is a requirement to change the viewing distance as part of an activity, since the ability of the eye to accommodate such change (Figure 6.10) reduces with age (Vassilief and Dain, 1986).

4.2. Contrast sensitivity Contrast is the difference in brightness between the foreground and the background. It is related to the size and illumination of the object to be detected. For example, the ability to distinguish a number key from the body of a mobile phone depends on the contrast between the two. If a person has low contrast sensitivity, it means that low contrast text (for example, gray on white) becomes very difficult or impossible to distinguish. The greater the contrast between foreground and background text, the more likely that it will be easily detected by people with low contrast sensitivity. The same holds true for controls on products. Buttons, knobs, indicators, and levers all need to be of sufficient contrast from the rest of the product body to be easily visible. Maximum contrast occurs with white on black or vice versa. For colors, the difference in lightness between the foreground and background colors determines the contrast. Contrast sensitivity is important for activities such as reading and recognizing low contrast text, moving around in the environment, and detecting the outlines of buildings, roads, and pavements.

4.3. Color perception Color can be described in terms of its hue, saturation and brightness. When choosing colors for product features, two important issues for consideration are color contrast (Arditi, 2006b) and color blindness.

Ocular accommodation (dioptres)

14 12

Mean

10 Upper limit

8 6 4 Lower limit

2 0 0

10

20

30

40

50

60

70

Age (years)

FIGURE 6.10 Changes in accommodation ability of the eye with age (Vassilief and Dain, 1986).

174

Product Experience

Certain combinations of text and background color result in poor contrast. High luminance foreground on a low luminance background (or vice versa) gives maximum contrast. Color blindness is the inability to distinguish between various colors: It does not mean that people with color blindness cannot perceive color (this is extremely rare). Color blindness refers to a wide spectrum of effects, but the most common form is red– green color blindness, where a person may be unable to effectively distinguish, under all lighting conditions, between colors from red to green in the color spectrum.

4.4. Usable visual field With increasing age and various eye conditions, the usable field of view can change. This can occur in two ways: The usable field of view can be reduced from the centre of the visual field and move out (central field loss), or it can start from the outside to the inside (peripheral field loss). The central visual field is used for focusing and perceiving detail. When the central field is obscured, tasks that require perception of detail (such as reading) become very difficult. People generally adapt to compensate for this loss and attempt to use the peripheral visual field. However, the peripheral field is less suited for these tasks. A reduction in the usable peripheral visual field results in tunnel vision and can affect mobility. The parking ticket machine in Figure 6.11A shows controls that are placed far away from the dispensing slot. For a user with peripheral field loss, the slot might be more difficult to locate. Conversely, the machine shown in Figure 6.11B, like most ATMs, is a good example where the dispensing slot is close to the display and controls.

4.5. Context of use The environment plays an important role in determining the ability of a person to have a successful visual interaction. It can affect interaction with the product by introducing effects such as glare, and can compromise the ability of the person using the product with distracting ambient lighting. Glare results if the intensity of light entering the eye off a surface is greater than the ambient intensity to which the eye is accommodated. Highly reflective surfaces can cause glare problems for people with reduced visual ability.

Controls

Dispensing slot

A Poor design?

B Better design?

FIGURE 6.11 The positioning of key controls can have a significant impact on the usability and accessibility of a complex interface.

Human capability and product design

175

The location of the product in the environment is also an important consideration for any design, since visual acuity depends on the distance from the product. For example, if signage is installed too far away or very high up (despite large text size and good contrast), problems may occur. Information on displays and screens can also suffer degradation when they are installed in locations that demand viewing at an angle.

4.6. Design guidance Better design can be achieved as a result of a better understanding of the above capabilities and associated capability losses (Evamy and Roberts, 2003; RNIB, 2006). The following guidance with regard to design of the product or service should assist in making designs more inclusive: • Attempt to make text as large as possible within the constraints of the design, and maximize the contrast between foreground text and the background. • Where possible use sans-serif fonts (such as Arial) at larger text sizes with plain instead of patterned backgrounds to increase clarity. • Avoid the use of decorative and cursive font styles (for example, fonts that mimic handwriting) in favor of clearer, more legible, sans-serif typefaces. • Attempt to make graphical symbols as large and clear as possible within the constraints of the design. • Attempt maximum contrast between product parts (such as buttons, keys and other controls) against the product body within the constraints of the color palette chosen for a design project. • Keep the different forms of color blindness in mind when choosing the color palette for a design project. If red and green are to be used together, try to provide an alternative clue (such as a text description) as to what the lights mean. • Attempt to avoid shiny and highly reflective surfaces that increase the likelihood of glare problems, using materials with matte finishes where possible. The following guidance with regard to design of the environment should also assist in making designs more inclusive: • Reduce glare by positioning light sources away from the user’s line of sight and by using shielding or diffusers on light sources. • Consider providing adjustable light sources (such as lamps) to allow different users to set the lighting environment to their needs. • Reduce glare and angle of view problems by providing displays and screens that can easily be repositioned.

4.7. Prevalence data Finally, for the purposes of examining products, it can be useful to measure vision capability (with any desired vision aids) on a four-point scale: 1. Full ability – can recognize a friend across the street with no difficulty; – can read ordinary news print with no difficulty. 2. Moderate ability – can recognize a friend across the room; – can read large print at an average reading distance.

176

Product Experience

GB 16⫹ population (%)

15

10

5

0 Full ability

Moderate ability

Partial ability

Minimal ability

FIGURE 6.12 Vision capability – cumulative frequency graph showing the proportion of GB population aged 16 or over that does not meet the specified ability level.

3. Partial ability – can recognize a friend at arms length away; – can read a newspaper headline. 4. Minimal ability – cannot tell where the windows are in daylight. Figure 6.12 presents cumulative values for vision capability, showing how many people in Great Britain have less than the specified level of ability, indicating how many people would have difficulty using a product with that level of demand. The vertical scale used matches that used of subsequent graphs to aid comparison of levels of loss for different capabilities.

5. HEARING The sense of hearing is the human response to sound vibration. We can identify simple sounds such as beeps and tones, and complex sounds such as speech and music. Hearing also allows for speech perception and understanding, which forms the basis of our ability to communicate with others. The human ear is divided into three parts: The outer ear, the middle ear and the inner ear (Figure 6.13). The outer ear collects sound energy and channels it to the middle ear via the ear drum. The sound energy is converted into mechanical vibrations of small bones in the middle ear. These vibrations are transmitted to the inner ear (cochlea) which is filled with fluid. A membrane in the cochlea picks up the fluid vibration and converts the vibrations into neural impulses. These impulses are then transmitted to the brain via the auditory nerve (see Chapter 3). When blockages exist anywhere in the passage from the outer ear to the middle ear, it results in conductive hearing loss. This results in the loss of ability to hear faint sounds. When the cochlea is affected by aging or disease, it results in sensory neural hearing loss. This affects the quality of the sound detected and results in loss of ability to understand speech and discriminate various sounds (Moore, 2003). Three major hearing functions are considered in design applications: 1. Sound detection. 2. Speech discrimination. 3. Sound localization.

177

Human capability and product design

Outer ear

Middle ear

Inner ear

Cochlea Cochlear nerve

Ear canal Eardrum

Auditory bones

FIGURE 6.13 The shape and structure of the human ear.

Changes in hearing function are known to occur with age. With conductive hearing loss, the ability to detect faint sounds is reduced, along with the ability to hear sounds of higher pitch. A person’s hearing threshold therefore increases. Speech discrimination becomes more difficult, especially in the presence of noise. Also, the ability to localize sounds decreases with age, particularly for sounds with relatively low volume and short duration (for example, watch beeps). These changes in hearing functions need to be considered in the design of products with auditory output.

5.1. Sound detection

Hearing loss (decibels)

Sound detection is the ability to detect beeps, tones and other sound output from various products. Sound vibrations can range from very low intensity to high intensity – this is perceived as loudness or volume. The sound vibration of simple tones can range from low frequency to high frequency – this is perceived as pitch. Complex sounds such as speech and music contain a range of frequencies at various levels of intensity. In order to detect a sound, it must be of a sufficient loudness and within a frequency range (Figure 6.14) that is audible to the listener, taking account of the variation that arises due to aging (Beales, 1965). 0 ⫺10 ⫺20 ⫺30 ⫺40 ⫺50 ⫺60 ⫺70 ⫺80 ⫺90 ⫺100

10–19 years 20–29 years 30–39 years 40–49 years 50–59 years 60–69 years 70–79 years

500

1000

2000

4000

8000

Frequency (Hertz)

FIGURE 6.14 Changes in frequency response of the ear with age (Beales, 1965).

178

Product Experience

Detecting sounds in the presence of noise is more difficult than detecting sounds in isolation. For example, hearing a phone conversation in a crowded restaurant or hearing a phone ring while the television is on require the ability to distinguish the sounds from background noise. Most real world tasks take place with some level of ambient noise.

5.2. Speech discrimination Speech discrimination is the ability to understand speech in quiet and noisy environments. Human ability to discriminate speech is an important consideration when designing products that facilitate verbal communication, and products that use speech output. Examples include communication products (such as telephones and mobile phones), ATMs, car navigation systems, and even accessible technology that aids people with low vision via alternative speech output. Speech sound can be detected if it is of a sufficient loudness, but discrimination of that speech can be difficult for some people, especially in the presence of noise (Moore, 2006). Male or female voices can be used for speech. Generally male voices are of lower pitch and are more likely to be within the range of hearing. However, the defining characteristics of speech, the consonants, may still be difficult to detect for those exhibiting a loss of high-frequency hearing capability. Various accents and intonation can also affect how well speech is understood. Speech output can be pre-recorded and replayed or speech can be synthesized. Synthesized speech is more difficult for older people to understand, especially if it is recorded, played back or synthesized at high speed (a large number of words per minute).

5.3. Sound localization Sound localization is the ability to tell the direction from which a sound is originating. When sound localization ability is low, it affects a person’s safe interaction in an environment. Lack of sound localization ability can influence safe movement in public spaces. If a person cannot tell the direction of an approaching vehicle, with or without accompanying horn or siren, it may lead to fatal consequences. Sound localization is important when interacting with products that warn the user and indicate their location by using sound output. People with low sound localization ability cannot tell that it is their own watch or reminder alarming and may miss, for example, a reminder to take medication. Hence, such products need to indicate in other ways to the user that it is a given time. People with loss of this ability also have trouble with their mobile phone, as they cannot tell that it is their own phone that is ringing. For other products in a home or office environment, it might be difficult to identify which appliance is giving sound output when a few appliances are sounding at the same time.

5.4. Context of use The most important factor that affects hearing is the presence of noise. Noise is essentially the ambient sound environment that interferes with the perception of the sound of interest. Noise introduces hearing demands, in that the user has to discriminate the sound of interest from a mixture of other sounds. Noise can range from low intensity to high intensity, and from general white noise to background speech and music (such as in a pub, restaurant or crowded area). Spaces that introduce large amounts of reflection and reverberation of sound can cause problems with hearing. The sound becomes less intelligible and it can be difficult to discriminate. This occurs in public spaces where announcements are important, such as train and underground stations, sports arenas and music halls. The increased reverberation

Human capability and product design

179

already affects people with average hearing abilities, and affects people with lower hearing abilities to a much larger extent.

5.5. Design guidance Better design can be achieved as a result of a better understanding of the above capabilities and associated capability losses (ASHA, 2006; RNID, 2006). The following guidance should assist in making designs more inclusive: • Make volume levels adjustable if possible and try to ensure that frequencies of sound are in the range 800 to 1000 Hz. • Avoid synthesized speech in favor of natural speech (recorded) if possible, and use lower pitched voices in preference to higher pitched voices. • Attempt to provide alternative feedback (such as visual or tactile) for people with very low hearing ability and facilitate connections with auditory aids. • Design environments and spaces to minimize background noises, sound reflection, and reverberation as much as possible to ensure clarity of sound transmission. • Attempt to ensure that when sounds of high pitch are used, they are of a long duration to maximize detection.

5.6. Prevalence data Finally, for the purposes of examining products, it can be useful to measure hearing capability (with any desired hearing aids) on a four-point scale: 1. Full ability – can hear a conversation in a moderate amount of background noise without great difficulty; – can follow a TV program at a volume that others find acceptable. 2. Moderate ability – can understand someone talking with a loud voice in a quiet room with no difficulty; – can hear someone talking over the telephone; – can hear a doorbell or alarm clock. 3. Partial ability – can follow a TV program if the volume is turned up. 4. Minimal ability – cannot hear any sounds at all. Figure 6.15 presents cumulative values for hearing capability, showing how many people in Great Britain have less than the specified level of ability, indicating how many people would have difficulty using a product with that level of demand.

6. INTELLECTUAL FUNCTIONING Most of the processes that underlie intellectual function occur in the brain, for example, memory and attention (Baddeley, 2004; Wickens and Hollands, 2000). The brain organizes incoming sensory information, processes it in the light of conscious awareness and attention, and initiates responses in the form of actions. Biological studies have shown that different regions of the brain are involved in different intellectual functions, such as: Perception, memory, and movement initiation as well as coordination and speech (Figure 6.16). However the precise nature of how the

180

Product Experience

GB population (%)

15

10

5

0 Full ability

Moderate ability

Partial ability

Minimal ability

FIGURE 6.15 Hearing capability – cumulative frequency graph showing the proportion of GB population aged 16 or over that does not meet the specified ability level. Basic movements Motor control areas

Skilled movements Emotions

Sensory areas

Memory, awareness

Visual object recognition

Making speech and language

Basic vision Balance and muscle coordination

Smell Hearing

Understanding speech and language

FIGURE 6.16 Intellectual functioning capability – summary of brain functions.

brain works is not observable or measurable, so cognitive psychologists have to rely on hypothetical models to help their understanding. Well-established memories and skills are usually unaffected with aging, whereas learning new things tends to take longer. The time required to make decisions and respond to sensory information also increases, as does the error rate for tasks requiring memory, numeracy and spatial orientation. The ability to perform simultaneous cognitive tasks also decreases. Degenerative brain disorders are more prevalent with age. These include Parkinson’s disease, Alzheimer’s, and vascular dementia, which can affect memory, attention, movement, perception, reasoning and social interaction. Although there are a wide range of functions carried out by the brain, the following are identified as the most significant functions for product design: • • • •

Working memory. Attention and performance. Visual-spatial thinking. Learning, recall and long-term memory.

Working memory describes the conscious ability to temporarily store, process and rearrange information (Figure 6.17). Attention and performance refers to the number

181

Human capability and product design

Long term memory Environment

Learning and recall Working memory Attention and performance Visual-spatial thinking

Perceptual processor

Ref le resp x level ons e

Response selection

Product in the world

Execution

FIGURE 6.17 Intellectual functioning capability – a hypothetical model.

of different things that can be kept in mind at once, and the speed at which information can be processed, decisions made or problems solved. Visual-spatial thinking refers to the ability to process, manipulate and rotate objects in mental space. Data can also be stored in the form of words, and sequences of linked past experiences can be remembered together. Information used repeatedly in working memory can be learnt, and stored in longterm memory indefinitely. Information can also be recalled into working memory when required. For example, a prior experience with similar products can help to learn the features of a new one. At a higher level, the sensory, cognitive and motor functions are integrated together within the brain, and understanding the interaction between all these functions is crucial for product design.

6.1. Working memory Working memory refers to the ability to manipulate and rearrange data within conscious thought, and these data can be in many different forms, such as numbers, images or words. In its simplest form, short-term memory refers to a fixed capacity number of bins that can simultaneously hold information. For example, when a person reads a random string of single digit numbers, each character takes up one bin, and the person can only correctly recall the number of characters equal to the number of bins they have. However, this process is further complicated by chunking, which allows information to be stored as chunks within each bin. A common example of chunking is letters joining together to form words, and words joining to form phrases.

182

Product Experience

In everyday tasks, working memory (as the fixed capacity number of bins) operates together with ‘attention and performance’, and ‘visual-spatial thinking’. Working memory can be thought of as a scratch pad or rough working area for items being attended to, where such data may have come from sensory perception, or by recall from long-term memory. The capacity of working memory in adults has been found to be around 5 to 9 chunks or items (Miller, 1956).

6.2. Attention and performance Attention can be consciously directed at specific tasks, or it can be ‘grabbed’ by an event such as a flashing light or sound. A person could be in deep concentration for a specific task, but their attention would still be distracted by a loud noise, a bright flashing light, or the sound of their name. This can be advantageous, to direct a person’s attention towards an impending hazard, or can be a nuisance if a flashing light disrupts attention away from the task at hand. Attention can be overloaded when too many things have to be kept in mind at once, and in this case the user may forget items or tasks. For example, if a person attempted to simultaneously cook with the grill, oven and four hobs while watching the news, it is likely that one of the foods will burn. Working memory performance is also affected by the time taken to process incoming sensory information. For example, when driving a car, sensory information is continually arriving in the form of road hazards, signs, and information from the vehicle. If the user cannot process and respond to the incoming information fast enough, then newly arriving information has to be ignored, or existing information forgotten to free up working memory space. Working memory performance (Figure 6.18) is also known to vary with age (Baddeley, 2004).

6.3. Visual-spatial thinking The most relevant aspects of visual-spatial thinking for product design are the mapping of controls to devices, and visual grouping. The mapping of controls can be illustrated with reference to domestic cookers. Minimum visual-spatial capability is required when the controls are laid out in an identical

130 120

Mean IQ

110

Verbal IQ 100 90

Performance IQ

80 70 0 20

30

40

50

60

70

Age (years)

FIGURE 6.18 Changes in intellectual function with age (Baddeley, 2004).

Human capability and product design

183

manner to the heating elements that they control, or when there is a visual or physical link between the control and device. Both of these allow the user to pick any control, and immediately work out which heating element it will operate, or pick any element, and immediately work out which control operates it. Yet the common practice for nearly all cookers is to relate the heating elements to their controls with a small diagram by each control. Typically the user knows which element they want to operate, but then has to perform a trial and error process to check each control and find out what it does: This requires higher visual-spatial ability, takes longer, and can often be mistaken, even by able bodied users. Understanding these explanatory diagrams is even harder if they need to be rotated to match the objects they represent. The pre-conscious process of visual grouping leads us to see objects as part of a group or separate set because of similarities in their shape, color or spatial positioning (see Chapter 1). This can help users to categorize buttons that have similar functions, thereby making the device easier to understand.

6.4. Learning, recall, and long-term memory Long-term memories require learning and repetition to become fixed, and once stored they can last a long time and be retrieved by triggers or cues. For example, seeing a specific door handle triggers a memory that the door should be pulled to open rather than pushed, without needing to read the sign on the door. Prior use of a product enables us to learn aspects of its features and use that are held in long-term memory, and make it easier to use in the future. To maximize learning the product must provide good feedback so that the users can immediately see the result of their actions and adjust them if necessary. The brain can learn to associate symbols, shapes and colors with particular features or functions, provided sufficient support is available by explanatory text and visual layout. For example, learning the symbols for the buttons used on the toolbar of a word editor is dramatically enhanced if explanatory text appears when the mouse is left over the symbol. Long-term memory may be transferred to help learn and understand other similar products. Our ability to learn decreases with age, and older generations that did not grow up with technology products lack the prior experience to understand functions that the designer may have considered obvious, which is why user testing is crucial to ensure usability.

6.5. Context of use Using products that place a high demand on working memory will be extremely difficult in a busy or noisy environment, and especially when attention can be distracted by other environmental cues such as flashing lights or beeps. Reducing the demand on working memory will make the product more usable in a wider variety of environments and situations. Memories are not perfect and can be forgotten or corrupted. The ability to correctly retain information can be affected by: • How the memory was learnt. A memory is not likely to be well stored if the information was not perceived to be of high importance at the time of learning, or if the learning process was distracted by other factors. • The nature of intervening experiences. Long-term memories decay slowly with time, but are refreshed each time they return to conscious thought.

184

Product Experience

• The conditions experienced when retrieving memories. Memories that had previously been unobtainable can suddenly be recalled when an appropriate cue is provided. Our ability to remember can be improved by clear, distinctive memory cues, such as icons or buttons that are easily distinguished by their shape, size, color, texture, and spatial layout.

6.6. Design guidance Better design can be achieved as a result of a better understanding of the above capabilities and associated capability losses (Huppert, 2002). The following guidance should assist in making designs more inclusive with regard to working memory: • Avoid overloading memory by reducing the number of simultaneous chunks that are presented at any time, and by not exceeding seven as the best practice; examples of this include the number of: – Buttons within any button grouping; – Sub-menu options within any menu; – Choices for any decision. • Try to minimize the levels of hierarchy used in any menu system and ensure that the current location within the overall hierarchy is always obvious; a distinctive spatial position for each menu option can be useful to aid learning and recall. • Avoid the need to scroll a screen to obtain more menu items and provide a ‘back-up’ button to aid menu navigation, ensuring it is as obvious as the ‘select’ button. • Consider the use of tabbed interfaces to provide a distinction between different levels of hierarchy, and a map of the current location within the hierarchy; drop down menus do not provide such distinction. • Use shapes, colors and alignment to group alike buttons and displays into chunks, thereby reducing the time and working memory required to locate a desired feature. The following guidance should assist in making designs more inclusive with regard to visual-spatial thinking: • Align controls in a spatial orientation that matches the objects they affect, minimizing the spatial transformations (for example, rotation) required for interpretation. • Ensure that the control needed is immediately obvious from each device, and that the device that will be affected is immediately obvious from each control. The following guidance should assist in making designs more inclusive with regard to time pressure, attention and learning: • Avoid unnecessary demands for time pressure in product interaction – for example, the expectation that the second of two digits should be entered within a particular time frame. • Ensure that attention is only required to be directed in one place at any one time – multiple simultaneous flashing lights or alarms will simply confuse the user. • Support learning by immediate feedback on any action and ensure all actions are easily and immediately reversible. • Provide error messages to guide the user to fix the problem.

185

Human capability and product design

GB population (%)

15

10

5

0 Full ability

Moderate ability

Partial ability

Minimal ability

FIGURE 6.19 Intellectual functioning capability – cumulative frequency graph showing the proportion of GB population aged 16 or over that does not meet the specified ability level.

6.7. Prevalence data Finally, for the purposes of examining products, it can be useful to measure intellectual capability on a four-point scale: 1. Full ability – can usually do something without forgetting what the task was in the middle of it; – can usually hold a conversation without losing track of what is being said; – can think clearly with no muddling thoughts; – can usually tell the time of day with no confusion; – can watch a 30-minute TV program and tell someone what it was about; – can read a short newspaper article; – can write a short letter to someone with no help; – can count well enough to handle money; – can remember a message and pass it on; – can usually remember to turn things off, such as cookers or taps; – can usually remember the names of friends/family that are seen regularly. 2. Moderate ability – can do seven of these tasks (cannot do four of them). 3. Partial ability – can do four of these tasks (cannot do seven of them). 4. Minimal ability – cannot do any of these tasks. Figure 6.19 presents cumulative values for intellectual functioning capability, showing how many people in Great Britain have less than the specified level of ability, indicating how many people would have difficulty using a product with that level of demand.

7. COMMUNICATION A communication simply requires someone that sends some information, and someone that receives and interprets it. Communication can occur through speech and text, but may also include non-verbal communication such as visual and iconic messages, together with sounds and gestures. In the context of product design, communication refers to the process of interaction between a person and a product. This involves the person’s perception of the product,

186

Product Experience

ENVIRONMENT Think Perceive

Product

Act

FIGURE 6.20 A model of product interaction.

and also their ability to perform actions such as pushing buttons. A person’s ability to communicate depends on their educational level and social skills. Communication can be affected by impairment of the physical capabilities necessary for interacting with a product, such as vision and movement (Wickens and Hollands, 2000). Environmental factors that affect these capabilities will also affect their use for communication purposes. Speech disabilities arise from both speech articulation and voice quality impairments, causing slowing, slurring and stuttering. Language disorders include reduced vocabulary, inappropriate use of words and faulty grammar. Cognitive impairment resulting from brain injuries, such as stroke or developmental disorders, can also affect linguistic capability. An interaction with a product (Figure 6.20) typically involves several cycles of: • Perceiving. • Thinking. • Acting. Both perceiving and acting are high level functions that may involve the combination of several lower level functions such as vision and dexterity. Although these functions are considered separately for the purpose of design guidance and prevalence data, in reality they are all combined during cycles of product interaction. The processes involved during thinking were covered in Section 6, while those relating to perceiving and acting are detailed here.

7.1. Perceiving Perceiving is the ability to comprehend information, which can be in forms such as speech, text, sounds, shapes or images. In addition to these specific outputs from a product, consumers are strongly influenced by the device’s general character. The form, color, and style of the product all influence the user’s assessment of its aesthetic, symbolic and practical value. This affects not only their willingness to interact with the device, but also their ability to do so successfully. In the context of product communication, perceiving refers to the ability of a person to receive and interpret information from a product. This information can be presented in many different forms, for example: • Speech, that can be used as an auditory output to convey detailed information. • Sounds, that may be used to enhance understanding when using a product, such as providing feedback for whether an operation was successful or not.

Human capability and product design

187

• Text, that can be used to describe what buttons do, provide detailed instructions, or be presented on a screen as output. • Images, that can be more powerful and quicker to understand than words, although some images can be too small, complex or abstract, in which case the meaning is lost. • The physical arrangement of a product, that can be used to convey information, such as a slider being adjacent to the device it controls, and the position of the slider indicating its current state. • The shape and form of a product, that should assist the user to understand which areas they can interact with, and in what way. Different people prefer to receive information in different forms, so understanding is most likely to be successful when information is presented in multiple different ways.

7.2. Acting The ability to transfer information to a product can occur in many different ways, such as the direct manipulation of physical controls, or more indirectly by speaking or entering text. In the context of product communication, acting refers to the ability to transfer information to a product, through the correct manipulation of its interface. Acting can be categorized according to whether the action is physical or symbolic. Physical actions refer to the manipulation of a control to achieve a prescribed physical effect. Such controls include: Discrete controls, for example, power and light switches; and analog controls, for example, dimmer switches and brake pedals. Symbolic actions refer to those that confer no physical effect, but can control a product after being interpreted. Such actions include: Selecting menu options; using touch screens or clicking on icons; and using a keyboard to enter text or speaking to a product. Symbolic actions can provide much more diversity than physical ones, yet must be carefully thought out by designers to ensure that they remain simple to use and provide an equivalent level of feedback.

7.3. Context of use A person’s ability to communicate can be influenced by their background, nationality or place of origin. Text and speech may require translating into different languages, whereas pictures have the potential to be understood independently of a person’s native language. A communication with a product uses lower level functions such as vision, hearing and dexterity, so environmental factors that hamper these functions will also hamper product communication.

7.4. Design guidance Better design can be achieved as a result of a better understanding of the above capabilities and associated capability losses. The following guidance should assist in making designs more inclusive: • Ensure the areas that the user can interact with, and the correct way to interact with them, are obvious from the overall form of the device. • Ensure that an uninitiated user can form a correct mental model of how the controls will affect the product and provide positive feedback so that the user can ascertain when their actions have been successful.

188

Product Experience

GB population (%)

15

10

5

0 Full ability

Moderate ability

Partial ability

Minimal ability

FIGURE 6.21 Communication capability – cumulative frequency graph showing the proportion of GB population aged 16 or over that does not meet the specified ability level.

• Ensure that the current state or mode of the device is obvious and avoid unnecessarily high demands on user capabilities during product interaction. • Provide helpful assistance in the event that the user has performed an incorrect action, detailing why their action was unsuccessful and what options are available. • Minimize the adverse consequences when errors or mistakes do occur and ensure all actions are reversible. • Provide the potential for information to be transferred by different modes, such as textual, verbal, pictorial, tactile, lights and sounds.

7.5. Prevalence data Finally, for the purposes of examining products, it can be useful to measure communication capability on a four-point scale: 1. Full ability – can communicate with strangers with no difficulty. 2. Moderate ability – can communicate with strangers with some difficulty; – can communicate with well known people easily. 3. Partial ability – can communicate with strangers with extreme difficulty; – can communicate with well known people with some difficulty. 4. Minimal ability – cannot communicate with strangers or well known people. Figure 6.21 presents cumulative values for hearing capability, showing how many people in Great Britain have less than the specified level of ability, indicating how many people would have difficulty using a product with that level of demand.

8. LOCOMOTION Locomotion is the ability to sit down and stand up, get up and down from the floor, or move and walk around in the environment. Activities such as walking, getting in and out of vehicles, on and off furniture, and maintaining balance are affected by loss of

Human capability and product design

189

locomotion ability. In order to move around, we require adequate muscle, motor control, and balance. Our legs and feet support the rest of our body and provide the force to propel us forward. Age related musculo-skeletal conditions cause loss of locomotion ability. In the case of arthritis joint mobility is limited, leading to reduced range of leg and lower body movement. The ability to maintain balance is also reduced, resulting in an increased risk of falling. Because muscle strength in the lower body such as the legs decreases with an increase in age, the arms are more frequently used to support the body when, for instance, sitting down and standing up. The reduction in visual ability with age also causes problems with moving around an environment. Coupled with reduced muscular strength, it can be difficult for older and disabled people to move around with speed and agility. The following locomotion functions are described in the context of product and service design: • Walking and balance. • Getting up, down, in and out.

8.1. Walking and balance In order to minimize the locomotion capability required to move around, good design should attempt to reduce the distance for continuous walking without rest, provide support to help people maintain balance, and ensure access is possible for those who cannot ascend or descend steps.

8.2. Getting up, down, in and out Actions such as kneeling or sitting down, standing up, or getting into and out of confined spaces or vehicles can be very difficult for people with low locomotion ability. Reducing the muscle strength and flexibility required to move the body into these required positions provides a more satisfying product for all, and helps to include those with reduced locomotion capability. This can be achieved through careful use of dimensions and shapes, and making allowance for the hands to help move the body around.

8.3. Context of use Provision of a suitable environment can dramatically affect those with reduced locomotion capability, who often use various aids such as walking sticks, crutches, frames and trolleys, wheelchairs and scooters, in order to increase their mobility. Product and environmental design should factor the use of these aids into design solutions. Eliminating steps and space constrictions to help wheelchairs and other mobility aids results in a design that also benefits those with push-chairs, bicycles, suitcases on wheels, etc. Balance aids are essential in transportation situations such as buses and trains, but can also reduce design exclusion and increase user satisfaction to aid any situation involving standing for long periods, such as when queuing. Environments should provide adequate capability for people to rest while getting from one place to another. Regular seating intervals can also be useful for those carrying heavy rucksacks, or who want to stop for a rest or to tie-up their shoelaces.

190

Product Experience

8.4. Design guidance Better design can be achieved as a result of a better understanding of the above capabilities and associated capability losses. The following guidance should assist in making designs more inclusive: • Attempt to provide adequate space for access and egress when designing doorways, entrances, and exits. • Consider the use of locomotion aids such as walkers, wheelchairs, and scooters in setting the dimensions of doorways, entrances and pathways. • Provide adequate seating at regular intervals in public spaces such as parks, airports, and shopping centres. • Furniture, shower, and toilet design should assist actions such as sitting down, standing up, getting in and out, by providing grab bars, handles or other means of support. • Design items such as seats, showers, and toilets to assist actions such as sitting down and standing up, or getting in and out, by providing grab bars, handles or other means of support. • Attempt to integrate grab bars and handles into the overall aesthetic appeal of the design and avoid designs that look ‘medical’ or ‘assistive’. • Reduce the need to bend the back or reach below waist level for any product interaction.

8.5. Prevalence data Finally, for the purposes of examining products, it can be useful to measure locomotion capability on a four-point scale: 1. Full ability – can walk 250 metres with no stopping or severe discomfort; – can ascend/descend a flight of stairs with no handrails and with no resting; – can use a dustpan and brush to sweep the floor and straighten up again. 2. Moderate ability – can walk 175 metres with no stopping or severe discomfort; – can ascend/descend 12 steps with no rests while using a handrail; – can balance for long periods without holding on to something; – can reach down to pick something up from the floor. GB 16⫹ population (%)

15

10

5

0 Full ability

Moderate ability

Partial ability

Minimal ability

FIGURE 6.22 Locomotion capability – cumulative frequency graph showing the proportion of GB population aged 16 or over that does not meet the specified ability level.

191

Human capability and product design

3. Partial ability – can walk 50 metres with no stopping or severe discomfort; – can ascend/descend 12 steps by using a rail and resting; – can balance for short periods without holding on to something; – can bend knees and straighten up again. 4. Minimal ability – cannot walk or use steps; – cannot balance even for short periods or perform any bending tasks. Figure 6.22 presents cumulative values for locomotion capability, showing how many people in Great Britain have less than the specified level of ability, indicating how many people would have difficulty using a product with that level of demand.

9. REACH AND STRETCH Different products require the ability to reach one or both arms out from the body. This reaching can either occur in front of and above the body, for example to put on a hat, or out to the sides, to reach objects while sitting at a desk (Figure 6.23). Reaching is more difficult to perform when the arm has to be moved further away from the rest position, and is more difficult when two arms have to reach simultaneously. The relative ability to position the hands compared to the head and body depends on the range of motion of the elbow and shoulder joints. The absolute position that the hands can reach to is also influenced by anthropometric data such as body height and arm length, together with other factors such as whether the person is in a wheelchair, although this absolute position is not considered here. Age related conditions such as arthritis can cause reductions in joint mobility and stiffness, leading to limited reaching ability. The distance a person might be able to reach out could be significantly less than the length of the arm. Temporary injuries such as a broken or bruised arm or collarbone will also affect the amount that a person can reach. The following reach and stretch functions are described in the context of product and service design: • Reaching out in front; • Reaching out to the sides.

9.1. Reaching out in front Most products require the ability to put one hand in front of the body to use them. Where possible, requiring both hands to be placed in front of the body simultaneously should be avoided, and using the product should be possible either by reaching the left, or right arm. Requiring a person to reach above their head can be more difficult than just reaching out in front.

A

B

C

FIGURE 6.23 A Some people can reach over their heads; B others only out in front; and C some cannot reach out at all.

192

Product Experience

9.2. Reaching out to the sides Reaching out to the sides is used to access items while sitting at a desk, or to put on a jacket. The further the arm has to reach out to the side or back, the more difficult it can be to reach items.

9.3. Context of use Environmental factors can additionally reduce a person’s reach and stretch capability. For example, clothing such as heavy jackets and multiple layers can reduce a person’s ability and comfort to reach out. Carrying things while using a product can reduce the mobility of one or both hands. Equally, it may not be desirable for the user to have to put down the baby, handbag or carrier bags to use the product.

9.4. Design guidance Better design can be achieved as a result of a better understanding of the above capabilities and associated capability losses. The following guidance should assist in making designs more inclusive: • Allow for single-handed operation where possible, by eliminating the need to reach both hands out simultaneously, and facilitating the option to reach either the left or right arm out to operate a product. • Ensure that products or services that require access by the public are able to cope with the range of heights that people can reach to, including those in wheelchairs. • Minimize the need to exert forces with the arms outstretched or, in particular, when reaching over the head. • Consult available data sources on reach ranges when setting the dimensions of products and environments.

9.5. Prevalence data Finally, for the purposes of examining products, it can be useful to measure reach and stretch capability on a four-point scale: 1. Full ability – can raise both arms up to the head, behind the back or out in front with no difficulty. 2. Moderate ability – can only raise one arm up to head, behind the back or out in front with no difficulty (has difficulty with other arm). 3. Partial ability – can only raise one arm up to head, behind the back or out in front with no difficulty (cannot use other arm). 4. Minimal ability – cannot raise either arm out in front of body or up to head; – cannot put either arm behind back. Figure 6.24 presents cumulative values for reach and stretch capability, showing how many people in Great Britain have less than the specified level of ability, indicating how many people would have difficulty using a product with that level of demand.

193

Human capability and product design

GB population (%)

15

10

5

0 Full ability

Moderate ability

Partial ability

Minimal ability

FIGURE 6.24 Reach and stretch capability – cumulative frequency graph showing the proportion of GB population aged 16 or over that does not meet the specified ability level.

10. DEXTERITY Most products require physical manipulation of controls and manual handling. The hands are used to grasp, move and exert forces to use and operate various products. Objects can be grasped, pushed and pulled. The human hand is composed of four fingers and a thumb (Figure 6.25). Muscles in the hand and arm control the movement of the fingers and the wrist. The presence of the opposable thumb is key to the many manual tasks that we can perform. We can exert clamping forces between fingers, and also clamp and hold larger objects in the palm of the hand. Individual fingers can also be used to exert pulling and pushing forces. Sometimes we use both hands at the same time to manipulate objects. This coordinated movement requires not only strength and dexterity in the fingers, but also visual and motor control ability. There are many causes of pain than can limit dexterity. Arthritis is a common cause of chronic pain, which is especially prevalent in older people. It can cause pain, stiffness, and swelling in the joints. This may affect not only the joints, but also other parts of the body, including muscles and organs. People vary in their ability to tolerate pain caused by those actions required for product use. This means that although the use of a product may be possible for those of a certain capability level, it will be fatiguing and painful. Continued use may lead to flare-ups of chronic pain, making product use impossible. Four major dexterity functions are considered in design applications: 1. 2. 3. 4.

Force exertion without grip. Precision gripping. Power gripping. Two-handed operations.

Various musculo-skeletal conditions such as various forms of arthritis cause joint stiffness, pain, and reduced joint mobility. This results in a reduction of movement and the ability to exert forces on objects. This reduction (Figure 6.26) generally occurs with increasing age (DTI, 2002; Burke et al., 1953).

10.1. Force exertion without grip Forces can be exerted on products with the fingers or the palm of the hand without requiring a grasping movement. Pushing buttons and depressing toaster sliders are examples of

194

Product Experience

Thumb Index Radius Middle Ulna

Ring Little

Wrist bones

Finger bones

FIGURE 6.25 The shape and structure of the human hand.

120 Grip strength

110

Pounds

100 90 80 70

Grip strength endurance

60 50

75–79

70–74

65–69

60–64

55–59

50–54

45–49

40–44

35–39

30–34

25–29

20–24

16–19

12–15

40

Age (years)

FIGURE 6.26 Changes in grip strength with age (Burke et al., 1953).

this type of action. A person must therefore have the ability to exert forces on the product controls and chassis in three planes: 1. Up-down hand motion and forces – adequate pushing force is required to depress levers and sliders of, for example, a toaster. 2. Right-left hand motion and forces – adequate pushing force is required to push the product from side to side on a surface, or push in a button at the side of the product. 3. Hand motion and forces away and toward the body – adequate pushing force is required to push the product away from or towards the body. In general, it is easier for forces to be exerted downward and toward the body, rather than upward and away from the body.

10.2. Precision gripping Precision gripping (or pinch gripping) is the ability to use the fingers and thumb to grasp product controls and exert the required forces to operate them. These grips are low force and high precision movements because they largely depend on the function of individual fingers. The resulting actions can be linear or rotational (twisting) movements. Everyday activities that use this fine manual dexterity include writing (pen gripping) and using chopsticks.

Human capability and product design

195

The pinch grip is used on many controls where the thumb and index finger provide a pinching force when in opposition. Examples include opening a coke can, grasping a small rotary dial, and picking up a small object.

10.3. Power gripping A power grip is a grip formed with the fingers and the palm of the hand in order to move or manipulate objects. This type of grip is used for grasping handles and holding smaller products such as mobile phones and remote controls. Power grips are high force and low precision movements because they use the bulk of the hand for force generation rather than the strength of individual fingers. The resulting actions lead to large arm movements for lifting and picking up larger and heavier objects. Everyday activities such as lifting a mug, a blender or a juice carton require power grips.

10.4. Two handed tasks Many tasks with everyday products require the use of both hands at the same time. Each hand will adopt either a power or precision grasp depending on the task. Tasks such as lifting a heavy casserole dish might require each hand having similar grasp. Tasks such as opening jars might require one hand to have a power grasp while the other hand adopts a precision grasp. The dominant hand (either right or left) is usually the preferred hand for accomplishing the more dextrous action (such as twisting the lid when opening a jar). Some people have only one functional hand. Certain tasks can be more difficult if this hand is not the dominant hand. Two handed tasks or tasks that require complex motions are also more difficult for older and disabled people to perform.

10.5. Context of use Ambient temperatures can affect hand function. Manual dexterity is reduced in colder environments, making it more difficult to perform manual tasks with products. The flexibility and sensitivity of the fingers are decreased in these circumstances. Wearing protective clothing such as gloves can also make it harder to operate controls on products. Activities such as making a call on a mobile phone in winter can be more demanding because of reduced dexterity. Surfaces that are smooth can be difficult to grasp and hold effectively. If such surfaces are placed in environments that are dusty, or where liquid can get onto grasping surfaces, the product becomes more difficult to hold. Products used in kitchen environments can get wet and covered with various cooking ingredients. Hand tools and other products used in workshop environments can become covered with dust particles or grease, or sweat in high-temperature conditions.

10.6. Design guidance Better design can be achieved as a result of a better understanding of the above capabilities and associated capability losses. The following guidance should assist in making designs more inclusive: • Consider the compatibility of grip and intended action on the product, to avoid situations where a product requires a certain type of grip or motion that is not compatible with the overall task.

196

Product Experience

• Attempt to lower all force requirements to operate the product controls (grasping, pushing, pulling, twisting and lifting forces), making allowance for older people and people with disabilities who generally have reduced strength compared to younger and fully able people. • Avoid, where possible, controls that require simultaneous manipulations such as pushing and twisting at the same time, such as those often used with dials and bottle caps. • Utilize pushing in preference to rotating, since for the latter a pincer grip is required in addition to the application of rotational force. • Cover surfaces to be gripped with materials that result in adequate friction between the surface of the product handle and the hand, since slippery or smooth surfaces are more difficult to grasp, whereas rubbery and slightly deformable surfaces are easier and more comfortable to hold.

10.7. Prevalence data Finally, for the purposes of examining products, it can be useful to measure hand function capability on a four-point scale: 1. Full ability – can pick up a safety pin or 2.5 kg bag of potatoes with either hand; – can tie a bow in laces with no difficulty; – can squeeze water from a sponge with either hand. 2. Moderate ability – can pick up a 2.5 kg bag of potatoes with one hand (but not the other); – can use a pen with no difficulty; – can unscrew the lid of a jar of instant coffee with no difficulty; – can wring out light washing with no difficulty. 3. Partial ability – can pick up and hold a safety pin or half a litre of milk in one hand (but not the other); – can turn a tap with one hand (but not the other); – can squeeze water from a sponge with one hand (but not the other). 4. Minimal ability – cannot hold and pick up relatively light objects such as cups and mugs; – cannot manipulate or control small objects; – cannot squeeze objects.

GB population (%)

15

10

5

0

Full ability

Moderate ability

Partial ability

Minimal ability

FIGURE 6.27 Dexterity capability – cumulative frequency graph showing the proportion of GB population aged 16 or over that does not meet the specified ability level.

Human capability and product design

197

Figure 6.27 presents cumulative values for dexterity capability, showing how many people in Great Britain have less than the specified level of ability, indicating how many people would have difficulty using a product with that level of demand.

11. SUMMARY Designing a product to minimize exclusion requires knowledge of the demands made by the product on its users, knowledge of the range of capabilities of the target users, and their prevalence, and knowledge of the context of use. This chapter has provided an introduction to such knowledge. The user capability cumulative frequency graphs indicate that some types of capability loss are much more prevalent than others. In addition, the shape of the graphs provides some indication of the design challenge required to maximize inclusion. For example, significant efforts are required to minimize exclusion due to dexterity capability losses (Figure 6.27) since the levels of exclusion do not decrease appreciably until the product demand is near minimal, thus accommodating those with partial ability. Such challenges can have a significant impact on user exclusion and the priorities for design, where ‘good’ design ensures that a product can operate within the capabilities of as many users as possible and meet their needs and desires. There are likely to be some aspects of a design or concept that can be improved at little cost, for example, the font type and size used on a safety label, while others may demand more radical change, such as the redesign of a menu-driven interface to avoid the use of multi-level menus. However, ‘good’ design is often compromised by conflicting requirements, for example, a small, light mobile phone cannot have both a large screen and large buttons with size 20 font labels. As a result, designers will make decisions that result in users being excluded; the challenge is to make sure that they knowingly do so. Knowledge of human physiology, its associated function and resulting capability variation can help designers to determine the costs and corresponding benefits for making products more inclusive. Consideration can also be given to the expected contexts of use of the product in order to increase its accessibility within an appropriate range of environments. Capability prevalence data encourages the setting of appropriate and achievable accessibility targets for each product, rather than generic targets that might encourage systematic exclusion. Inclusive design is an aspiration that should drive designers to better understand the diversity of the users of their products and services, enabling them to delight more customers.

ACKNOWLEDGMENTS The contents of this chapter have drawn on the collective efforts of the Inclusive Design team at the Cambridge Engineering Design Centre, including: Nicholas Caldwell, Carlos Cardoso, Susannah Clarke, Nathan Crilly, Joy Goodman, Mari Huhtala, Patrick Langdon, Tim Lewis, Umesh Persad, Nick Reddall, Sam Waller and Suzanne Williams.

REFERENCES Arditi, A. (2006a). Making text legible [Online article]. URL http://www.lighthouse.org/print_leg.htm Arditi, A. (2006b). Effective color contrast [Online article]. URL http://www.lighthouse.org/color_contrast.htm ASHA. (2006). Type, degree and configuration of hearing loss [Online resource]. American Speech-Language Hearing Association. URL http://www.asha.org/public/hearing/disorders/types.htm

198

Product Experience

Baddeley, A. D. (2004). Your memory: A user’s guide (4th Ed.) London: Carlton Books. Beales, P. H. (1965). Noise, hearing and deafness. London: Michael Joseph. Burke, W. E., Tuttle, W. W., Thompson, C. W., Janney, C. D. and Weber, R. J. (1953). The relation of grip strength and grip strength endurance to age. Journal of Applied Physiology, 15, 628–630. DTI. (2002). Specific anthropometric and strength data for people with dexterity disability [Online report]. London: Consumer and Competition Policy Directorate, Department of Trade and Industry. URL http:// www.berr.gov.uk/files/file21811.pdf Evamy, M. and Roberts, L. (2003). In sight: A guide to design with low vision in mind. Mies, Switzerland: Rotovision. Grundy, E., Ahlburg, D., Ali, M., Breeze, E. and Sloggett, A. (1999). Disability in Great Britain. London: Department of Social Security, Corporate Document Services. Huppert, F. A. (2002). Designing for older users. In: P. J. Clarkson, R. Coleman, S. Keates and C. Lebbon (Eds.) Inclusive design: Design for the whole population, pp. 30–49. London: Springer Verlag. Microsoft. (2003). The market for accessible technology: The wide range of abilities and its impact on computer use [Online report]. Accessible technology market research commissioned by Microsoft, conducted by Forrester Research, Inc. URL http://www.microsoft.com/enable/research/phase1.aspx and http://www. microsoft.com/enable/research/phase2.aspx Miller, G. A. (1956). The magical number seven, plus or minus two: some limits on our capacity for processing information. The Psychological Review, 63, 81–97. Moore, B. C. J. (2003). An introduction to the psychology of hearing (5th Ed.) San Diego, CA: Academic Press. Moore, B. C. J. (2006). Factors affecting speech intelligibility for people with cochlear hearing loss. In: S. Greenberg, and W. A. Ainsworth (Eds.) Listening to speech: An auditory perspective, pp. 273–287. Mahwah, NJ: Erlbaum. Norris, B. and Wilson, J. R. (1995). Childata: The handbook of child measurements and capabilities – data for design safety. London: Department of Trade and Industry. Peebles, L. and Norris, B. (1998). Adultdata: The handbook of adult anthropomorphic and strength measurements – data for design safety. London: Department of Trade and Industry. Philips. (2004). The Philips index: Calibrating the convergence of healthcare, lifestyle and technology. URL http://www.design-council.org.uk/Documents/About design/Design techniques/Inclusive design/Philips Index(us version).pdf RNIB. (2006). Good design guidelines [On-line resource]. URL http://www.rnib.org.uk/xpedio/groups/public/ documents/code/public_rnib003460.hcsp RNID. (2006). RNID information resources [Online resource]. URL http://www.rnid.org. uk/information_resources/productsandequipment/productnews/2006/?ciid⫽286419 Smith, S., Norris, B. and Peebles, L. (2000). Older adultdata: The handbook of measurements and capabilities of the older adult – data for design safety. London: Department of Trade and Industry. UN. (1990). Disability statistics compendium: Series Y. No. 4. New York: United Nations. Vassilief, A. and Dain, S. (1986). Bifocal wearing and VDU operation: A review and graphical analysis. Applied Ergonomics, 17(2), 82–86. Wickens, C. D. and Hollands, J. G. (2000). Engineering psychology and human performance (3rd Ed.) Upper Saddle River, NJ: Prentice-Hall.

7

CONNECTING DESIGN WITH COGNITION AT WORK DAVID WOODS The Ohio State University, OH, USA

AXEL ROESLER University of Washington, Seattle, WA, USA

1. INTRODUCTION In design, we either hobble or support people’s natural ability to express forms of expertise. There is no neutral. (Woods, 2002)

What is the relationship between design and cognition? This question is especially important because new technological capabilities have been appearing at an accelerating rate. First, technologies for more autonomous machines create the image of future machines that do things as substitutes for our own activities, or that even become alternatives to us (Kiesler and Hinds, 2004). Secondly, the technology for connecting people at a distance provides new opportunities to coordinate activity across large distances and to integrate activities that go on at different places or at differing time scales (Hinds and Kiesler, 2002). Thirdly, the technology of collecting, sorting, and focusing data has grown so rapidly that people have available to them hyper-massive fields of data for analyzing situations and generating new or revised plans of action (Woods et al., 2002; Thomas and Cook, 2005). Fourthly, there are new ways to visualize data sets, to fuse feeds from different sensor types, to look at aspects of the world as if we were standing in places that are difficult, expensive, dangerous or impossible to stand in. Sensors connect us and project us into remote situations where we can assume and change different points of observation. All of these vectors of technological change have profound implications for what people do, why they want to do these things, and how they will do these activities in the future. How can designers use the new powers to accelerate human skill acquisition and amplify human performance? Product Experience Copyright © 2008 Elsevier Ltd.

199

200

Product Experience

What is the relationship between cognition and design? One longstanding perspective on this relationship is that people have severe limits on their memory, attention and problem-solving capabilities. People are prone to illusions and biases of many kinds (Gilovich et al., 2002; Kahneman and Tversky, 1996; but see Gigerenzer, 2000; Zsambok and Klein, 1997). In this view, design then would use new technological capabilities to develop prostheses that overcome the inherent weaknesses of people. For example, some people want to design machines that watch over people in action to see how badly their limitations may be impacting their performance (e.g. Schmorrow, 2005). Depending on the machine’s assessment of how people’s mental or physical state is changing, the machine will decide when and how to change the interface, and change user tasks so that the demands remain within people’s limited capabilities. But equally longstanding is a completely different perspective about cognition and the impact of design. People are active, adaptive, goal seeking agents who re-design objects through their activities so that devices work for them (Winograd and Flores, 1986; Flach and Hoffman, 2003; Woods and Hollnagel, 2006). This view starts with people as meaning seeking, explanation generating, attention focusing, learning agents (Bruner, 1990; Weick, 1995). Actually, people, when acting expertly, are a model of competent action, and often the only model of competence available to study when one wants to understand how these processes of cognition work (see Feltovich et al., 1997; Ericsson et al., 2006; Roesler and Woods, in Chapter 8 of this volume). People will learn to express various forms of expertise with experience, and they will shape the artifacts they encounter and interact with as resources to cope with the demands of the situations they face as they seek to meet their goals. Designed artifacts and systems then are resources that stimulate people to adapt to achieve their purposes – triggering adaptive behavior that expands what people can achieve (Woods and Dekker, 2000; Woods et al., 2002). These expansive adaptations are not simply good or bad. Rather they transform systems leading to new roles, new ways of doing things, new reasons to do things. Technology change and new designs are one set of drivers in these processes of organizational transformation and human adaptation. The changes that are triggered result in new levels of performance on some dimensions, new squeezes on performance in other places, new side effects when things that were separate become connected, and new forms of complexity. The basic dynamic is captured in the Law of Stretched Systems (Woods, 2002; Woods and Hollnagel, 2006). The Law of Stretched Systems: every system is continuously stretched to operate at its capacity. People will exploit ‘improvements,’ for example, in the form of new technology, to better achieve goals by pushing the system out to operate near the edge of its new capacity boundaries. The process of adapting to exploit the improvement results in a new intensity, complexity, and tempo of activity.

The following empirical summary captures this dynamic, adaptive process and represents the findings from several researchers (Carroll et al., 1991; Winograd and Flores, 1986). Much of the equipment deployed … was designed to ease the burden on the operator, reduce fatigue, and simplify the tasks involved in operations. Instead, these advances were used to demand more from the operator. Almost without exception, technology did not meet the goal of unencumbering the personnel operating the equipment … Systems often required exceptional human expertise, commitment, and endurance … There is a natural synergy between tactics, technology, and human factors … Effective leaders will exploit every new advance to the limit. As a result, virtually every advance in ergonomics was exploited to ask personnel to do more, do it faster and do it in more complex ways … One very real lesson is that new tactics and technology simply result in altering the pattern of human stress to achieve a new intensity and tempo of operations. (Cordesman and Wagner, 1996, p. 25) [Edited to rephrase domain referents generically].

Connecting design with cognition at work

201

In adaptive cycles, designs act as stimulants in two major ways (Woods and Hollnagel, 2006). First, designs can trigger expansive adaptations by users and stakeholders that exploit capabilities as they seek to achieve their ends. When these expansive adaptations occur, we discover that people have exploited designs in ways typically unforeseen by the designers (Woods and Dekker, 2000). But, second, design change also can introduce impediments that create complexities to be adapted around and overcome – the workarounds often captured in compilations of designs with poor usability (Norman, 1988; Koopman and Hoffman, 2003). Though specific designs often create a mix of both affordances to be exploited and complexities to be worked around, this mix can vary greatly for different people in different roles across an organization. The epigraph to this section captures the basic concept: Fielded designs have real effects on people in various roles, squeezing or enhancing their ability to express expertise. Designs can be examined in terms of how they release or undermine the cognitive skills of people as they carry out roles, handle changing situations, coordinate with others’ activities, and adapt to meet goals. Good designs then release these skills, and poor designs hinder or undermine people’s ability to acquire or demonstrate cognitive skills (Norman, 1993; Bruner, 2003). As Christopher Alexander (1977) noted, design has the special responsibility to connect technological capability to the adaptive power of people as goal-directed agents. Design should understand how skills, expertise, and adaptive capacity develop, how the work of designing can release these basic human processes, and how designs inadvertently can thwart these processes, leaving people to act as though they were cognitively impaired (Woods et al., 1994, Chapter 5). This chapter provides an introduction to some of the basic concepts about design and cognition. Mastery of these concepts will help design processes to enable or release human expertise, or, as Don Norman (1993) put it, how to innovate ‘ things that make us smart’. A note about terminology – artifacts – People who write about cognition and design often use a neutral word for a designed interface, visualization, product, or object. They refer to these as artifacts since the object is just that, an object, until put in use by some person in some role. In use the physical thing or artifact is adapted to provide support. Through use the artifact becomes a cognitive tool, in that it supports or enables people to exhibit skilled and coordinated activity. But many artifacts introduce complexities which add burdens to the people trying to carry out some activity in the world.

2. DESIGN AND COGNITION AT WORK: IMPAIRED OR UNIMPAIRED MICRO-COGNITION To understand how properties of artifacts shape or influence cognition and activity, one has to appreciate some basics about cognition at work. One dominant perspective about cognition focuses on mental processes, skills, and reasoning strategies as things that happen only ‘in the head’. This is a microcognitive perspective (Klein et al., 2003). Micro-cognition refers to basic mental processes, such as working memory, that are believed to act as building blocks for more complex information processing. When cognition occurs only in someone’s head, the processes are invisible. As a result, research methods focus on overcoming this difficulty using various techniques to make internal mental processes explicit and observable to outsiders. For design, micro-cognition is concerned with the impact of a proposed or new artifact on the mental costs of carrying out the details of specific tasks and sub-tasks

202

Product Experience

for their role. For example, designs can act as external memory aids helping people call to mind relevant knowledge, or designs can act as memory barriers forcing people to remember various codes and abbreviations in order to carry out their basic activities through that device. A quite common and relatively simple case occurs with alarm codes: Consider this example from NASA’s mission control dialog over voice loops as the lunar lander descended to the moon during Apollo 11 (Murray and Cox, 1989). ‘1202.’ Astronaut announcing that an alarm buzzer and light had gone off and the code 1202 was indicated on the computer display. ‘What’s a 1202?’ ‘1202, what’s that?’ ‘12 … 1202 alarm.’

The alarms did not direct attention to the trouble, or jump start diagnosis of the source of the anomaly, or direct mission control’s response to the anomaly. Instead, the alarm codes resulted in a memory bottleneck. People, without help from the external displays or representations, had to remember what each code stood for (Woods and Hollnagel, 2006). This case is not just about alarms (or other examples of specific inscrutable codings), nor is it just a historical curiosity describing past designs. Many books continue to collect examples where the design of a device, interaction, or display turned out to increase memory demands on those using the artifact to accomplish something (e.g. Cooper, 1999). The impact of a design is summarized as a specific version of a very general story line – how a new artifact inadvertently created a bottleneck in micro-cognition. Often the demonstration of new memory bottlenecks produced by a design includes pictures of paper notes that users added to electronic systems as memory workarounds (e.g. users adding interpretive keys to overcome a bottleneck introduced by poor design). From a micro-cognitive perspective, poor designs create or exacerbate bottlenecks in elemental cognitive processes. Inadvertently increasing memory demands is just one class of negative consequences of poor design on micro-cognition. Potential bottlenecks in micro-cognition include: • Knowledge bottlenecks, i.e. the design makes it harder for people to call to mind knowledge relevant to the task at hand (Feltovich et al., 1997). • Memory bottlenecks (Byrne and Bovair, 1997). • Attention bottlenecks, i.e. the design makes it harder for people to switch attention between different threads of activity or lines of reasoning, such as interruptions (Czerwinski et al., 1991). • Workload bottlenecks, i.e. the design makes it harder for people to shift or prioritize workload to avoid risk of failure at workload peaks. • Oversimplified mental models, i.e. the design makes it harder for people to develop appropriate and accurate models of how a device or process works (Kieras and Bovair, 1984; Cook et al., 1991). • Intent bottlenecks, i.e. the design makes it harder for people to form coherent intentions to act given uncertainty and multiple goals which can conflict (see Kieras et al., 2001 on the modeling the translation of intention into action and see Cook, 2006, or Woods et al., 1994, Chapter 4 for illustrative cases of managing goal conflicts in health care). A good example is the bottlenecks that arise when poor design of transitions from one display to another as people perform interrelated tasks through a computer interface

Connecting design with cognition at work

203

creates the keyhole effect (Woods and Watts, 1997). The keyhole effect occurs when people can only see a small portion of the relevant information space at one time. The keyhole effect creates a lack of peripheral awareness that can leave people lost in the display set and not know where to look next to find relevant information (an attentional bottleneck). The navigation workload goes up dramatically when there is a keyhole effect which leads to memory and knowledge bottlenecks, as they must call to mind more information and remember more about where important data might be found among the many displays hidden behind the keyhole of the computer screen (Watts-Perotti and Woods, 1999; Doherty and Upton, 2005). Another example is how the principles for designing direct manipulation interfaces have been validated by studies that show such designs eliminate micro-cognitive bottlenecks for memory and workload by connecting intentions to actions when people perform tasks via classic computer interfaces (Kieras et al., 2001). Thinking about the relationship between cognition and design in terms of microcognition helps, but only within some very definite limits. Assessing the micro-cognitive consequences of designs is good for certain aspects of usability evaluation. If one has access to a prototyped or designed device that includes a partial specification of the tasks and sequences, of the procedures in different cases, and of the associated interactions and screen layouts, then one can use models of micro-cognition to detect memory and other bottlenecks. For example, micro-cognitive models have been used to predict long learning times and some types of basic task sequencing errors (Byrne and Bovair, 1997; Gray, 2007). In these cases, the specification of tasks is translated into a computerized cognitive model of micro-cognition (e.g. how working memory load varies as tasks are performed, and how chunking occurs as a mechanism of learning from consistent experience over repeated tasks). When the computer simulation is able to carry out the task, one can use parameters of the underlying cognitive model to detect bottlenecks on working memory or long learning times (Bovair et al., 1990 provides one illustration of these tests). But note there are several ‘catches’ in the relationship of micro-cognition to design. First, for design to use information about how an artifact impacts micro-cognition requires one first has a design (or at least a partial design), since the tests require specification of the details of interaction. In other words, micro-cognition can be used to evaluate or test the impact of designs, but cannot play a direct role in the processes of innovation necessary to generate useful or promising possibilities (Woods, 1998; Lee et al., 2006). Secondly, micro-cognitive tests have difficulty detecting cumulative complexity that builds up as more and more features are added to a product over time in a competitive business environment (Lee et al., 2006). Each feature, looked at alone, appears to pass usability tests and to provide something for a use case or to compete with another product. Inadvertently, however, there is a cumulative complexity effect that comes to dominate any benefits added by each individual feature or option. Pragmatically, businesses have begun to notice the costs of cumulative complexity and the resulting risk of poor product performance in the competitive business landscape (Rust et al., 2006). Several companies and commentators have begun to call for reductions in the number of features on products in a search for a kind of simplicity (Pogue, 2005; Phillips Co., 2005; Wallace, 2006). Thirdly, by using micro-cognitive tests one can discover how a design hobbles cognition and retards learning, but one cannot discover new affordances that may lead to useful adaptations by practitioners that expand their ability to reach goals. These tests can identify how a design creates bottlenecks by forcing cognition to the micro-level so that the specific features that create complexity can be modified or repaired. But the tests do not contribute to the innovation processes that would lead to new concepts for products. When users face complexities such as bottlenecks in micro-cognition, they work around them (e.g. Woods et al., 1994; Cook and Woods, 1996; Cook et al., 2000). It is interesting

204

Product Experience

to note that this is one of the ways that the complexities are noted, not by measuring memory or workload bottlenecks directly, but by observing the workarounds people devise to avoid being hobbled too much by these sorts of complexity. The ways that users tailor their strategies, and the ways that they re-shape artifacts to workaround complexities and fill gaps act as pointers to other adaptive processes. Designs can help foster expansive adaptations. But to do so we need to look beyond bottlenecks in micro-cognition, and examine the functions of macro-cognition that link what people do, as responsible agents carrying out roles, to larger processes of coordinated activity and to larger functions such as anomaly recognition, sense making, and replanning.

3. DESIGN AND COGNITION AT WORK: EXPANDING THE IMPACT OF MACRO-COGNITION A broader perspective on cognition looks at how mental processes support coordinated activities in the world. Most vividly, it has been described as cognition in the wild (Hutchins, 1995) or cognition in natural settings (Klein, 1998), though other labels are joint cognitive systems (Hollnagel and Woods, 2005); situated cognition (Suchman, 1987), or distributed cognition (Hutchins, 1995). This approach considers how macro-cognitive functions such as joint attention, anomaly recognition, replanning, sense making, and others are situated in the context of human goals, desires, activity, and work (Klein et al., 2003; Klein et al., 2006a,b). Macro-cognition moves away from an ‘in the head’ view of the internal processing of an individual because such a perspective loses sight of the active role of people in constructing meaning and exploring their environment. Micro-cognitive approaches to interaction design tend to get lost in a single person in front of a computer screen, missing how people engage in broader activities for larger purposes that move well beyond what appears on computer screens. Putting cognition in the world, or in the situation, emphasizes how people actively explore and find meaning in the world (Gibson, 1979). The ability to look, listen, smell, taste, or feel requires an animal capable of orienting its body so that its eyes, ears, nose, mouth, or hands can be directed toward objects and relevant stimulation from objects. Lack of orientation to the ground or to the medium surrounding one, or to the earth below and the sky above, means inability to direct perceptual exploration in an adequate way. (Reed, 1988; p. 227 on Gibson and perceptual exploration.)

People as agents exist in an environment that is more than what can be seen directly. The actor’s environment includes social structures, shared concepts, conventions, and human-fashioned artifacts. The active exploration process operates just as strongly when the people’s experiences, observations, and actions are mediated by properties of digital media. Heuristically, macro-cognitive functions can be grouped into a few categories of anomaly recognition, sense making, joint activity, and replanning. Anomaly recognition – Noticing when events do not fit (or are beginning to deviate from) the current assessment or expectations (Woods and Hollnagel, 2006, Chapter 8). An anomaly is an event that produces a ‘break’ or interruption in the smooth flow of activity (Winograd and Flores, 1986). Attention, information search, explanation building (sense making), and problem solving then follow as one tries to understand the anomaly, what it means, and how to modify future actions. A simple example is software problems

Connecting design with cognition at work

205

that occur when one is preparing material to meet a critical deadline – the break in flow occurs when the word processing software freezes. The crash produces a break in the activities enabled or empowered by the software; people’s attention and resources have to flow to the software as an object in itself and away from the activity and its purpose. Macro-cognitive studies examine what constitutes an anomaly, how people may overlook or discount anomalous indications or be very sensitive to early signs of trouble (Klein et al., 2005b), and how recognizing an anomaly influences information search, explanation building and explanation revision (Woods, 1995a). Sense making – How people find meaning in the data they see around them (Weick, 1995; Pirolli, 2007). People are explanation machines looking for meaning in what they experience (Bruner, 1990). Klein et al. (2006a,b) have proposed that sense making is built on how people frame and re-frame the events around them, where ‘frame’ refers to the perspective or context used to interpret the data or event in itself. The framing effect in psychology is the much replicated finding that changing the context can change how people see and interpret the very same data or event. In Klein’s model, sense making refers to how people generate and modify the frame or context given the flow of experience (as opposed to the usual study that examines how the interpretation of the flow of experience changes if the larger frame called to mind changes). For Klein et al. (2006b) sense making consists of how people elaborate a starting frame as new information comes in or new events occur, how they begin to question the current frame, and, eventually, how they look for a new frame to understand the events in question – reframing. Sense making, then, is concerned with how people maintain a ‘big picture’ and avoid getting lost in the details surrounding a situation (or to use an aviation analogy, how to stay ‘heads-up’ when there are many individual sub-tasks to be carried out which tend to take you ‘heads-down’ in each), how people can demonstrate what Clausewitz in 1832 on the art of war called coup d’oeil, that is, how people can see at a glance the essential characteristic or leverage point in a complex situation (insight), how people are able to move from mere information gathering to develop syntheses and alternative explanations, how people are able to revise assessments and avoid being stuck in one view, how people are able to develop broadening checks to help avoid premature closure on a weak assessment or explanation. Interestingly, many of the aspects of sense making work best when they do not rely just on an expert individual, but when they are part of collaborative interactions with others. For example, it can be very difficult for a single person to revise an assessment even as new evidence begins to accumulate that the assessment is incorrect (Feltovich et al., 2004). It is surprisingly easy to discount or rationalize away the new evidence and remain stuck in a previous assessment. Revision generally occurs when a fresh point of view enters the situation (Woods and Hollnagel, 2006). Errors in plans are best detected when cross-checks are built into a system by providing review over multiple and diverse perspectives (Patterson et al., 2004). Joint activity – How to coordinate and synchronize activities that are distributed over multiple parties and artifacts (Hutchins, 1995; Sebanz et al., 2006). Moments of micro-cognition are part of, and contribute to, larger flows of interaction and coordination. For example, one often thinks of attention as a characteristic of an individual at some point in time – focused on one out of many activities, cues, or lines of thought. However, people demonstrate the ability to assess where another person’s attention is focused and to follow another’s focus, even at very young ages (Moore and Dunham, 1995). In addition, people signal to others where their attention is focused in order to influence where the other’s attention will focus. This ability for joint attention illustrates

206

Product Experience

how interpersonal cognitive capabilities play critical roles in processes of interaction and coordination across people. Joint attention turns out to be critical when different agents need to modify what they are doing when new events occur (Woods and Hollnagel, 2006). Another part of joint attention is judging the interruptibility of others. Without this capability, miscoordination occurs where interruptions have high costs (Czerwinski et al., 1991; Ho et al., 2004). How technology is used influences joint attention as one kind of macro-cognitive function, for example the ability to listen in on other conversations by being part of a voice loop has been shown to be a critical contributor to very high levels of coordination in space shuttle mission control, even though from a micro-cognitive perspective the voice loop seems very noisy, confusing, and distracting (Patterson et al., 1999). Klein et al. (2005a) describe various other functions that are necessary for joint action over multiple parties and roles. For example, maintaining and repairing common ground – a shared frame of reference about the activities and intentions going on – is a critical aspect of joint activity (Clark and Brennan, 1991; Monk, 2003). Replanning – How to modify a plan in progress when disruptions occur or opportunities arise (Shattuck and Woods, 2000). Plans are always in motion at some time scale, and there are degrees of commitment to planned actions ahead of a continuous rolling horizon. To modify plans in progress, participants need to be able to recognize unexpected events that disrupt the unfolding plans or unexpected events that provide new opportunities to achieve goals behind the plan. It can be quite difficult to recognize when new events affect a plan in progress or to see all of the implications of a new event for the plan in progress. Replanning is an adaptive exercise adjusting or re-formulating a plan when new events introduce impasses or undermine the assumptions behind the plan. It may be even more difficult to recognize when new events provide new opportunities to achieve the goals behind the plan. In replanning situations, people can often under-adapt by trying to continue the plan despite the disruptions (Woods and Shattuck, 2000). Or they can over-adapt – that is, they recognize events challenge the literal plan sequence, but their actions fail to respect many important constraints that had been considered in developing the original plan. New actions are taken to get around the specific disrupting event work around the disruption, but these actions often have undesirable side effects because they fail to conserve other important aspects of the situation, original plan, and goals. This failure mode is referred to as ‘missing the side effects of changes to planned activities following changes in the external situation’. Replanning becomes more difficult as the plan involves more groups or organizations over wider ranges of space and time (Smith et al., 2007). Changing one aspect of a plan in progress can produce a cascade of modifications or even disruptions for other activities in motion by other groups. Assessing the impact and coordinating the adjustments is increasingly difficult as the time and spatial scale of the planned activities increases. Effective replanning depends on being able to see and track the cascades, reverberations, and side effects of new events and possible plan modifications.

4. CONTRASTING MICRO- AND MACRO-COGNITIVE VIEWPOINTS Note the contrast of the list of macro-cognitive functions and the list of micro-cognitive bottlenecks. The macro-cognitive list refers to difficult capabilities that can be enabled or expanded through design of interactions, displays, and devices. The micro-cognitive list refers to troubles that arise when design hobbles people.

Connecting design with cognition at work

207

One can see the contrast in how each view approaches basic topics in workload and attention. Micro-cognition focuses on the limits of an individual to carry out mental operations and keep track of multiple lines of reasoning. Studies examine how people can time share or divide attention over different kinds of processing that could be required to execute various tasks, for example, listening to one conversation while visually controlling a dynamic process (such as occurs in driving a car while talking on a cell phone). The studies can help identify bottlenecks in workload and attention which could lead to poor performance (the cell phone conversation takes mental resources away from driving-related tasks, leading to poor vehicle control and slow responses to changes in the driving situation). On the other hand, macro-cognition is interested in how people adapt to manage their tasks over time to avoid bottlenecks or how they prepare coping strategies to use should they be forced into a bottleneck situation. The starting point is in the world of activity, and the macro-cognitive observer notes that activity ebbs and flows, with periods of lower activity and more self-paced tasks interspersed with busy, high-tempo, externally paced operations where the quality of task performance is more critical. As people learn where and when bottlenecks that threaten performance may occur, they can invoke some basic adaptive strategies (Woods and Hollnagel, 2006). They can develop a strategic response by either shifting some of the activity to other times to lower workload or by recruiting more resources, for example by bringing other people into the situation (for example, critical care physicians carry out these kinds of responses to reduce the chances of being trapped in a workload bottleneck should a patient’s status deteriorate surprisingly; e.g. Cook and Woods, 1996). These adaptive responses depend on the ability to anticipate potential upcoming bottleneck points and the changing pressures on task performance as situations ebb and flow or vary, being more routine or more exceptional. With anticipation one can recruit more resources such as extra staff, special equipment, additional expertise, or additional time, but since this strategy consumes organizational resources there can easily be pressures from above that restrict this strategic response. A part of expertise, then, consists of being able to anticipate potential bottlenecks and high tempo periods and to invest in certain tasks, or conserve certain resources, which reduce the risk of being caught in a bottleneck (and the accompanying risk of prioritizing the wrong parts or doing the whole task poorly) or being unable to cope with an anticipated bottleneck when it does occur. This becomes an illustration of the sense making macro-cognitive function at work and the sub-function of problem detection (Klein et al., 2005b, 2006b). People may not always be good at sense making in situations with the potential for significant workload bottlenecks, but we can investigate the strategies experts use and which strategies work well in some situations to envision new promising possibilities for design. From this short exploration of a single issue we can see the general contrast between the two perspectives on cognition and design. The micro-cognitive perspective provides concepts and tools to detect when design limits performance, and can provide an impetus or justification for (but not innovate in) new design work – it initiates repair. In the case of cell phone distractions, if others have developed new designs, micro-cognitive tests can be carried out to see if attentional bottlenecks are reduced. The macro-cognitive perspective generates directions to explore in the conceptual and ideation phases of design – it participates in innovation. And importantly, macrocognitive results provide criteria that can be used to see if early design work will prove to be a promising direction that merits further investment of energy, talent, time and other design resources (rather than waiting for a prototype to be well specified enough to begin testing). Finally, macro-cognition provides a guide to create concrete scenarios that

208

Product Experience

instantiate challenges for anomaly recognition, sense making, joint activity, and replanning as relevant to the scope of the design project. These scenarios can be used in envisioning and participatory design techniques and as test cases for evaluation. From a macro-cognitive viewpoint, cell phones are just one of many additional tasks that can take driver’s attention away from vehicle control and anticipating upcoming events. It is one example of a general problem created by new technology in the car – how to maintain a ‘heads-up’ or big picture view of the driving situation, despite the need or desire to perform many other ‘heads-down’ or secondary tasks. Heads-up is a label that comes from aviation. It refers literally to looking out the window and figuratively to the sense making processes that allow one to maintain a big picture view of the priorities for the context. Heads-down literally refers to looking at one of the many interfaces or devices used for secondary tasks that require drivers or pilots to look down inside the cockpit or cab; the label has the connotation of getting lost in details. New tools are being devised especially for aviation that help pilots shift attention to the big picture by using tactile signals to indicate the need to shift out of secondary tasks and re-examine the strategic situation (achieving joint attention while judging interruptability; see Ho et al., 2004).

5. MACRO-COGNITION AND EXPANSIVE ADAPTATIONS Often new technological developments are important because they have the potential to change how macro-cognitive functions are carried out. Effective leaders and innovators will discover ways to exploit the new technological power to better achieve goals which will either require or result in changes in macro-cognitive functions. One ongoing trend is new sensor, robotic, and networking technology that makes it possible for macrocognitive functions to be extended over much larger fields of available information and over a larger set of relevant stakeholders. This power – broader connection across groups that are physically separated – is exploited to achieve more, more quickly (e.g. global product engineering networks that speed product development). This new organization enabled by the technology for connectivity at distance expands the scope of sense making and replanning activities over new groups and new relationships. But the tools that support sense making and replanning also change when the scope changes. How will engineers in Bangalore, Seattle, and Moscow collaborate to maintain seeing the big picture as the work encounters obstacles? How will the groups track the reverberations of design changes to avoid missing side effects of change decisions? How will one group realize the new opportunity represented by an advance developed by another group in another part of the world? The process of transformation triggered by new technology changes the demands placed on macro-cognition, changes the affordances for support of macro-cognition, and requires new innovative design efforts to build new support tools. For example, simply copying the devices and displays that support collocated teams and re-using them when teams are distributed has led to lower performance on macrocognitive functions (Hinds and Kiesler, 2002). Macro-cognitive functions are resources people also use to cope with change. To the degree a team or organization is better able to do anomaly recognition, sense making, joint activity, and replanning, they will also be better able to adapt to change by avoiding unnecessary complexity. When people can perform macro-cognition effectively, they are better able to exploit change to achieve more while managing the side effects and tradeoffs produced by those changes.

Connecting design with cognition at work

209

When macro-cognitive functions are effective, what dimensions of adaptive capacity expand? The answer is three-fold; macro-cognitive capability expands the following: 1. The ability to be resilient when situations challenge the boundary conditions of normal practices or when surprising situations occur; macro-cognitive functions support how systems can be resilient when surprises occur and how to manage the unexpected (anomaly recognition, reframing). 2. The scope of potential control, by better coordinating activity over multiple people and groups in different roles; macro-cognitive functions support coordinated and not fragmented activity as multiple parties become more closely tied together (joint activity, replanning). 3. The scope and distances over which people can search for or create meaning and then project their intent from a distance; macro-cognition supports how people can project themselves into, and make sense of, remote settings relative to different purposes (sense making and replanning). Each of these three kinds of adaptive capability is the necessary complement to a vector of technology change. Inevitably, new technological powers are adapted to re-shape what we do, how we do it, and why we do it. Each form of technological power constitutes an infrastructure which demands a new superstructure (of resilience, coordination, and sense making, respectively) that works in parallel. The first dimension of adaptation – resilience – is a critical complementary partner to technological developments that tightly couple together different parts of a larger process or system for hyper-efficiency. A variety of technological powers are tapped in order to squeeze systems of human activity to be more and more optimal. In parallel, these systems become more brittle and can fail dramatically when surprising situations arise – situations which challenge the boundary conditions of textbook operation (Hollnagel et al., 2006). As systems of activity become larger but more tightly coupled, surprising conditions arise in part because a single anomaly will cascade and spread quickly through the system (e.g. modern supply chains in business). Rather than be trapped in workload bottlenecks, experts add resilience to systems by anticipating to avoid workload bottlenecks or by anticipating and preparing coping strategies so that the appropriate priorities are maintained, even if outside events should force workload bottlenecks onto the people responsible. The second dimension of adaptation – coordination – is a critical complementary partner to the technological development of ubiquitous connectivity. The technology of connection by itself does not create coordination, rather it represents underlying powerful infrastructure that is adapted in ways that re-shape how and when we interact with others in what we seek to accomplish. The infrastructure for connectivity, then, challenges our ability to design superstructures that coordinate non-co-located activities and asynchronous activities over greater ranges (Hinds and Kiesler, 2002). Good designs will trigger adaptations that support coordination; poor designs will lead to fragmentation and breakdowns at the boundaries of the different groups or roles (e.g. coordination surprises; Woods and Hollnagel, 2006). The third dimension of adaptation – meaning finding – is a critical complementary partner to the technological developments that combine virtual worlds, new sensor technologies and the ability to shift virtual point of observation with new ‘3-D’ technologies. These powers allow people to virtually stand in places that would be too difficult, expensive, dangerous or impossible to reach physically. How do we make sense of these remote situations? For example, the current video feeds from robots in search and rescue tasks

210

Product Experience

undermine all of the perceptual skills people naturally bring to bear when they are present at a scene (Murphy and Burke, 2005; Woods et al., 2004). The infrastructure of mobile remote sensing (and how to combine these different types of data about the world) demands a new superstructure for sense making as people can ‘forage’ more widely for relevant data and project themselves into distant situations (Pirolli and Card, 1999; Woods et al., 2002). Smith et al. (2007) provide a good example of the connections between infrastructure change and superstructure response that depends on designing affordances to support macro-cognitive functions. Under increasing economic pressure, air traffic control needed to change in order to provide greater flexibility in tailoring and modifying flight plans to match varying congestion points and the changing costs of routing aircraft to destinations. Technological changes made adopting new digital communication infrastructure readily obtainable, and this new capability appeared to allow for much greater flexibility in flight planning and replanning. However, the infrastructure change, while necessary, was insufficient alone without a parallel and complementary change in the superstructure for coordination across multiple groups, especially given changes in weather and failures that could reduce airport capacity. Changing the patterns of coordination over the multiple parts of the national air traffic system was needed so that the system could adapt gracefully and intelligently when disrupting events occurred. Designing the collaborative information and display systems that support distributed work across airline dispatch and strategic and regional air traffic control centers, as situations change, required new design to support macro-cognitive functions (Klein et al., 2005a): • Joint activity required tools for building common ground to help recognize anomalies. • Replanning required synchronized joint activity to consider all of the ramifications of a disrupting event or plan modification. • Shared sense making needed new forms of exchange of information and perspective to recognize when strategic adjustments were necessary.

6. INVENTING THE FUTURE OF COGNITION AT WORK Design changes cognition. By introducing artifacts that mediate information pick up, interaction, and goal-directed action, design influences both micro- and macro-cognition. This is a fundamental finding in cognition – often called the representation effect (Norman, 1993; Zhang and Norman, 1994; Woods, 1995b). Design can impair micro-cognition creating bottlenecks that make it easy to see people as highly limited processors. Adding complexities makes interaction clumsy, makes attention flow to the device itself, rather than the larger goal-directed activity. These impairments add significant costs that limit people and lead to workaround and gap-filling forms of adaptation. Finding bottlenecks and workarounds initiates repair of past design. Alternatively, design produces affordances – support for the difficult macro-cognitive functions of anomaly recognition, sense making, joint activity, and replanning. These advances release the growth of expertise as people learn and share ways to manage the complexities and tradeoffs that challenge all aspects of human activity in our uncertain, changing worlds. These twin effects of designs – clumsiness relative to micro-cognition and affordances relative to macro-cognition – will trigger people, acting in various roles to achieve multiple goals, to adapt. These adaptations may be workarounds to cope with complexities

Connecting design with cognition at work

211

and bottlenecks – gap-filling adaptations. The adaptations may be transformations that exploit new capabilities and that result in new demands and pressures on systems and on human roles in those systems – expansive adaptations. Ironically, these latter act as transformative adaptations (Winograd and Flores, 1986; Woods and Dekker, 2000) which feed back on cognition to create new demands and constraints on carrying out macrocognitive functions, and new costs for micro-cognitive bottlenecks. As has been noted before, in adaptive cycles, today’s design solutions will produce surprises that become tomorrow’s design challenges.

REFERENCES Alexander, C. (1977). A pattern language: buildings, towns, and cities. New York: Oxford University Press. Bovair, S., Kieras, D. E. and Polson, P. G. (1990). The acquisition and performance of text-editing skill: a cognitive complexity analysis. Human-Computer Interaction, 5(1), 1–48. Bruner, J. (1990). Acts of meaning. Cambridge, MA: Harvard University Press. Bruner, J. (2003). The narrative construction of reality. In: M. Mateas and P. Sengers (Eds.) Narrative intelligence. Philadelphia, PA: J. Benjamins. Byrne, M. D. and Bovair, S. (1997). A working memory model of a common procedural error. Cognitive Science, 21, 31–61. Carroll, J. M., Kellogg, W. A. and Rosson, M. B. (1991). The task-artifact cycle. In: J. M. Carroll (Ed.) Designing interaction: Psychology at the human-computer interface. Cambridge: Cambridge University Press. Carroll, J. M. (Ed.) (2003). HCI models, theories, and frameworks: Toward a multidisciplinary science. Boston: Morgan Kaufmann. Clark, H. and S. Brennan, S. (1991). Grounding in communication. In: L. Resnick, J. M. Levine and S. D. Teasley (Eds.) Perspectives on socially shared cognition, pp. 127–149.Washington, DC: American Psychological Association. Clausewitz, C. (1968, original 1832). On war, p. 141. London: Penguin. Cook, R. I., Potter, S., Woods, D. D. and McDonald, J. S. (1991). Evaluating the human engineering of microprocessor controlled operating room devices. Journal of Clinical Monitoring, 7, 217–226. Cook, R. I. and Woods, D. D. (1996). Adapting to new technology in the operating room. Human Factors, 38(4), 593–613. Cook, R. I., Render M. L. and Woods, D. D. (2000). Gaps in the continuity of care and progress on patient safety. British Medical Journal, 320, 791–794. Cook, R. I. (2006). Being bumpable. In: D. D.Woods, and E. Hollnagel. Joint cognitive systems: Patterns in cognitive systems engineering. Boca Raton, FL: Taylor and Francis. Cooper, A. (1999). The inmates are running the asylum: Why high tech products drive us crazy and how to restore the sanity. Indiana: Sams. Cordesman, A. H. and Wagner, A. R. (1996). The lessons of modern war, Vol. 4: The Gulf War. Boulder, CO: Westview Press. Czerwinski, M., Chrisman, S. and Schumacher, B. (1991). The effects of warnings and display similarities on interruptions in multitasking environments. SIGCHI Bulletin, 23(4), 38–39. Doherty, G. and Upton, C. (2005). Designing displays for mobile decision support. L. MacKinnon, O. Bertelsen and N. Bryan-Kinns (Eds.) Proceedings of 19th BCS Conference on Human Computer Interaction (HCI 2005), Vol. 2, pp. 83–88. Ericsson, K. A., Charness, N., Hoffman, R. R. and Feltovitch, P. J. (Eds.) (2006). The Cambridge handbook on expertise and expert performance. Cambridge: Cambridge University Press. Feltovich, P. J., Ford, K. M. and Hoffmann, R. R. (Eds.) (1997). Expertise in context: Human and machine. Cambridge, MA: MIT Press. Feltovich, P. J., Hoffman, R. R., Woods. D. D. and Roesler, A. (2004). Looking at cognitive engineers doing cognitive engineering: Implications of the reductive tendency for the design of complex sociotechnical systems. IEEE Intelligent Systems, May/June, 90–94. Flach, J. and Hoffman, R. R. (2003). The limitations of limitations. IEEE Intelligent Systems,18(1), 94–97. Gibson, J. J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin. Gigerenzer, G. (2000). Adaptive thinking. New York: Oxford University Press. Gilovich, T., Griffin, D. and Kahneman, D. (Eds.) (2002). Heuristics and biases: The psychology of intuitive judgment. New York: Cambridge University Press.

212

Product Experience

Gray, W. D. (Ed.) (2007). Integrated models of cognitive systems. New York: Oxford University Press. Hinds, P. and Kiesler, S. (Eds.) (2002). Distributed work. Cambridge, MA: MIT Press. Ho, C.-Y., Nikolic. M., Waters, M. and Sarter, N. B. (2004). Not now: Supporting interruption management by indicating the modality and urgency of pending tasks. Human Factors, 46(3), 399–409. Hoffman, R. R., Roesler, A. and Moon, B. M. (2004). What is design in the context of human-centered computing? IEEE Intelligent Systems, 19(4), 89–95. Hollnagel, E. and Woods, D. D. (2005). Joint cognitive systems: foundations of cognitive systems engineering. Boca Raton, FL: Taylor and Francis. Hollnagel, E., Woods, D. D. and Leveson, N. (Eds.) (2006). Resilience engineering: Concepts and precepts. Aldershot: Ashgate. Hutchins, E. (1995). Cognition in the wild. Cambridge, MA: MIT Press. Kahneman, D. and Tversky, A. (1996). On the reality of cognitive illusions: A reply to Gigerenzer’s critique. Psychological Review, 103, 582–591. Kiesler, S. and Hinds, P. (Eds.) (2004). Human-robot interaction: A special double issue of human–computer interaction. Hillsdale, NJ: Lawrence Erlbaum. Kieras, D. E. and Bovair, S. (1984). The role of a mental model in learning to operate a device. Cognitive Science, 8, 255–273. Kieras, D. E., Meyer, D. and Ballas, J. (2001). Towards demystification of direct manipulation: Cognitive modeling charts the gulf of execution. In: M. Beaudouin-Lafon, and R. J. K. Jacob (Eds.) Proceedings of the ACM CHI 2001 Human Factors in Computing Systems Conference, pp. 128–135. Seattle, Washington, USA. Klein, G. A. (1998). Sources of power: How people make decisions. Cambridge, MA: MIT Press. Klein, G. A., Ross, K. G., Moon, B. M., et al. (2003). Macrocognition. IEEE Intelligent Systems, 18(3), 81–85. Klein, G., Feltovich, P., Bradshaw, J. M. and Woods, D. D. (2005a). Common Ground and Coordination in Joint Activity. In: W. Rouse and K. Boff (Ed.) Organizational simulation, pp. 139–178. New York: Wiley. Klein, G., Pliske, R., Crandall, B. and Woods, D. (2005b). Problem detection. Cognition, Technology, and Work, 7(1), 14–28. Klein, G. A., Moon, B. and Hoffman, R. R. (2006a). Making sense of sensemaking 1: Alternative perspectives. IEEE Intelligent Systems, 21(4), 22–25. Klein, G. A., Moon, B. and Hoffman, R. R. (2006b). Making sense of sensemaking 2: A macrocognitive model. IEEE Intelligent Systems, 21(6), 22–26. Koopman, P. and Hoffman, R. R. (2003). Work-arounds, make-work, and Kludges. IEEE Intelligent Systems, 18(6), 70–75. Lee, D.-S., Woods, D. D. and Kidwell, D. (2006). Escaping the design traps of creeping featurism: Introducing a fitness management strategy. Usability Professionals’ Association Annual Conference, Broomfield, Colorado, June 12–16, 2006. Moore, C. and Dunham, P. (Eds.) (1995). Joint attention: Its origins and role in development. Hillsdale, NJ: Lawrence Erlbaum Associates. Monk, A. (2003). Common ground in electronically mediated communication. In: J. M. Carroll (Ed.) HCI models, theories, and frameworks: Toward a multidisciplinary science, pp. 265–290. Boston: Morgan Kaufmann. Murphy, R. R. and Burke, J. L. (2005). Up from the rubble: Lessons learned about HRI from search and rescue. In: Proceedings of the 49th Annual Meeting of the Human Factors and Ergonomics Society, Orlando, FL, Sep. 26–30, 2005. Human Factors and Ergonomics Society. Murray, C. and Cox, C. B. (1989). Apollo, The race to the moon. New York: Simon and Schuster. Neisser, U. (1976). Cognition and reality. San Francisco: W. H. Freeman. Norman, D. (1988). The psychology of everyday things. New York: Doubleday. Norman, D. A. (1993). Things that make us smart. Boston: Addison-Wesley. Patterson, E. S., Watts-Perotti, J. and Woods, D. D. (1999). Voice loops as coordination aids in space shuttle mission control. Computer Supported Cooperative Work, 8(4), 353–371. Patterson, E. S., Cook, R. I., Woods, D. D. and Render, M. L. (2004). Examining the complexity behind a medication error: Generic patterns in communication. IEEE SMC Part A,34(6), 749–756. Philips. (2005). Why simplicity? Available from: http://www.philips.com/about/brand/whysimplicity/ Pirolli, P. and Card, S. K. (1999). Information Foraging. Psychological Review 106(4), 643–675. Pirolli, P. (2007). Information foraging: Adaptive interaction with information. New York: Oxford University Press. Pogue, D. (2005). The cellphone that does everything imaginable, at least sort of. NY Times, May 12, 2005. Reed, E. S. (1988). James J. Gibson and the psychology of perception. New Haven, CT: Yale University Press.

Connecting design with cognition at work

213

Rust, R. T., Thompson, D. V. and Hamilton, R. W. (2006). Defeating feature fatigue. Harvard Business Review, February 2006. Schmorrow, D. D. (Ed.) (2005). Foundations of augmented cognition. Hillsdale, NJ: Lawrence Erlbaum. Sebanz, N., Bekkering, H. and Knoblich, G. (2006). Joint action: Bodies and minds moving together. Trends in Cognitive Sciences, 10, 70–76. Shattuck, L. G. and Woods, D. D. (2000). Communication of intent in military command and control systems. In: C. McCann and R. Pigeau (Eds.) The human in command: exploring the modern military experience, pp. 279–292. New York: Kluwer Academic/Plenum Publishers, Smith, P. J., Spencer, A. L. and Billings, C. E. (2007). Strategies for designing distributed systems: Case studies in the design of an air traffic management system. Cognition, Technology and Work, 9, 39–49. Suchman, L. (1987). Plans and situated action. New York: Cambridge University Press. Thomas, J. J. and Cook, K. A. (2005). Illuminating the path: The research and development agenda for visual analytics. New York: IEEE Computer Society. Wallace, R. (2006). Be smart, be simple. DMI Review Article, 17(2), Spring 2006. Watts-Perotti, J. and Woods, D. D. (1999). How experienced users avoid getting lost in large display networks. International Journal of Human-Computer Interaction, 11(4), 269–299. Weick, K. E. (1995). Sensemaking in organizations. Thousand Oaks, CA: Sage Publications. Winograd, T. and Flores, F. (1986). Understanding computers and cognition. Norwood, NJ: Ablex. Woods, D. D., Johannesen, L., Cook, R. I. and Sarter, N. (1994). Behind human error: Cognitive systems, computers and hindsight. Human Systems Integration Information and Analysis Center, WPAFB, Dayton OH. (available at http://www.hsiiac.org/hsi/products.do?action⫽detailandcode⫽HS-1994-2). Woods, D. D. (1995a). The alarm problem and directed attention in dynamic fault management. Ergonomics, 38(11), 2371–2393. Woods, D. D. (1995b). Towards a theoretical base for representation design in the computer medium: Ecological perception and aiding human cognition. In: J. Flach, P. Hancock, J. Caird, and K. Vicente (Eds.) An ecological approach to human machine systems: A global perspective. Mahwah, NJ: Lawrence Erlbaum. Woods, D. D. and Watts, J. C. (1997). How not to have to navigate through too many displays. In: M. G. Helander, T. K. Landauer and P. Prabhu (Eds.) Handbook of human-computer interaction (2nd Ed.) Amsterdam, The Netherlands: Elsevier Science. Woods, D. D. (1998). Designs are hypotheses about how artifacts shape cognition and collaboration. Ergonomics, 41, 168–173. Woods, D. D. and Dekker, S. W. A. (2000). Anticipating the effects of technological change: A new era of dynamics for human factors. Theoretical Issues in Ergonomic Science, 1(3), 272–282. Woods, D. D. and Shattuck, L. G. (2000). Distant supervision – local action given the potential for surprise. Cognition, Technology and Work, 2, 242–245. Woods, D. D. (2002). Steering the reverberations of technology change on fields of practice: Laws that govern cognitive work. Proceedings of the 24th Annual Meeting of the Cognitive Science Society. [Plenary Address], Atlanta, GA. Woods, D. D., Patterson, E. S. and Roth, E. M. (2002). Can we ever escape from data overload? A cognitive systems diagnosis. Cognition, Technology, and Work, 4(1), 22–36. Woods, D. D. and Hollnagel, E. (2006). Joint cognitive systems: Patterns in cognitive systems engineering. Boca Raton, FL: Taylor and Francis. Zhang, J. and Norman, D. A. (1994). Representations in distributed cognitive tasks. Cognitive Science, 18, 87–122. Zsambok, C. E. and Klein, G. (Eds.) (1997). Naturalistic decision making. Mahwah, NJ: Lawrence Erlbaum.

8

DESIGNING FOR EXPERTISE AXEL ROESLER University of Washington, Seattle, WA, USA

DAVID WOODS The Ohio State University, Columbus, OH, USA

1. INTRODUCTION Designers are often surprised when innovative new systems are introduced to the field. Suddenly, the new products or systems encounter people, and with them practices and operations that have been established before the design of a new product was envisioned. As the new designs are fielded, users are confronted with the new products and wonder how they afford new possibilities and how they can be integrated into current activities. How artifacts are put to use depends in large part on the perspectives, initiatives, and inquiries of these users. The people who will engage in the application of the new products are in many instances more than users; they are practitioners of trained skills. They are professionals who contribute their knowledge to the application of the new products and systems as they utilize these in order to accomplish their goals. Practitioners utilize products and systems with purposes in mind. Do the practitioners consider a product useful? Can they use it to advance their goals? Does it make sense to them in terms of how it influences or changes their activity? As designers, we have learned to study design challenges from the perspectives of the people who will put the new designs to use: How do practitioners know what to do with the new design? What are the intentions and judgments that guide operations? How do practitioners anticipate what will likely happen next as they engage in a particular operation with a new product? How can we – as designers – design artifacts and systems that support practitioners, so that they excel in what they are already good at? How can we integrate technologies by design to support the expertise of others or to help create new forms of expert activity? Understanding how practitioners make decisions and formulate plans for action is a prerequisite for designing systems that facilitate domain expertise in context. We will illustrate in this chapter that expertise is comprised of many factors, and relies on the training and experience of practitioners in their domain. Expertise represents a convergence of Product Experience Copyright © 2008 Elsevier Ltd.

215

216

Product Experience

knowledge, skills, and experience that results in competence – the ability to make appropriate decisions and assessments with regard to the situations at hand. Experts operate in a variety of domains. We find them in serious domains where they act in high-stakes functions as surgeons, pilots, judges, commanders, and high-level decision makers. Many of us are experts in everyday environments, where we converge special understanding and particular skills as technicians, designers, cooks, musicians, wine connoisseurs, and athletes (Hoffman, 1992). Can just anyone who is particularly knowledgeable in a specific domain of human activity be considered an expert by others who depend on the consequences of these decisions or actions? On a day-to-day basis, we use the label ‘expert’ both as an indicator for a high level of proficiency, training, and knowledge that is generally associated with extended practical experience, and as a social construct, where a group of stakeholders identify those who are more expert among them. As more radical innovations in the form of technology-intense systems, softwaredriven products, and digital services continue to transform our daily lives, being expert has acquired a new meaning. Those among us who manage to cope with novelty can learn and adapt to be able to catch up with the continuous series of iterations, updates and improvements in a continuously changing environment. In contrast we often find lab researchers and developers speaking of novice and expert users; in other words, acting as if one could divide the audience of new technological artifacts into those who are in the know, and those who do not know. This has always been a distortion, as expertise is tied to a context and depends on more than just cumulative experience. What is it that experts know and how is it relevant to design? Expertise is rooted in a particularly deep-level understanding of a domain of practice. This understanding utilizes explanations and reasoning strategies that are extracted from a conceptual model of the domain, constructed and continuously consulted, tested, and updated by the domain expert in the course of practicing in the domain by engaging in activities in order to achieve goals. The conceptual model takes the form of an abstract representation (Rappaport, 1997; Heiser and Tversky, 2005). Briefly, a conceptual model is composed of functional and physical relationships in the domain that were identified by the expert and that allow him or her to simulate what is likely to happen next (Klein and Crandall, 1995). The conceptual model assists the expert in finding explanations for observations. The expert has constructed this conceptual model in the course of exploring the domain along various storylines and getting feedback about the different events that follow in various situations. Conceptual models as mental representations serve as structures for reasoning strategies that utilize actual observations in the course of sense-making (Klein et al., 2006). Design influences the acquisition of these mental conceptual models by providing external representations that influence how people see the world and obtain feedback about the world. Design influences the use of these mental conceptual models by changing the mental workload (memory and attentional loads) associated with making sense of situations and planning how to act. These external representations mediate the user’s cognitive work to develop and use conceptual models to make sense of the world accessed through the external devices and displays. Research on expertise is concerned with issues that arise in the course of work with representations in real settings – where people trained to different levels of proficiency work at tasks to different levels of skill. We will, in the following, focus on factors that determine the knowledge and performance of experts – in roles as operators of artifacts and systems, and as stakeholders that face the complexities posed by novel technology. Expert operators and decision-makers form intentions – what needs to be accomplished,

Designing for expertise

217

how to transform these intentions into plans and actions, how to recognize disruptions to these plans, how to cope with complexities and adapt their activities to achieve goals. Note that we replace the common terminology of ‘user’ with the label practitioner. We use the latter term to emphasize that people are not passive recipients of products designed for them or passive rule followers. Au contraire – people actively make things into tools and adapt plans to achieve goals as conditions and environments change. Practitioners will modify unsatisfactory design, devise workarounds, or simply abandon artifacts that do not enable their purposes. Our objective as designers is to support the work of practitioners in context by utilizing innovative technologies that are useful, usable, and easily understood. Only if designers understand the role and nature of expertise can this integration be possible and successful (Ericsson et al., 2006).

2. PERSPECTIVES ON EXPERTISE The designer’s focus is on how technological novelty can affect people. By envisioning how new technologies could take shape in the form of artifacts, systems, and environments, designers mediate technological possibilities with what future practitioners identify as useful, useable, and understandable. As human-centered design, this perspective on design practice dates back to the formation of work studies in human factors engineering and industrial psychology after World War II (Dreyfus, 1955). The initial application of human-centered design techniques in aviation, in industrial manufacturing, and on the battlefield, illustrates that the focus of the practitionercentered view in design has originated in domains where things can go wrong, with serious consequences when designers have solved the wrong problems even when using the right techniques (Woods et al., 1994, Chapter 5). To avoid these design failures, designers have learned to study the prospective users of the proposed design: Who are these practitioners, what are their expectations, and what do they know? As we move into more advanced work domains that encompass considerably more complex technological environments, we deal with highly trained and knowledgeable practitioners who perform as experts in their domain. In order to design from their perspective, we need to understand what identifies their expertise. Plus, technology advances are making more everyday activities considerably complex environments that place increasing demands on people’s expertise – often expertise in working with poor interface designs. Common to all design process models is that they demand that designers understand the domain of human activity before they can be able to integrate useful design innovation into existing contexts (Bayazit, 2004). The question is: If the domain relies on the performance of expert practitioners, do the designers have to become experts in that domain before they can understand the needs of practitioners and thus create appropriate design support? On one hand it simply requires too much time to become a domain expert. But besides the time constraint, it is quite ambitious for a designer to presume that he or she could become an expert surgeon, a pilot, or an intelligence analyst at all. Such achievement could take long training periods, extended apprenticeships, and extensive experience with difficult situations. And expertise at design may have little in common with the kinds of knowledge or skills that experts possess in other domains of human activity. What does it take to become an expert? Is there a fast lane for designers to understand the expertise of practitioners so that they can apply human-centered design techniques that do not fall prey to oversimplifications about what experts in different domains do?

218

Product Experience

Being an expert, on the other hand, can hinder design work where it comes to rethinking existing products. The expert might not be able to see why a routine activity from his or her point of view may pose a challenge to practitioners who perform at different levels of proficiency. Experts may be biased by how things are done today when evaluating diverse alternative strategies and design concepts that would change how things would be done tomorrow. Doing design work in expert domains, designers face the Ethnographer’s Challenge: In order to make interesting observations, they have to be, in part, insiders in the setting they are observing, while remaining, in part, outside the domain in order to have insights about how practice works, how practice fails, and how it could work better given future change. Design observations in the field of practice, where designers watch experts doing cognitive work, relies on being prepared to be surprised in order to distinguish unexpected behaviors that reveal how expertise works and how these experts work (Woods and Hollnagel, 2006). We have encountered examples where user analyses in interaction design contexts mystify rather than clarify domain expertise. It is very difficult to characterize expertise of people in the cost- and time-pressured setting of usability testing environments. It is easy to oversimplify the role of expertise, especially when there are limits on access or high costs to contact with actual practitioners. For example, the need for rapid design input might lead to situations where one might ask software specialists to evaluate the usability of a piece of software designed by other software experts. Dangers like this often lead to the formation of interdisciplinary design teams. But simply juxtaposing specialists from different backgrounds does not make an expert design team. When working in such teams on products for experts, designers in interaction design, industrial design, and visual communication design often feel unprepared to assess complexity in technical domains. They may even feel intimidated by the excessive detail in specifications and the complex nature of underlying processes. Visual designers often state that good solutions need to be simple solutions. Systems engineers, on the other hand, know that only complexity can cope with complexity; that often, there cannot be a simple solution to a complex problem – the problem is complex because an appropriate response can take many forms. Confronting the open possibilities and the critical uncertainties, designers draw, build, and explore where in contrast engineers tend to simulate, analyze, and plot results. Both model, but in completely different ways. Design builds tangible models people can play off as they envision future possibilities; engineers model how processes work fundamentally in order to run simulations that would predict exactly how things would work under postulated different, that is, future, conditions. For the former, design is discovering new possibilities (and this requires generating a wide range of novel approaches and combinations); for the latter, design is applying a set of constraints on what is a solution, to narrow in on a solution provided directly once one has the results from the right series of analyses or simulations. Working only beside each other these two perspectives are both needed and are both incomplete. Simplicity cannot be designed without confronting complexity (e.g. there are limits to using techniques like out-of-the-box testing where the designer assumes the role of the outsider, the non-specialist, and the novice that walks up to an artifact and is assumed to operate it without prior instruction). Analysis alone cannot guide how people adapt new capabilities into real tools, and in this process transform roles, processes, decisions, difficulties and even what is expertise. One challenge that transcends traditional design and engineering perspectives is that new technology changes what it means to be a practitioner. The future practitioner will not be the same in terms of knowledge, skill, expertise, and collaborative interplay as technology changes (Mark et al., 2005). To understand and guide the process of how

Designing for expertise

219

new technology and new designs change what people need to learn and do to be experts, various other experts need to collaborate and transcend their individual areas of knowledge. This collaboration over different kinds of experts is targeted at enhancing the expertise in action in particular domains of human activity. But the targets of the design have a role and a stake in terms of how new products affect their abilities, their roles, and their goals. Participatory design is more than generating collaboration across different specialists in design. The practitioner also participates in design. Practitioners are the experts at applying information to make decisions for actions that will affect states and behaviors in their environment – they are the experts in the use of artifacts. Designers, on the other hand, are experts in the design decisions that steer the development of products to support the decision making of practitioners (Woods, 1995). Ultimately, in participatory design processes practitioners, technologists, and innovators co-mingle, each with critical but incomplete roles as designer (Roesler, Woods and Feil, 2005). The questions that surround the nature and acquisition of the expertise are central to understanding design, both the mixture of experts who contribute to product designs and the expertise in context as people use new devices to do things in the world. The following sections are explorations into the nature of expertise: Observations of expertise in action, how we discover or elicit the knowledge organizations behind expert performance, and how the design of new products affects the ability to acquire and demonstrate expertise.

2.1. A history of the study of expertise Practitioners base their reasoning strategies on creating and interpreting artifacts as a form of external knowledge representation. Designers support practitioners in choosing, structuring, and using these representations rather than relying totally on basic individual mental capabilities without any external tools. Depending on their expertise in a task, area, or field, practitioners will engage in activities differently from what the designers expect them to do, and may or may not arrive at anticipated outcomes used to justify the investment in the development project. This makes expertise an important issue to consider in design, where patterns of interaction between practitioners, artifacts, and the world vary with different degrees and kinds of expertise. The study of expertise has a long history that reaches back to performance evaluations conducted by experimental psychologists in World War I-era studies of skilled machinists (Hoffman, 1992). The methods applied in these observations date back into the 1890s and focus basically on the timing of observable actions in the course of given tasks. Binet’s work at the beginning of the twentieth century set virtually all of the standards that would be applied by post-1950s studies of expertise. Typically the researcher would identify a task – for example, the selection, lifting, and alignment of a part in a machining task – and identify a typical begin and end state of the task. The researcher then would time a worker engaging in the process necessary to complete the task from its beginning to end. The hypothesis was typically that the more experienced worker could accomplish the task faster or engage in more tasks in parallel, given time pressure. In favoring the duration and number of tasks as central measures, these studies fall short in capturing, discussing, and exploring the nature of the individual differences among subjects, the situation that has preceded the test observation, and the context (such as motivations and social structure). These broader factors were initially considered distortions, and by transporting the work task into a controlled laboratory, the researcher could exclude (control) the

220

Product Experience

unwanted contextual circumstances in order to obtain objective results. It soon became apparent that time measures of performance provided a quite limited view into the nature of expertise. Measuring performance times in isolated sub-tasks soon evolved into observing related activities, and in the course of this shift in focus, it emerged that many aspects of expert decision-making reside in the context, social structure, and demands posed by different situations. In addition, it was evident that individual differences and training played a central role in the trade of experts. Cognitive psychologists began to study cognition and reasoning in field settings. One of the first subjects studied was the expertise of chess masters (De Groot, 1946). The reason why chess was a good subject for the study of expertise was that metrics for the determination of chess proficiency were already in place (Ericsson and Smith, 1991). Furthermore, the chess board and configuration of chess pieces serve as both physical representation of states on the board and abstract representation in the chess player’s conceptual model of possible – and best – next moves that would affect the constellations of chess pieces on the board. Simon and Chase’s classic study on expert chess players (1973) found that chess masters can quickly detect meaningful patterns or configurations of chess pieces and can remember a very large number of configurations if they form meaningful patterns relative to different situations that confront the experienced chess player. To demonstrate some of De Groot’s observations about the capabilities of experts, Simon and Chase tested players’ ability to recognize and memorize chess positions. They presented players either with game-meaningful configurations of pieces or with random configurations of pieces. They found that expert players were capable of memorizing, and hence planning ahead, a greater number of proper configurations than chess novices. When presented with random chess piece configurations, however, the chess experts did not perform significantly better than the novices. According to Simon and Chase, chess masters differ from players at lower degrees of proficiency in that they can recall a much higher number of game-meaningful configurations. This allows them to plan further ahead, given that a significant constellation forms the starting point. Compared to other chess players, the experts can assess the consequences of many different options for moves from a current constellation in parallel. Simon and Chase have shown that chess masters apply pattern recognition strategies in their assessment of what to do next. They can do this not because of superior memory for elements (each piece), but because they have a different knowledge organization based on relationships that connect pieces into configurations, and connect configurations into tactical and strategic sequences of action and counter-action. The paradigm shift from studying behavior to studying cognition in American psychology was stimulated and grew along with the introduction and growth of computerized systems into more and more realms of human activity. The development of computer applications quickly grew through attempts to reproduce human decision-making or to create computers that could make decisions on their own (called expert systems). Optimistically, these researchers thought they would be able to replace human cognition with computerized expert systems. While this agenda has repeatedly failed (and been repeated and failed several more times), the goal of autonomous computers required intense study of human expertise. Deployment of these expert computer systems quickly led to situations where the computerized expert and the human expert were both present in the actual decision situation. The developers expected that the computer expert system would improve the decisions of the human practitioner – hence they would be decision support systems. However, providing decision support and building team work between human and machine agents turned out to be a much more sophisticated topic, and the early computer expert systems turned out to be very poor team players. It was found that sophisticated decision support

Designing for expertise

221

systems often had ended up being underutilized or unused by experts during their firsthand reasoning (Woods, and Roth Bennett, 1990; Shanteau, 1992; Hollnagel and Woods, 2005; Woods and Hollnagel, 2006). The reasons for this are many – computer expert systems are inherently brittle, non-cooperative, opaque, disembodied, and designed based on erroneous models of expert knowledge (Hoffman, Feltovitch, and Ford, 1997). The original attempt to understand expertise in production measures of isolated subtasks had failed; the much later attempt to put expertise into a computer system and use that system to replace people as an isolated component also failed. A third line of work focused on studying the reasoning strategies behind expert decisions, usually with a goal of developing new training systems to speed up the acquisition of expertise as people moved from novice to journeyman to expert. Simon and Chase (1973) have shown that valuable insight on expert decision making can be collected by techniques such as think-aloud verbal protocols. Hoffman (1987) reminds us that using this and other knowledge elicitation techniques to study expert reasoning is time consuming and leaves out important factors. Variations on think-aloud protocols have been developed, such as critical incident analysis and cognitive task analysis. Today there is a wide and rich range of techniques that can be used to gain insight about the basis of expertise in particular domains of human activity (Christoffersen et al., 2007). By using a broad array of techniques researchers were able to see how social constructs and organizational factors affected expertise (Wright and Bolger, 1992; Hoffman, et al., 1997; Klein, 1997). One of the new lines of inquiry on expertise is called naturalistic decision making (NDM). NDM is the way people use their experience to make decisions in field settings, or, in a more comprehensive description, the study of NDM asks ‘how experienced people, working as individuals or groups in dynamic, uncertain, and often fast-paced environments, identify and assess their situation, make decisions and take actions whose consequences are meaningful to them and to the larger organization in which they operate’ (Zsambok and Klein, 1997; Klein, 1998). Naturalistic decision making research underlines the role of context, social organization, and domain-specific knowledge in the decision making of experts. NDM directs decision making in high-stake domains such as fire fighting, military command and control, mission control, aviation, and planning. One of the key results is that expertise uses external artifacts to support the processes that contribute to expert performance – expertise is not all in the head, rather it is distributed over a person and the artifacts they use, and over the other agents they interact with as they carry out activities, avoid failure, cope with complexity, and adapt to disruptions and change. For an illustration on how found characteristics in the environment become cognitive artifacts by being associated with cognitive tasks, see Edwin Hutchins’ observations of Micronesian and Western navigation (Hutchins, 1995a). Another key finding is the role of context (Woods et al., 2002). Hoffman, Feltovitch, and Ford (1997) describe context as composed of: • The analyst’s purposes and goals. • The analyst’s views, assumptions, and theories. • The analyst’s methods. What data people regard as meaningful depends on these kind of contextual factors. Experts are very sensitive to context shifts and how these shifts change what knowledge and what mental conceptual models are relevant to the task at hand (Feltovich et al., 1997). Context cannot be neglected in the study of decision making, expertise, and innovation. This has led to the macro-cognitive view on cognitive work (Klein et al., 2003), and with it an extended focus in the study of expertise – one that incorporates context,

222

Product Experience

the nature of the tasks, and the social and organizational factors that identify experts. Macro-cognition addresses the study of decision making and expertise in context, and looks at activities that are unfolding in organizations, situated in the environment of real tasks (Klein et al., 2003; and see Woods and Roesler, in Chapter 7 of this volume). Expertise is situated in real work settings where practitioners have experience with variations on routine performance and can access a rich set of cues that help them anticipate what could happen next. Expert reasoning is more than a specialized algorithm which could be captured in a computer system; expertise adapts reasoning strategies, plans, and actions in the face of changing conditions in the environment in order to achieve goals (Woods and Hollnagel, 2006). Overall, expertise does not reside simply in a single individual, but is situated in the relationships between practitioner, task, environment, and the organizational characteristics that direct and constrain practice in the domain.

2.2. What is expertise? A first assessment of expertise considers a person’s levels of proficiency at a task. Research then asks the question how do people move from one level of proficiency to another. A basic answer is that more knowledge about the task and associated context supports greater expertise. But this view of expertise as more extensive and better organized knowledge does not take into account the role of hands-on practice. Donald Schön’s (1983) work on the intersections of knowledge and practice extends the expertise-as-knowledge position. He focuses on the role of reflection-in-action that occurs when knowledge is put into practice. In his view, expertise is acquired by combining exposure to task-relevant reasoning with task-specific knowledge. Proficiency in domains such as law, medicine, and design is the integration and mutual refinement of knowledge, skill, and experience that can only occur through doing. The traditional assumption in psychology is that expertise is a phenomenon that can be accounted for in terms of the quantity, quality, and organization of the domainspecific knowledge of experts. In this view, ‘expert knowledge is based on deep reflection, allowing experts to do well because they know more, know better, and know in a more useable way’ (Hoffman et al., 1997). Hoffman et al. continue to question this position by reminding us that knowledge and meaning can be seen as extra-personal, and is located in the community rather than just inside a person. This leads us back to context, to addressing the differences between different domains, where knowledge is constructed and maintained in different communities. Expertise, from this angle, can be read as a repository of strategies for applying knowledge to context by taking into account the nature of the task and the social aspects surrounding the task. This view leads us to three characteristics of expertise: 1. First, expertise is domain-specific. 2. Secondly, experts adapt to changes in their environment. 3. Thirdly, experts rarely act as isolated individuals. Summarizing various studies of expertise in the agricultural, medical, weather forecasting, and law sectors, Shanteau (1992) identifies five psychological strategies that help experts in making decisions: 1. Expert judgment shows a willingness to make continuous adjustments in initial decisions. 2. Experts get help from others in order to make better decisions when coping with uncertainty. They can identify the experts in sub-domains.

223

Designing for expertise

3. Experts make use of both formal and informal external decision aids. 4. Even though experts may make small errors in judgment, they avoid making large mistakes. ‘The focus is not on being exactly right, but on avoiding making bad decisions’. To do this, experts first conduct a broad estimate of a problem situation and then conduct a more careful analysis of components in the situation that were identified as more critical in the first pass. 5. Experts decompose complex situations into manageable chunks, and can then re-construct the larger situation from its individual aspects. These five strategies indicate that experts have access to a quite refined conceptual model of their domain of expertise to identify functional relationships, dependencies, and likely outcomes. This model of the domain includes scenarios of what not to do (Minsky, 1997). The expert’s conceptual model also includes an understanding of relationships of their domain with sub-domains that are part of the challenges ahead. Experts know who the experts are in these sub-domains, and when to consult them. Expertise forms a decision-making capacity in the neighborhood of other forces. Hoffman, Feltovitch and Ford illustrate that expertise entails context as a set of other factors that surround and direct expertise with their TEMPEST model (Figure 8.1). The tempest model has evolved from previous models as new research results have accumulated. The original base is James Jenkins’ (1979) tetrahedron model that captured results from learning research. Honeck and Temple (1992) adapted a version to incorporate new results on the cognitive underpinnings of expertise. Building on these models, the TEMPEST model maps expertise as the tetrahedral set of relationships between the expert’s strategies, goals, background experience, and materials used as decision support.

the ‘wind’

materials

the ‘kite’

driving forces

data, tools used in the familiar tasks

the expert

societal needs and expectations, personal motives

cognition, perception

strategies used in familiar tasks

goals of the familiar tasks

background experience knowledge and skills

the ‘tail’

stabilizing forces selection criteria, training methods, professional standards

the ‘line’

controlling forces lawfulness in the world, practical constraints, accountability mechanisms and performance expectations

FIGURE 8.1 TEMPEST model of expertise in context (modified from Hoffman, Feltovitch and Ford, 1997).

224

Product Experience

As a diagram, the two-dimensional tetrahedron resembles a kite, and represents the expert. In taking the kite analogy further, Hoffman et al. illustrate the context of expertise in showing that it takes more to fly a kite than the kite alone. • The wind carries the kite – in the TEMPEST model, the wind resembles the driving forces on expertise, rooted in societal needs and expectations and personal motives. • The kite is steered by the line – controlling forces on expertise that are institutionalized by lawfulness in the world, practical constraints, accountability mechanisms and performance expectations. • The kite is stabilized in the wind by its tail – the stabilizing forces on expertise are selection criteria, training methods, and professional standards for experts. • Tail and line make sure the kite stays stable in the wind and can be controlled. • Together with the wind as the driving force, they form the context of expertise, and if we look at the labels of the contextual forces, we encounter many of the sociological parameters that identify expertise. Note how it takes the interactions between all factors in the diagram to constitute, identify, and steer expertise, and how the four subsystems (kite, wind, line, and tail) can partially compensate for deficiencies in other aspects. If the kite, for example becomes unstable due to a tail that is too short, more reactive steering with the line is required to keep the kite in the air. The TEMPEST model illustrates expertise as a macro-cognitive system of interrelated factors of organizational structure, environmental conditions, decision making, knowledge, and planning ahead. Characteristic of this macro-cognitive representation of expertise is that experts balance multiple factors in parallel, and that all factors represent diversity and contrasting forces that mutually reinforce another. Growing expertise to act effectively requires orchestrating the interplay over this set of factors and driving forces.

2.3. Who are the experts? Expertise is both a social construct and the context-specific application and adaptation of knowledge. Studies on the quality of expert decisions have shown that experts do not generally make better decisions than do non-experts (Shanteau, 1992). This contrasts with the general perception that experts can be identified by their outstanding performance, and hints at a social dimension behind the identification of experts. In virtually every organization, certain people are known as the best ones to consult for specific types of problems. Though experts do not always make the best decisions in a given situation, they have been shown to avoid making decisions that they expect to have particularly negative consequences (Shanteau, 1992). Expert decision-making seems to be driven by appropriateness, not necessarily by correctness. Non-experts in similar situations are prone to making wrong decisions because, often, they cannot assess the repercussions of alternative decisions. Experts will likely avoid making wrong decisions, but will opt for a solution with the least trade-off if they do not see the possibility for an optimal solution. Since experts have a conceptual model of their domain, they know what not to do (how to avoid errors of over-control of complex dynamic processes). Societal institutions play a role in the accreditation of experts. While some experts have been identified by peers for their remarkable performance in a number of situations, other specialists are accredited through licensing systems such as university degrees and academic titles. Certificates document that an expert has access to an agreed-upon body

Designing for expertise

225

of knowledge and that he or she has demonstrated performance in a number of practice situations, as well as passed examinations of expected performance. Knowing who the experts are in a domain can be a challenge. While domain practitioners seem to have no problem in identifying the experts among themselves, outsiders have difficulties identifying who the experts are in a particular domain (Sternberg, 1997). Utilizing insights from the study of expertise, let us refer to a list of nine characteristics that identify an expert (Shanteau, 1992). Experts have access to a vast body of up-to-date content knowledge; as a basis for judgment, this deep level understanding of their domain is characterized by: • Highly developed perceptual/attention abilities: Experts can extract information in their domain that non-experts miss (Fiore and Hoffman, 2007). • Case-based reasoning in context: Experts have a sense of what is relevant when they are making decisions. Where novices often get sidetracked by irrelevant data, experts base their decisions on less information, but ensure that their judgment rests on relevant information. • The ability to simplify complex problems: Experts have developed strategies to decompose complicated situations in their domain into smaller chunks. While novices attempt decomposition, they often fall into making oversimplifications. This is not true for experts, because they know that their domain is composed of several sub-domains. The ability to decompose complicated context is guided by a conceptual model of the domain that the expert has constructed (Feltovich et al., 1997). • The ability to communicate expertise: Experts have no problem communicating information in the areas where they are experts. They assume that their assessments should be preferred over those of others. This outspoken expertise goes hand-in-hand with the social construction of expertise; that confidence in claiming oneself as an expert can accredit a person as expert. Let us keep in mind that this statement is put to the test until our expert unites all nine characteristics in this list. • The ability to handle diversity more successfully than non-experts: Experts continue to make effective decisions even when things are not going well. This characteristic is again supported by the expert’s comprehensive conceptual model of his or her domain. This model does not only include best case scenarios, but also includes forks in the road where things can go wrong, together with scenarios of how not-so-optimal sequences will play out and how to recover from them. • The ability to identify and adapt to exceptions: Experts are not likely to walk down ‘garden paths’. They can identify when an expected situation deviates from the expectation. • Knowing how and when to adapt their decision strategies to changing task conditions: Experts detect when changes in the context of the task demand adapted reasoning strategies. This is possible because experts have access to many alternative response strategies and understand that different situations cannot all be approached with the same strategy. • Strong self-confidence: Experts trust their decisions. • A strong sense of responsibility: Experts can assess how situations will likely play out. They understand their role as expert decision-makers and are aware of the consequences of a wrong decision. In any case, experts know that they must make decisions because they can control the situation at hand. Naturally, these characteristics of expertise make experts difficult people to work with. They are confident in their expertise, and they can assess the expertise of others in their domain. This makes expertise an interesting topic from the training and learning

226

Product Experience

perspective, where domain experts help novices in eliciting domain knowledge to eventually become experts themselves.

2.4. How expertise is acquired? Dreyfus and Dreyfus (1986) have identified five stages in the acquisition of expertise, where a novice starts at a beginner level, moves continually through levels of competence and proficiency, and finally arrives at the expert level. The five stages illustrate increasing levels of proficiency in performance, as well as the role of context-free rules, the realization of patterns in context, and the hierarchical organization of task-relevant clues for decision making or intuitive response. Stage 1: Novice – The beginner engages in an instruction sequence where he or she learns context-free features of the domain that can be recognized without experience in the domain. Beginners in chess, for example, learn a numeric value for each type of piece regardless of its playing value. They also learn a rule that is associated with the piece, as well as the positioning constraints on the piece. Beginners are slow performers during this stage because they need to remember and recall the rules that guide their actions. Stage 2: Advanced beginner – At this stage, the beginner is pointed to specific features that provide situational aspects of the activity. He or she begins to see patterns in the context of the task that provide additional information in support of the rules. Stage 3: Competent – At the competent level, the student continues to identify more task-relevant features in the context of the activity. This eventually results in a large number of patterns that he or she recognizes, culminating in the sheer overload of potentially task-relevant data. To cope with complexity, the student begins to structure patterns and strategies by installing hierarchical reasoning sequences. The competent performer can address a broad spectrum of situations. Performance is slow and depends on the student’s seeking new rules and reasoning strategies in order to devise a plan or take a perspective. Stage 4: Proficient – In the proficient stage, intuitive engagement in the activities continually replaces reasoned responses. Instead of being rule-based, the reasoning strategies of the performer are driven by adapting decision support structures to the current context of a situation. The performer simply knows what to do instead of deciding in the course of a calculative response. Stage 5: Expert – While the proficient performer sees what needs to be done and decides to do it, the expert performer not only knows what needs to be done, but also knows how to achieve the desired state. This allows the immediate response to a clearly identified situation. This immediate and appropriate response is characteristic of expertise. Dreyfus and Dreyfus’ five stages in acquiring expertise focus on the student’s journey. If the student were required to rely solely on his or her individual experience, the extent of lessons learned would depend on the variety of failures that confronted the student, so that limitations would have to be overcome by practice. Ericsson and Charness (1997) point out the importance of deliberate practice in the acquisition of expertise. In deliberative practice, students work with a teacher or coach whose job is to identify individual weaknesses and to design training programs that repeatedly confront students with the weak aspects of their performance. The continuous confrontation engages the student in developing response strategies for overcoming individual weakness. In their study of the training and development of expertise in outstanding musical performers and athletes, Ericsson and Charness identify four developmental stages, from the novice stage to that of eminent expert. Particularly interesting in their observations

Designing for expertise

227

is that they look at an extended temporal frame of expert development, encompassing the three to five decades spanning acquisition of knowledge, skill, and endurance, in the development from the beginner stage towards the highest level of expertise. The extremely long training period required to achieve the mastery of musical performance or world class competition level in a sport is necessary to provide for the mutual adaptation between the performer’s or athlete’s physical capabilities, and the advancement of expertise and endurance during practice. In the course of their classic study on chess, Simon and Chase (1973) have identified that it typically takes ten years of intensive deliberative practice to arrive at an expert level of performance in chess. Studies in other domains (Ericsson, Krampe and TeschRömer, 1993) have confirmed this estimate. Ericsson and Charness’ four phases in the course of becoming an eminent expert pass through the following proficiency levels: 1. In Phase one, musical performers and athletes, at a young age, engage in playful activities in the domain. This phase ends with the identification of potential in the performer, indicating that he or she can move the acquired skills onto the next level. 2. Phase two is an extended preparation. It ends with the commitment of engaging in practice full-time. Usually, this phase entails the consultation of specialized trainers or coaches, and often even requires moving into another geographic region where the required training with an advanced teacher or coach is available. 3. In Phase three, the performer makes a commitment to full-time engagement in the training and practice of the skill. This phase ends either with the performer being able to make a living as a professional, or with the complete termination of the full-time activity (Bloom, 1985). At this stage, the performer is practicing or training with an expert teacher or coach. During the first three stages, students integrate the knowledge and skill that master teachers and coaches can convey (Ericsson and Charness, 1997). 4. To achieve the highest level of expertise – the fourth phase – performers must attain a level of performance where they make an original contribution to the domain of their expertise. In this phase, they reach a level of proficiency that goes beyond that of their teachers. At the eminent level, where the highest level of expertise is innovation, empirical studies of these original contributions have been sparse. This is attributed to the fact that the moment at which the contribution will occur is hard to predict. Ideas, conditions, and decisions that have led to major breakthroughs in thinking and practice can be traced by retrospective process tracing methods (Gruber, 1981; Wallace and Gruber, 1989).

3. INNOVATION AND THE EMINENT LEVEL OF EXPERTISE Ericsson and Charness’ (1997) observation that innovation is an indicator of an eminent state of expertise draws parallels to the difficulties that grip the concept of innovation in design circles. As practitioners, designers tend to know what innovation is, yet a comprehensive description of innovation in design is still largely missing. Like important discoveries by scientists and creative leaps by artists and performers, innovation in design can often only be studied in retrospect, since it is difficult to experimentally set up the conditions that would allow an innovative breakthrough to be observed as it unfolds. Such staging techniques would be more than desirable to designers. Perhaps design methods could be considered as a direction in the quest to demystify innovation; a methodic design approach enables designers to observe, provide, and steer environments, and design conditions so that they provide fertile grounds for innovations to ‘happen’.

228

Product Experience

The design process is the structuring of resource pools that enables designers to envision promising design responses for future iterations of a product or system, based on observations of the performance of current design solutions for that product or system in the field. These observations eventually result in a domain understanding for the designer. In its sequence of formation, this development of insight shows many parallels to the sequence of acquiring expert understanding. Although the sequence of learning from observations plays out in a much shorter timeframe and is conducted from an emic perspective of the designer (from a view inside the culture of design), the insight won by designers makes it possible for them to create novel interventions relative to the target domain of the expert practitioner. As a consequence of the shorter time frame spent in the expert domain, the insight won by the designers is much less comprehensive than the deep level domain understanding acquired by an expert over a span of many years. One possible explanation of these limitations is that the designers are limited to observations and often cannot practice the observed operations of use, since they are not sufficiently prepared. Knowing this restriction, the challenge for designers is in knowing how observations in the field can be linked to the design of concepts that respond to identified problems or needs in the domain of practice. In the design process, the acquisition of expertise can be viewed as an iterative cycle of feedback, concept generation, and selection. In the cycle, observations are intertwined with modeling of a conceptual representation of the field of practice under observation; these conceptual representations are providing the grounds for creating products that can be fielded, and then in turn can be observed. Figures 8.2A and B illustrate a cyclical framework that can be used to chart this design process, the de:cycle model (Roesler and Woods, 2005). Design activity, to be able to respond to the challenges provided by the call for novelty balanced with pressures on realizablity, coordinates expertise from three perspectives – observe, explore, create. These provide linkage points between three broad roles that draw on pools of experience – pools of research patterns, technological possibilities, and experience in the field. Each of these roles Provides their knowledge, processes, and artifacts with respect to the future under design: Practitioner – how they adapt to complexity; innovator – how they envision what would be useful; and technologist – how they bring the anticipated change into the world of practice. The three roles intersect, and design activity within them is conducted in parallel – which provides for numerous interactions across the center of the de:cycle. In a typical design project, design activity begins with observations in the field of practice where practitioners identify limitations of previously fielded objects. In the de:cycle graphic, this point of departure is located at the 4 o’clock position: Design intent is represented in clockwise movements; design effects are projected in counterclockwise rotation. Moving counterclockwise for exposition purposes, in the observation arc (northeast section), design researchers plan studies, collect data records from protocols, carry out ognitive task analyses, and undertake other process tracing techniques. Functional syntheses based on the data and observations lead to abstract models that capture patterns, provide explanations, and lead to proposed alternative directions. For example, researchers may design observations to reveal how practitioners have adapted products to work for them in ways different from the designer’s model (design intent is represented in clockwise movements). Different explanations and characterizations of the basis for these adaptations then become seeds for exploring possible new design directions. In the exploration arc (northwest section), there is movement from the models, patterns, and data of research to design seeds that capture promising concepts hypothesized

229

Designing for expertise

innovator: pools of research patterns

e

ze

e l)

at e r and © 2004 by Axel Roesle

technologist: pools of technological (A) possibilities

i ll r ,a ds oo id W Dav

im p lem

en

t)

bj ec

cr e

gh ts res erv ed.

o b se r v e

de

e lo r

od

xp

a li

(m

n (h y p o t h e s sig is )

co n ce p t u

o t(

practitioner: pools of experience in the field

abstract model explanations abstracted patterns leverage points

developing concepts

developing scenarios

affordances representation techniques

c on c e p t u

stories of use

protocols data records

listening to stories suggestions for improvement

ea

te

,

final design model

im p le m

t en

ec

cr

t)

unsatisfactory product

bj

is ) (h y p o t h e s s ig n

de

model of the ideal design

(

o b s e r ve

building prototypes

e

e l)

animock

lo r

od

hypotheses of what is useful

p ex

m

design seeds

observations of adaptations

aliz e

(o

established field product in field revision 2 in field revision 1

design documentation

detailing for production

(B)

adjusted final release

reproduction model tooling test realese plans and documentation

deployment into the field

FIGURE 8.2 (A) The de:cycle – The three roles in design and their respective interests and expertise. Counterclockwise moves represent design synthesis, as objects are created. Clockwise moves represent the analysis of design that identifies the origins, basis, and expectations associated with design decisions. (B) Handover marks and design process products make design activity tangible and allow designers to plan for deadlines and coordinate with other stakeholders.

to lead to useful change in the target field of practice. We refer to this arc in the de:cycle as the ‘Northwest Passage’ as the processes of innovation and ideation are not supported systematically, but rather have tended to be supported or marked by design artifacts personal to specific designers or teams (at least as compared to the more formal artifacts produced by design in the realization process, e.g. those that mark software development). In the Northwest Passage, ideation mediates between research insights about the situation under design (rotating clockwise) and technological capacity for change (rotating counter-clockwise) as related to judgments about desired improvements from the point of view of practitioners (cutting across the center of the cycle from left to right).

230

Product Experience

The techniques of those proficient in innovation generate insights to generate promising concepts that can be turned into prototype demonstrations that support commitment of resources for realizing these concepts in fieldable forms. In the implementation arc (southern section), technologists choose from a pool of available technologies those that are appropriate for the realization of selected design concepts. The appropriateness of applied technologies is evaluated against factors of functionality, reliability, feasibility, safety, and economic feasibility – and how necessary modifications will reflect on the original design program. Figure 8.2B shows how handovers across the roles serve as demarcations and how materials produced in the process of design, such as scenarios, sketches, mock-ups, and prototypes, make these processes tangible, helping designers reflect on their progress and share ideas with other stakeholders in the design process. The processes of design, focusing on the need to create innovations, can be seen as a type of expertise that demonstrates the same regularities as have been summarized in the preceding sections of the chapter. As designers become acquainted with a situation under design they incrementally attain a deeper level understanding of the design domain. A deepening understanding facilitates the quest for innovation as in the definition of the eminent level of expertise. Figure 8.3 shows the cycle of expertise acquisition and domain change, overlaid on the de:cycle framework. In design, one moves from novice, relative to the target field of practice, to greater expertise, relative to how that domain could function. The design process meets the criteria for eminent expertise with the generation of innovative concepts, though these will turn out to have surprising effects when fielded which will release a new round of adaptation through use. In the observation arc, designers who had entered the design situation as novices have moved through stages of advanced beginner, competence, and proficiency. Starting at the stage of proficiency, they have begun building a conceptual representation around the acquired knowledge of the domain as research findings. While the evolving

proficient extracting What’s interesting (reasoning backward)

identifying what’s promising (reasoning forward) expert

el)

directing observations

sle

)

Wo ods, all ri ghts reser

e Ro el Ax by 006 ©2

im plem

competent

ac t

te

ra nd

Da vid

applied

cr ea

ved.

observe

conceptual c

ice

e

ali ze

od

xp

e lor

(m

testing hypotheses

(hypot he sis sign ) de

concept u

pr t( en

eminent expert detailling responses novice

implementing change advanced beginner

FIGURE 8.3 The cyclical model of the acquisition of expertise is overlaid on the de:cycle, a cyclical model of the design process. The overlay illustrates how innovation as a result of the build up to eminent stage of expertise produces change that leads to effects that require the growth of new expertise.

Designing for expertise

231

conceptual model provides the proficient designer with explanations for what has been observed, it allows the capacity to make predictions by simulating the outcomes of proposed changes that would be part of new products. Because the conceptual model is a representation of the relationship, dependencies, and causal sequences identified during observations in the field, it allows experts to reason forward by manipulating variables in the model. Moving counter-clockwise is a process of projecting ahead toward future states. These predictions form hypotheses about how investments in additional design work will produce change in the target field of human activity. These hypotheses are tentative and open to revision as evidence comes in about the likely impact of the proposed changes. One difficulty is that getting this evidence usually requires movement into activities that make design ideas tangible in the form of sketches, mock-ups, and refined prototypes. These types of design representations allow designers to share their observations with the domain practitioners and design clients. Taking this as a starting point, let us return to the nature of expertise in a more general frame of reference. Practice is characterized by an experimental component, where the expert follows through with promising actions based on decisions that were supported by simulations performed in the framework of relationships of the conceptual model. Activities are subsequently evaluated in the context of the real situation before being adapted to unanticipated conditions and modified in a later iteration. Any expert is designing response strategies based on a hard-won conceptual understanding of the situation at hand. The eminent expert knows that, in order to identify the boundaries of their understanding, they have to manipulate the situation at hand. This alteration of existing conditions is done responsibly, as he or she can identify manipulations that would lead to unmanageable outcomes. The eminent expert also knows the scope and possible results of unmanageable states, and will avoid losing control of the situation. The stage of eminent expertise is characterized by innovation that had originated in the course of many previous observations of such experiments. These previous experiments do not need to be repeated for the current decision task. The past outcome of the original experiment or practice is sufficient for making the current decision. The eminent expert recycles past insight, accessing and applying patterns from a knowledge base of past storylines. The expert has learned to calibrate projections, based on appropriate interventions that conform to the constraint space identified as boundary conditions. Yet the eminent expert carries out an intervention to provide more insight into the situation at hand and does not rely completely on his or her knowledge of the relevant constraint space. This is the process of envisioning innovation. Eminent expertise is characterized by the continual refinement of a conceptual model of the expert’s domain. This occurs by directing observations with the intent to actively manipulate conditions in the domain in order to arrive at innovative new outcomes from given and known situations. The eminent expert does this repeatedly, as he or she purposefully steers activity away from routine. The eminent expert knows about the limits of routines and knows how to apply slight changes to the repeated activity in order to scope new boundaries. As he or she initiates new conditions in the known field, the expert will eventually find him/herself in new terrain, hence being less than an eminent expert in the newly created domain area. In a cyclical configuration of gathering understanding and generating novelty by applying the new understanding, the knowledge base of the domain expert grows with each iteration of change. As with designers who first need to learn about the domain under design before being able to initiate change, the eminent expert explores the boundaries of the current

232

Product Experience

conceptual models by purposefully manipulating conditions in his or her domain. Designing – the act of manipulating representations – is necessary in order to arrive at refined representations. Eminent expertise thus is not the mastery of analytic deductions in the found domain, but instead incorporates steering the domain by changing it. The original contribution of the eminent expert is a necessity in this process of gathering advanced understanding to correlate actual results with anticipated outcomes. This process is the engagement of the expert in reflective practice (Schön, 1983) where strategies for activities, as well as the activities themselves, are subject to adaptations in the face of a changing environment. Domain experts have gone through many iterations of reflection upon action, initiated by changes in the conditions within the domain environment and driven by results and problems that arise from these changes. These iterations are necessary to reinforce mutual dependencies between conceptual assessment and practical application on the basis of a series of interventions into ongoing activities. The eminent expert’s conceptual model of the domain has reached a level where it can represent the patterns of change that steer the adaptation of response strategies to the changing context of the situation.

4. THE IMPLICATIONS OF DIFFERENCES IN USER EXPERTISE FOR PRODUCT DESIGN Expert decision making is driven by context. From the perspective of the designer who faces the challenges of designing artifacts as decision support, this has many implications. First, the nature of expertise is rooted in the social structure of the domain and is driven by the changing nature of the domain context as captured in the TEMPEST model of expertise as a kite in the wind (Figure 8.1). Expertise is generally instantiated as an interpersonal collaboration where designers can identify experts in certain stages of the activities, while expert roles may change during different stages of tasks. This same pattern is relevant to the design process, where the lead expert roles shift between practitioners, innovators and technologists (Roesler et al., 2005). In order to conduct the design process, all three pools of expertise are necessary in parallel. No person can play all roles; designers cannot completely put themselves into the seat of practitioners, practitioners cannot take over design work, and technologists cannot replace designers – nor can they take on the role of practitioner. Problems arise in the design process when one pool of expertise attempts to stand in for others on the basis of inferences from a distance. Key to an understanding of expertise in design is coordinating the convergence of these different types of expertise as a particular product development cycle matures and restarts. There are many examples where the handover of role-specific expertise is central. Consider, for example, the changing meanings of artifacts in the course of cognitive work with the model of a door key. While a technologist is primarily concerned with designing a secure key that can reliably lock and unlock a door, the innovator might question the concept of a key to lock a door at all. The question might not stop at this point. Why a door at all, what is it for? What are other possibilities for the concept of a key? From the perspective of a practitioner, the existence of the key becomes meaningful only as specific episodes arise during the execution of tasks. On the way to the site of a task, the practitioner encounters a locked door. Unlocking the door is not the central task for the practitioner – getting to the other side of the door where the task is located is the central task.

Designing for expertise

233

Let’s put this apparently simple issue in a context: During an emergency at a power plant a pump that should provide a critical function to maintain safe conditions fails to start. The person on the scene is inexperienced and is unable to start the pump; the experienced person is in a different location in the plant (the control room) and attempts to get to the failed pump to perform the restart. However, during an emergency for security reasons physical access from one part of the plant to another is restricted. The key expert is blocked from reaching the critical location; if the pump is not restarted an extreme and untried safety procedure must be started. Can the expert find a way through the blocked doors in time to restart the critical pump in time? (He did in the actual nuclear power plant emergency in 1985 that is the basis for this example.) This unique event captures a general tradeoff between security and collaboration – blocking access when it is undesirable or blocking some groups access versus enabling collaboration and connection to enhance performance – that is a central dilemma in current debates about computer security in a world of web-based interactions at a distance. One pool of expertise is about different ways to block access; another pool of expertise is about ways to enable more collaboration at a distance. Without coordinating the two, cycles of mal-adaptation develop where for security reasons access is hobbled or blocked and people who can take advantage of the connection devise workarounds to communicate. In reverse, we can deduct levels of expertise from artifact characteristics in the work environments of experts. The expert work of aircraft pilots is mapped into the complex structure of the aviation cockpit (which in turn is a representation of the complexities underlying the task of flying a plane – a representation of the joint system of different people and devices that underlies flying; see Hutchins 1995b). There is a link between expertise and the complexity of technology applications. Alexander (1964) has argued that, for emerging specializations, the emergence of experts is the consequence of technological advances and part of the formation of new domains. Understanding the context-dependent nature of expertise is a prerequisite for designing artifacts that support decision making – that allow people to become familiar with patterns of use and that support identifying constraints necessary to detect and resolve irregularities in the course of action. From the cognitive systems design perspective, the level of expertise of domain practitioners directs how much of the guidance for use must be built into the artifacts of a system, since the training of experts may serve as a substitute in the absence of self-explanatory clues in expert environments. As an example, compare a pilot with the everyday driver of an automobile. For the former, flight training typically takes a few hundred hours, with frequent deliberate refresher sessions in the flight simulator. As a consequence, the pilot has an extensive conceptual representation of the plane as a system of artifacts, and of the plane’s boundaries of performance in the face of environmental condition changes and other events. This expertise acts as a source of resilience to overcome design deficiencies such as mode confusions so that they rarely turn into plane crashes (but see the mode confusion issue in, for example, Woods et al., 1994). Now consider the car example. Driver training is much less elaborate and consists of a rough understanding of traffic rules, performance of the vehicle, and control of the vehicle. The domain of driving a car in roadway traffic is less complex than that of flying an airplane in air traffic. Compared to the domain of flying a plane, the driving domain consists of factors such as slower speed and simpler degree of technology in the groundbound vehicle, greater visibility of traffic and obstacles, and a greater proximity margin of other traffic participants. In contrast, the airplane pilot must think further ahead, due to the higher speed of the plane, and must cope with the complex technology of a vehicle that cannot stop

234

Product Experience

without losing aerodynamic lift. In addition, the pilot is a participant in a larger collaborative organization consisting of cockpit crew, airport crew, and air traffic controller – all in different roles – where limited flight, take-off, and landing pathways must be shared with other planes within very tight schedules. Add to this complex organizational conglomerate automated on-board systems, replanning due to delays, invisible flight routes, and eventual technical problems, and it’s easy to assess why pilots are experts. We have learned how difficult the assessment of typical practitioner expertise is. Expert users can draw resources for interpretation of gaps from their individual knowledge, while novice users rely on knowledge in the world that is provided by the design of the task environment (Norman, 1986). While providing an extended range of performance, advanced technological applications introduce new forms of complexity, and with them new possible failures. During the past thirty years, advanced computerized systems in high-stakes domains have demonstrated the new realities of increased sophistication in the form of new possibilities. Unfortunately, they also have demonstrated the new realities of increased complexities that have contributed to new types of incidents and accidents in aviation, process control, monitoring, medicine, mission control, and military applications (Woods and Hollnagel, 2006). Due to the public consequences of failures, extensive studies about the nature of expertise in the monitoring and design of technology have generated invaluable insight into the relationship between domain knowledge, training, and design. In all instances, highly trained experts are paired with advanced computerized technology in joint systems (Hollnagel and Woods, 2005). In facing the challenges of designing for these expert domains, it is essential that designers realize that their objective is not to design for users, but to support experts by design. Learning how to reinforce innovation is a form of cognitive work as reflectionin-action. Design requires understanding, exploration, and experimentation; during ideation phases in design projects, this cognitive work leads to novel artifacts and new approaches to doing things, always coupled with the initiation of change. New technological artifacts will change the nature of practice (Winograd and Flores, 1986). Novel artifacts will require practitioners to engage in new types of operations that need to be learned. In the long run, innovation will form new types of expertise, while experts will make contributions that lead to new inventions (Woods and Dekker, 2000). Designers are confronted with the challenge of understanding the expertise in the domain under design; this does not require that they become experts in the domain themselves. Studies of expertise in various fields of practice, conducted during the past 50 years – among them studies on the expertise of designers by Cross (2003) – illustrate the nature of expertise as a characteristic feature in cognitive work that is tightly coupled with the nature of the field of practice. The most recent view on expertise is that it exists in the context of a specific domain and encompasses the social structure of the field, individual differences, training, and personal experience. Another important finding is that expertise is subject to continuous change. As technologies advance, they transform the field of practice and confront experts with the necessity of keeping their knowledge updated. In the face of these characteristics of expertise, it seems quite illusionary that designers could become experts in a field they encounter during a design assignment. However, designers may find it helpful to become familiar with the patterns of knowledge elicitation and work practice that instantiate expertise at work (see the earlier section on ‘What is expertise’). By designing support for expert reasoning strategies in collaboration with practitioners and technology developers, designers make their contribution to

Designing for expertise

235

expert work. Key to the design of systems that support practitioners in being experts is a design process model that allows the convergence of the perspectives of practitioners, innovators and technologists.

5. SUMMARY AND CONCLUSION After excursions through the nature, study of, and various views on expertise; through the implications of expertise research on design; and through the strategies for a design process that addresses the introduction of novelty into the domains of experts, we can now summarize expertise in a few sentences to illustrate how aspects of expert practice, decision making, and innovation-building affect designers. Our view on expertise enables designers to design with the objective of supporting the work of experts and acquiring expertise. Expertise as a focus for design presents an interesting though somewhat moving target for design activity, since expertise is driven by the context and social structure of the domain. One of the consequences of this is that expertise continuously re-invents itself in a sequence of adaptations to a changing environment. At the same time, events that drive these changes are, in part, outcomes of expert practice. We summarize expertise as follows: • Experts have observed, learned and practiced in their domain for a long time. Their expertise is domain-specific, driven by the context of the domain, and instantiated in the social structure of the domain. • Expertise is both knowledge and skill in the adaptation of understanding to observations in the context of a situation. • Expert practitioners capture and model their domain understanding and approach strategies in representations that they refine while eliciting the knowledge required to practice as experts in their domain. • Expertise changes; the agreed-upon standards of outstanding performance are subject to continuous improvement. Improvements may include refined reasoning strategies, better decision support representations, or more advanced adaptations to the changing nature of practice. In addition, social standards on how to assess expertise may change. • Expertise is a form of contextual understanding that guides the formation of strategies in making sense of observations in a given context. • Expertise is limited by the perspective of the expert. Complex decision-making tasks, such as designing, require reasoning strategies that utilize methods of coordination between the different types of expertise of design stakeholders. It is not only advisable to consult other perspectives, it is necessary (Klein, 1997). • At the eminent of level of expertise, an expert generates an original contribution to the standards of the domain. As innovators, experts shape the future of succeeding experts in their field. In the course of learning to understand their domain, the eminent experts change their domain, and the changed domain requires and forms a new generation of experts, often trained by the eminent experts, and a product of the new conditions. This eminent level shows parallels to the key challenge in design work: The generation and creation of novelty, and the responsibilities to steer the changed conditions. Expertise, in conclusion, evolves as a consequence of adaptations in a changing environment. In parallel, expertise forms the basis from where practitioners and designers envision and implement changes in the form of novelty into the domain. Novel technologies, in turn, will initiate new forms of expertise. Innovation and expertise are mutually interrelated.

236

Product Experience

Assessing the expertise of future practitioners is both a requirement and an outcome of design support. Expertise does not reside in artifacts or people, but is the result of cognitive work by practitioners who build a correspondence across artifacts with their goals, their environment, the perspectives of others in their field of practice, and the stakeholders who depend on the outcomes produced.

REFERENCES Alexander, C. (1964). Notes on the synthesis of form. Cambridge, MA: Harvard University Press. Bayazit, N. (2004). Investigating design: A review of forty years of design research. Design Issues, 20(1), 16–29. Bloom, B. S. (Ed.) (1985). Developing talent in young people. New York: Ballentine Books. Christoffersen, K., Woods, D. D. and Blike, G. T. (2007). Discovering the events expert practitioners extract from dynamic data streams: The mUMP technique. Cognition, Technology, and Work, 9, 81–98. Cross, N. (2003) The expertise of exceptional designers. In: N. Cross and E. Edmonds (Eds.) (2003). Expertise in design, creativity and cognition, pp. 23–35. Sydney: University of Technology. De Groot, A. D. (1978). Thought and choice in chess. The Hague, Netherlands: Mouton. (Original work published 1946.) Dreyfus, H. L. and Dreyfus, S. E. (1986). Mind over machine. New York: Free Press. Dreyfus, H. L. (1997). Intuitive, deliberative, and calculative models of expert performance. In: C. E. Zsambok and G. Klein (Eds.) Naturalistic decision making. Mahwah, NJ: Lawrence Erlbaum. Dreyfus, H. (1955, 1967). Designing for people. New York: Allworth Press. Dunbar, K. (1999). How scientists build models: In vivo science as a window on the scientific mind. In: L. Magnani, N. Nersessian and P. Thagard. Model-based reasoning in scientific discovery, pp. 89–98. New York: Plenum Press. Ericsson, K. A. and Smith, J. (Eds.) (1991). Toward a general theory of expertise. Cambridge: Cambridge University Press. Ericsson, K. A. and Charness, N. (1997). Cognitive and developmental factors in expert performance. In: P. J. Feltovitch, K. M. Ford, R. R Hoffmann (Eds.) Expertise in context: Human and machine. Cambridge, MA: MIT Press. Ericsson, K. A., Krampe, R. T. and Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363–400. Ericsson, K. A., Charness, N., Hoffman, R. R. and Feltovitch, P. J. (Eds.) (2006). The Cambridge handbook on expertise and expert performance. Cambridge: Cambridge University Press. Feltovitch, P. J., Ford, K. M. and Hoffmann, R. R. (Eds.) (1997). Expertise in context: Human and machine. Cambridge, MA: MIT Press. Fiore, S. F. and Hoffman, R. R. (2007). Perceptual (Re)learning: A leverage point for human-centered computing. IEEE Intelligent Systems, 22(3), 79–83. Gruber, H. E. (1981). Darwin on man: A psychological study of scientific creativity (2nd Ed.) Chicago: Chicago University Press. Heiser, J. and Tversky, B. (2005). Characterizing diagrams produced by individuals and dyads. In: T. Barkowsky (Ed.) (2005). Spatial cognition: Reasoning, action, interaction, pp. 214–223. Berlin: Springer-Verlag. Hoffman, R. R. (1987). The problem of extracting the knowledge of experts from the perspective of experimental psychology. The AI Magazine, 8(2), 53–66. Hoffman, R. R. (Ed.) (1992). The psychology of expertise: Cognitive research and empirical AI. New York: Springer Verlag. Hoffman, R. R., Feltovitch, P. J. and Ford, K. M. (1997). A general conceptual framework for conceiving of expertise and expert systems. In: P. J. Feltovitch, K. M. Ford, R. R. Hoffmann (Eds.) Expertise in context: Human and machine. Cambridge, MA: MIT Press. Hollnagel, E. and Woods, D. D. (2005). Joint cognitive systems: Foundations of cognitive systems engineering. Boca Raton, FL: Taylor and Francis. Honeck, R. P. and Temple, J. G. (1992). Metaphor, experiences, and a PEST. Metaphor and symbolic activity, 7, 237–252. Hutchins, E. (1995a). Cognition in the wild. Cambridge, MA: MIT Press. Hutchins, E. (1995b). How a cockpit remembers its speeds. Cognitive Science, 19, 265–288.

Designing for expertise

237

Jenkins, J. J. (1979). Four points to remember: A tetrahedral model of memory experiments. In: L. S. Cermak and F. I. M. Craik (Eds.), Levels of processing in human memory. Hillsdale, NJ: Lawrence Erlbaum, 429–446. Klein, G. A. and Crandall, B. W. (1995). The role of mental simulation in naturalistic decision making. In: P. Hancock, J. Flach, J. Caird and K. Vicente (Eds.) Local applications of the ecological approach to human-machine systems, Vol. 2, pp. 324–358. Mahwah, NJ: Lawrence Erlbaum Associates. Klein, G. (1997). An overview of naturalistic decision making applications. In: C. E. Zsambok and G. Klein (Eds.) Naturalistic decision making. Mahwah, NJ: Lawrence Erlbaum. Klein, G. (1998). Sources of power: How people make decisions. Cambridge, MA: MIT Press. Klein, G. A., Ross, K. G., Moon, B. M., et al. (2003). Macrocognition. IEEE Intelligent Systems, 18(3), 81–85. Klein, G. A., Moon, B. and Hoffman, R. R. (2006). Making sense of sensemaking. 1: Alternative perspectives. IEEE Intelligent Systems, 21(4), 22–25. Mark, G., Gonzalez, V. M. and Harris, J. (2005). No task left behind? Examining the nature of fragmented work. Proceedings of CHI 2005, pp. 321–330, ACM, Portland OR, April 2–7, 2005. Minsky, M. (1997). Negative expertise. In: P. J. Feltovitch, K. M Ford, R. R. Hoffmann (Eds.) Expertise in context: Human and machine. Cambridge, MA: MIT Press. Norman, D. (1986). The design of everyday things. New York: Doubleday. Rappaport, A. T. (1997). Context, cognition, and the future of intelligent infostructures. In: P. J. Feltovitch, K. M. Ford, R. R. Hoffmann (Eds.) Expertise in context: Human and machine. Cambridge, MA: MIT Press. Roesler, A. and Woods, D. D. (2005). Inventing the future of cognitive work, Proceedings of the 6th international conference of the European Academy of Design. University of the Arts, Bremen, Germany, March 29–31 2005. Roesler, A., Woods, D. D. and Feil, M. (2005). Inventing the future of cognitive work: Proceedings of the 6th international conference of the European Academy of Design. University of the Arts, Bremen, Germany, March 29–31 2005. Schön, D. (1983). The reflective practitioner. New York: Basic Books. Shanteau, J. (1992). The psychology of experts: An alternative view. In: G. Wright and F. Bolger (Eds.) Expertise and decision support. London: Plenum Press. Simon, H. A. and Chase, W. G. (1973). Skill in chess. American Scientist, 61, 394–403. Sternberg, R. J. (1997). Cognitive conceptions of expertise. In: P. J. Feltovitch, K. M. Ford, R. R. Hoffmann (Eds.) Expertise in context: Human and machine. Cambridge, MA: MIT Press. Wallace, D. B. and Gruber, H. E. (Eds.) (1989). Creative people at work. New York: Oxford University Press. Winograd, T. and Flores, F. (1986). Understanding computers and cognition. Norwood, NJ: Ablex Publishing Corporation. Woods, D. D., Roth, E. M. and Bennett, K. B. (1990). Explorations in joint human-machine cognitive systems. In: S. Robertson, W. Zachary and J. Black (Eds). Cognition, computing and cooperation. Norwood, NJ: Ablex Publishing. Woods, D. D., Johannesen, L., Cook, R. I. and Sarter, N. (1994). Behind human error: Cognitive systems, computers and hindsight. Human Systems Integration Information and Analysis Center, WPAFB, Dayton OH. (available at http://www.hsiiac.org/hsi/products.do?action⫽detailandcode⫽HS-1994-2). Woods, D. D. (1995). Towards a theoretical base for representation design in the computer medium: Ecological perception and aiding human cognition. In: J. Flach, P. Hancock, J. Caird and K. Vicente (Eds.) An ecological approach to human machine systems I: A global perspective. Mahwah, NJ: Lawrence Erlbaum. Woods, D. D. and Dekker, S. W. A. (2000). Anticipating the effects of technological change: A new era of dynamics for human factors. Theoretical Issues in Ergonomic Science, 1(3), 272–282. Woods, D. D., Tinapple, D., Roesler, A. and Feil, M. (2002). Studying cognitive work in context: Facilitating insight at the intersection of people, technology and work. Cognitive Systems Engineering Laboratory, Institute for Ergonomics, The Ohio State University Columbus OH at url: http://csel.eng.ohio-state. edu/woodscta Woods, D. D. and Hollnagel, E. (2006). Joint cognitive systems: Patterns in cognitive systems engineering. Boca Raton, FL: Taylor and Francis. Wright, G. and Bolger, F. (Eds.) (1992). Expertise and decision support. London: Plenum Press. Zsambok, C. E. and Klein, G. (Eds.) (1997). Naturalistic decision making. Mahwah, NJ: Lawrence Erlbaum.

13 PRODUCT EXPRESSION: BRIDGING THE GAP BETWEEN THE SYMBOLIC AND THE CONCRETE THOMAS J.L.VAN ROMPAY Faculty of Behavioural Sciences, Enschede, The Netherlands

1. INTRODUCTION Browsing through a home improvement magazine, overhearing a conversation about the latest design trend, spotting an ad for the new Peugeot; it is all too clear that those qualities we refer to as symbolic (i.e. qualities that are not ‘literally’ part of product appearance such as distant, inviting and firm) take on increased importance in today’s market. In design and marketing literature, this trend is often traced to the fact that many (or most) products appearing on the market are very similar in function and price, making it hard or simply irrelevant for people to differentiate products on such primary criteria (Postrel, 2003; Veryzer, 1995). As a result, products are ever more evaluated in terms of their experiential benefits: ‘What does this product say about me?’, ‘Is this really me?’, etc. Indeed, a recent study confirmed such speculations by revealing that consumers’ preferences for product appearance are by and large motivated by symbolic (product) meaning (Creusen and Schoormans, 2005). Although we all perceive objects as expressive of symbolic meanings, it does not follow that we understand what it is that makes us, for instance, experience a certain product as adventurous, modest or trustworthy. On the contrary, attempts to relate symbolic meanings to formal product features often reach no further than global attributes such as size or color. And even with respect to stereotypical, seemingly obvious relations between form and expression, one may find it is not that easy to explain why, for instance, organic or rounded form features are generally perceived as secure or emotional. In other words, although perceiving what products express comes most natural, Product Experience Copyright © 2008 Elsevier Ltd.

333

334

Product Experience

accounting for a product’s expression is less straightforward. Arguably, it is precisely this ‘accounting for’ that is a prerequisite for successful design. In this chapter, different perspectives on (product) expression or symbolic meaning (synonyms used alternately) will be presented. Although different in scope and focus, they all share the, sometimes hidden, assumption that an object’s perceived expression results from the interaction between object and perceiver. However, in accounting for symbolic meanings, researchers usually stress the role of either the perceiver or the object perceived (Crozier, 1994; Crozier and Chapman, 1984; Dewey, 1934). In order to provide a rough classification of relevant studies in expression (varying widely in focus and scope) they will be presented along these lines. Another reason for doing so is to clarify the rationale underlying a third approach, discussed consecutively. This approach underlines the necessity to simultaneously and explicitly take into account the mutual contributions of both perceiver and object perceived. In the first part of this chapter we will discuss studies in which the object is at the center of investigation. In this type of study, relations between formal object features and symbolic meanings are explored. In the second part of this chapter we will review research that places primary emphasis on the role of the perceiver in the coming about of an object’s expression. According to this approach, symbolic product meanings can be traced to cognitive or biologically centered processes. Following our discussion of object- and individual-centered perspectives, we will present a third perspective, originating in the writings of the philosophers John Dewey (1934) and Merleau-Ponty (1962). Both stress the fact that (symbolic) meaning can only be studied in the light of interactions between individual and environment. By consequence, explicit and equal emphasis should be placed on the interdependent contributions of both object and perceiver. Since object-perceiver interactions are constrained by the peculiarities of the human body, both view meaning as essentially embodied. In the last two decades, this approach has ‘resurfaced’ in cognitive psychology (e.g. Johnson, 1987; Lakoff and Johnson, 1999), and has proven to be very successful in accounting for symbolic meanings of all kinds. In line with recent studies in cognitive psychology, we will argue that symbolic meanings exemplified by products are rooted in our own embodied experiences arising from interactions with the environment. Before launching our review it should be stressed that it is not our intention to attest to the merits of one approach or perspective over another. In the course of this chapter, it will become clear that the studies under review vary widely with respect to the type of symbolic meanings addressed, withholding us from any such endeavors. Nonetheless, the relative lack of studies addressing product expression from an interactional perspective justifies, we feel, the more elaborate discussion of this approach presented in this chapter.

2. THE EXPRESSIVE OBJECT At the far end of the spectrum, theories stressing the object perceived in order to account for its expression embrace the assumption that our world and its objects are intrinsically meaningful. As such, expression resides in the object and can be described without taking into account the role of the perceiver, except to say that he or she can be more or less receptive or equipped to perceive an object’s expression. The artist Wassily Kandinsky, for instance, considered visual elements such as point, line, and plane (the basic elements of abstract painting) to be fully ‘alive’, and therefore expressive in and of themselves. In discussing the expressive characteristics of the basic plane, Kandinsky (1926) argues that its four lines, i.e. the two verticals and the two horizontals, are bound up with different

Product expression: Bridging the gap between the symbolic and the concrete

335

‘sounds’. Whereas the ‘above’ and, although to a lesser extent, the ‘left’ are expressive of lightness, emancipation, and freedom, the ‘below’ and the ‘right’ express condensation, heaviness, and constraint (Kandinsky, 1926). In line with his claim that the basic elements of painting are ‘alive’ in and of themselves, Kandinsky motivates his assertions in terms of the presumed ability of every great artist to ‘feel the breathing of the still untouched plane’ (Kandinsky, 1926, p. 116). Discussing the works of Kandinsky immediately brings to mind the name of Paul Klee; artist, visionary, and, like Kandinsky, teacher at the Bauhaus, the famous school of design in Weimar. Klee considered communication with nature as the most essential condition for the artist. Forms, Klee argued, are important in so far as they symbolize man’s relation to the cosmos (Klee, 1953). In symbolizing this relationship, Klee considered line, tone value (the degrees of shading between white and black), and color as the basic formal elements an artist has at his disposal: Certain proportions of line, the combination of certain tones from the scale of tone values, certain harmonies of color carry with them at the time quite distinctive and outstanding modes of expressions (Klee, 1948, p. 37).

Dependent on the combination of these factors, objects strike us as ‘serene or severe, tense or relaxed, comforting or forbidding, suffering or smiling’ (Klee, 1948, p. 33). In many of Klee’s sketches his attempts to create what he refers to as ‘expressive contrasts’ can be witnessed. Straight rigid lines, for instance, may be combined with curved smooth lines. The use of medium shades of grey, implying weakness, may be alternated with the wide use of all tones from black to white, implying strength, and with regard to color; ‘what tremendous possibilities for the variation of meaning are offered by the combination of colors’ (Klee, 1948, p. 39). Although Klee regarded the origin of artistic expression as intrinsically linked to human life and the relation of man to the cosmos, Klee, in his role as teacher, discussed the symbolic characteristics of his sketches primarily in terms of their formal properties. Likewise, the perspectives to follow, although acknowledging the role of the perceiver, primarily stress the object perceived in order to account for its expression.

2.1. Bodily expression Without doubt, the most basic form of expression is the kind we refer to as ‘body language’. Varying from facial expressions to bodily postures, the ways in which people express themselves through their ‘body language’ are numerous. The posture of an old man may express defeat and resignation, the movements of a child playfully walking over a thin line victory and temptation, and the facial expression of an abandoned lover despair and sorrow. But apart from bodily states or activities in relation to which expression is not an end in itself, but a natural autonomic reflection of one’s state of mind, at other times people deliberately engage in bodily activities to express themselves. The type of bodily expression that comes most readily to mind in this context is dance. In general terms, form (i.e. bodily shapes or postures) and motion (i.e. bodily movements) are the main carriers of meaning in dance (Kreitler and Kreitler, 1972). Kandinsky (1926) regarded a dance as an uninterrupted composition of lines and shapes, and, therefore, subject to the same ‘rules’ applying to formal elements of works of art. But what distinguishes a dance most clearly from, for instance, a painting is its dynamic character; the interaction of form and movement. Most researchers have, for that reason, studied the ways in which dancers express meanings through bodily movement. Oskar

336

Product Experience

Schlemmer, Kandinsky’s colleague and director of the Bauhaus stage in Dessau (1925– 1929) is particularly known for his stage plays and workshops in which the relations between stage, body, and space were explored: We shall observe the appearance of the human figure as an event and recognize that at the very moment it has become a part of the stage, it is a ‘space-bewitched’ creature, so to speak. With a certainty that is automatic, each gesture and each movement is drawn into the sphere of significance (Schlemmer, 1927. Cited in Wingler, 1974, p. 474).

More recently, Sawada, Suda and Ishii (2003) looked into relations between armmovement characteristics (i.e. speed, force, and directness) and emotional expression, based on Laban’s classification of movement in terms of time, weight, space, and flow (Laban, 1988). They showed that dancers’ expressions of anger are reflected in arm movements fast in velocity and strong in force, as opposed to slower and weaker arm movements indicative of sadness and joy. Expressions of joy differ from those of sadness in that the former are characterized by a longer traveled distance with the arms (i.e. more ‘expansiveness’) and a more varied trajectory (i.e. more ‘indirectness’). In a similar study, De Meijer (1989) demonstrated that expressions of positive emotions, e.g. joy, are characterized by upward directed movements (i.e. stretching trunk movement), whereas downward directed movements (i.e. bowing trunk movement) are characteristic of negative emotions. These results are in line with Osgood’s finding that people across the world perceive downward-directed curves as sad and upward-directed curves as merry or joyful (Osgood, 1960). Earlier on, we noted that Kandinsky proposed a similar structuring when relating the ‘above’ to positive connotations like lightness and freedom, and the ‘below’ to negative connotations such as heaviness and constraint (Kandinsky, 1926). Although the studies discussed in the remainder of this chapter all center on static, non-moving objects such as products and works of art, aforementioned studies are of relevance in so far as both moving and static objects are perceived as infused with symbolic qualities, as will be shown next. A second distinction concerns the difference between two-dimensional and three-dimensional forms, discussed alternately in the remainder of this chapter. Whereas human bodies, products, and sculptures may literally enclose space or convey depth, two-dimensional forms such as paintings are never literally ‘open’ or ‘closed’, ‘near’ or ‘distant’. It should be kept in mind, however, that our interest is not in what products literally convey, but in their symbolic or expressive meaning. In that regard, products and two-dimensional objects are alike; they are both perceived as expressive of symbolic meanings through their visual appearance.

2.2. Arnheim and the Gestalt school In the early twentieth century, Gestalt psychology developed as a response to the traditional method of scientific analysis advocating the analysis of complex phenomena in terms of their separate parts. According to the Gestalt psychologists ‘the whole is more than the sum of its parts’. A triangle, for instance, is perceived as an independent entity (i.e. a whole), and not just as a collection of three lines plus three angles. In order to account for such primary perceptual phenomena, the Gestalt psychologists proposed a number of innate tendencies, i.e. ‘Gestalt principles’, guiding our perception of the world and its objects (e.g. Koffka, 1935; Wertheimer, 1938). The ‘similarity principle’, for instance, predicts that things that share visual characteristics (e.g. shape, size, color, and orientation) are perceived as belonging together

Product expression: Bridging the gap between the symbolic and the concrete

337

(i.e. forming one whole). The ‘continuity principle’ reflects the finding that perceptual organization favors continuity over abruptness. Next to proposing gestalt principles, the Gestalt psychologists also assumed a preference for good or ‘prägnant’ gestalts, a term used to designate those gestalts which are the best organizations of stimuli in a given situation. In general terms, good gestalts are characterized by regularity, symmetry, inclusiveness, unity, harmony, maximal simplicity, and conciseness. Perhaps the most famous proponent of the Gestalt school is Rudolf Arnheim, best known for his enlightening discussions of works of art. He demonstrated that works of art are perceived as gestalts that can be more or less ‘balanced’ dependent on a wide variety of factors (Arnheim, 1974). Paintings, for instance, may express a sense of stillness or striving dependent on the placement of their constitutive elements within the frame (i.e. dependent on their composition). Whereas compositions properly balanced are generally perceived as still, compositions lacking balance are perceived as restless or as striving towards equilibrium. Arnheim’s conception of symbolic or expressive qualities motivates this brief discussion of his writings under an object-centered perspective: ‘Expressive qualities are authentic and objective qualities conveyed by perceptual shape, size, movement, intensity, rhythm, and so on’ (Arnheim, 1992, p. 205). The ability to directly perceive expressive characteristics is explained in terms of the organization of the nervous system, a notion referred to as ‘isomorphism’: ‘If gestalt processes are observed in perceptual experience, analogous processes are likely to account for them in the brain’ (Arnheim, 1992, p. 201). It is this claim, however, that has been subject to criticism for lack of empirical support (e.g. Berlyne, 1971; Crozier and Chapman, 1984). Nonetheless, many of Arnheim’s predictions on the perception of form, and in particular on the role of perceived balance and movement herein, have been supported by controlled studies (e.g. Locher, Gray and Nodine, 1996; Locher and Stappers, 2002). In his works ‘The dynamics of architectural form’ (1977) and ‘The power of the centre’ (1988) Arnheim also (explicitly) acknowledges the role of the perceiver in the experience of objects and architectural spaces. In the former, he introduces the concept of anisotropy to explain that different directions in space are perceived unequally (Arnheim, 1977). According to Arnheim, this perceived inequality relates to experiences that arise from moving through space; going up, for instance, takes more effort than going down since we have to overcome the force of gravity, or in Arnheim’s words: The symbolic endowment of architectural shape is compelling only because the humble daily experience of climbing stairs reverberates with the connotations of overcoming the weight of gravity and rising victoriously toward the heights (Arnheim, 1977, p. 210).

As a result of this anisotropy, directions in architecture and works of art are also perceived ‘unequally’; an extension in the vertical dimension is perceived as more pronounced in comparison to an equal extension in the horizontal dimension. It is for this reason that a tree or skyscraper standing up straight ‘looks’ more impressive than the same one brought down. Locher and Stappers (2002) demonstrated that anisotropy also influences the perception of visual displays; designs with greater ‘weight’ above the horizontal were rated as significantly more dynamic than designs with greater ‘weight’ below the horizontal.

2.3. The ecological approach The ecological approach to perception originates in the writings of J. J. Gibson (1979). According to Gibson, perception of the environment is direct and unmediated. In this

338

Product Experience

regard, the ecological approach can be seen as congruent with the notions put forward by the Gestalt school. But whereas the Gestalt psychologists explained this capability in terms of the functioning of the nervous system, Gibson assumed meanings are conveyed through information in the light, reflecting textures in the optic array. He considered this hypothesis radical ‘for it implies that the values and meanings of things in the environment can be directly perceived’ (Gibson, 1979, p. 127). Central to the theory of direct perception is the concept of an ‘affordance’, defined as a specific combination of invariant properties of an environment relative to a particular organism (Gibson, 1979). Gibson argued that people perceive things in terms of what actions they afford, i.e. in terms of what we can do with them. A chair, for instance, affords ‘sitting’, a cup affords ‘holding’, and a tree affords ‘climbing’. Clearly, for a bird or elephant none of these affordances holds; it is in this sense that affordances are relative to a particular organism. The notion that objects are perceived in terms of what they afford is reminiscent of the Gestalt school (Bruce, Green and Georgeson, 1996), and in particular of Koffka’s concept of ‘demand characteristics’: To primitive man each thing says what it is and what he ought to do with it: a fruit says ‘Eat me’; water says ‘Drink me’; thunder says ‘Fear me’, and woman says ‘Love me’ (Koffka, 1935, p. 7).

Although Gibson was not primarily interested in objects’ symbolic or expressive characteristics, his writings are of interest for the present context in so far as affordances may guide the attribution of symbolic meanings. Upon perceiving a chair, for instance, an individual may not just realize that it affords sitting; based on specifics of the chair he may understand that it affords a particular type of sitting. A chair with extended armrests, for instance, affords a more relaxed type of sitting than a chair with no armrests. Accordingly, the former may be perceived as more dignified or stately than the latter. In a similar fashion, distinguishing between motorbikes in terms of symbolic qualities such as toughness or ferociousness may be based on appreciation of the bodily position afforded by shape and positioning of the motorbike seat (Kreuzbauer and Malter, 2005).

2.4. Discussion The studies on expression reviewed in this section place primary emphasis on the object perceived. Although at times certainly successful in relating symbolic or expressive characteristics to the visual appearance of objects, a difficulty of studies originating in this approach is that they often fall short in explaining the relations uncovered. Arnheim’s assertion, for instance, that the proposed relations can be accounted for in terms of the workings of the nervous system is hard to sustain in the absence of empirical data. It is only when he explicitly acknowledges the role of the perceiver in the experience of space and objects, that one ‘understands’ why, for instance, the vertical connotes qualitatively different meanings than the horizontal. From a philosophical point of view, object-centered theories on expression are problematic in so far as they embrace the assumption that the world ‘possesses’ fixed properties that can be ‘picked up’. When looking at our own experience it may indeed seem natural to consider objects, for instance, impressive, natural or playful in and of themselves. But as it turns out, attempts to account for such expressive or symbolic qualities are frustrated as long as one concentrates primarily on the object of perception. To address the difficulties encountered by an object-centered approach, both practical and philosophical, researchers increasingly shifted the emphasis of their projects from stimulus properties of objects to the processes underlying their perception and understanding.

Product expression: Bridging the gap between the symbolic and the concrete

339

3. THE CONSTRUCTIVE INDIVIDUAL Paralleling the rise of cognitive science, researchers became increasingly interested in the ways in which processes related to the workings of body and mind partake in the establishment of meaning. Whereas in object-centered studies symbolic qualities were primarily studied in terms of objects’ formal properties, in the perspective discussed next emphasis is on the individual and all that influences his or her experience of the world.

3.1. Arousal and the aesthetic experience By far the most influential biologically centered theory on the aesthetic experience of visual stimuli is Berlyne’s psychobiological approach (Berlyne, 1971). Assuming people prefer an optimal level of arousal engaging the nervous system to the right extent, Berlyne considered the potential to introduce a rise in this arousal level the most characteristic feature of works of art. This potential, Berlyne argued, is contingent on stimulus properties such as ‘perceived novelty’, ‘surprisingness’, and ‘complexity’, properties he refers to as ‘collative’. Asymmetrical and irregular forms, for instance, are more ‘complex’ in comparison to symmetrical and regular forms and, therefore, have greater arousal potential. Berlyne was primarily interested in stimulus qualities capturing attention, increasing arousal, and affording exploration, and not so much in symbolic or expressive meanings of stimuli. Berlyne’s findings, however, are of relevance to our project in so far as they suggest a relation between the arousal potential of stimuli and their expressiveness. Underscoring this relation, Berlyne argues: Art biased towards heightening arousal is called dramatic, dynamic or stirring. If the arousalmoderating devices have the upper hand, art is said to be static, harmonious or serene (Berlyne, 1971, p. 254).

Obviously, people may prefer different levels of arousal, and dependent on the context in which the stimulus is perceived its arousal potential will vary. In paintings, for instance, a visual element is always embedded within a complex structure comprising both stylistic and semantic layers that codetermine its arousal potential, and hence to some degree its symbolic meaning or expressiveness (Cupchik, 1994). With respect to product design, Berlyne’s writings inspired Coates (2003) to explore the relation between perceived novelty and product expression. Departing from Osgood’s framework for the measurement of meaning (Osgood, 1957), Coates showed that higher degrees of novelty trigger perceptions of products as more ‘potent’ and ‘active’, as opposed to products presenting less novelty. Hence, they are perceived as expressing higher degrees of related characteristics such as excitement and emotionality (‘activity’), and dominance and toughness (‘potency’).

3.2. Dynamization and empathy Another biologically anchored phenomenon relevant to our discussion is the ‘dynamic’ or ‘dynamizing’ mode of response (Werner and Kaplan, 1963). Dynamization originates in enhanced kinesthetic or motoric reactivity, and relates to the fact that people, when asked to explain why they relate a certain form to a particular feeling or concept, often refer to a hypothetical quality of motion along its lines. One may, for instance, relate a circle to the concept of infinity because movement along its contour is endless and repetitive. Likewise, a sharp angle may be referred to as shocking or thrilling for its acute

340

Product Experience

and sudden change in direction of motion. And long curves may be said to express calm because their ‘motion’ is slow (Kreitler and Kreitler, 1972). In contrast to studies on bodily expression discussed earlier, in this line of reasoning the origin of symbolic expression is not sought in properties of the object or (simulated) shape as such, but primarily in the motoric activity associated with the stimulus. German psychologist Theodor Lipps (1897), for instance, discusses dynamization in relation to ‘empathy’ or ‘einfühlung’. According to Lipps, people show a tendency to imitate perceived movements or dynamic postures of people and objects. Since specific motoric movements automatically give rise to their emotional counterparts, Lipps argued, we come to feel or experience the emotion of another person (i.e. empathize with another person) or experience an object as expressing a particular symbolic concept. Sadness, for instance, is commonly reflected in curved, earth-bound bodily postures. Upon observing a like-formed object, Lipps claimed, we ‘unknowingly’ imitate its posture. And since a curved, earth bound posture gives rise to the emotion it corresponds with, i.e. sadness, we come to feel the emotion and locate its origin in the object perceived, whereby it becomes expressive (Lipps, 1897). Although research indeed shows that people tend to imitate behaviors, e.g. movements and emotional expressive behaviors of others (Bandura, 1969), Lipps’ claim that the emotional experience ‘automatically’ follows from such simulations has not withstood the test of time. The works of Schachter and Singer (1962), for instance, clearly show that physiological changes alone do not suffice to evoke a full blown emotional experience; of equal importance are the ways in which these sensations are interpreted.

3.3. Metaphor and analogy In interacting with products, people frequently draw implicit comparisons between products belonging to different categories, or between products and other phenomena (e.g. objects of nature, people or animals). The reason for doing so is that one may learn about a particular object by relating it to another. This ability is referred to as metaphorizing, defined as experiencing or understanding one thing, i.e. the target domain, in terms of another, i.e. the source domain (Lakoff and Johnson, 1980). The most popular computer interface is the ‘desktop’ on which the user drags items into folders, moves these to a desired location, or drops them in the trashcan (just to name a few of the actions the ‘desktop’ affords). The ‘desktop’ is successful because virtually everyone knows how to operate it by relying on existing knowledge from daily office work involving real desktops. For instance, one easily understands the purpose of placing related Microsoft Word-files in one folder, since placing paper documents in (physical) folders is a familiar ‘office activity’. It is in this sense that designers can allow their users to understand a relatively new or complex domain more easily by presenting it in terms of a domain they are familiar with. Next to using metaphor foremost as a means to render a new or complex product intelligible, i.e. to reduce the cognitive workload, designers may also employ a metaphor in order to promote other kinds of user experiences. At Delft University, for instance, a project was initiated addressing the design of a copier (Figure 13.1), departing from the metaphor ‘Interacting with a machine is a dance’ (Hekkert, Mostert and Stompff, 2003). One aspect of a ‘dance for two’ is that the participants feel and respond to each other’s moves, an aspect labeled ‘resonance’. The designers mapped this aspect onto the copier by reconsidering the (traditional) ways in which copiers react to user behavior. Agitated movements, for instance, cause this copier to offer more resistance in handling its different parts, whereas smooth movements evoke less resistance. In resonating with

Product expression: Bridging the gap between the symbolic and the concrete

341

FIGURE 13.1 Copier (from Hekkert et al., 2003).

FIGURE 13.2 CD player (IDEO).

its user, the product breaks free from inflexible interaction patterns that so often make traditional copiers a burden to deal with. Hence, the copier is no longer a frustrating machine but rather a dance partner and, therefore, experienced in terms of those qualities people commonly ascribe to dancers (e.g. supportiveness, flexibility, and sensitivity). Whereas metaphor involves the transfer of meaning from one domain to another, a product analogy typically involves a functional similarity between an object and another object or phenomenon. In contrast to metaphor, the transfer of meaning is usually not considered a defining characteristic of a product analogy. IDEO’s CD player may serve as an example (Figure 13.2). The analogy with a traditional light switch is clear; pulling the chord will either turn on or off the CD player similar to how a light may be switched on or off. But this is

342

Product Experience

where the similarity stops; it is, supposedly, not the designer’s intention to map meanings from the domain of lights or light switches onto his product, but merely to point out a surprising or witty functional similarity. One could argue, however, that the difference between metaphor and analogy is not as straightforward, or generally agreed upon, as perhaps suggested in this brief discussion. The designer of IDEO’s CD player, for instance, might object that his design was in fact centered on the transfer of meaning, grounded in the metaphor ‘Music is light’ (i.e. ‘music’ is a source of ‘light’ in the lives of people). Based on such considerations, several authors have pointed out that our understanding of a symbolic expression rests on the intention we ascribe to its creator, i.e. on our figuring out what it is the designer tries to communicate (Cupchik, 2003; Forceville, 1996; Gibbs, 1999; Sperber and Wilson, 1995). In case we assume, for the moment, the designer of IDEO’s CD player did in fact have the metaphor ‘Music is light’ in mind (and not just a functional analogy), his product may very well acquire a different, perhaps ‘deeper’, meaning, changing the experience of the product as a whole. Of particular interest in this regard are Cupchik’s writings on metaphor: Metaphors are generated through intentional acts and presume a point of view on the part of their creators, prompting readers to adopt a perspective in their ‘effort after meaning’ (Cupchik, 2003, p. 19).

Readers adopting an appropriate perspective, Cupchik argues, spontaneously experience the unity of the metaphor, brought about by a merging of the two domains involved (Cupchik, 2003). Arguably, this ‘merging’ takes on special significance in relation to product design where source and target literally merge, i.e. they literally occupy the same space (cf. Forceville, Hekkert and Tan, 2006). As a result of this merging, we usually do not distinguish between source and target in the experience of products; what is experienced is an integrated, seemingly novel phenomenon. Elaborating on the aforementioned ‘desktop metaphor,’ Fauconnier and Turner (2002) point out: The user manipulates this computer interface not by means of an elaborate conscious analogy but, rather, as an integrated form with its own coherent structure and properties. From an ‘objective’ point of view, this activity is totally novel – it shares very few physical characteristics with moving real folders … Yet the whole point of the desktop interface is that the integrated activity is immediately accessible and congenial (Fauconnier and Turner, 2002, p. 24).

Fauconnier and Turner (2002) refer to integrated forms of the kind discussed, i.e. novel structures that arise from the merging of two or more domains, as ‘blends’. Although this brief discussion might suggest metaphors are special, stylistic devices a designer has at his or her disposal, in the next section we will discuss a more fundamental way in which metaphor partakes in our experience of the world.

3.4. Learned meanings Product experience is also influenced by conventions specific to a particular group of people, culture or geographical region; conventions that we learn and familiarize with through interacting with our environment and others within our culture. The role of conventions is particularly stressed in ‘semiotics’, the theory of signs, defined as: ‘Everything that, on the grounds of a previously established social convention, can be taken as something standing for something else’ (Eco, 1976, p. 16). I have learned, for instance, that red flashing lights and high beeping sounds stand for, i.e. signal, danger. Likewise, specific gestures stand for anger or contempt (I gave him the finger), and specific kinds of

Product expression: Bridging the gap between the symbolic and the concrete

343

clothes or haircuts for membership of, or adherence to, a specific social class or ideology. As indicated by these examples, semiotics is basically concerned with communication between people living within a society by means of some system of shared conventions (Smythe, 1984). As for industrial design, the theory of signs gave rise to a discipline called product semantics, defined by Krippendorff and Butter (1984; chapter 14, this volume) as: An effort to understand and to take full responsibility for the symbolic environment into which industrial products are placed and where they should function by virtue of their own communicative qualities (Krippendorff and Butter, 1984, p. 4).

For the study of these communicative qualities, it is constructive to distinguish between ‘denotation’ and ‘connotation’. Whereas the former refers to the communication of a product’s primary function, the latter reflects the communication of socio-cultural values and behavioral standards (Muller, 2001). A chair’s primary function, for instance, is ‘sitting’; the communication of this function by means of the conventionalized form elements laid down in the prototype of the category is what we refer to as ‘denotation’. Next to communicating its primary function, a chair also communicates, i.e. connotes, supplementary or secondary meanings. Dependent on features of its design, a chair may be typified as an everyday office chair or a majestic throne. Conventions play a role herein in that particular materials or forms are readily associated with specific sociocultural values. Materials like silver and gold, for instance, are easily associated with luxury and refinement, recycled materials and plastic, on the other hand, are generally perceived as cheap or trashy. Designers, for this reason, may be considered communicators who have at their disposal a repertoire of forms and materials connoting agreed upon, shared meanings. Following this conceptualization of design as a process of communication, designers and researchers at times advocate the development of a suitable form language in which to talk about the symbolic qualities of products (e.g. Krippendorff and Butter, 1984). In car and theater design, for instance, designers often rely on the idea that consumers recognize a certain number of established, standard characters. Organic form features, for instance, are readily recognized as expressing cuteness and friendliness within Western culture, granting a car an overall friendly ‘character’ (Janlert and Stolterman, 1997).

3.5. Discussion The studies reviewed in this section place primary emphasis on biological or cognitive processes involved in the experience of objects, hence the emphasis on biologically motivated levels of arousal in the nervous system, learned meanings, metaphorizing, etc. In its strong form, the individual-centered perspective considers an object’s expression to be foremost a construct of the mind; objects merely present ‘information’, what really matters are the ways in which we process that information. It is this assumption underlying a so-called ‘idealist’ approach, one could argue, that is at times problematic. For by focusing primarily on the perceiver, they often fall short in providing us with the means to trace the symbolic or expressive meanings people give to objects. Regardless of the emphasis on either object or perceiver, it is of course true that the studies discussed so far in one way or the other acknowledge the contributions of both. In the perspective discussed next, however, it is stressed that explicit and equal emphasis should be placed on the interdependent contributions of both object and perceiver by focusing on the way they interact. This third perspective, coined ‘embodied realism’

344

Product Experience

(Lakoff and Johnson, 2002), originates in the writings of the American philosopher John Dewey and the French philosopher Merleau-Ponty. Both stress the fact that meaning can only be studied in the light of the interactions between individual and world. Since these interactions are constrained by the peculiarities of the human body, both view meaning (in opposition to theories in which meaning has nothing or little to do with the human body) as essentially embodied. Although notions that conceive of meaning as embodied are thus not ‘new’, it is only in the last two decades that a comprehensive scientific theory centered on embodiment and its relation to symbolic meaning has gradually emerged within the field of cognitive semantics.

4. THE INTERACTIONAL STANCE After reviewing both object- and individual-centered perspectives bearing on the subject of expression, Dewey concludes: ‘Both of the theories considered separate the live creature from the world in which it lives; lives by interaction through a series of related doings and undergoings’ (Dewey, 1934, p. 103). According to Dewey, meaning is not a more or less fixed property of world or mind, but arises in interactions between individual and environment. Accordingly, Dewey argues, in order to account for an object’s expression, one has to focus on the fusion of qualities directly present in the object and meanings extracted from prior experience: Different lines and different relations of lines have become subconsciously charged with all the values that result from what they have done in our experience in our every contact with the world about us. The expressiveness of lines and space relations in painting cannot be understood upon any other basis (Dewey, 1934, p. 101).

In stressing interactions