Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Moshe Y. Vardi Rice University, Houston, TX, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany
4206
Paul Dourish Adrian Friday (Eds.)
UbiComp 2006: Ubiquitous Computing 8th International Conference, UbiComp 2006 Orange County, CA, USA, September 17-21, 2006 Proceedings
13
Volume Editors Paul Dourish University of California, Irvine Donald Bren School of Information & Computer Sciences Irvine, CA 92697-3440, USA E-mail:
[email protected] Adrian Friday Computing Department Lancaster University InfoLab 2, South Drive, Lancaster LA1 4WA, UK E-mail:
[email protected]
Library of Congress Control Number: 2006932025 CR Subject Classification (1998): C.2, C.3, D.2, D.4, H.4, H.5, K.4 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13
0302-9743 3-540-39634-9 Springer Berlin Heidelberg New York 978-3-540-39634-5 Springer Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com © Springer-Verlag Berlin Heidelberg 2006 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 11853565 06/3142 543210
Preface Fifteen years after the publication of Weiser’s seminal paper in Scientific American, ubiquitous computing is a large, thriving field of academic research and commercial innovation. The proceedings of the 8th International Conference on Ubiquitous Computing reflect the strength, diversity and vigor of the research program that Weiser initiated. We hope that you will find much in the papers contained herein to stimulate new and exciting work as we reach forward toward ubiquitous computing together. The location of the conference, in Orange County, California, was particularly apposite given recent directions in the field. A Pacific Rim location linked the conference to last year’s venue in Tokyo, Japan, and provided an appropriate space to reflect on the different implementations and visions of ubiquitous computing in different parts of the world. Two significant elements of local industry – the software industry and the entertainment industry – reflect concerns heavily represented in recent research in ubiquitous computing, with mobile and ubiquitous gaming, for instance, an obvious link. The unique metropolitan setting of the Los Angeles region provided a useful setting in which to develop research agendas in urban computing. Finally, while there are no papers in the proceedings on surfing-based Ubicomp, the cultural focus on healthy lifestyles in Southern California relates to concerns with aging and health as important application areas for ubiquitous computing technologies and solutions. This year, the conference received a total of 232 paper submissions. These reflected contributions from 31 countries, drawn almost equally from Europe (38%), the USA (32%), and Asia (26%). From these, the Program Committee selected 30 papers for inclusion in the conference, on the basis of a rigorous double-blind review process. Every paper received at least three independent reviews; most received four, and some more. Reviewers were selected as experts in the field, reflecting a range of specific expertise for the content of each paper as well as a general Ubicomp readership. Our primary concern in the review process was to form a program that reflects the diversity of research activity in ubiquitous computing. The field gains much of its strength from the breadth of interdisciplinary work that it engenders, encompassing computer science, engineering, social science, humanities, arts, and more. As it reflects a maturing discipline, we are as concerned with studies of real-world practice and the realities of ubiquitous computing today as we are with future directions and the potential for ubiquitous computing tomorrow. We are concerned, too, with critical reflection upon our own intellectual and practical histories and research practices. Accordingly, we sought to form a Program Committee that drew broadly from the larger ubiquitous computing community, and the result, we believe, is a vigorously interdisciplinary program that reflects a wide range of contemporary research and intellectual inquiry. Like many conferences in computer-related fields, the Ubicomp series has seen its submission rate rise significantly over the last few years, from 117 papers in 2003 to 144 in 2004, 230 in 2005, and 232 this year. This trend marks the increasing interest in ubiquitous computing as a topic, the growing reputation of the Ubicomp conference as a forum, and the growing relevance of ubiquitous computing topics to other areas of research endeavor. On the other hand, it poses its own challenges. In order to
VI
Preface
remain vibrant and in order to grow, it is important that the conference have space for new ideas and for challenges to received wisdom. The inherently normalizing tendency of peer review creates a tension between innovation and assessment. Reflecting the record submission rate, this year’s roster of 30 accepted papers is also a new record, and we have worked with Conference Chair Cristina Lopes to create space in the conference format for an enlarged program. At the same time, with an acceptance rate of just 13%, Ubicomp remains an extremely competitive venue, making it clear that a commitment to openness and innovation does not come at the expense of quality and rigor in the reviewing process. Given the large number of submissions and the broad nature of their content, the reviewing process was a challenging one. The members of the Program Committee and the reviewers worked extremely hard under severe time pressure to evaluate papers and deliberate on their merits. The Program Committee members were responsible for finding, assigning, and monitoring reviewers for each paper, as well as for writing reviews themselves and for representing papers at the Program Committee meeting in Irvine in early June. The feedback from reviewers enabled us to make balanced assessments of papers’ contributions as well, we hope, as providing useful feedback to authors that will assist in the revision of those papers that could not be included in the program. We are immensely grateful to all who volunteered significant amounts of time and energy to helping us create a strong program for Ubicomp 2006. Conferences of this sort depend entirely on volunteer labor, and we owe a huge debt of thanks to the committee members and reviewers. Even more, we are grateful to the 639 authors who submitted papers to the conference. Although space restrictions limit the number of papers that can be accepted, this represents a tremendous amount of effort, and the conference depends crucially on the outstanding strength of the submitted work. Finally, we would like to extend our gratitude to Conference Chair Cristina Lopes and to Local Arrangements Chair Don Patterson for their contributions both to the papers review process and, more generally, to making Ubicomp 2006 an exciting and enjoyable event. We hope that you find in the program much to inspire and support your own research efforts. Paul Dourish Adrian Friday
Conference Organization Organizers General Chair and Treasurer Cristina Videira Lopes
University of California, Irvine, USA
Program Chairs Paul Dourish Adrian Friday
University of California, Irvine, USA Lancaster University, UK
Demonstrations Chairs Julian Bleecker Khai Truong
University of Southern California / Techkwondo, USA University of Toronto, Canada
Posters Chairs Itiro Shiio Beverly Harrison
Tamagawa University, Japan Intel Research, USA
Videos Chairs Marc Langheinrich Louise Barkhuus
ETH Zurich, Switzerland University of Glasgow, UK
Doctoral Colloquium Chairs Beki Grinter Gregory Abowd
Georgia Institute of Technology, USA Georgia Institute of Technology, USA
Workshops Chairs John Krumm Ken Anderson
Microsoft Research, USA Intel Research, USA
Student Volunteers Chairs Amir Haghighat Gillian Hayes
University of California, Irvine, USA Georgia Institute of Technology, USA
Publications Chair Florian Michahelles
ETH Zurich, Switzerland
Publicity Chairs Timothy Sohn Matthias Kranz Takashi Matsumoto
University of California, San Diego, USA University of Munich, Germany Keio University, Japan
Webmaster Amanda Williams
University of California, Irvine, USA
Local Arrangements Don Patterson
University of California, Irvine, USA
VIII
Organization
Sponsors Silver Sponsor: Bronze Sponsors:
Intel Microsoft Google IBM Nokia Samsung
Supporting Societies UbiComp 2006 enjoys in-cooperation status with the following special interest groups of the Association for Computing Machinery (ACM): SIGBED (Embedded Systems) SIGCHI (Computer-Human Interaction) SIGSOFT (Software Engineering) SIGMOBILE (Mobile Computing)
Program Committee Elizabeth Belding-Royer Genevieve Bell A.J. Brush Vinny Cahill Matthew Chalmers Sunny Consolvo Joelle Coutaz Gregory Abowd Boriana Koleva John Krumm Anthony LaMarca Eyal de Lara Rainer Malaka Henk Muller Moira Norrie Naohito Okude Jun Rekimoto Yvonne Rogers Albrecht Schmidt Mirjana Spasojevic Beatriz da Costa Khai Truong
University of California, Santa Barbara, USA Intel, USA Microsoft Research, USA Trinity College Dublin, Ireland University of Glasgow, UK Intel Research Seattle, USA IMAG, France Georgia Tech, USA Nottingham University, UK Microsoft Research, USA Intel Research Seattle, USA University of Toronto, Canada European Media Laboratory, Ireland University of Bristol, UK ETH Zurich, Switzerland Keio University, Japan Sony Computer Science Laboratory, Japan Indiana University, USA University of Munich (LMU), Germany Yahoo! Research Berkeley, USA University of California, Irvine, USA University of Toronto, Canada
Organization
Reviewers Karl Aberer Mark Ackerman Luis von Ahn Fahd Albinali Brian Amento Ken Anderson Ian Anderson Paul Aoki Trevor Armstrong Yuji Ayatsuka Chris Baber Maribeth Back Jean Bacon Ravin Balakrishnan Rajesh Balan Rafael Ballagas Luciano Baresi Louise Barkhuus John Barton Agathe Battestini Martin Bauer Christian Becker Richard Beckwith Ashweeni Beeharee Michael Beigl Marek Bell Steve Benford Abraham Bernstein Jan Beutel Nina Bhatti Nilton Bila Alan Blackwell Julian Bleecker Sara Bly Marc Bohlen Gaetano Borriello Andrew Boucher Michael Boyle Patrick Brezillon Harry Brignull Barry Brown Ciarán Bryce Tilo Burghardt Gary Burnett Andreas Butz
EPFL, Switzerland University of Michigan, USA Carnegie Mellon University, USA University of Arizona, USA AT&T, USA Intel Research, USA University of Bristol, UK Intel Research, USA University of Toronto, Canada Sony Computer Science Laboratories, Japan University of Birmingham, UK Fuji Xerox Palo Alto Laboratory, USA University of Cambridge, UK University of Toronto, Canada Carnegie Mellon University, USA RWTH Aachen University, Germany Politecnico di Milano, Italy University of Glasgow, UK IBM Research, USA Nokia Research Center, Finland NEC Europe Ltd., Germany Universität Stuttgart, Germany Intel Research, USA University College London, UK University of Karlsruhe, Germany University of Glasgow, UK University of Nottingham, UK University of Zurich, Switzerland ETH Zurich, Switzerland HP Laboratories, USA University of Toronto, Canada University of Cambridge, UK University of Southern California, USA Sara Bly Consulting, USA University at Buffalo, USA University of Washington, USA Goldsmiths University of London, UK Smart Technologies, Canada Université Pierre et Marie Curie, France Amber Light, UK University of Glasgow, UK University of Geneva, Switzerland University of Bristol, UK University of Nottingham, UK University of Munich, Germany
IX
X
Organization
Jean Camp Neill Campbell Roy Campbell Scott Carter Michele Chang Yu-Chung Cheng Adrian Cheok Keith Cheverst Patrick Chiu Elizabeth Churchill Angus Clark Siobhán Clarke Kay Connelly George Coulouris Scott Counts Michael Covington Andy Crabtree Lorrie Cranor Allen Cypher Anwis Das Alan Dearle Christian Decker Corsin Decurtins Anind Dey Alan Dix Kelly Dobson Simon Dobson Gavin Doherty Jim Dowling Nicolas Ducheneaut Paul Duff Schahram Dustdar Keith Edwards Stefan Egglestone Rana el Kaliouby Christian Elting Mattias Esbjörnsson Jérôme Euzenat Pascal Felber Danyel Fisher Kenneth Fishkin Geraldine Fitzpatrick James Fogarty Jodi Forlizzi Brooke Foucault Jon Froehlich Kentaro Fukuchi
Indiana University, USA University of Bristol, UK University of Illinois at Urbana-Champaign, USA University of California, Berkeley, USA ReD Associates, Denmark UCSD, USA Nanyang University, Singapore Lancaster University, UK Fuji Xerox Palo Alto Laboratory, USA Palo Alto Research Center (PARC), USA University of Bristol, UK Trinity College Dublin, Ireland Indiana University, USA University of Cambridge, UK Microsoft Research, USA Intel Corporation, USA University of Nottingham, UK Carnegie Mellon University, USA IBM Research, USA Rice University, USA University of St. Andrews, UK University of Karlsruhe, Germany ETH Zurich, Switzerland Carnegie Mellon University, USA Lancaster University, UK Massachusetts Institute of Technology, USA UCD Dublin, Ireland Trinity College Dublin, Ireland MySQL, Sweden Palo Alto Research Center (PARC), USA University of Bristol, UK Vienna University of Technology, Austria Georgia Institute of Technology, USA University of Nottingham, UK Massachusetts Institute of Technology, USA European Media Lab GmbH, Germany Interactive Institute, Sweden INRIA Rhône-Alpes, France University of Neuchatel, Switzerland Microsoft Research, USA Google, USA University of Sussex, UK Carnegie Mellon University, USA Carnegie Mellon University, USA Intel, USA University of Washington, USA UEC, Japan
Organization
Masaaki Fukumoto Krzysztof Gajos Areti Galani Eric Garcia Anurag Garg William Gaver Erik Geelhoed Hans Gellersen Andreas Girgensohn Patrick Goddi Elizabeth Goodman Paul Grace Tom Gross Michael Grossniklaus Dennis Groth Dirk Haehnel Joseph Hall Paul David Hankes Drielsma Vicki Hanson Robert Harle Eric Harris Beverly Harrison William Harrison Michael Harrison Lonnie Harvel Gillian Hayes Mike Hazas Cina Hazegh William Hazlewood Jong Hee Kang Urs Hengartner Jeffrey Hightower Tad Hirsch Larissa Hjorth Paul Holleis Jason Hong Yih-Chun Hu Elaine Huang Richard Hull Giovanni Iachello Liviu Iftode Takeo Igarashi Masahiko Inakage Stephen Intille Jun Ito Shahram Izadi Amit Jardosh
XI
NTT DoCoMo Multimedia Laboratories, Japan University of Washington, USA University of Newcastle, UK University of Washington, USA University of Trento, Italy Goldsmiths College, UK HP Labs, UK Lancaster University, UK FX Palo Alto Laboratory, USA HP Labs, USA Intel Corporation, USA Lancaster University, UK Bauhaus-University Weimar, Germany ETH Zurich, Switzerland Indiana University, USA Intel Research Seattle, USA University of California Berkeley, USA ETH Zurich, Switzerland IBM, USA University of Cambridge, UK University of Sussex, UK Intel Research Seattle, USA Trinity College Dublin, Ireland University of Newcastle upon Tyne, UK Georgia Gwinnett College, USA Georgia Institute of Technology, USA Lancaster University, UK University of California, Irvine, USA Indiana University, USA University of Washington, USA University of Waterloo, Canada Intel Research Seattle, USA Massachusetts Institute of Technology, USA RMIT, Australia University of Munich, Germany Carnegie Mellon University, USA University of Illinois at Urbana-Champaign, USA Georgia Institute of Technology, USA Hewlett-Packard Laboratories, UK Georgia Institute of Technology, USA Rutgers University, USA University of Tokyo/JST PRESTO, Japan Keio University, Japan Massachusetts Institute of Technology, USA Sony, Japan Microsoft Research, UK University of California, Santa Barbara, USA
XII
Organization
Zhu Ji Matthias Joest Wouter Joosen Oskar Juhlin Susanne Kaiser Achilles Kameas Epaminondas Kapetanios Victor Kaptelinin Karrie Karahalios Fumihiro Kato Fumitoshi Kato Henry Kautz Nobuo Kawaguti Tatsuyuki Kawamura Joseph Kaye Julie Kientz Marc-Olivier Killijian Tim Kindberg David Kirk Thomas Kirste Scott Klemmer Ralph Kling Kenji Kohiyama Naohiko Kohtake Hideki Koike Yasuyuki Kono Panu Korpipää Ilpo Koskinen Matthias Kranz Matthias Krauss Albert Krohn Reto Krummenacher Antonio Krüger Thomas Kunz Andrew Kurtz Andrew Ladd Horacio Lagar Cavilla Jennifer Lai Spyros Lalis Marc Langheinrich Jonathan Lester Julie Letchner Lin Liao Dave Lillethun Gitte Lindgaard Daria Loi Rose Luckin
University of Maryland, College Park, USA EML, Germany University of Leuven, Belgium Interactive Institute, Sweden University of Geneva, Switzerland Computer Technology Institute, Greece University of Westminster, UK Umea University, Sweden University of Illinois, Urbana Champaign, USA Keio University, Japan Keio University, Japan University of Washington, USA Nagoya University, Japan NAIST, Japan Cornell University, USA Georgia Institute of Technology, USA LAAS-CNRS, France Hewlett-Packard Laboratories, UK University of Nottingham, UK University of Rostock, Germany Stanford University, USA Intel, USA Keio University, Japan Keio University/JAXA, Japan University of Electro-Communications, Japan NAIST, Japan VTT Electronics, Finland University of Art and Design Helsinki, Finland University of Munich, Germany University of Bremen, Germany University of Karlsruhe, Germany DERI Innsbruck, Austria University of Münster, Germany Carleton University, Canada Indiana University, USA Rice University, USA University of Toronto, Canada IBM, USA University of Thessaly, Greece ETH Zurich, Switzerland University of Washington, USA University of Washington, USA University of Washington, USA Georgia Institute of Technology, USA Carleton University, Canada RMIT University, Australia Institute of Education, UK
Organization
Jay Lundell Evan Magill Scott Mainwaring Lena Mamykina Jennifer Mankoff Panos Markopoulos Natalia Marmasse Herve Martin Kenji Mase Maristella Matera Friedemann Mattern Walterio Mayol-Cuevas Rene Mayrhofer Michael McCarthy David McDonald René Meier Brian Meyers Danius Michaelides Masateru Minami April Mitchell Kimaya Mittal Jun Miyazaki Michael Mock Iqbal Mohomed Margaret Morris Mor Naaman Kris Nagel Yasuto Nakanishi Junya Nakata Andronikos Nedos Steve Neely Carman Neustaedter Mark W. Newman Jeff Nichols Robert Nideffer Chris Nippert-Eng Yoshifumi Nishida Toyoaki Nishida Paddy Nixon Kenton O'Hara Yoosoo Oh Gérald Oster Veljo Otsason
XIII
Intel Corporation, USA University of Stirling, UK Intel Research, USA Siemens Corporate Research and Georgia Institute of Technology, USA Carnegie Mellon University, USA Eindhoven University of Technology, The Netherlands IBM Research, Israel Université Joseph Fourier, France Nagoya University, Japan Politecnico di Milano, Italy ETH Zurich, Switzerland University of Bristol, UK Lancaster University, UK University of Bristol, UK University of Washington, USA Trinity College Dublin, Ireland Microsoft Research, USA University of Southampton, UK Shibaura Institute of Technology, Japan Hewlett-Packard, USA University of California Santa Barbara, USA Fuji Xerox, Japan Fraunhofer Institute for Autonomous Intelligent Systems, Germany University of Toronto, Canada Intel Corporation, USA Yahoo! Research Berkeley, USA Georgia Institute of Technology, USA Keio University, Japan Japan Advanced Institute of Science and Technology, Japan Trinity College Dublin, Ireland UCD Dublin, Ireland University of Calgary, Canada Palo Alto Research Center (PARC), USA Carnegie Mellon, USA University of California, Irvine, USA IIT, USA Sony Computer Science Laboratories, Japan Kyoto University, Japan University College Dublin, Ireland HP Laboratories, UK GIST, Korea ETH Zurich, Switzerland University of Tartu, Estonia
XIV
Organization
Raju Pandey Joseph Paradiso Shwetak Patel Sameer Patil Donald Patterson Simon Penny Trevor Pering Matthai Philipose Claudio Pinhanez Kevin Ponto Robert Porzel Ivan Poupyrev Aaron Quigley Rakhi Rajani Krishna Ramachandran Umakishore Ramachandran Cliff Randell Pierre-Guillaume Raverdy Madhu Reddy Stuart Reeves Josephine Reid Patrick Reignier Derek Reilly Heather Richter Thomas Rist Toni Robertson Tom Rodden Jennifer Rode Manuel Roman Troy Ronda Åsa Rudström Enrico Rukzio Lee Sanggoog Bernt Schiele Bill Schilit Chris Schmandt Holger Schnädelbach Odilo Schoch Jean Scholtz Heiko Schuldt Jamie Schulte Jamieson Schulte Gerrhard Schwabe James Scott Jean-Marc Seigneur Abigail Sellen
University of California, Davis, USA Massachusetts Institute of Technology, USA Georgia Institute of Technology, USA University of California, Irvine, USA University of California, Irvine, USA University of California, Irvine, USA Intel Research, USA Intel Research Seattle, USA IBM T.J. Watson Research Center, USA University of California, Irvine, USA University of Bremen, Germany Sony CSL, Japan University College Dublin, Ireland Central Saint Martins College of Art and Design, UK University of California, Santa Barbara, USA Georgia Institute of Technology, USA University of Bristol, UK INRIA, France Penn State University, USA University of Nottingham, UK HP Labs, UK University Joseph Fourier/INRIA, France Dalhousie University, Canada University of North Carolina at Charlotte, USA FH Augsburg, Germany University of Technology Sydney, Australia University of Nottingham, UK University of California, Irvine, USA NTT DoCoMo Labs USA, USA University of Toronto, Canada Swedish Institute of Computer Science, Sweden University of Munich, Germany SAIT, Korea TU Darmstadt, Germany Intel Research, USA Massachusetts Institute of Technolgy, USA University of Nottingham, UK Architect, Switzerland NIST, USA University of Basel, Switzerland Stanford University, USA Stanford University, USA University of Zurich, Switzerland Intel Research Cambridge, UK University of Geneva, Switzerland Microsoft Research, UK
Organization
Shondip Sen Aline Senart Phoebe Sengers Steven Shafer Irfan Sheriff John Sherry Beat Signer Itiro Siio Brooke Singer Morris Sloman Alan Smeaton Ian Smith Joshua Smith Timothy Sohn Xiang Song João Pedro Sousa Flavia Sparacino Alexandre de Spindler Cormac Sreenan Danäe Stanton Fraser Thad Starner John Stasko Anthony Steed Kilian Stoffel Erik Stolterman Thomas Strang Mark Stringer Jing Su Yasuyuki Sumi Jay Summet Yan Sun Martin Svensson Shigeru Tajima Kaz Takashio Desney Tan Hiroya Tanaka Karen Tang Alex Taylor Lucia Terrenghi Michael Terry Sotirios Terzis Hiroaki Tobita Hideyuki Tokuda Niraj Tolia Kentaro Toyama Quan Tran Mike Twidale
XV
CSIRO, Australia Trinity College Dublin, Ireland Cornell University, USA Microsoft Research, USA UC Santa Barbara, USA Intel Corporation, USA ETH Zurich, Switzerland Ochanomizu University, Japan SUNY Purchase, USA Imperial College London, UK Dublin City University, Ireland Intel Research Seattle, USA Intel Research Seattle, USA University of California, San Diego, USA Georgia Institute of Technology, USA Carnegie Mellon University, USA Sensing Places/MIT, USA ETH Zurich, Switzerland University College Cork, Ireland University of Bath, UK Georgia Institute of Technology, USA Georgia Institute of Technology, USA University College London, UK University of Neuchatel, Switzerland Indiana University, USA DLR, Germany University of Sussex, UK University of Toronto, Canada Kyoto University, Japan Georgia Institute of Technology, USA University of Rhode Island, USA Swedish Institute of Computer Science, Sweden Sony CSL, Japan Keio University, Japan Microsoft Research, USA Keio University, Japan Carnegie Mellon University, USA Microsoft Research Cambridge, UK Microsoft Research Cambridge, UK University of Waterloo, Canada University of Strathclyde, UK Sony CSL, Japan Keio University, Japan Carnegie Mellon University, USA Microsoft Research, India Georgia Institute of Technology, USA University of Illinois, USA
XVI
Organization
Nancy Van House Kristof Van Laerhoven Alex Varshavsky Juergen Vogel Amy Voida Roy Want Andy Ward Mark Weal Stefan Weber Matt Welsh Andy Wilson Woontack Woo Kun Yang Alex Zafiroglu
University of California Berkeley, USA Lancaster University, UK University of Toronto, Canada European Media Laboratory, Germany Georgia Institute of Technology, USA Intel Research, USA Ubisense, UK University of Southampton, UK Trinity College Dublin, Ireland Harvard University, USA Microsoft Research, USA GIST, Korea University of Essex, UK Intel, USA
Table of Contents
A Quantitative Method for Revealing and Comparing Places in the Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ryan Aipperspach, Tye Rattenbury, Allison Woodruff, John Canny
1
Principles of Smart Home Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Scott Davidoff, Min Kyung Lee, Charles Yiu, John Zimmerman, Anind K. Dey
19
Historical Analysis: Using the Past to Design the Future . . . . . . . . . . . . . . . Susan Wyche, Phoebe Sengers, Rebecca E. Grinter
35
Extending Authoring Tools for Location-Aware Applications with an Infrastructure Visualization Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Leif Oppermann, Gregor Broll, Mauricio Capra, Steve Benford
52
Automated Generation of Basic Custom Sensor-Based Embedded Computing Systems Guided by End-User Optimization Criteria . . . . . . . . . Susan Lysecky, Frank Vahid
69
An Experimental Comparison of Physical Mobile Interaction Techniques: Touching, Pointing and Scanning . . . . . . . . . . . . . . . . . . . . . . . . . Enrico Rukzio, Karin Leichtenstern, Vic Callaghan, Paul Holleis, Albrecht Schmidt, Jeannette Chin
87
An Exploratory Study of How Older Women Use Mobile Phones . . . . . . . . 105 Sri Kurniawan Farther Than You May Think: An Empirical Investigation of the Proximity of Users to Their Mobile Phones . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Shwetak N. Patel, Julie A. Kientz, Gillian R. Hayes, Sooraj Bhat, Gregory D. Abowd No More SMS from Jesus: Ubicomp, Religion and Techno-spiritual Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Genevieve Bell Scribe4Me: Evaluating a Mobile Sound Transcription Tool for the Deaf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Tara Matthews, Scott Carter, Carol Pai, Janette Fong, Jennifer Mankoff
XVIII
Table of Contents
SenseCam: A Retrospective Memory Aid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 Steve Hodges, Lyndsay Williams, Emma Berry, Shahram Izadi, James Srinivasan, Alex Butler, Gavin Smyth, Narinder Kapur, Ken Wood Development of a Privacy Addendum for Open Source Licenses: Value Sensitive Design in Industry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Batya Friedman, Ian Smith, Peter H. Kahn Jr., Sunny Consolvo, Jaina Selawski Mobility Detection Using Everyday GSM Traces . . . . . . . . . . . . . . . . . . . . . . 212 Timothy Sohn, Alex Varshavsky, Anthony LaMarca, Mike Y. Chen, Tanzeem Choudhury, Ian Smith, Sunny Consolvo, Jeffrey Hightower, William G. Griswold, Eyal de Lara Practical Metropolitan-Scale Positioning for GSM Phones . . . . . . . . . . . . . . 225 Mike Y. Chen, Timothy Sohn, Dmitri Chmelev, Dirk Haehnel, Jeffrey Hightower, Jeff Hughes, Anthony LaMarca, Fred Potter, Ian Smith, Alex Varshavsky Predestination: Inferring Destinations from Partial Trajectories . . . . . . . . . 243 John Krumm, Eric Horvitz Fish’n’Steps: Encouraging Physical Activity with an Interactive Computer Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 James J. Lin, Lena Mamykina, Silvia Lindtner, Gregory Delajoux, Henry B. Strub Hitchers: Designing for Cellular Positioning . . . . . . . . . . . . . . . . . . . . . . . . . . 279 Adam Drozd, Steve Benford, Nick Tandavanitj, Michael Wright, Alan Chamberlain Embedding Behavior Modification Strategies into a Consumer Electronic Device: A Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 Jason Nawyn, Stephen S. Intille, Kent Larson Instrumenting the City: Developing Methods for Observing and Understanding the Digital Cityscape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 Eamonn O’Neill, Vassilis Kostakos, Tim Kindberg, Ava Fatah gen. Schiek, Alan Penn, Dana¨e Stanton Fraser, Tim Jones Voting with Your Feet: An Investigative Study of the Relationship Between Place Visit Behavior and Preference . . . . . . . . . . . . . . . . . . . . . . . . . 333 Jon Froehlich, Mike Y. Chen, Ian E. Smith, Fred Potter
Table of Contents
XIX
Lo-Fi Matchmaking: A Study of Social Pairing for Backpackers . . . . . . . . . 351 Jeff Axup, Stephen Viller, Ian MacColl, Roslyn Cooper Experiences from Real-World Deployment of Context-Aware Technologies in a Hospital Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369 Jakob E. Bardram, Thomas R. Hansen, Martin Mogensen, Mads Soegaard Doing Community: Co-construction of Meaning and Use with Interactive Information Kiosks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387 Tom Hope, Masahiro Hamasaki, Yutaka Matsuo, Yoshiyuki Nakamura, Noriyuki Fujimura, Takuichi Nishimura Moving on from Weiser’s Vision of Calm Computing: Engaging UbiComp Experiences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 Yvonne Rogers Ferret: RFID Localization for Pervasive Multimedia . . . . . . . . . . . . . . . . . . . 422 Xiaotao Liu, Mark D. Corner, Prashant Shenoy PowerLine Positioning: A Practical Sub-Room-Level Indoor Location System for Domestic Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441 Shwetak N. Patel, Khai N. Truong, Gregory D. Abowd UbiREAL: Realistic Smartspace Simulator for Systematic Testing . . . . . . . 459 Hiroshi Nishikawa, Shinya Yamamoto, Morihiko Tamai, Kouji Nishigaki, Tomoya Kitani, Naoki Shibata, Keiichi Yasumoto, Minoru Ito Instant Matchmaking: Simple and Secure Integrated Ubiquitous Computing Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477 D.K. Smetters, Dirk Balfanz, Glenn Durfee, Trevor F. Smith, Kyung-Hee Lee A Wirelessly-Powered Platform for Sensing and Computation . . . . . . . . . . . 495 Joshua R. Smith, Alanson P. Sample, Pauline S. Powledge, Sumit Roy, Alexander Mamishev Automated Application-Specific Tuning of Parameterized Sensor-Based Embedded System Building Blocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507 Susan Lysecky, Frank Vahid Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525
A Quantitative Method for Revealing and Comparing Places in the Home Ryan Aipperspach1,2 , Tye Rattenbury1 , Allison Woodruff2 , and John Canny1 1
Berkeley Institute of Design, Computer Science Division, University of California, Berkeley, USA {ryanaip, rattenbt, jfc}@cs.berkeley.edu 2 Intel Research Berkeley, USA
[email protected]
Abstract. Increasing availability of sensor-based location traces for individuals, combined with the goal of better understanding user context, has resulted in a recent emphasis on algorithms for automatically extracting users’ significant places from location data. Place-finding can be characterized by two sub-problems, (1) finding significant locations, and (2) assigning semantic labels to those locations (the problem of “moving from location to place”) [8]. Existing algorithms focus on the first sub-problem and on finding city-level locations. We use a principled approach in adapting Gaussian Mixture Models (GMMs) to provide a first solution for finding significant places within the home, based on the first set of long-term, precise location data collected from several homes. We also present a novel metric for quantifying the similarity between places, which has the potential to assign semantic labels to places by comparing them to a library of known places. We discuss several implications of these new techniques for the design of Ubicomp systems.
1
Introduction
The importance of understanding users’ contexts is widely accepted in the Ubicomp community. Understanding context can help systems behave appropriately in a variety of situations and integrate more seamlessly into everyday life. Recently, the increased availability of position sensing technologies has resulted in a focus on finding significant places based on traces of a user’s position [3,8,9,10,12,13,23]. Places are stable contexts within which social practices are situated [7], playing a significant role in broader definitions of context. Hightower et al. [8] list two problems in developing algorithms for finding significant places for individual users: (1) finding significant locations and (2) assigning semantics or names to those significant locations (“moving from location to place”). The primary contributions of this work are a solution to the first problem in the domain of the home and the introduction of a novel method for addressing the second problem, based on a measure for comparing the similarity between places. Most existing algorithms for finding significant locations target building-sized locations within city-scale areas, since most available location data sets exist at a building-scale resolution. We have collected the first P. Dourish and A. Friday (Eds.): Ubicomp 2006, LNCS 4206, pp. 1–18, 2006. c Springer-Verlag Berlin Heidelberg 2006
2
R. Aipperspach et al.
long-term, high-precision home location traces [2], enabling us to begin developing place-finding algorithms for the home. The distribution of places within the home differs markedly from the distribution at city-scale, motivating the development of new techniques for finding places in the home. To validate our algorithm choice, we discuss the places it finds in several homes. Most existing place-finding algorithms focus on the problem of finding significant locations rather than on assigning semantic labels to the places they find. Instead, they rely on geocoding repositories (e.g. Microsoft MapPoint) or on hand-labeling by people [26] to apply labels. Hightower et al. [8] also note that geocoded information, like raw coordinates, “does not correspond to someone’s mental model of their personal routine nor to the terminology they use when discussing the places they go.” We provide a step toward solving this problem. Concretely, we describe a method for measuring the similarity between different places that should enable automatic labeling of new places. By measuring the similarity between places, it is possible to find recurring structures that reveal the shared types of place within a particular culture (e.g., bedrooms in one home should be similar to bedrooms in other homes within the same culture). While it is difficult to learn the specific details of a place, we claim that the shared attributes of places can be automatically detected. Using these attributes, new places can be compared to a library of known places resulting in an (indirect) understanding of the new place. The limited availability of location traces from homes makes it difficult to build and test a library of similar places. In this work, we make use of the largest high-precision home location data set currently available to highlight the potential of such techniques. Our similarity metric is based on emergent properties of places resulting from people’s patterns of behavior. Consequently, we claim that it satisfies some of the notions of context constructed through actions, rather than imposed via a fixed structure. In section 2 we describe existing definitions of place and argue for the importance of measuring the similarity between different places. In section 3 we discuss related work. Section 4 describes our algorithm and the results of applying it to a set of location data collected in several homes. Finally, section 5 describes potential design implications and section 6 concludes and provides directions for future work.
2 2.1
A Definition of Place Existing Definitions of Place
The definition of place that we use in this paper is most aligned with the work of Harrison and Dourish [7]. They observe that designers of interactive systems and virtual environments often build notions of “space” – the three dimensional environment in which objects interact and events occur – into their systems with the goal of facilitating interaction by framing users’ behavior in the same way that it is framed in the physical world (i.e., by relying on spatial metaphors). However, they argue that behavior in the physical world is framed not only by space but also by the actions normally performed in that space.
A Quantitative Method for Revealing and Comparing Places in the Home
3
Harrison and Dourish define place as “a space which is invested with understandings of behavioral appropriateness, cultural expectations, and so forth,” saying that “we are located in space, but we act in place”. That is, a place is useful for framing behavior because a user can make use of culturally embedded knowledge to determine what activities are appropriate for particular places. Whether walking into a church, a post office, a living room, or a kitchen, people use cues based both on both the architecture and configuration of the space and on the observable behaviors of others to understand what kind of place they are entering and how they should act. According to this definition, a place’s meaning is jointly constructed by the physical structure of a location and the activities that regularly occur there. This definition of place is a useful definition for designers of Ubicomp systems. Many recognize the need for determining users’ context to better support their activities through appropriate and relevant actions [4]. They look for cues that determine appropriate behavior in particular situations, which is exactly what Harrison and Dourish suggest that “place” encodes. Interestingly, Harrison and Dourish’s definition of place is similar to definitions in other fields, such as place settings in environmental society [27]. However, a discussion of these similarities is beyond the scope of this paper. 2.2
Working Definition of Place
From the existing definitions of place, we highlight six attributes of place along two different dimensions: – Attributes relating to time-invariant physical configuration • Position, a bounded region in three-dimensional, physical space • Physical structure, the architectural components of a space • Object co-presence, the consistently present artifacts in a space (e.g. furniture) – Attributes relating to the historical behavior within a place • Time, the distribution of time spent in a particular place • Stable patterns of behavior, the set of activities that occur in a place • Person co-presence, the patterns of presence of other people within a place Drawing on activity theory [11], we argue that common patterns of interaction between people, objects, and locations (as captured in the six attributes above) embody the set of common behaviors characteristic to each place as suggested by Harrison and Dourish. The categorization of these attributes into two dimensions is a matter of conceptual convenience. In general, the physical configuration attributes and the historical behavior attributes are deeply intertwined. (Notions like “affordance” attempt to capture this connection.) In fact, their interconnection can be considered both a consequence of and a generator of social culture [7]. Practically, however, the distinction between physical and behavioral attributes prescribes a basis for the algorithms we present later.
4
R. Aipperspach et al.
Shared Notions of Place. Because places are intertwined with culture, there exist common notions of place within specific cultures (e.g., “kitchen”, “meeting room”, “bedroom”). This fact is captured in the following argument: because strangers can enter spaces and, at a glance, assess both the names of the places and the activities likely to occur there, there must exist a shared notion of types of places and the appropriate actions for each of those types. (Of course, strangers may miss more idiosyncratic activities, e.g. eating cereal for breakfast while standing in the kitchen). As a result of these shared cultural definitions of place, it should be possible to measure the similarity between particular “instantiations” of those places in individual homes – e.g., bedrooms should appear similar to other bedrooms, and kitchens should appear similar to other kitchens. In this work, we present an initial metric for comparing places in the home. We utilize the fact that different types of places tend to support different typical activities, basing our comparison metric on the patterns of use in the spaces we compare. The correlation between types of place and typical activity patterns is supported by previous work. For example, in a study of 100 Brazilian homes, Monteiro [15] found a correlation between the types of activities carried out in different places and the relative distance of each place from the entrance of the home. In section 5 we discuss several implications of a system for measuring shared notions of place.
3
Previous Work
Recently, a number of authors within the ubiquitous computing community have considered the problem of finding significant locations based on a person’s location history. We describe some of the existing methods below. Most algorithms focus on finding significant locations at the city-scale. In agreement with previous studies of place within the home [18,28], we argue that places in the home are significantly different from larger-scale places. Consequently, different methods are required for finding places in the home. 3.1
Existing Place-Finding Algorithms
Existing place-finding algorithms can be divided into two classes, geometry-based and fingerprint-based [8]. Examples of geometry-based algorithms include those by Marmasse et al. [13], Ashbrook and Starner [3], Liao et al. [12], and Kang et al. [9]. Each of these algorithms takes a history of locations from a single person (e.g. from GPS readers) and finds locations where the person spends significant periods of time. The algorithms vary based on the type of sensor data they use and on the specific clustering algorithm they use. For example, Ashbrook and Starner search GPS data for locations where position changed slowly or readings dropped out completely, indicating times when the user was not moving or was inside a building. They apply a hierarchical variant of k-means to cluster the candidate locations. Kang et al. use sensor-agnostic location traces to determine
A Quantitative Method for Revealing and Comparing Places in the Home
5
when users remained within a (specified radius) “space-sized” region for at least some minimum amount of time to find significant locations. Examples of fingerprint-based algorithms include those by Trevisani and Vitaletti [23], Laasonen et al. [10], and Hightower et al. [8]. These algorithms calculate the “fingerprint” of a person’s location (e.g., a vector of currently visible cell towers and wireless access points), find recurring fingerprints where the person spends a significant amount of time, and return a list of places based on significant fingerprints. These algorithms make the assumption that there exists a stable mapping between the fingerprints they find and the three-dimensional physical space that humans inhabit. Existing place-finding algorithms find significant locations for different individuals, but they do not attempt to label types of places or to compare places across different people. For example, existing algorithms do not address the problem of automatically labeling places as belonging to the class “work” or “home” or of determining that two people’s “work” places are similar unless those places are in the same physical location. Without knowledge of these distinctions, it is more appropriate to treat these methods as finding significant locations, not places. 3.2
Place in the Home
There has been little work done in determining significant places in the home based on location histories, primarily because high-resolution (sub-meter accuracy) location traces for residents within the home have not been available. However, several observational studies have explored the concept of place in the home. Oswald et al. [18] found that elders tend to spend much of their time in favored places. These centralized locations, which often provide easy access to essential (e.g., medication and food) and important objects (e.g., pictures and diaries) are crucial in allowing elders to remain autonomous as moving about their homes becomes more difficult. Based on qualitative studies in a range of homes, we found similar use of favored places within a more general population (see [28]). Like Oswald, we found that study participants spent the majority of their time in favored places. However, we found that participants also visited a range of kinetic places, or places used for shorter duration activities often involving physical manipulations, such as the bathroom, a mirror for doing one’s hair in the morning, or a kitchen counter used to make sandwiches for lunch. The work cited above suggests that places in the home exist at a variety of levels, ranging from small places like a single mirror to entire rooms. In this work, we are interested in finding places that are large enough to capture generic usage patterns across people and homes (e.g. cooking, sleeping), yet small enough to reveal differences in these patterns. Places satisfying these constraints should be comparable between people and be semantically meaningful. Often, different sized places exist even at the same semantic level (e.g., living rooms and bathrooms). The range of place sizes makes it difficult for existing algorithms that assume places at a single level have similar sizes (e.g. [9,13]).
6
R. Aipperspach et al.
Additionally, places in the home are much closer together in relation to their size than are places at the building scale (consider the distance between work and home, compared to the difference between two rooms in a home). The small size and close proximity of places in the home means that the separation between places is close to the scale of errors in location sensor readings, even when using very accurate sensors. (The size of places and typical error bounds for our sensor data are both on the order of meters.) As a result, a more probabilistic view of places is necessary than is taken in many existing place-finding algorithms. For example, we applied Kang et al.’s place-finding algorithm to the data for each participant (with a place-size of 2 meters and a time window of 2 minutes). Because place sizes and noise in the home are at similar scales, readings taken while a user remains in a single place frequently fall outside the bounds of that place. Of data while users where home and awake, the Kang algorithm labeled an average of 45% of points as “moving between places” and 55% as belonging to a specific place. However, qualitative data analysis and interviews based on our data found that most people spend the majority of their time in a small number of favored places and little time moving around [2], suggesting this large portion of unlabeled data is an undesired artifact of the algorithm. Consequently, we developed a new geometry-based algorithm for finding significant places in the home. This algorithm must work for environments where the ratio of place size to inter-place distance is high. Therefore, the algorithm must be capable of finding places without relying on the detection, either explicitly or implicitly, of movement paths between places.
4
Finding Place in the Home
As discussed earlier, we treat place as an emergent quality of the physical structure of a location and of the routine activities that people carry out there. Instead of directly processing the rich details intrinsic to both of these dimensions, we use simple proxies. For physical structure we use the set of locations that people have occupied. For action and activity we use the first-order temporal patterns in each person’s position data. Both proxies are common input features for existing place-finding algorithms (e.g. [8,9]), and the tunable parameters used by our algorithm are similar to those used by other algorithms. However, while these algorithms are reliable and accurate in their intended domains (city-scale areas), they do not work at the scale of places within a home (see section 3.2). Like other place-finding algorithms, our main algorithm described in section 4.2 identifies significant locations rather than places. However, the similarity metric described in section 4.3 considers the activity patterns that differentiate places from locations, helping to address the problem of moving from location to place. While the location data in our data set does not capture the full range of attributes of place described in section 2.2, the spatial and temporal patterns provide a good starting place for such analysis. As richer data sets become available, the algorithm can be expanded to include additional attributes of place.
A Quantitative Method for Revealing and Comparing Places in the Home
4.1
7
Data Set
There are currently few data sets providing accurate in-home location information. Typical examples are Rowan and Mynatt [20], who collected one year of data from a single woman using in-floor pressure sensors, providing meter-level accurate location traces, and Tapia et al. [22], who collected 2 weeks of data from 2 single-person apartments using ∼80 binary state-change sensors with known locations, providing sub-room level accuracy. In this study, we make use of highly-precise (sub-meter) sensor data which we collected from 3 homes using ultra wideband (UWB) sensors from Ubisense, Ltd. [25]. To our knowledge, this is the first long-term, high-precision set of location data collected in the home. The data set includes location traces for each person (7 total) in the study, over a time period ranging from three weeks to several months in each home. Each data point is a 4-tuple containing: (1) x position, (2) y position, (3) time, and (4) the duration of time that the measurement covers. For more information about the data collection process, see [2]. The participants from each home are described below: 1. Home 1: Brad and Jacqueline. A one-bedroom apartment occupied by Brad and Jacqueline, two graduate students from Australia. 2. Home 2: Jack and Margaret. A one-bedroom apartment occupied by Jack and Margaret, a recently married couple from England. 3. Home 3: Sierra, Gaby, and Cathy. A three-bedroom, single story home occupied by Sierra and Gaby, a female couple, and their roommate, Cathy. 4.2
An Algorithm for Detecting Significant Locations in the Home
Before describing the details of our algorithm, we introduce the following terminology as a matter of convenience: candidate place: a bounded region proposed as a likely place merge: the process of combining two candidate places into one candidate place Candidate places are the primary data element that our algorithm uses. Merging is the primary action performed on these elements. Algorithm Description. The algorithm we use in this work is an agglomerative clustering version of the standard Gaussian Mixture Model (GMM). Our choice was influenced by the fact that we needed to find robust clusters with respect to noisy position readings (which led us toward GMMs) and the fact that we did not want to hand-code the number of places to find (which led us toward an agglomerative process). To ground our discussion, we first describe the standard GMM algorithm. The GMM algorithm is an iterative process for fitting Gaussian probability distributions to data, consisting of two sub-steps. The pseudo-code is as follows:
8
R. Aipperspach et al.
1. choose a fixed number of clusters, M 2. initialize these clusters to cover different portions of the data 3. loop until convergence (a) for each data point and each cluster, calculate the probability that the cluster generated the data point (b) for each cluster, calculate the best Gaussian parameters to explain the data that is strongly associated with it Usually, the initialization is performed using k-means. We refer to the Gaussian clusters in our algorithm as candidate places. Each candidate place is a three dimensional Gaussian distribution. Two of the dimensions correspond to position and one to time.1 Our algorithm augments the basic iterative architecture of the standard GMM with one additional sub-step: an agglomerative step that merges clusters meeting various merging criterion. This additional step occurs after step 3 in the pseudocode above. We employ three different merging criteria during three phases of our algorithm. At a high level, these phases are (1) merge data points adjacent in space and time into “visits”, (2) merge temporally adjacent visits that do not individually meet a minimum duration, (3) merge spatially similar visits into significant locations. Each of these merging steps has an associated parameter that relies on a measure of closeness between candidate places. In describing the algorithm below, we will introduce these parameters, the values we used for the parameters, and the measure of closeness that they rely on. The algorithm starts by creating one candidate place for each data point. Because each candidate place is initially generated from one data point, the associated Gaussian distribution will have zero variance in the two position dimensions. To combat this issue, we require all candidate places to have a minimum variance along each dimension. Although our position data has sub-meter accuracy, the sensors can give sporadic noise readings that are several meters away. Consequently, we set the minimum variance to be 1.0 for each dimension of the Gaussian distribution. The choice of minimum variance determines the minimum size of a candidate place. For our choice of parameters, this means that no candidate place can be smaller than about 1 meter in radius. For our data, this is a reasonable constraint. Obviously, the minimum variance values will have to be adjusted for data covering different temporal and spatial scales. Now, we discuss the three phases of the algorithm and their associated merging steps. In the first phase, we use the following heuristic for merging candidate places: if two candidate places are temporally adjacent and cover similar positions, they are likely covering data from the same place and should be merged into one visit. “Similar spaces” is captured in a parameter, λ1 . λ1 is a thresholdon the 1
To simplify the calculation performed by our algorithm, we treat each dimension in the Gaussian distribution as independent.
A Quantitative Method for Revealing and Comparing Places in the Home
9
joint KL-divergence2 between the two candidate place probability distributions. This heuristic is utilized in most existing place-finding algorithms, used to determine when two location readings come from the same place, and is similarly enforced using a threshold on some distance metric between candidate places. In the second phase, we continue to apply the heuristic used in phase one and also apply the following heuristic: a place visit should explain a minimum duration of data. We encode this heuristic in the parameter τ . Again, many place-finding algorithms encode this heuristic in a parameter as well. Generally, it is described as the minimum time that a person must spend in a candidate place before it is treated as a true place. Finally, in the third phase we apply the following merging heuristic: if two candidate place visits cover similar spaces but are not temporally adjacent, they are likely encoding two different visits to the same place and should be merged. This heuristic is captured in the parameter λ2 , which, like λ1 from phase 1, is a threshold on the joint KL-divergence between candidate places. Many existing place-finding algorithms also utilize this heuristic, and there has even been discussion on different metrics for capturing it numerically [8]. There is an important distinction between this merging step and the previous two. When merging two candidate places, instead of averaging the distributions for each place in the time dimension, we create a new candidate place whose distribution is the union of the two candidate place’s distributions. As a result, candidate places take on a slightly new form during phase three: they have a single two-dimensional Gaussian distribution of x and y positions but potentially many Gaussian distributions over time. This allows each candidate place to account for many disjoint visits to the same place. The space and time complexity of this algorithm depends on both the parameter values and the input data. In the best case, the algorithm will use O(N log(N )) time and space, where N is the initial number of data points. In the worst case, it will require O(N 2 ). We implemented the algorithm in Matlab and ran it on a Pentium 4 with 1GB of RAM. The algorithm took between 30 minutes and 2.5 hours to run (we had between 38K and 110K points for each person). Setting Parameter Values. The algorithm has three parameters: λ1 , τ , and λ2 corresponding to the three different phases of merging. λ1 encodes the conditions for merging time-adjacent points into candidate places and is constrained by the geometry of the data and the minimum variance values that are chosen for the initial candidate places. We could treat the minimum variance values as additional parameters; however, we found the following steps for adjusting λ1 provide sufficient control over phase one of the algorithm. The first step is to set the minimum variance values based on the geometry of the data and a common-sense idea of minimum place size. We used 1.0 meter for both position 2
KL-divergence, or Kullback-Leibler divergence, is a calculation of the similarity between two probability distributions. It is written DKL (P ||Q), where P and Q are probability distributions over the same space. Since KL-divergence is not symmetric in its arguments, we calculate DKL (P ||Q) + DKL (Q||P ), the joint KL-divergence.
10
R. Aipperspach et al.
dimensions and 1.0 millisecond for the temporal dimension. The second step is to consider the distribution of joint KL-divergences between the initial candidate places. Although not strictly bi-modal, this distribution exhibited a strong peak of values such that choosing λ1 = 1.0 merged most of the candidate places that covered similar positions and avoided merging candidates from different places. We fixed λ1 = 1.0 for the remainder of this discussion and for all the results presented in section 4.4. To set the second parameter, τ , which encodes the minimum duration of a visit to a place, we tested many possible values ranging from 30 seconds up to 10 minutes. As discussed in [8], this parameter needs to satisfy conflicting constraints. First, it needs to be short enough that good candidate places are not discarded for not having been visited long enough. Second, it needs to be long enough so that spurious places, created by noisy sensor readings or by brief pauses in an insignificant location, are discarded. Since we are interested in finding places with relatively short visits, like bathrooms, a short minimum duration seemed appropriate. For our data, we found that a τ value of 1 or 2 minutes worked well for most participants. Finally, λ2 , which encodes the conditions for merging spatially overlapping places during phase three, needs to be set so that candidate places covering approximately the same place are merged while candidate places covering different places are not. The value of this parameter depends on the ratio between the size of single place to the distance between places. Unlike at the city scale, where sensor readings rarely exhibit noise that makes a point intended for one place, e.g. “work”, appear near a different place, e.g. “grocery store”, our sensor data at the scale of a single home often exhibits noise characteristics that make readings ambiguous as to which place a person is in. Although not a rigorous test, we used two out of the three houses to observe the effects of this parameter (treating the third house as a held-out test set). λ2 = 2.0 works well for the data used in this study. 4.3
Measuring the Similarity Between Places
The similarity score that we calculate is derived from the output of our placefinding algorithm. We calculate the score as follows. First, we use each data point’s posterior distribution (over candidate places) to label it with its most likely candidate place. In general, the posterior distributions for data points are very peaked, so taking the maximum value is appropriate. Next, using the sequence of data point labels, we calculate the duration of each visit made to each place. For example, if ten successive data points covering 45 seconds all received the same place label, and the preceding and succeeding data points received different labels, then these ten data points constitute a single visit lasting 45 seconds. We also record the time of day that this visit covers. From these visit segments, we calculate a joint distribution over durations (discretized into exponentially growing bins, e.g. 1-2min, 2-4min, 4-8min) and times-of-day (discretized into twenty-four overlapping 5 hour windows) for each place. We claim that these joint distributions roughly characterize the shared attributes of a place, based
A Quantitative Method for Revealing and Comparing Places in the Home HH01: Brad
Bathroom
HH03: Sierra
HH01: Jacqueline
Bedroom
Bedroom
Bathroom
Moving Around (1)
Bathroom
HH03: Cathy
Bedroom
Office Moving Around
Couch
Kitchen
Moving Around (2) + Not Home
Office Moving Around
Living Room Couch
Kitchen Table
Master Bedroom
Bedroom
Bedroom Moving Around
Kitchen
Kitchen Table "Not Home"
11
Couch
Kitchen "Not Home"
Kitchen + Gone
Living Room Entry
"Not Home"
Fig. 1. Places found for several study participants. Each place is centered on the Gaussian distribution found for that place; the length of the cross lines shows one standard deviation in the X and Y directions. Places plotted with dashed lines account for less than 5% of the data, and italic labels show regions not labeled as a place for each person.
on the notion that activity patterns follow regularities in terms of their duration and the time-of-day during which they occur. The logical measurement to make between distributions is the joint KL-divergence. Although the KL-divergence between the complete joint probability distributions (over durations and timeof-day) provided reasonable results, we found that using only the distribution over durations resulted in better matchings between the places we found in our data set, and we use this distribution in calculating the similarity scores reported below. 4.4
Results
We ran our place-finding algorithm on the data from each study participant, throwing out places accounting for less than 1% of all data. This resulted in an average of 6.6 places per person (ranging from 4 to 9). In general, each person had one place accounting for “not home” data (participants placed their sensors at a predetermined location, typically by the front door, when they were away from home) and one place accounting for sleeping. The remaining places accounted for activity patterns while participants were at home during the day. Sample places from Households 1 and 3 are shown in figure 1. The fit of these places to the true place in the home, based on experiences in interviews with participants, is discussed in section 4.5. The similarity measures between all pairs of places were also calculated as described previously. We used these measures to find the optimal many-to-one bipartite matching between the places for each pair of people (matching each place from one person to the “closest” place from each other person). Sample matchings are shown in table 1.
12
R. Aipperspach et al.
Table 1. Similarity-based place mappings. Italic place-names indicate unexpected mappings. Within household Brad Table Living Jacqueline Table Living Margaret Jack
Table Table
Kitchen Gone
Between households Sierra Office Bed Brad Table Moving(2)
4.5
Gone Gone
Moving(1) Bath
Bath Bath
Couch Living
Couch Couch
Bath Bath
Gone Kitchen
Spare room Spare room
Living Gone
Moving Moving(1)
Bath Bath
Couch Couch
Bed Bed
Gone Bed
Evaluation
Evaluation of Place-Finding Algorithm. Quantitatively evaluating the “correctness” of places in the home is a particularly difficult process. Systems designed to operate at the city scale are often evaluated based on the percentage of places in the data that they correctly identify. However, this evaluation metric requires that it is possible to identify each important place by hand. For a city, this is straightforward (Hightower et al. [8] had participants carry around a clipboard and write down every place they visited), but places in the home are less well defined, depending on subtle patterns of use. We argue that the “correct” places in the home should not be identified to match a set of a priori hand-labeled locations but rather allowed to emerge from the behavior of users, constrained only by the limitations of the characteristics of place that we defined and built into our algorithm. We evaluate the found places by considering how well they match places that would be expected from a taxonomic labeling of people’s places (e.g. bedrooms, living rooms, etc.) and by looking for the presence of interesting places that emerge from unexpected patterns of use. We base these evaluations on the qualitative interviews we conducted in each household [2,28]. Figure 1 illustrates examples of typical and atypical places in Households 1 and 3. For each person, typical places such as couch, kitchen table, bedroom and bathroom often appear. Because these places emerge only when there are significant lengths of time spent in them, the set of labeled places is different for each individual. This is best shown an example from Household 3, where Sierra and Gaby live together and rent their spare room to Cathy. The sets of places for Sierra and Cathy reflect the partitioning of the house between the different people: Sierra’s places include the master bedroom, office, living room, and kitchen; and Cathy’s places include her bedroom and the kitchen. This distribution of places highlights the difference in space use for the different people, as well as highlighting the kitchen as the primary shared space in the house. The benefit of allowing other, unexpected, places to emerge, rather than evaluating them with respect to an a priori metric can be seen in an example from Household 1. In the main living area of the house, both Brad and Jacqueline
A Quantitative Method for Revealing and Comparing Places in the Home
13
reported that there were two significant places: the living room couch and the kitchen table. However, for both Brad and Jacqueline, our algorithm found three places covering portions of the main living area: the living room, the kitchen table, and a third “moving around” area that included both places.3 Looking at the periods when this larger space was used, we found that it was used during the night of Brad’s birthday party. During that time, the two were moving around in the entire main living area of the house as if it were a single place. In this case, our place-finding algorithm revealed a significant pattern of place use that would not have been identified in a hand labeling of places and that was not apparent from our interviews. Looking more closely at the sequence of places visited by Jacqueline during the party, we discovered that she began the party in the “moving around place” and as the end approached began to spend more time in the “kitchen table” and “couch” places, suggesting that her use of space became much more localized at the end of the party. Similar unexpected places are revealed in data from the other households. During the study period, Jack and Margaret in Household 2 had just moved into their apartment and had not yet purchased a bed, sleeping instead on the futon in the main living room of the house each night. As a result, neither labeled the bedroom as a significant place during the study. However, the spare bedroom did show up as a significant place for both Jack and Margaret. In Margaret’s case, this was because she used the room to do exercises. In Jack’s case, this was because he had an ant farm in the spare bedroom that he would visit periodically throughout the day to take pictures of the ants. (He was creating a time-lapse video of their behavior.) Again, our algorithm revealed patterns of activity and place use that were not immediately apparent upon examination of the architectural space or reported by participants but instead arose from activity. Evaluation of Similarity Measure. The limited amount of in-home location data available makes it difficult to rigorously test our similarity measure. Our goal here is to highlight the potential usefulness of comparing the similarity between places. We evaluate our measure by showing that the resulting mappings between the places of different people are semantically appropriate (e.g., bedrooms are mapped to bedrooms). We also discuss instances where this mapping is unintuitive. Finally, we consider the usefulness of the mappings in capturing common structures by using them to improve predictive models of people’s movement patterns in their home. Table 1 shows the optimal bipartite mappings between the places of different participants. In general, the mappings between the places of two people from the same household are good. For example, a comparison between Brad and Jacqueline fits the expected semantic mapping between places except for mapping “Moving” to “Bathroom” (the couch to living room mapping is reasonable because Jacqueline does not have a separate couch place). Similarly, the 3
Brad’s moving around place was split and partially merged with a “not home” place because his sensor frequently produced readings in the main living room when it was actually near the front door (where he hung his tag while away).
14
R. Aipperspach et al.
mapping between Margaret and Jack is as expected, except for mapping between “Kitchen” and “Gone”. This incorrect mapping can be attributed to sensor noise – Jack and Margaret’s kitchen and door (where they hung their tags) were so close together that readings often bounced between them. Mappings between participants from different households fit with expected mappings in most instances but show more unexpected mappings than within household comparisons. In the mapping between Sierra and Brad, bedrooms are incorrectly matched. However, this mapping also reveals an interesting behavior pattern: Sierra’s office and Brad’s kitchen table appear to be incorrectly mapped together. But, interviews with both participants revealed that Sierra tended to work in the office and Brad tended to work at the kitchen table more often than eating there. Because our similarity score compares patterns of behavior between places, it captures the fact that, while Sierra’s office and Brad’s table appear on the surface to be different, the two in fact function as similar places. The similarity measures between places can also be used to capture and share common patterns of behavior between different people. Many researchers have attempted to predict movement patterns of people between places. For example, Ashbrook and Starner [3] use a second-order Markov model to predict transitions between significant places, Aipperspach et al. [1] use higher-order Markov models to predict transitions, and Liao et al. [12] use a hierarchical activity model to predict transitions. We suggest that good measures of the similarity between places will help to capture “typical” transition patterns between different types of places (e.g., the pattern “people often go from sleeping to the bathroom”). As a method of evaluating the benefit that can be gained by using typical patterns of movement between places to aid in learning movement patterns for an individual in the home, we have built a 3rd-order Markov model over the sequence of place-visits for each individual, as in [1] and [3]. We then build a model that is “bootstrapped” with data from other individuals, using the mappings between places generated based on our similarity measure. This model is built using the Bayesian mixture method implemented in the SRI Language Modeling Toolkit (SRILM) [21]. In evaluating the performance of each model, we use cross-validated perplexity as reported by the SRI toolkit. (A model that perfectly matches the data has a perplexity of 1; models with less than perfect matches have higher perplexity.) Following Mehta et al. [14], we define F0 to be the area between the learning curve for the first (single person) model and its optimal value (e.g., the area below each dotted line in figure 2). We then define F0|H as the area between the learning curve for the second (multi-person) model and its optimal value (e.g., the area below each solid line in figure 2). Each F measures the performance of its model based on rate of learning and maximum performance, and the transfer 0 ratio, FF0|H , measures the performance gained when using information about typical movement patterns between different genres of place. Figure 2 shows the results of running the predictive models on the data from Brad in Household 1, showing the performance when the model is bootstrapped with data from Jacqueline (in-household transfer). The bootstrapped model has
A Quantitative Method for Revealing and Comparing Places in the Home
15
6
5.5
5
Model perplexity
4.5
4
3.5
3
2.5
2
"Bootstrapped" with data from Jacqueline Brad’s data only
1.5
1 50
100
150
200
250
300
350
400
450
500
Training set size
Fig. 2. Learning curves showing transfer between Brad and Jacqueline in Household 1. The dashed line shows the model performance for a model trained only on data from Brad, and the solid line shows the model performance of the same model initialized with data from Jacqueline. The model has a transfer ratio of 1.20.
a transfer ratio of 1.197 when compared with the model using only Brad’s data, indicating successful transfer of shared movement patterns when using the similarity-based mappings between places. When evaluating transfer between other study participants, we discovered that our mappings resulted in significantly lower transfer ratios. The primary reason for this is that two people’s places do not always have an isomorphic mapping. Hence, requiring a single set of place names to function for two people is a strong assumption to make. In our data set, only Brad and Jacqueline had isomorphic place mappings. We identify two potential solutions to this. First, one could increase the library of known places to include enough cases that an isomorphism is likely to exist between two sets of places. Second, we could allow for probabilistic place name mappings.
5 5.1
Design Implications Predicting Activities
Several projects are focusing on building models of human activity, which is a difficult process involving the collection and labeling of large histories of data from different types of sensors [6,19,22]. However, the shared context of culture can significantly simplify this problem. For example “common sense” repositories (like the Internet) can be automatically mined for information about common tasks [29]. Similarly, since culturally shared notions of place encourage certain activities and not others, an algorithm which can determine how similar a place is to “typical” places of various types could enable context-aware applications to determine the activities most likely to occur in that place. Given a broad enough sample of places from different homes, applications could potentially learn common types of places and then learn typical behaviors for each. Additionally, Ubicomp systems that reference places by name (e.g. [24]) could use names derived from similarity to places with known names to replace hand-labeled names
16
R. Aipperspach et al.
for locations. Currently, the limited availability of accurate location and activity data from multiple homes makes it impossible to fully validate the possibility of learning typical places and associated activities from sensor data. As more data becomes available, we intend to explore more fully the improved activity prediction that can be carried out using shared notions of place. 5.2
Place, Similarity, and Context
An understanding of a user’s context is an important aspect of Ubiquitous computing. Dourish [5] describes several problematic assumptions made by traditional views of context. He argues that context is not a stable object, definable separately from the activities of an individual. Rather, he suggests that context is a dynamically defined phenomenon, emerging from and maintained by the patterns of activity collectively carried out by individuals. According to Dourish, our goal as technologists is not “to support particular forms of practice, but to support the evolution of practice.” Our notions of place in the home are in part an attempt to distill meaningful information encoded in people’s patterns of movement in a manner compatible with the goals described by Dourish. In particular, both the places that we find and the relationships between them are allowed to emerge entirely from activity. We make few assumptions about the structure of those patterns and instead look for an emergent structure based on people’s actions. The ability to find and compare emergent contextual structures suggests several design possibilities. We consider an example in elder care. One common task in elder care applications is to detect changes in behavior that provide early warning of changes in a person’s health [16]. Our similarity metric could be applied to monitor how a single person’s place usage changes over time (e.g., by calculating the similarity between the same person’s places over two time periods). A low similarity between the sets of places found for an elderly individual may show changes in sleeping patterns or activity levels. These changes in place use could be used to highlight potential health issues. The changes could also be visualized as part of more reflective technologies, such as the ambient health feedback displays developed by Morris [17].
6
Conclusions and Future Work
In this paper, we have presented the first (to our knowledge) algorithm for finding significant places in the home based on location sensor data, and have explored the novel technique of calculating the similarity between places as an aid in understanding the places found by our algorithm. This work is based on the first sub-meter accuracy location traces collected in the home. In order to more fully validate our techniques, particularly in light of the relative uncertainty in the boundaries and definition of place in the home, it will be necessary to continue collecting accurate location traces from a larger set of homes. This will allow us to explore the range of places that exist in the home and to begin to categorize and understand the different types of places that exist across homes, quantifying the
A Quantitative Method for Revealing and Comparing Places in the Home
17
work done in [28]. We also plan on collecting and analyzing more diverse types of sensor data. By deploying a broader range of sensors to detect both location and activity, we will be better able to explore the relationship between place and activity and to determine if culturally shared notions of place and appropriate activities can be used to improve activity recognition algorithms. Currently, the only long-term high-precision location data from the home is in the form of (X,Y) location traces. With current location technology, this data is difficult to collect, requiring a lengthy sensor installation and calibration process. We expect that in-home location traces will become more readily available as location-tracking technologies improve. Another possibility is to explore fingerprint-based place-finding algorithms for the home. Because fingerprint algorithms rely only on recurring patterns of sensor readings, not on detecting the absolute position of each person, the sensor installation process would be greatly simplified. For example, a typical installation may involve only the placement of radio beacons throughout a home instead of the careful installation and calibration of sensors in known positions. Given such fingerprint data for the home, it would be possible to find significant locations and compare their similarity to each other using a variant of our algorithms, at the expense of losing the mapping between the places found and absolute position within the home. Finally, while we developed our current metric for comparing the similarity between places based on the types of patterns we saw in place usage in the home, we are interested in trying these similarity metrics on place usage at other scales. We plan on applying our similarity measures to existing place data available at the city scale.
References 1. Aipperspach R., Cohen, E., Canny J.: Analysis and Prediction of Sensor Data Collected from Smart Homes. In: Proc. Pervasive ’06 (2006) 2. Aipperspach, R., Woodruff, A., Anderson, K., Hooker, B.: Maps of Our Lives: Sensing People and Objects Together in the Home. Tech. Report EECS-2005-22, EECS Department, UC Berkeley (2005) 3. Ashbrook, D., Starner, T.: Using GPS to Learn Significant Locations and Predict Movement Across Multiple Users. In: Personal and Ubiquitous Comp., Vol. 7 (2003) 4. Dey, A., et al.: A Conceptual Framework and a Toolkit for Supporting the Rapid Prototyping of Context-Aware Applications. In: Human-Computer Interaction, Vol. 16 (2001) 5. Dourish, P.: What We Talk About When We Talk About Context. In: Personal and Ubiquitous Computing, Vol. 8 (2004) 19-30 6. Dragunov, A.N., et al.: TaskTracer: A Desktop Environment to Support Multitasking Knowledge Workers. In: Int’l. Conf. on Intelligent User Interfaces (2005) 7. Harrison, S., Dourish, P.: Re-Place-ing Space: The Roles of Place and Space in Collaborative Systems. In: Proc. CSCW ’96 (1996) 8. Hightower, J., et al.: Learning and Recognizing the Places We Go. In: Proc. Ubicomp ’05 (2005) 9. Kang, J.H., Welbourne, W., Stewart, B., Borriello, G.: Extracting Places from Traces of Locations. In: Proc. Workshop on Wireless Mobile Applications and Services on WLAN Hotspots, WMASH (2004)
18
R. Aipperspach et al.
10. Laasonenm K., Raento, M., Toivonen, H.: Adaptive On-Device Location Recognition. In Proc. Pervasive ’04 (2004) 11. Leont’ev, A. N.: Activity, Consciousness, and Personality. Prentice-Hall, Englewood Cliffs, NJ (1978) 12. Liao, L., Fox, D., Kautz, H.: Learning and Inferring Transportation Routines. In: Proc. AAAI ’04 (2004) 13. Marmasse, N., Schmandt, C.: Location-Aware Information Delivery with Commotion. In: Proc. Symposium on Handheld and Ubiquitous Computing, HUC (2000) 14. Mehta, N., Natarajan, S., Tadepalli, P., Fern, A.: Transfer in Variable-Reward Hierarchical Reinforcement Learning. In: Proc. of the Inductive Transfer workshop at NIPS 2005 (2005) 15. Monteiro, C.: Activity Analysis in Houses of Recife, Brazil. In: Proc. First Int’l. Space Syntax Conf (1997) 20.1-20.13 16. Morris, M., Intille, S.S., Beaudin, J.S.: Embedded Assessment: Overcoming Barriers to Early Detection with Pervasive Computing. In: Proc. Pervasive ’05 (2005) 17. Morris, M.: Social Networks as Health Feedback Displays. In: IEEE Internet Computing, Vol. 9(5) (2005) 29-37 18. Oswald, F., et al.: The Role of the Home Environment in Middle and Late Adulthood. In: Wahl, H.-W., et al. (eds.): The Many Faces of Health, Competence and Well-being in Old Age: Integrating Epidemiological, Psychological and Social Perspectives. Springer, Heidelberg (2006) 19. Philipose, M., et al.: Inferring Activities from Interactions with Objects. In: IEEE Pervasive Computing (October 2004) 50-57 20. Rowan, J., Mynatt, E.D.: Digital Family Portraits: Providing Peace of Mind for Extended Family Members. In: Proc. CHI ’01 (2001) 21. Stolcke, A.: SRILM - An Extensible Language Modeling Toolkit. In: Proc. of the Int’l. Conf. on Spoken Language Processing (2002) 22. Munguia Tapia, E., Intille, S.S., Larson, K.: Activity Recognition in the Home Setting Using Simple and Ubiquitous Sensors. In: Proc. Pervasive ’04 (2004) 23. Trevisani, E., Vitaletti, A.: Cell-Id Location Technique, Limits and Benefits: An Experimental Study. In: Proc. IEEE Workshop on Mobile Computing Systems and Applications, WMCSA (2004) 24. Truong, E., et al.: CAMP: A Magnetic Poetry Interface for End-User Programming of Capture Applications for the Home. In: Proc. Ubicomp ’04 (2004) 25. Ubisense, Ltd. Website, http://www.ubisense.net. 26. Wang, J., Canny, J.: End-User Place Annotation on Mobile Devices : A Comparative Study. In: Extended Abstracts, CHI ’06 (2006) 27. Wicker, A.: An Introduction to Ecological Psychology. Brooks/Cole Publishing Company, Monterey, California (1979) 28. Woodruff, A., Anderson, A., Mainwaring, S.D., Aipperspach, R.: Portable, But Not Mobile: A Study of Wireless Laptops in the Home. Tech. Report EECS-2006-88, EECS Department, UC Berkeley (2006) 29. Wyatt, D., Philipose, M., Choudhury, T.: Unsupervised Activity Recognition Using Automatically Mined Common Sense. In: Proc. of AAAI ’05 (2005)
Principles of Smart Home Control Scott Davidoff, Min Kyung Lee, Charles Yiu, John Zimmerman, and Anind K. Dey Human-Computer Interaction Institute + School of Design Carnegie Mellon University {scott.davidoff, mklee, cmyiu, johnz, anind}@cmu.edu http://smarthome.cs.cmu.edu
Abstract. Seeking to be sensitive to users, smart home researchers have focused on the concept of control. They attempt to allow users to gain control over their lives by framing the problem as one of end-user programming. But families are not users as we typically conceive them, and a large body of ethnographic research shows how their activities and routines do not map well to programming tasks. End-user programming ultimately provides control of devices. But families want more control of their lives. In this paper, we explore this disconnect. Using grounded contextual fieldwork with dual-income families, we describe the control that families want, and suggest seven design principles that will help end-user programming systems deliver that control.
1 Introduction In order for smart homes to achieve their promise of significantly improving the lives of families through socially appropriate and timely assistance, they will need to sense, anticipate and respond to activities in the home. Interestingly, expanding system capabilities can easily overstep some invisible boundary, making families feel at the mercy of, instead of in control of that technology [1][5][6][21][30]. An important goal of smart home research then becomes how to appropriately expand system capabilities to produce more control – both perceived and actual. Researchers often frame the problem of control as one of end-user programming [13][21][27][34]. However, end-user programming as it is typically conceived requires a priori specificity and rigidity that conflict with a large body of ethnographic research on the organic, opportunistic, and improvisational ways that families construct, maintain, and modify their routines and plans, e.g. [3][14][15][16][17][25] [26][38][42][45]. In exploring the issue of control and the smart home, we have chosen to focus on dual-income families, a large and growing segment of the population of the United States. These families often feel out of control due to the complex and rapidly changing logistics that result from integrating and prioritizing work [41], school, family [10], and enrichment activities [3][15][17][25]. While end-user programming returns some measure of control to the user over preprogrammed or learning-only systems [1][21], ultimately, framing the problem as end-user programming leads researchers to view the research and evaluation in terms P. Dourish and A. Friday (Eds.): Ubicomp 2006, LNCS 4206, pp. 19 – 34, 2006. © Springer-Verlag Berlin Heidelberg 2006
20
S. Davidoff et al.
of control of devices. Our fieldwork on dual-income families, however, shows that more than control of their devices, families desire more control of their lives. We suggest that the problem of home control should be widened to include not just control of artifacts and tasks, but control of the things families most value – their time, their activities, and their relationships. We argue that the way to do this is to enable end-user programming systems to more flexibly and appropriately reflect the complex nature of observed human interaction. In this paper, we discuss how a smart home might provide dual-income families with more control over their lives. We describe the approach current end-user programming systems take to the problem of control. We then describe an ethnography we conducted with this population, and then situate our findings within the existing body of social research to outline a rich description of the kind of control families value, and the flexibility they utilize to manage their daily lives. Finally, based on these findings, we define seven principles to help researchers operationalize this concept of families being in control of their lives, as applied to end-user programming systems.
2 Related Work As research in ubiquitous [48] and context-aware [18] computing moves into the home, and systems begin to take action on behalf of users [22] either in a proactive or reactive sense, researchers are finding that users feel unexpectedly challenged by these systems [30]. In traditional desktop models of human-computer interaction, systems are slaves to human input. But advanced context-aware systems can break this interaction model [6], and, in doing so, they challenge or even invert the user’s sense of who is actually in control. Expanding system capabilities while maintaining users’ sense of control then becomes one of the central concerns of context-aware computing. One approach views the problem of smart home control in terms of end-user programming (in this paper we use the terms end-user programming and smart home control interchangeably). These end-user programming systems explore various interfaces to provide end-users control of home devices, including natural language [27], interlocking puzzle pieces [34], visual programming [35], programming by demonstration [19], and magnetic refrigerator poetry [46]. The end-user programming approach has several benefits. It provides users control over an unpredictable confederation of interoperating devices [40], and allows users to customize services as they might see fit [34], even inventing new services [46]. Existing end-user programming systems, however, also face certain limitations, especially in relation to how they might apply to dual-income families. First, families resist categorization as users. The user tends to be singular, while families are, by definition, plural. Unlike users, families both individually and collectively resist clearly-defined goals [15][17][38]. Also, family relationships are complex and evolve organically over time [10][16], whereas conceptions of the user tend to be static. In addition, programming requires definition, formality, and specification which conflict with a large body of ethnographic research on how families operate. Responsibility or ownership of tasks in a family is rarely clearly-defined [3][16], while the
Principles of Smart Home Control
21
majority of programming tasks connote clear lines of ownership. Also, various individuals have multiple [26] (often competing [25]) methods of completing tasks and metrics for the quality of their performance [16], while programming requires clear boundaries for task completion. Lastly, most programming tasks assume complete access to relevant information, and the focused attention of the programmer. But rapid context switching between home and work oblige families to make decisions with incomplete information [2] and marginal attention [41]. But most importantly, end-user programming studies define success in terms of usability, and overlook other core metrics contained in the human value proposition [43]. Systems are designed to facilitate the successful completion of tasks that often involve configuration of and complex interaction with multiple devices. But we suggest that usability measures should be contained within the larger discussion of value, which include utility and likeability [43]. We are suggesting that end-user programming consider not just the usability of the system, and how the users manage the artifacts that constitute it, but what utility the system is ultimately responsible for delivering. Existing end-user programming systems that focus on artifacts cannot repair the diminished sense of control that dualincome families already experience. But by expanding their evaluation metric to include the utility smart home systems provide, end-user programming systems might be able to help families regain a sense of control over their lives. To understand how to appropriately support families in making that transition, we undertook an ethnography of dual income families. Here, our work builds upon the comprehensive studies of Darrah [14][15][16] and Beech et al [3]. While our work focuses on issues of routines, planning and control, other studies of family life have focused on household communication [13] and its relationship to locations [24], time management [25], communication technology [26], use of refrigerator magnets [44], knowledge specialization [42], routines [45], overall technology use [47], or calendaring [7].
3 Ethnography of Dual-Income Families 3.1 Participants Our study began with an exploration of the needs of 12 dual-income families with school age children. These families represent an interesting population for several reasons. First, at 43% of the population of the United States and growing [32], dualincome families represent a significant demographic whose sheer magnitude merits attention. In addition, dual-income families show a particular need for support. By moving away from the single-income model, these families are exposed to a surprising variety of stressors [14][15][32]. Taxing schedules and no anchor presence at home impose particularly high coordination costs. Parents are also exposed to role strain, where they feel pressures in trying to comply with the expectations attached to their roles as parents [2][12], which adds to an already stressful existence.
22
S. Davidoff et al.
Lastly, we believe dual-income families are realistically poised to receive the benefits of smart home advances. The challenges dual-income families face often drive them to aggressively adopt and experimentally use new communication and coordination technologies [26]. Their demonstrated desire for increased flexibility and control over their own lives, and this historical precedent of using technology to satisfy those needs, we believe places dual-income families as prime candidates to become smart home early adopters. Our families were solicited through bulletin board advertisement, and in person at shopping malls. We pre-screened families in order to include a wide range of professions, ages of parents and children, and economic class. Since our interest includes end-user programming, we also disqualified families where one of the parents had a job involving programming. Table 1 contains information about each of the families we recruited. Table 1. Detailed description of participating families
Family A
Father Not provided, Carpenter
B
Mother Not provided, Administrative assistant 47, Department manager
C
41, Professor
39, Teacher
D
38, Business manager
41, Marketing manager
E F G H
Not provided, Professor 45, Secretary 32, Surgeon 36, Project manager
Not provided, Carpenter 46, Truck driver 31, Graduate student 34, Graduate student
I
52, Nurse
53, Steam fitter
J
49, Administrative assistant
50, Manager
K L
54, Events coordinator 43, Legal secretary
55, Salesman 46, Landscaper
48, Art gallery director
Children 15, son 18, son 9, daughter 15, daughter 1, son 5, daughter 5, son 8, daughter 10, son 15, son 15, daughter 5, son 1, daughter 5, daughter 15, son 19, daughter 15, daughter 20, son 21, son 11, daughter 15, daughter 17, daughter 19, daughter
Participant homes averaged 2.3 bedrooms, and 6.6 rooms in total. Five homes had two floors, while the remaining seven had one floor. A typical footprint would include a modest family space near the front door connected to a small dining area. The kitchen area would form the hub of the home, while bedrooms were situated on the periphery or on the second floor.
Principles of Smart Home Control
23
3.2 Method Fieldwork began with three-hour contextual interviews in the homes of the families. Time was organized to include directed storytelling, artifact walkthrough [8], and role-playing activities. The entire family was asked to participate in all the research activities. Fieldwork pilots identified the “wake up” and “arrive home” times as key opportunity areas for smart home assistance, so we focused our investigation on those time windows. During directed storytelling we solicited personal accounts of waking up or arriving home from particular family members. In addition, we asked them to act out the scenes so we could better understand the relationship between their routine and its context. During the artifact walkthrough, we asked families to demonstrate the use of their main coordination artifacts, often a large kitchen calendar, and their various personal calendars. During role-playing, we asked families to pretend to coordinate for a fictitious school field trip, for which we provided simulated school paperwork. The interviews covered both predictable days, like weekdays and weekends, predictable exceptions like business trips and holidays, and unpredictable exceptions such as sick days or miss-the-bus days. To gain additional insight into the wake-up and arrive-home activities, we left families with a cultural probe package [29]. The package explored the stressors and pleasures of waking up and arriving home. We also asked parents to comment on the parts of their lives that make them feel like good moms and dads. The package included a book of stimuli questions, free response text space, and a camera to photograph related vignettes. Finally, we asked each family to log their wake-up and arrive-home activities for one week. We captured stress and rush levels, principal activities, immediate needs, and preoccupations. 3.3 Data Analysis To analyze our data, we mapped each home, photographed objects and locations of interest, and recorded photo locations on our maps. We coded our interview notes for emergent themes. We also evaluated our photographic record, grouping images to provide both visual support for themes identified during interviews, and themes that contained uniquely visual information, often place- or object-related. We created individual schedules for each family member, including typical days (often weekdays), and regular exceptional days (like weekends), and occasional exceptional days (e.g. snow days, holidays). Schedules included both an order of activity and, if available, a corresponding time. Multiple timelines allowed us to compare order of operations both within and between families. The cultural probes helped us identify emotional connections between families and their homes, and how families define themselves through their products and their uses with them. Finally, the activity logs allowed us to identify specific stress triggers and compare stress levels between weekdays and weekend days. They also provided rich description of family priorities when faced with competing needs.
24
S. Davidoff et al.
4 Findings Many of the logistical challenges that produce this feeling of “life out of control” can be traced to the enrichment activities that children participate in. While work and school add to the complexity of daily living, they tend to present predictable logistical needs. But many qualities of enrichment activities make them resistant to simple logistical management. Enrichment activities present rapid seasonal changes. They also require constantly evolving transportation needs. And they often present both anticipated and unanticipated attendant responsibilities. These factors make them much more resistant to the development of a consistent routine and much more likely to cause a breakdown requiring improvisation to the underlying logistical plan. Other sources of loss of control can be attributed to unpredictable events and breakdowns, like sick children, or missing the school bus. Generally our families address this loss of control by increasing their flexibility. We consider this situation in more detail. 4.1 Less Than Ideal Control Enrichment activities frame the day. Dual-income families fill their children’s lives with enrichment activities. These activities benefit the children in many ways such as teaching values, providing physical fitness, teaching competition and teamwork, supporting existing social structure and providing supplemental education. Families often select activities based on long-term goals such as preparing their kids for a successful career or increasing the chance of college admission to selective schools [15]. In addition, the activities often serve as de facto “babysitting” to help cover the time parents are at work. Every child in every family we interviewed participated in at least one (and on average two) enrichment activities. While children form the principal participants, enrichment activities affect every member of the household. In addition to the management of their households and the completion of whatever work they might have brought home, parents are charged with the successful logistical management of their children as they relate to these activities. Even the simple is complex. Family J’s soccer practice shows how much logistical complexity can be contained within a single event. First, getting the kids wherever they need to be represents a challenge. Practices are held in one of two locations. Games are held in any number of locations. Locations are printed on the team schedule, which is kept on the family refrigerator. This sheet has no directions. Parents who carpool have to coordinate who picks up and who drops off. Practices start at consistent times, but games start at one of three times. Kids also have to come prepared. If the event is on turf, kids will need to bring their flat shoes. If the event is on grass, they will need to bring their cleats. Kids always need to bring their shin guards and knee pads. Games require either the home or away uniform. Practices require practice jerseys. All clothes need to be laundered, which often means washing them the night before so that they are clean for the day of use.
Principles of Smart Home Control
25
Even parents have homework for soccer games. Three families bring refreshments to each game – juice for thirsty players during the game, juice for after the game, and oranges for halftime. This information is also encoded on the schedule on the fridge. Forgetting comes with a high social cost. Either the team goes thirsty, and the child is embarrassed, or the parent has to face the panic and stress of racing to get kids to the field on time while running to the store to buy drinks and fruit. A simple activity, like leaving the house, is rarely “simple.” To get the kids to school, parents need to make sure children are awake, washed, dressed, fed, and ready for their rides. These activities depend on the (sometimes unwilling) participation of the child(ren), the coordinated use of (or competition for) scarce resources (e.g., bathroom time) with other family members, and the presence of these and other resources (e.g., school bag) along with the knowledge of their whereabouts. Our Family H’s Dad described a successful morning as one where “we all get out the door, and there are no major disasters.” Activities resist routinization. Routines allow families to function without having to carefully consider every option at every moment [45]. However, routines for enrichment activities prove difficult to construct. And even when possible, these routines provide many opportunities to break down. Their high variability in both detail and responsibility make construction of a “normal” routine difficult. In addition, most of these activities do not run for an entire year. Instead they are “seasonal,” forcing families to constantly re-adjust schedules as seasons end and new seasons begin. Transportation to activities such as team sports almost always involves variability in the routine. Most dual-income parents needed to rely on others to provide some transportation due to other commitments in their complex schedules and due to the different locations and times of the events. In addition, most activities required inconsistent use of special equipment such as the types of shoes and uniforms mentioned above. Responsibilities bleed past fixed boundaries. Timing and order also play major roles in the capable execution of enrichment activities, adding further to their logistical complexity. Many activities, like musical groups, require special equipment. And since children often go to their activities directly from school, children must take this equipment with them in the morning, extending the time window of responsibility to the morning of the event. For Family H, washing soccer uniforms extended responsibility to the night before the event. Responsibility can even extend significantly further than a single day. Family H, for example, has to bring in snacks for the entire class one day each month. Parents have to remember this day, and make sure to have enough snacks for the entire class in the house on the morning of the day on which the snack is their responsibility. This extends the time window for the event as far back as the weekend before the event, when Family H would go food shopping for the week. Breakdowns cause cascade effects. Because children and parents are so interdependent, their schedules are united by a chain of dependency. Small failures that affect one individual can extend individual failures into multiple, shared coordination failures. We found this scenario to be common among our participant families: Mom
26
S. Davidoff et al.
might be running behind for a business meeting, so she needs the bathroom first. Her Son is forced to shower second, and misses his bus. Dad then has to drive him to school, which makes him late for his morning meeting. The chain of dependency can also become more complex when parents, out of necessity, divide the jobs surrounding certain tasks. For example, in Family H, it is Dad’s job to get their Daughter ready for ballet, and it is Mom’s job to take their Daughter to ballet as part of a carpool. When Mom goes away on a business trip, and the carpool parent providing the ride calls with a cancellation, Dad lacks the resources to easily arrange a new ride. He does not know which other parents he can lean on, as this task is not part of his regular responsibilities. Not enough gas in the car, traffic, a forgotten briefcase, an extra trip to the grocery store – all simple events that might delay one person – cascade into the schedules of other family members, who all depend on one another. “Busyness” is a moral good. It would seem that if enrichment activities caused so much distress and made families feel “out of control”, then a simple strategy to regain control would be to do less. However, this strategy is conspicuously absent in dualincome families. The parents in our ethnography value the enrichment their children receive through participation in these activities. In addition, participation allows them to demonstrate their mastery of “busyness”, and the ability to master busyness is one of the values these families who have generally self-selected to be dual-income wish to pass on to their children [14][16]. 4.2 Flexibility as Coping Strategy Families exhibited many behaviors that allow them to manage this hyper-busyness. Some families imposed simple rules. Family L, for example, limited their four children to no more than three activities each. Many families assigned responsibilities for specific events to particular parents, liberating the other parent from dealing with those details. All our families leveraged some technological infrastructure [26] and had routines surrounding its use. In general, we observed across almost all coping strategies, a quality of flexibility. We explain in more detail by example. Detail is acquired when necessary. One flexibility strategy involved incrementally adding details to plans as they became necessary. This tendency causes long-term plans to differ substantially from short-term plans. Long-term plans tend to resemble rough sketches. When Family H learns their day for snacks is weeks away, Mom puts the snack calendar on the fridge. She does not consider what snack to get at that time because what she buys will depend in part on what her daughter wants and what other families have provided that week. She doesn’t know if she will purchase the snack when food shopping the weekend before or if Dad will have time to pick it up on his way home the night before. Only summary knowledge is either known or even possible to be known. But on the day of certain activities every logistical detail has to be covered. The night before a soccer game, just before dinner, for example, Family H consults the various media that include information relevant to that game: the location, home or away, who is dropping off, and who is picking up. Mom knows it is her job to pick
Principles of Smart Home Control
27
up, so she confirms with her Daughter where to wait for her. Mom also confirms who else she will be taking in the carpool. Mom confirms all the details and places just to be sure that nothing had changed since its codification on the central artifacts. Improvise. Many events never acquire any a priori detail at all. Sometimes, this was due to constraints of memory. Family H, for example, reported keeping “only two or three [plans] in our head at one time.” And so on days when there were multiple events after school, Mom and Dad would speak on their cell phone multiple times and watch many plans evolve without preplanned conception of their ultimate order of operations. In fact, we observed that many plans started out as successful improvisations, and then were adopted as routines because of that success. Frissen [26] and Darrah [16] also describe parents adopting new communication technology so that they could become more available and ultimately increase their flexibility. Because they could be reached by their families at a moment’s notice, plans were able to evolve and be improvised even closer to their target times. Work and home blend. Mom and Dad also tend to bring their work life home, and arrange their home life at work [41]. By bringing the two contexts together, parents don’t have to constrain all their home planning to the house, or all their work planning to the office. This ability to seamlessly move between contexts allows for plans to dynamically evolve, and allows for more flexibility in how time is used. In Family F the Mother worked at home in the evenings close to her daughter to allow her to focus both on work and her family simultaneously. This flexible changing between home and work also extends to artifacts. Parents would seamlessly shift between whatever media was most immediately available [3] [13][26]. We found that our families stored work information on their home calendar, and home information on their work calendar [25]. This opportunistic use of media helps parents master their busyness while, at the same time, complicates their lives by requiring synchronization between the many artifacts that impact the different aspects of their lives. Lifestyle choices. We observed families making deliberate choices about highly important parts of their lives so that they could have more flexible routines. Family J chose to live close to Dad’s work so that Dad could be available in emergencies. In [15], Darrah found families who would consolidate their children into one school so as to simplify logistics. Family L staggered their work schedules so that one parent could be more available for unexpected events, like children being sick.
5 Design Principles The homes that we observed were inhabited by complex families full of contradictory human beings in need of support. To help these families regain control of their lives, researchers might try to help them make better lifestyle choices, including simply reducing the number of activities in which they are actively involved. But our findings suggest that these activities play a central role in the social identity of families and modifying them would damage this fundamental aspect of family life. Instead of proposing a social engineering solution, we approach the problem by exploring the role of technology in supporting families’ existing lifestyles and identities.
28
S. Davidoff et al.
Currently, we observe a disconnect between the nuanced lifestyles our study demonstrates, and the smart homes that many technologists envision. This suggests that for technology to help families regain control over their lives, we refocus the end-user programming of smart home systems to more appropriately reflect that complexity. To begin this investigation into a new kind of end-user programming, we provide a list of principles that such a system would need to adhere to. The principles include the following: 1. 2. 3. 4. 5. 6. 7.
Allow for the organic evolution of routines and plans Easily construct new behaviors and modify existing behaviors Understand periodic changes, exceptions and improvisation Design for breakdowns Account for multiple, overlapping and occasionally conflicting goals The home is more than a location Participate in the construction of family identity
We address each principle in turn. 5.1 Allow for the Organic Evolution of Routines and Plans Routines form the basis of normalcy in many domestic situations, and allow family members to focus their attention on other, more demanding activities [3][15][25][45]. Plans allow families to coordinate the substantial number of activities they carry out during an average day. Plans and routines represent an important and powerful way families manage to resist entropy and regain elements of control over their lives. Thus a smart home system that intends to provide families some measure of control over their lives must account for the concept of the routine. But routines, as we and others [45] have observed, contain many subtle inflections that resist the simple models of computation imposed by many end-user programming systems. First of all, describing all the routines of a family would constitute an enormous undertaking. Next, our families would likely not recall all their routine behaviors, as some have simply become so routine that they have passed below consciousness [15]. But even known routines rarely conform to the notion that they might be able to be completely specified a priori. Many routines themselves were once successful improvisations that have been incorporated into a family’s daily activities. Other plans begin with only cursory detail, and gain in precision as the day of their enacting encroaches. On the surface, existing smart home control systems attempt to provide support for routines. But, their concept of routines is too rigid, requiring a priori specification of all the routines that the user can remember. To give families a sense of control over their lives, a smart home system will have to both support the concept of routine, but not bind families to that notion. Such a system will need to allow plans and routines to evolve organically. Systems that enforce rigidity in routines, including most end-user programming systems built for the home, do not match our observations of how people behave and will not be successful at helping families take control of their lives. This inflexibility will lead to families losing control to the technology itself, replacing control with an inflexible algorithmic tyranny.
Principles of Smart Home Control
29
While an extreme case, evidence from the MavHome, a smart home research environment [11] illustrates this point well. The MavHome had a lighting model that turned off its lights late at night because its occupant did not have visitors late at night and was asleep. When the occupant did have visitors, they literally had to “remain in the dark” because of the difficulty in controlling and changing the home’s behavior. Systems that support this design principle would have to incorporate the flexible, organic nature of routines that people adhere to themselves. 5.2 Easily Construct New Plans and Routines, and Modify Existing Ones Many of our families engaged in planning and coordination tasks daily. We fully expect that families would interact just as frequently with a smart home system that supported the concepts of plans and routines. Such a smart home system would need to offer a low-cost way for families to describe plans and routines. Additionally, given the need to support the organic evolution of routines, including occasional improvisation, and breakdowns and exceptions in those routines, this system will also need to provide a low-cost way to modify existing behaviors. 5.3 Understand Periodic Changes, Exceptions and Improvisation Even if routines could be specified entirely beforehand, people both intentionally and necessarily deviate from routines. Some deviation is seasonally mandated, such as children participating in soccer in the fall, and in basketball in the winter. Some deviation occurs due to exceptions – Mom wants to get a surprise gift for her daughter. Many routines are substantially improvisational, representing constantly shifting targets. A smart home system that enforced a single, rigid correct model of routines would find that few routines were ever capably executed according to that model, and would be constantly interrupting and misinterpreting what was actually happening. Once again, smart home technology that reflected observed behavior would have to understand that exceptions to routines are actually not exceptional, and be prepared to interpret them correctly. This means that a representation of a routine or plan cannot be rigid, and the system has to understand concepts like periodicity, what an exception is and how to recover from or adapt to it, and how to deal with improvisation. Simply put, any computational model that is expected to take action on behalf of families would be confronted with the fact that routines can prove to be anything but. 5.4 Design for Breakdowns Even when routines are capably carried out by family members, and the family machine marches forward like clockwork, we found many instances where the surrounding world simply did not cooperate. Family I’s Dad gets the kids ready for school on time, but his Son’s shoes have gone mysteriously missing. The carpool stops while Dad scours the house for the delinquent footwear. Family J’s Dad has to bring his Daughter to physical therapy after school. But she forgets, and goes to her friends’ house. Mom has to drive out of her way to pick her Daughter up. Plans just don’t always work. Routines sometimes fail. A smart home system will need to expect, account for, understand, and appropriately respond to these eventualities.
30
S. Davidoff et al.
Also, a smart home would need to recognize when a breakdown is occurring and understand the kind of cascading failures that can occur and learn or know how to react to them and ideally avoid as many of the attendant failures as possible. 5.5 Account for Multiple, Overlapping and Occasionally Conflicting Goals Sometimes it is not the world, but the family structure itself that causes complexity or breakdowns in plans or routines. Laundry may be Mom’s job on Monday, but it is Dad’s job on Tuesday. These collaborative tasks suggest another aspect of the flexibility that a smart home system would have to reflect. Many tasks simply do not fit a simple model of ownership. Family F’s Daughter’s is responsible for walking the dog every morning, except when she’s running behind. In these circumstances, Mom was happy to help out. Other tasks resist the concept of ownership all together. Everybody in Family L shares in the laundry duties. Other tasks move between individuals on particular days of the week, muddling the notion that responsibility is owned by any single individual. We also might find open conflict regarding a chore or routine [38]. Preferences vary widely for what constitutes a comfortable temperature, and many families battle over the thermostat. Other conflicts may surround varying methods for task completion, varying tolerances and metrics of quality, or the priority of a particular task. For example, Mom may make a rule that says “don’t use the scrubber on the pots.” But everybody in the family knows the rule really means “don’t use the scrubber on the pots when Mom’s in the room.” A smart home system would need to participate in this ambiguous notion of responsibility. A smart home may need to face, interpret and react to a situation where goals conflict and the situation may still be valid. In such a situation, the system would have to provide a way to maintain internal consistency in data structure in the face of external disagreement. 5.6 The House Is More Than a Location As opportunistic planners, our families did not limit their coordination activities to the physical constraints of any single location within the house, or even the house itself. Planning and coordination also occur at work, at school, at the place of the activities themselves, and during transportation between any of these locations. The functional boundaries of the smart home will need to extend beyond the physical parameters of the home itself, and be accessible anywhere planning for or activities receiving support occurs. A smart home that participates in a family’s planning dialog will have to allow for the family to enter and update information from these various locations as easily as if they were at home. This specification extends to routines as well. A smart house would need to support planning activities, and also be able to monitor everything else going on outside the home to better understand how routines are changing and whether breakdowns, exceptions and/or organic changes are taking place. The blending of spaces also extends to information space. Families enter information in calendars at both home and work. Paper school flyers and sports schedules find their way onto refrigerators. Calendars also cross the boundaries of individuals.
Principles of Smart Home Control
31
A smart home system that allows its inhabitants to feel in control of their information will need to be able to blend elements across contexts, media and individuals and aggregate them into a unified model of what is happening. A smart home that supports this principle would support planning tasks and maintenance of routines everywhere the family goes, monitor and react to family activity in and out of the house and be able to leverage all available artifacts that the family uses. 5.7 Participate in the Construction of Family Identity Potential smart home services might also potentially collide with how individual family members derive their identity. Consumption and use of products contribute to how people formulate their social identity [4][23][37]. And since the home represents the largest purchase most people ever make, to many families the home itself makes the largest product-based contribution through which they realize their social identity. By extension, we can expect that the services a smart home provides will in many ways contribute to how individuals formulate their social identity. Here, grocery list support provides a good example. While cooking dinner every night might be challenging and a source of frustration for Mom, automating the cooking process could also remove an opportunity for Mom to feel like a good Mom [44]. The act of preparing food constitutes an important part of Mom’s identity. Instead, automating the creation of a grocery list is an assistive task that aids Mom without challenging her identity. Existing smart home systems often focus on the usability or efficiency of a particular task, but remove that task from the larger context in which it is so importantly embedded. Many tasks are time-intensive but are vital to our identities as Moms, Dads and Families. As researchers search for new services a smart home can provide, they will have to extend their evaluation metrics to include this important component of the human value proposition. This understanding of how smart home services contribute to our identity also extends to situations where the smart home may take action on behalf of users. When a smart home system is making a decision about when it is appropriate to take an action, it has to be aware of these invisible value lines, and have some understanding of social protocol, so that it can decide when it is appropriate to cross those lines, and when it is not.
6 Conclusions In this paper, we have presented seven principles that we believe smart home researchers should address in the systems they build, in order for families to take control of their lives. These principles were derived from an ethnography we conducted with dual-income families that illustrated the loss of control these families feel over their lives. Enrichment, work and school activities and the exceptions and breakdowns that can occur, combined with parents trying to instill their values into children all contribute to families’ sense of loss of control. Much of the smart home research that addresses control, leverages end-user programming. However, in practice, these systems promote control over artifacts and
32
S. Davidoff et al.
devices, rather than promoting control over the routines, planning and time (in short, the lives) of families. From our ethnography, we found that the key to regaining control over complex lives is to support flexibility. Our design principles focus on providing flexibility in the planning of activities, construction and modification of routines, through support for organic formation of routines, along with support for exceptions, improvisation and breakdowns in those routines. They also suggest the need to accommodate conflicting goals and responsibilities among family members, to push the boundaries of the home beyond the physical construct of the home itself, and to participate in the construction of a positive family identity. We acknowledge that it will be difficult to embed these principles into working end-user systems, but our contribution is in pointing out that an end-user control system will likely fail unless it adheres to the principles. Our next step is to build and evaluate a smart home system that adheres to these principles and makes families be and feel more in control of their lives. An important goal in sharing these principles is to initiate a discussion with other smart home researchers about smart home occupants and their feelings of control. In particular, we hope to bring attention to the social and often chaotic nature of family life, the impact it has on smart home technology and the impact smart home technology can have on it.
References [1] Barkhuus, L., & Dey, A.K. (2003) Is context-aware computing taking control away from the user? Three levels of interactivity examined, Proceedings of Ubicomp 2003, 159-166. [2] Barnett, R.C. (1994). Home-to-work spillover revisited: A study of full-time employed women in dual-earner couples, in Journal of Marriage and the Family, 56: 647-656. [3] Beech, S., Geelhoed, E., Murphy, R., Parker, J., Sellen, A. & Shaw, K. (2004) Lifestyles of working parents: Implications and opportunities for new technologies, HP Tech report HPL-2003-88 (R.1). [4] Belk, R.W. (1988) Possessions and the extended self, Journal of Consumer Research, 15(2): 139-168. [5] Bellotti, V., Back, M., Edwards, W.K., Grinter, R.E., Henderson, A., & Lopes, C. (2002) Making sense of sensing systems: Five questions for designers and researchers, Proceedings of CHI 2002, 415-422. [6] Bellotti, V. & Edwards, W. K. (2001) Intelligibility and accountability: Human considerations in context-aware systems, Human-Computer Interaction, 16(2-4): 193-212. [7] Bernheim Brush, A.J. & Combs Turner, T. (2005) A Survey of personal and household scheduling, in Proceedings of Group 2005, 330-331. [8] Beyer, H. & Holtzblatt, K. (1998) Contextual design: Defining customer-centered systems, San Francisco: Morgan Kaufman Publishers. [9] Blackwell, A.F., Burnett, M.F. & Peyton Jones, S. (2004) Champagne prototyping: A Research technique for early evaluation of complex end-user programming systems, in Proceedgings of VLHCC 2004, 47-54. [10] Cherlin, A. J. (1988) The Changing American family and public policy, Washington, D.C.: Urban Institute Press.
Principles of Smart Home Control
33
[11] Cook, D.J., Youngblood, M., Heierman, E., Gopalratnam, K., Rao, S., Litvin, A., & Khawaja, F. (2003) MavHome: An agent-based smart home, in Proceedings of PerCom 2003, 521-524. [12] Cowan, R. S. (1989) More Work for Mother, London: Free Association Books. [13] Crabtree, A., Rodden, T., Hemmings, T. & Benford, S. (2003): Finding a place for ubicomp in the home, in Proceedings of Ubicomp 2003, 208-226. [14] Darrah, C.N. & English-Lueck, J.A. (2000) Living in the eye of the storm: controlling the maelstrom in Silicon Valley, in Proceedings of the 2000 Work and Family: Expanding the Horizons Conference. [15] Darrah, C. N., English-Lueck, J. & Freeman, J. (2001) Families at work: An ethnography of dual career families, Report for the Sloane Foundation (Grant Number 98-6-21). [16] Darrah, C.N. (2003) Family models, model families, in Proceedings of the 2003 American Anthropological Association Annual Conference. [17] Davidoff, S., Lee, M.K., Zimmerman, J. & Dey, A.K. (2006) Socially-aware requirements for a smart home, in Proceedings of the International Symposium on Intelligent Environments, 41-44. [18] Dey, A., Abowd, G., & Salber, D. (2001) A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications, Human-Computer Interaction, 16(2-4). [19] Dey, A.K., Hamid, R., Beckmann, C., Li, I., & Hsu, D. (2004): a CAPpella: programming by demonstration of context-aware applications, in Proceedings of CHI 2004, 33-40. [20] Dey, A.K., Sohn, T., Streng, S., & Kodama, J. (2006) iCAP: Interactive Prototyping of Context-Aware Applications, in Proceedings of Pervasive 2006, 254-271. [21] Dey, A.K., Newberger, A.N., & Chau, E. (2006) Support for context monitoring and control, in submission. [22] Dourish, P. (2004) What we talk about when we talk about context, Personal and Ubiquitous Computing, 8(1):19-30. [23] Dittmar, H. (1989) Gender identity-related meanings of personal possessions, in British Journal of Social Psychology, 28(6):159-171. [24] Elliot, K., Neustaedter, C., & Greenberg, S. (2005) Time, Ownership and Awareness: The Value of Contextual Locations in the Home, in Proceedings of Ubicomp 2005. [25] Fleuriot, C. (2001). An Investigation into the management of time in complex lifestyles, Ph.D thesis, University of the West of England. [26] Frissen, V.A.J. (2000) ICTs in the rush hour of life, The Information Society, 16: 65-75 [27] Gajos, K., Fox, H., & Shrobe, H. (2002) End user empowerment in human centered pervasive computing, in Proceedings of Pervasive 2002, 1-7. [28] Garfinkel, H. (1967) Studies in Ethnomethodology, Engelwood Cliffs, NJ: Prentice Hall. [29] Gaver, B., Dunne, T., & Pacenti, E. (1999) Design: Cultural probes, interactions, 6(1): 21-29. [30] Grinter, R.E. & Edwards, W.K. (2001) At Home with ubiquitous computing: Seven challenges, In Proceedings of Ubicomp 2001, 256-272. [31] Harper, R. (2003) Inside the smart home: Ideas, possibilities and methods. in Richard Harper (Ed.) Inside the smart home. New York: Springer, 1-14. [32] Hayghe, H. V. (1989) Children in 2 worker families and real family income, in Bureau of Labor and Statistics’ Monthly Labor Review, 112(12): 48-52. [33] Hindus, D. (1999) The importance of homes in technology research, in Proceedings of CoBuild 1999, 199-207.
34
S. Davidoff et al.
[34] Humble, J., Crabtree, A., Hemmings, T., Åkesson, K., Koleva, B., Rodden, T., &Hansson, P. (2003) “Playing with the bits”: User-configuration of ubiquitous domestic environments, in Proceedings of Ubicomp 2003, 256–263. [35] Jahnke, J.H., d’Entremont, M., & Stier, J. (2002) Facilitating the programming of the smart home, IEEE Wireless Communications, 9(6): 70-76. [36] Kidd, C., Orr, R.J., Abowd, G.D., Atkeson, C., Essa, I., MacIntyre, B., Mynatt, E., Starner, T., & Newstetter, W. (1999) The Aware Home: A living laboratory for ubiquitous computing research, in Proceedings of CoBuild 1999, 191-198. [37] Kleine, R.E., Kleine, S.S., & Kernan, J.B. (1993) Mundane consumption and the self: A social identity perspective, in Journal of Consumer Research, 2(3): 209-235. [38] McCalley, L. T., Midden, C. J. H. & Haagdorens, K. (2005) Computing systems for household energy conservation: Consumer response and social ecological considerations, in Proceedings of CHI 2005 Workshop on Social Implications of Ubiquitous Computing. [39] Mozer, M. (1998) The neural network house, in Proceedings of AAAI Symposium on Intelligent Environments, 110-114. [40] Newman, M., Sedivy, J. Z., Neuwirth, C. M., Edwards, W. K., Hong, J. I., Izadi, S., Marcelo, K., & Smith, T. F. (2002) Designing for serendipity: Supporting end-user configuration of ubiquitous computing environments, in Proceedings of DIS 2002, 147-156. [41] Nippert-Eng, C. (1995) Home and work. Negotiating boundaries through everyday life. Chicago: University of Chicago Press. [42] Rode, J.A., Toye, E.F. & Blackwell, A.F. (2005) The domestic economy: A broader unit of analysis for end user programming, in Proceedings of CHI 2005, 1757-1760. [43] Shackel, B. (1991). Usability – context, framework, definition, design and evaluation, in Shackel, B. & Richardson, S. (Eds.) Human Factors for Informatics Usability, Cambridge, UK: Cambridge University Press, 21-37. [44] Taylor, A.S. & Swan, L. (2005) Artful systems in the home, in Proceedings of CHI 2005, 641-650. [45] Tolmie, P., Pycock, J., Diggins, T., MacLean, A. & Karsenty, A. (2002) Unremarkable computing, in Proceedings of CHI 2002, 399-406. [46] Truong, K. N., Huang, E. M., & Abowd, G. D. (2004) CAMP: A magnetic poetry interface for end-user programming of capture applications for the home, in Proceedings of Ubicomp 2004, 143-160. [47] Venkatesh, A., Chuan-Fong E.S. & Stolzoff, N.C. (2000) A Longitudinal analysis of computing in the home based on census data 1984-1997, in Proceedings of HOIT 2000, 205-215. [48] Weiser, M. (1991) The computer for the 21st Century, Scientific American, 265(3): 94-104.
Historical Analysis: Using the Past to Design the Future Susan Wyche1, Phoebe Sengers2, and Rebecca E. Grinter 1 1 GVU Center College of Computing Georgia Institute of Technology Atlanta, GA, USA {spwyche, beki}@cc.gatech.edu 2 Information Science Cornell University Ithaca, NY, USA
[email protected]
Abstract. Ubicomp developers are increasingly borrowing from other disciplines, such as anthropology and creative design, to inform their design process. In this paper, we demonstrate that the discipline of history similarly has much to offer ubicomp research. Specifically, we describe a historicallygrounded approach to designing ubicomp systems and applications for the home. We present findings from a study examining aging and housework that demonstrate how our approach can be useful to sensitize ubicomp developers to the impact of cultural values on household technology, to reunderstand the home space, and to spur development of new design spaces. Our findings suggest that historically-grounded research approaches may be useful in more deeply understanding and designing for context both in and outside of the home.
1 Introduction As ubicomp moves beyond the work environment and into a broader social and cultural world, researchers are drawing on an expanding set of disciplinary perspectives to inform design. Ubicomp developers commonly employ anthropological methods, most notably ethnography [e.g., 24,26,27]. Similarly, researchers borrow from art and design to develop novel ways to explore the home, such as cultural probes [13]. In this paper, we describe how ubicomp developers can borrow from another discipline useful for exploring domestic environments: history. Examining the past has previously been used to inspire new form factors and styles such as retro; we suggest that history can be further used to provide strategies that, like anthropology, unpack the culture of the home and, like art-inspired design, defamiliarize the home [2]. In this paper we present a study examining housework by older adults and describe how we integrate historical analysis into the design process. We then present findings from a study of older adults’ experiences with housework that suggest history can be beneficial in understanding the culture of the home, in defamiliarizing the home, and in spurring designers’ imaginations, thereby opening new design spaces. P. Dourish and A. Friday (Eds.): Ubicomp 2006, LNCS 4206, pp. 35 – 51, 2006. © Springer-Verlag Berlin Heidelberg 2006
36
S. Wyche, P. Sengers, and R.E. Grinter
Specifically, our findings demonstrate that historical analysis sheds new light on recurring cultural themes embedded in domestic technology, and by extension, ‘smart homes.’ Questioning these themes has the potential to lead designers to rethink assumptions about domestic technology use. For example, rather than using “ease of use” as a guiding principle, elders described difficult, yet enjoyable aspects of housework that technology removed. Older adults fondly recalled products that were durable, contradicting the consumption-driven theme that arguably underlies many of the systems and devices being developed for smart homes. This leads to new heuristics for design; for example, do we assume users will be interested in constant software upgrades and stylistically new gadgets and devices or would it be more appropriate to develop products that last for decades? Historical analysis and elders’ personal accounts of their histories revealed the importance of sensual aspects of housework lost with the introduction of new technologies. Participants described the isolating impact of technologies introduced to the home, specifically electric dishwashers and washing machines. Developing technologies to support one person rather than multiple people or families is a historical theme repeating itself in current domestic systems. In each of these examples, understanding how technology has changed for better or worse in the past suggests new options for contemporary technology design. We believe using historical analysis could benefit other designers by providing an additional way to understand context and by spurring their imaginations.
2 Background Why study history, if our goal is to design the future? One answer can be found in philosopher George Santayana’s famous proclamation, “Those who cannot remember the past are condemned to repeat it” [28]. This quotation is widely used to argue that exploring the past helps us understand who we are today and where we are going. For ubiquitous computing, historical awareness can deepen designers’ understanding of the context they are designing for. In addition, history can spur designers’ imaginations by revealing the contingency of the present situation, rendering it less obvious and inevitable. As Bell et al. suggest [2], using history to defamiliarize the present supports designers in envisioning future domestic life less constrained by present-day cultural assumptions embedded in technology. Historical awareness could also prompt ubicomp developers to make design decisions that have more positive social and cultural ramifications. As Bell and Kaye have argued [3], new designs for ‘smart homes’ often repeat themes from the past that, with reflection, designers may not wish to propagate. Critics of smart home prototypes similarly suggest that technologists’ visions of the future tend to look backwards rather than forwards. Spigel [31] describes this as “yesterday’s future.” She uses surveillance systems to demonstrate how familiar uses of technology persist in past and present visions of the smart home. Systems that give parents the ability to survey their children’s activities and to monitor unusual behavior have been touted as “the future” for the past 60 years. Even the architectural styles of smart homes demonstrate such repetition; Spigel describes how Tudor, Spanish, and colonial styles have been consistently used for smart home prototypes since their inception as a
Historical Analysis: Using the Past to Design the Future
37
marketing tool in the 1920’s. Indeed, distinguishing the exterior of older “home of tomorrow concepts” presented at fairs and conventions from today’s newer “smart homes” is difficult. If designers recognize such themes at the time of development, they can consciously choose whether they should be repeated or altered [29]. There has been some mention of history’s relevance in designing for the future in ubicomp and related literature, the most notable being Blythe et al.’s “technology biographies” [6]. These are a set of questions that ask participants to reflect on their present, past, and future experiences with technologies. One element of the technology biographies, “personal histories,” are questions aimed at uncovering users’ feelings of loss and nostalgia as they relate to technological change. For instance, a participant may be asked to remember their first home computer or how they communicated at work prior to using e-mail. These historical reflections are integrated into an ethnographic study approach. In this paper we present a historically grounded approach that complements and reinforces history as an element of ubicomp design. Our goal in this paper is to show how history can be integrated into the early stages of design of ubicomp systems through a case study of early design for housework technology. In the following section, we outline the process by which historical analysis was integrated into early design in our case study. This is followed by findings from our study examining aging and housework. We conclude with a discussion about how historically grounded research approaches can benefit the design process.
3 Using Historically Informed Approaches to Explore the Home Our case study was motivated by two major goals. Topically, our objective was to examine housework as a dimension of the smart home. Housework is a domestic activity largely absent from current smart home discourse (with a few exceptions [4, 5, 11, 25]). Indeed, housework is often rendered obsolete in visions of the future [4], despite the fact that even after more than a century of automation the number of hours women work in the home has remained remarkably stable [10]. Methodologically, our goal was to integrate existing ubicomp data collection methods with sensitivity to history. We intend for these techniques to supplement commonly used ubicomp data gathering methods, such as interviewing and design ethnography [7, 26]. In this section, we describe our approach. First, we describe the historical analysis we engaged in as background research for our study, which included examining historical texts, first-hand sources of popular culture such as magazines and catalogues, and patents. Then, we describe how this historical research led to the development of a new data collecting tool, the ‘memory scrapbook’, used to elicit additional historical data from study participants. Finally, we describe how our inhome study was structured to leverage historical awareness. 3.1 Historical Analysis A history is an account of some past event or combination of events. Historical analysis is, therefore, a method of discovering, from records and accounts, what happened in the past [20]. In historical analysis, researchers consider various sources
38
S. Wyche, P. Sengers, and R.E. Grinter
of historical data such as historical texts, newspaper reports, diaries, and maps. The method is commonly used by historians to gain insights into social phenomena. Designers can similarly use historical analysis to identify themes embedded in their work, avoid re-inventing systems that already exist, and establish background prior to user observation or interviewing. Indeed, leading design firm IDEO recognizes this and includes historical analysis in the early stages of their design process [18]. As we will describe below, in our work, we drew on three particular kinds of sources to establish common themes and design opportunities for housework: we reviewed the historical literature to find trends that historians have already identified as relevant to domestic technology; we studied patents to identify previously attempted technologies and to spark inspiration for new design, and we immersed ourselves in primary sources from popular literature that give an experiential sense of the past and provide design resources. History is not culturally universal. Because we were interested in domestic design in US contexts, we focused our study on the American history of domestic technology. Our results will hold to some extent for other Western contexts which have a similar history, but different histories would need to be told for other cultural contexts. 3.1.1 Reviewing Historical Texts We began our work by reviewing relevant literature on the history of housework. Although this step took time, it helped establish a background prior to the project’s next phases. We took advantage of historians at our university, who specialize in the history of American homes, to point us to seminal works in the field. Our analysis was limited to historical texts written after 1900, because the decades following the industrial revolution are widely considered a time of dramatic change in American homes [9]. We describe here three themes that emerged from the literature as particularly important to understanding the last 100 years of housework in the US: 1) the “labor saving” debate, 2) domestic technology’s gendered character, and 3) loss of sensual and emotional qualities that accompanied housework. Designers often conceive of products thinking they will make tasks easier or faster to perform. However, domestic technologies which are proposed as labor-saving and efficient historically have had a different impact. Research suggests new technologies have often increased time spent doing housework rather than decreased it [9,34]. In part this was due to the rising cleanliness standards that accompanied electric technology into homes during the twentieth century. This created higher expectations for women to produce spotless and hygienic bathtubs, sinks, and toilets. With the introduction of the electronic washer, laundering increased because there was greater demand for clean clothes. Indeed, novel cleaning approaches often divert time from one task to another, thus creating ‘more work for mother’ [9]. Today, women remain largely responsible for maintaining a home. The drawbacks of assuming housework is “women’s work” are well documented [4,9,19,32]. Sweeping, washing, vacuuming, and tidying-up, arguably confine women to the “domestic sphere,” thus making it more difficult for them to participate in the socially influential “public sphere.” Finally, as technology makes its way into our domestic lives, some of the felt qualities embedded in everyday experiences become lost. McCarthy and Wright describe feltness as the emotional and sensual aspects that make up humans
Historical Analysis: Using the Past to Design the Future
39
experiences using technology [22]. For example, before dryers, women hung laundry in their backyards where they would talk and exchange gossip with neighbors. Today, dryers are confined to laundry rooms or basements, isolating those who use them from others and thus diminishing some of laundries’ felt qualities. We are careful not to downplay the technologies’ contributions to removing much of the drudgery associated with housework, but use this example to suggest there are subtle characteristics that shape users’ experiences with technology that we risk losing if efficiency and production drive technology development. Historical awareness enabled us to consciously choose which of these themes deserved repeating, and which we wanted to resist in our designs. For instance, we understood how housework has arguably contributed to woman’s marginalization in society and acknowledged this was not a theme we wanted to perpetuate in the smart home. The final benefit of conducting a historical analysis during the initial design phase was that it helped us develop the protocol for our study’s interview stage, to be described later. 3.1.2 Patent Search In order to better understand the historical design space for domestic technology, we engaged in a patent search. The United States Patent office represents a tremendous body of original knowledge and technological innovation.1 Online databases such as the one found on the Unites States Patent and Trademark Office’s website (www.uspto.gov) and freepatentsonline.com make exploring issued patents, patent applications, and expired patents, dating back to 1790 accessible to anyone with internet access. We searched patents from a variety of years, but focused on those issued between 1940 and 1965, because this is considered the height of America’s preoccupation with domestic cleanliness [17]. We broadly looked for issued patents related to cleaning technology such as vacuums, dishwashers, irons, and washing machines.
Fig. 1. US Patent no. 3,771,192 Combination Toy Dog and Vacuum Cleaner 1
We chose USPTO because we were interested in designs that had the US market in mind.
40
S. Wyche, P. Sengers, and R.E. Grinter
Patent searches were useful in tracking the historical roots of many common cleaning technologies used in homes today and making us aware of inventions not mentioned in the history of housework literature. Archived in patent databases were ideas that were never made commercially available, for reasons we did not explore, including a patent for a dinner table that converts into a dishwasher [35], a vacuum cleaner that is disguised to look like a dog [37], and a prototype for a self cleaning house [1]. These forgotten examples spurred us to imagine wildly different ways to think about housework. 3.1.3 Popular Magazines and Catalogues Recognizing that housework was not only interesting from the standpoint of technological development, but also from the perspective of consumer culture, we examined back issues of magazines and catalogues. This technique, also found in historical accounts of household technology, provides an opportunity to learn about how appliances were sold to and perceived by the public, typically using advertising, problem pages, and articles to elicit that information. We chose to look at Good Housekeeping, Ladies Home Journal, and copies of Sears Roebuck catalogues because they are considered valid sources for understanding the nature of domestic work in relation to consumer culture in the first half of the twentieth century [8,21]. Wanting to understand housework through the consumer’s lens during this time, we looked through randomly selected copies of early magazines. Libraries typically have bound volumes of old magazines shelved chronologically. These primary sources supported a different kind of historical awareness than historical texts, less intellectual and more experiential. We felt like we were traveling back in time, looking at the ads and glancing at the articles in the format in which they originally appeared. Just as with the patent search, there were things to be learned about the history of housework not revealed in historical texts. It was housework that led, in part, to the creation and rise of these magazines. At the end of the twentieth century, changing attitudes towards cleanliness and the decline of domestic servants led to the proliferation of magazines like Good Housekeeping and Ladies Home Journal. This was the time of “the great hygienic boom.” Breakthroughs in germ theory were taking place and middle-class women were becoming increasingly concerned with germs and their potential to spread disease [17]. The resulting increased standards of cleanliness prompted manufacturers to develop a myriad of products to help homemakers disinfect every room in their homes. The magazines provided a forum for manufacturers to advertise their new products and to offer advice on how to properly maintain a home. Good Housekeeping and Ladies Home Journal have been in continuous publication since the late 1800’s and are considered indicators of the social and technical change that have occurred in American, middle-class homes [8]. We had read about the electrification of homes during the 1920’s, but observing the transition from laundry tubs to electric washers in catalogues added another dimension to our understanding. The decline in household assistants, or maids, is frequently discussed in historical texts, but viewing advertisements demonstrated the significance of this change. The
Historical Analysis: Using the Past to Design the Future
41
sharp decline of images picturing maids to ones depicting housewives cheerfully touting various new products was clear. A particular strength of these ads and images was that they provided a rich illustration of changes occurring in the home between the years 1920-1960, in a way that could not be gotten from historical accounts alone. 3.2 Home Studies to Elicit Histories: Elders and the Memory Scrapbook Reviews of historical literature, patents, and magazines identified a variety of themes and design opportunities for domestic technologies that we decided to explore in an empirical study. Again, we turned to another historically grounded approach to gather empirical evidence: oral histories. Oral histories are verbal testimonies about past events or simply stories from any individual’s life [16]. We recognized that, at a time when computer networks are entering homes, much can be learned from those who experienced the past wave in which electricity was introduced into the home. Inspired by Blythe et al.’s technology biographies [6], we decided to integrate oral histories into an ethnographic home study, similar to those already used in ubicomp [7]. There were two core elements to our home study: the selection of appropriate participants and the development of a ‘memory scrapbook’ to help elicit stories from our informants. 3.2.1 Selection of Project Participants In order to elicit oral histories, a decisive factor was our selection of participants. We chose to work with older adults who would be able to share their housework experiences from the years 1940 to 1965. Interviews with elders became an important way to breathe life and personal meaning into the historical data we had collected. Specifically, we did home studies with 11 female homemakers who described themselves as being largely responsible for housework in their homes during the years 1940 to 1965. During these ethnographic-style home studies, we toured their homes and intensively interviewed the homemakers on their housekeeping practices. With the exception of one participant, all had lived in single-family household structures with children, environments similar to what we saw in the advertisements. The age range was 69-84 years old, with a mean age 76 years. Thus, the older participants in our study were approximately 20 to 54 years old during the years we focused on, while younger participants ranged from being children to young adults. Younger participants frequently recalled memories from their adult years as well as their childhoods. We recruited participants by asking colleagues and acquaintances if we could talk to their parents and/or grandparents. Participants were predominantly from the northeastern part of the US. They were compensated for their participation with $50 or a gift certificate to a popular restaurant in their area. 3.2.2 Eliciting Histories Using the Memory Scrapbook In addition to home tours and intensive interviewing, we developed a research instrument called the ‘memory scrapbook’ to support collection of historical experiences. In this section, we will explain how the memory scrapbook was designed, then describe its deployment as part of our home studies.
42
S. Wyche, P. Sengers, and R.E. Grinter
Design of the memory scrapbook The memory scrapbook design is based on photo elicitation, or the use of photographs to provoke a response. Photo elicitation is based on the simple idea of inserting photographs into a research interview [15]. It is used by anthropologists and historians to stimulate recollection of personal and public memories. The difference between using photos and typical interviewing techniques is that images can evoke deep elements of human consciousness, thus helping uncover richer aspects of a user’s experience [20]. In addition, we hoped that photo elicitation might help offset problems with memory that typically occur as people grow older. We chose to put the images in a scrapbook because it was an attractive and familiar medium for eliciting memories. Scrapbooking is a hobby that emerged in the late 1800s when Americans began pursuing the art of pasting letters, photos, greeting cards and other mementoes in books, as a way of preserving them for future generations [10]. We could expect our participants to be familiar with scrapbooks as a genre for stimulating discussion of the past. Additionally, we hoped the scrapbook’s informal and personal nature would be useful in establishing a rich dialogue with our informants while keeping the interview grounded in housework and history. We drew on our previous historical analysis for design inspiration, returning to the back issues of Good Housekeeping, Ladies Home Journal, and Sears Roebuck catalogues we had looked through earlier. In the magazines we looked for advertisements related to housework. This included products used for doing laundry, washing dishes, polishing silver, and scrubbing toilets, countertops, and floors. We avoided advertisements related to childcare and personal hygiene because we were specifically interested in house cleaning. In the Sears Roebuck catalogues we looked for images of cleaning instruments such as brooms, irons, and washing machines. Pictures were selected according to their graphical interest. Selected images were then scanned and catalogued according to their date and what they depicted. Approximately 100 ads and photographs were chosen to potentially be in the scrapbook. They were cropped to take out irrelevant background details so that the focus would be on the advertisement or image. We purchased an 8.5 x 11 inch, fabric-bound scrapbook at a chain crafts store. There are a variety of scrapbook styles, ranging from ones covered in multi-colored flowers to costly leather bounds ones. Wanting to keep in line with the dated imagery inside the scrapbook, we chose a book covered in a simple black fabric with gold embossed trim. Selected images were color printed on white card stock and arranged on each page with attention to visual composition. There were 12 pages in the book and a total of 52 variously sized images from the years 1940 and 1964 (see Figure 2). Deployment of the memory scrapbook The scrapbook was used during our home visits. Following the home tours we returned to the living room or kitchen and asked participants to interact with the scrapbook. It was introduced after the home tour because we were interested in creating a contrast between elders’ current housework experiences and their past ones. We wanted to collect stories rather than facts, which prompted us to develop something more engaging than a typical survey. As with cultural probes [12], the scrapbook was meant to question the preconception of researchers as authorities.
Historical Analysis: Using the Past to Design the Future
43
Fig. 2. Page from Memory Scrapbook
Indeed, older adults were the experts in our case because no one on our research team was alive during the time period we were examining. We were also sensitive to the fact that we were using an unfamiliar implementation—the scrapbook—of the oral history research technique. Participants were told that the scrapbook was an experiment on our part: This is a “memory scrapbook,” its a way for us to find out more about you and how technology has changed during your lifetime, ignore things you are not familiar with, and discuss things you are. There is no right or wrong answer and I hope it is fun. –study’s principal investigator We let participants hold the scrapbook and flip through the pages at their own pace. While thumbing through the scrapbook, we stressed that we were interested in memories related to the advertisements and that, if there were images that they were unfamiliar with, not to worry and to go on to the next page.
4 Findings of the Study In this section, we present findings from our study organized into three topic areas, including issues raised by our study and the resulting design implications. First, we examine the tension between making domestic tasks easier and the challenging aspects that accompanied housework in the past. This suggests that designers should not always make reduction-of-effort a central focus of housework technology, instead retaining some of the challenging and rewarding aspects that elders enjoyed. Second,
44
S. Wyche, P. Sengers, and R.E. Grinter
we discuss the decline in the quality of manufactured cleaning products and suggest that designers should work towards developing systems that are as durable and longlasting as Fuller brushes. Finally, we analyze the historical tendency of technology to individualize domestic tasks rather than preserving appealing social aspects, and propose ways entertainment and communication devices in the home could bring people together. 4.1 “Hands and Knees” Using the scrapbook during our interviews, we asked elders to discuss how cleaning had changed over time. Unsurprisingly, many described technology’s role in making tasks easier during their lifetimes. Images of wringer washers sparked stories about preferring today’s electric washing machines to the earlier manual versions. Wringer washers were commonly used to clean clothes in the 1930’s and early 1950’s; they were physically demanding and dangerous to use. Advertisements for the laundry soaps like Bon Ami, Duz, and Tide caused some to recollect when they needed to scrape and boil their own soap; all were pleased that they could buy these items at the store rather than having to make their own. Despite the positive changes brought by making household tasks easier, many informants described missing difficult aspects of it that became less common with the introduction of efficiency driven domestic technologies. For instance, many participants preferred using their “hands and knees” to clean floors as opposed to using a floor mop. I was a hands-and-knees washer. And I still am basically hands andknees, if it really has to be done, because just swishing the mop around, you don't get the corners, you really don't do the job. - 72 year old woman Prior to the introduction of floor mops and the more recent Swiffer™ cleaners, hands and knees was a common way to clean floors. Though it is physically demanding and time-intensive, some elders preferred this technique because of its thoroughness, the challenge of making a dirtied floor “sparkle,” and the sense of satisfaction that followed finishing the job. It’s a rather superficial job, using the mop. I would prefer a more thorough job like you get when you get down on the floor yourself. 81 year old woman or I like to have things clean. Like, I enjoy working and cleaning the kitchen floor, you know, making things look good. I feel like I have accomplished something. - 77 year old woman Others preferred cleaning on their hands and knees because it was a form of exercise. Indeed, rather than removing the physical effort required to clean a floor, some enjoyed it and considered it a source of physical activity.
Historical Analysis: Using the Past to Design the Future
45
When its cold or raining outside I can’t go out walking, getting down and cleaning the floors is how I exercise; it’s good for me. - 81 year old woman We have no intention of implying that technology needs to make tasks more difficult and agree that efforts to decrease the amount of time spent on housework are beneficial. However, our findings suggest that by focusing on making tasks effortless, other positive aspects of the experience may be lost. In addition to developing systems to make tasks easier, perhaps other aspects should be considered. For instance, rather than being a chore, computational systems and devices could be developed that treat housework like a game [5]. Or designers could imagine scrubbing floors as a form of exercise and develop cleaning systems that monitor how many calories users burn, similar to the feedback workout equipment provides. If thoroughness in cleaning is a concern, perhaps that signals a need to develop systems that communicate to users how clean their floors are. If anything, the finding suggests that by exploring how housework was done in the past, designers can consider new ways of envisioning how it can be done in the future. 4.2 Designing Durable Systems and Devices Included in the scrapbook were images of cleaning supplies taken from Sears Roebuck catalogues, like mops, feather dusters, wringer washers, and vacuum cleaners. These images inspired comments about how the quality of cleaning instruments had declined over time. The “Fuller Brush” ads exemplified this trend, because they elicited comments about how durable some products were in the past. The Fuller Brush Company has continuously manufactured cleaning brushes since 1906 and touts itself as creating “the best products of their kind.” Indeed the quality and durability described in the brushes’ ads proved true. Informants dug into their closets and drawers and eagerly showed us brushes from years ago. Despite frequent use, the participants described how the brushes’ bristles always stayed in place. They were high quality you know . . .very good quality . . . they say bristles don’t come out, well they don’t. Of course years ago most of our products were well made you know . . . its different today. - 79 year old woman This sharply contrasted with informants’ attitudes regarding products today, which many described as lacking the quality evident in older products. I remember the Fuller Brush man; his brushes were expensive but wonderful. And they lasted forever. Not like the ones you get at [a chain department store] today. -68 year old woman And Oh yes, Fuller Brushes, they were the best brushes ever made, I still have two of them, they are nice, not made out of that cheap plastic you get today. – 71 year old woman
46
S. Wyche, P. Sengers, and R.E. Grinter
This claim was further supported by elders’ comments during our in-home tours. We discovered that participants continued to use some of the same tools to clean that they had used decades earlier. When asked about why they chose to use these older versions, many describe quality as being key. I think back then we care more about things and make them last. Today they don’t. You just go buy new ones. I mean I could use this to death [referring to feather duster] and it wouldn’t fall apart. Now they throw ’em away. Everything is replaceable today. And back when I was growing up, it wasn’t. - 70 year old woman One design implication is clear. Rather than designing computational devices and systems that will become outmoded in a limited amount of time, i.e. incorporate planned obsolescence, designers should consider borrowing from Fuller’s Brush Company’s mantra of making “making things last.” Today, computers are becoming almost as disposable as toilet brushes. Monitors fill up landfills and hard drives abound in thrift stores because they no longer work or are out of fashion. The opportunities for new devices in future homes is an opportunity to subvert planned obsolescence and potentially develop ubiquitous computing devices that are sustainable and long-lasting. 4.3 Design to Support Togetherness Five of the 11 participants had fond memories related to washing dishes when they were growing-up. They described missing the social interaction that accompanied this common household chore. This finding personalizes a theme common in historical accounts, that technology has made housework more isolating. Indeed, as technology made its way into homes, the unplanned interactions that were a common part of housework tended to disappear. Prior to the electric dryer, garments were dried outside on clotheslines. When electric dryers were introduced, homemakers ceased to go outdoors to hang their washing out, instead going to their own indoor basements or laundry rooms to dry clothes. This led to decreased opportunities to serendipitously interact with neighbors. We lived in a row house. So of course backyards were backyards. . . . but every Sunday, there would be a lot of wash on the line. And I would go and help my grandmother bring it all in. And there would be neighbors there and we would all talk. And my mother, when she would go out and hang wash, we would always talk to our neighbors outside. We always hung wash - we didn’t have dryers. -81 year old woman Others fondly recalled domestic life prior to electric dishwashers. For many, manual dishwashing was an activity that supported informal conversations among family members.
Historical Analysis: Using the Past to Design the Future
47
Well, one of the things I apparently like doing, oddly enough, is washing dishes. I hardly use my dishwasher. Because when I was growing up, the only time I really had a relaxed communication with my mother was when she would be washing dishes, and I would be drying. That became a – and also with my kids, when they were growing up, it became a nice, easy communication – non-threatening communication time. - 69 year old woman And I think we have lost something because of the dishwasher that used to be kind of a good time to talk about things and discuss what had been going on in the day and what was bothering you. If you were doing it with your sisters and brothers, or your mother or maid or anybody, it was a nice time. We didn’t realize it at the time. We didn’t care for doing the dishes, but we have lost that now. - 77 year old woman Indeed, we see this trend of designing home entertainment and communication technologies for one user rather than multiple ones repeated today. We use personal computers at home and TiVo® recording systems note preferences for one user, instead of multiple family members, something which has been empirically seen to cause tension in the group-oriented setting of the home [14]. Rather than having a shared landline, we increasingly see people relying on their personal cell phones when communicating, thus limiting the opportunities for serendipitous interactions with others. Once again, we are aware that advances in technology have vastly improved how housework is done today, but we also want to draw attention to the sensual qualities of users’ experience that have been lost over time. We are arguing that ubicomp developers should design systems and devices to support collective as well as individual activity in order to preserve the social interaction that were an important part of our users’ domestic chores.
5 Discussion and Future Work What became clear from our use of historical methods to understand housework was how different domestic life was 50 years ago. It was effective in helping us understand the subtle changes that have resulted with the introduction of new domestic technologies and in opening new space for design. Although the historical texts already revealed themes pertinent to ubicomp design (i.e. labor-saving debate and technology’s gendered character), by drawing on popular texts, patents, and interviews with elders as well, we learned things that could not easily be gleaned from texts alones. For instance, cleaning fluids used to be packaged in glass rather than plastic bottles, the “super sized” packages that detergent are sold in today did not exist, and rather than having dozens of brands of window cleaner to choose from people used to have two or three. Indeed it was primarily the sensual or felt aspects of the domestic experience that appear to have been lost with the introduction of domestic technologies motivated by efficiency. With current interest in restoring felt experience as central to design [22], we believe that historical analysis is an important
48
S. Wyche, P. Sengers, and R.E. Grinter
source for becoming aware of sensual aspects of experiences that have become lost but could be addressed in new forms of technology design. In particular, the multidimensional aspects of our analysis—not only reading historical texts but also looking at patents and images and talking to elders with images as a stimulus— support the development of a rich sense of the felt experience of the activity we are seeking to redesign. In addition to revealing how felt qualities are altered with the introduction of new technologies, another benefit of our historically grounded approach is its potential to inspire radically novel design concepts. A collection of speculative design proposals resulted from our process [see 30 and 36 for details]. Like ethnography, history forces designers to become more aware of their preconceptions about a topic. Because of its ability to defamiliarize the present, history can be a powerful recourse for inspiring innovative computational devices and systems. 5.1 Broader Implications Although our study focused on housework, we believe the same strategies would be useful for other aspects of the domestic environment, such as cooking, childcare, or entertainment, as well as for other activities outside the home. For example, during our interviews, many older adults described massive changes in how they shopped for household goods; exploring this changing history of shopping could reveal design opportunities for e-shopping today. Even the design of workplace systems could arguably be inspired by an understanding of how work has changed over the last century. Central to our techniques for historical analysis is a carefully designed, material artefact which stimulates oral histories. Other than a scrapbook, we believe other mediums have potential for eliciting stories from the past for the purpose of inspiring design. For instance, a view-master with a slide reel from the early games and toys could be an evocative way to elicit stories about how gaming systems have evolved over the last 40 years. Specially-designed recipe books could be deployed to understand how kitchen technologies have changed over time, while researchers exploring urban computing could use old maps and atlases to elicit stories about how urban space has changed. In all these cases, carefully designed artefacts can provide an evocative and engaging focus for conversation with users.
6 Conclusion In recent years ubicomp researchers and developers have increasingly been influenced by methodologies deployed by designers, sociologists, and anthropologists. In this paper we extend this trend to include methods from the discipline of history and demonstrate how doing so can contribute to understanding the domestic environment for the purpose of design. The process of historical analysis that we developed involved four major steps. First, we analysed historical texts to identify major themes in the development of technologies (often automation) for the activities under investigation, in our case housework. Second, we gained a broader understanding of
Historical Analysis: Using the Past to Design the Future
49
the existing technological design space through the search of patents. Third, we developed a personal sense of the changing nature of housework through examination of primary sources from popular culture. Finally, as part of broader fieldwork we gathered oral histories from older people, using a designed, material artefact that reflected the popular history of housework to stimulate memories and reflections. Through these steps, we both developed a better understanding of the activities under design and defamiliarized ourselves from the standard technology design process, opening up new spaces for technology design in the home.
Acknowledgments We would like to thank all of the homemakers who kindly let us into their homes to talk about housework. We would also like to thank Paul Dourish, Giovanni Iachello, Kris Nagel and Amy Voida for their supportive advice on how to structure this paper, and the anonymous reviewers for helpful comments, criticisms, and suggestions. This work was supported by a research grant from the S.C. Johnson and Son Co. and by NSF Grant IIS-0238132.
References 1. Bateson, F.G. (1984) “Self-cleaning Building Constructions.” United States Patent and Trademark Office. Patent no. 4,428,085. 2. Bell, G., Blythe, M. and Sengers, P. (2005) “Making by Making Strange: Defamiliarization and the Design of Domestic Technology.” Transactions on ComputerHuman Interaction (TOCHI), Special issue on Social Issues and HCI, vol. 12, no. 2, pp. 149-173. 3. Bell, G. and Kaye, J. (2002) “Designing Technology for Domestic Spaces: A Kitchen Manifesto.” Gastronomica, vol. 2, no. 2, pp.46-62. 4. Berg, A.J. (1999) “A gendered socio-technical construction: the smart house,” in The Social Shaping of Technology, D. MacKenzie and J. Wajcman (eds.), Second ed. Buckingham: Open University Press, pp. 301-313. 5. Blythe, M., and Monk, A. (2002) “Notes Towards an Ethnography of Domestic Technology.” Proc. DIS 2002, pp. 279-282, London: ACM Press. 6. Blythe, M., Monk, A., and Park, J. (2002) “Technology Biographies: Field Study Techniques for Home Use Product Development.” Extended Abstracts CHI 2002, pp. 658659, Minneapolis, MN: ACM Press. 7. Consolvo, S., Arnstein, L., and Franza, B.R. (2002) “User Study Techniques in the design and Evaluation of a Ubicomp Environment.” Proc. UbiComp ’02, pp. 73-30, Göteborg, Sweden: Springer. 8. Cowan, R.S. (1976) “The ‘Industrial Revolution’ In the Home: Household Technology and Social Change in the 20th Century.” Technology and Culture, vol. 17. pp.1-23. 9. Cowan, R.S. (1983) More Work for Mother: The Ironies of Household Technology from the Open Hearth to the Microwave. Basic Books Inc. 10. Fram, E. (2005) “The Booming Scrapbooking Market in the USA: Despite Phenomenal Growth, the Future’s Unclear.” International Journal of Retail & Distribution Management. vol. 33, no. 3, pp. 215-225.
50
S. Wyche, P. Sengers, and R.E. Grinter
11. Forlizzi, J., and DiSalvo, C. (2006). “Service Robots in the Domestic Environment: A Study of the Roomba Vacuum in the Home,” Proc. Of Human-Robot Interaction 2006. Salt Lake City, Utah: ACM Press. 12. Gaver, W. (2001) The Presence Project. Computer Related Design Research, RCA London. 13. Gaver, W., Dunne, A., and Pacenti, E. (1999) “Cultural Probes,” Interactions. vol. 6, no. 1, pp 21-29. 14. Grinter, R.E., Edwards, W.K., Newman, M.W., and Ducheneaut, N. (2005) “The Work to Make a Home Network Work,” Proc. of the Ninth European Conference on ComputerSupported Cooperative Work (ECSCW’05). Paris, France. 15. Harper, D. (2002) “Talking About Pictures: A Case for Photo Elicitation.” Visual Studies. vol. 17, no.1. pp. 13-26. 16. Hoopes, J. (1979) Oral History: An Introduction for Students. Chapel Hill, NC: The University of North Carolina Press. 17. Hoy, S. (1997) Chasing Dirt: The American Pursuit of Cleanliness. Oxford University Press. 18. IDEO (2003) “IDEO Method Cards: 51 Ways to Inspire Design.” San Francisco, CA: W. Stout Architectural Books. 19. Lupton, E. (1993) Mechanical Brides: Women and Machines from Home to Office. New York, NY: Cooper-Hewitt National Museum of Design. 20. Marshall, C. and Rossman, G.B.(1998) Designing Qualitative Research. Newbury Park, CA.: Sage Publications, Inc. 21. Martens, L. and Scott, S. (2005) “The Unbearable Lightness of Cleaning: Representations of Domestic Practice and Products in Good Housekeeping Magazine: 1951-2001.” Consumption, Markets and Culture, vol. 8, no.4, pp. 379-401. 22. McCarthy P., and Wright, P. (2004) Technology as Experience. Cambridge, Mass.: MIT Press. 23. Modell, J. and Brodsky, C. (1994) “Envisioning Homestead: Using Photographs in Interviewing” in Interactive Oral History Interviewing, E. McMahan and K.L. Rogers (eds.), Hillsdale, NJ: Erlbaum. 24. Rode, J.A., Toye, E.F. and Blackwell, A.F. (2004) “The Fuzzy Felt Ethnography Understanding the Programming Patterns of Domestic Appliances.” Personal and Ubiquitous Computing 8 , pp. 161-176 25. Rode, J. (2005) “Appliances for Whom? Considering Place.” Proc. of the 3rd International Conference on Appliance Design. Bristol, UK. 26. Salvador, T., Bell, G. and Anderson, K. (1999) “Design Ethnography.” Design Management Journal, vol. 10, no. 4, pp. 35-41. 27. Salvador, T., and Anderson, K. (2003) “Practical Considerations of Context for Context Based Systems: An Example from an Ethnographic Case Study of a Man Diagnosed with Early Onset Alzheimer's Disease.” Proc. UbiComp ’03, pp. 243-255. Seattle, WA: Springer. 28. Santanaya, G. (1953). “Reason in Common Sense.” The Life Of Reason, Rev. Ed. New York: Charles Scribner’s Sons. 29. Sengers, P. (2004) “Doomed to Repeat? How History Can (and Should!) Inform Home Technology.” Position paper presented at CHI 2004 Workshop on “Designing Culturally Situated Technologies for the Home.” Available online: www.cemcom.infosci.cornell.edu/ papers/doomed-to-repeat.pdf 30. Sengers, P., Kaye, J., Boehner, K., Fairbank, J., Gay, G., Medynskiy, E., Wyche, S. (2004) “Culturally Embedded Computing.” IEEE Pervasive Computing, vol. 03, no.1, pp. 14-21.
Historical Analysis: Using the Past to Design the Future
51
31. Spigel, L. (2001) “Yesterday’s Future, Tomorrow’s Home.” Emergences, vol. 11, no. 1., pp. 29-49. 32. Strasser, S. (2000) Never Done: A History of American Housework. New York, NY.; Henry Holt. 33. The Sears Roebuck Catalogues of the Thirties: Selected Pages from the Entire Decade. (1978) Franklin Square, N.Y.: Nostalgia, Inc. 34. Vanek, J. (1974) “Time Spent in Housework.” Scientific America, vol. 231, pp. 116-120. 35. Wilson, W. (1961) “Combined Dining Table and Dishwasher.” US Patent no. 2,971,519. 36. Wyche, S. (2005) “Designing Speculative Cleaning Products for the Home.” Proc. DUX ’05. San Francisco, Ca.: ACM Press. 37. Zaleski, A.M. (1973) “Combination Toy Dog and Vacuum Cleaner.” United States Patent and trademark Office. Patent no. 3,771,192.
Extending Authoring Tools for Location-Aware Applications with an Infrastructure Visualization Layer Leif Oppermann1, Gregor Broll2, Mauricio Capra1, and Steve Benford1 1
Mixed Reality Laboratory, University of Nottingham Wollaton Road, Nottingham, NG8 1BB, UK {lxo, mxc, sdb}@cs.nott.ac.uk 2 Embedded Interaction Research Group, University of Munich Amalienstraße 17, 80333 Munich, Germany
[email protected]
Abstract. In current authoring tools for location-aware applications the designer typically places trigger zones onto a map of the target environment and associates these with events and media assets. However, studies of deployed experiences have shown that the characteristics of the usually invisible ubiquitous computing infrastructure, especially limited coverage and accuracy, have a major impact on an experience. We propose a new approach in which designers work with three layers of information: information about the physical world, information about digital media, but also visualizations of ubiquitous infrastructure. We describe the implementation of a prototype authoring tool that embodies this approach and describe how it has been used to author a location-based game for mobile phones called Tycoon. We then outline the key challenges involved in generalizing this approach to more powerful authoring tools including acquiring and visualizing infrastructure data, acquiring map data, and flexibly specifying how digital content relates to both of these.
1 Introduction As location-aware applications continue to emerge and mature so attention increasingly turns to the question of how best to author and configure them. Applications as diverse as tourism, games, navigation and mobile information retrieval require the creation of content in which digital assets – images, sounds, text, video and graphics – are attached to locations in the physical world to be triggered by participants who enter, leave or dwell in them. Dedicated authoring tools and techniques for location-aware applications have begun to emerge in response. All of the examples produced to date share a common overarching approach – the idea of taking a map of the physical area in which the experience is to take place and then somehow placing or drawing a series of regions, locales or hotspots on top of this that are triggered according to a set of basic events such as participants entering and leaving them. The Mediascape tool (see figure 1, left) from the Mobile Bristol project is one such tool in which designers can place different sized and shaped triggers on a map and associate these with a range of events. Mediascape [1] has been used to support a P. Dourish and A. Friday (Eds.): Ubicomp 2006, LNCS 4206, pp. 52 – 68, 2006. © Springer-Verlag Berlin Heidelberg 2006
Extending Authoring Tools for Location-Aware Applications
53
variety of location-aware applications including: Scape the Hood [2] in which users explored an area of San Francisco, triggering stories as they entered different city blocks and Riot! [3], an audio experience in which participants triggered spatialized sounds as they explored a large city-centre square. A variant on this approach is the idea of Colour Maps (figure 1, right), in which artists directly paint location triggers onto a map, affording a high degree of flexibility in terms of their size and shape and also enabling them to use existing and familiar paint tools. This approach was used to support the projects Uncle Roy All Around You in which a group of artists defined a trail of text clues around a city for players to follow as part of an interactive performance [4], and Savannah in which groups of six children at a time role played being lions on a virtual savannah that appeared to be overlaid on their school playing field [5]. As a third example, Caerus [6], enables a designer to administer trigger areas that overlay a digital map. The user can add further maps, points of interest and multimedia streams.
Fig. 1. The design of Scape the Hood in Mediascape (left), The design of Uncle Roy All Around You using Colour Maps (right)
2 Designing for the Uncertainty of Positioning and Communications However, published studies of location-aware applications suggesting that there is more to designing them than attaching media assets to a map. Rather – in strong contrast to traditional wired applications – designers need to be aware of the inherent uncertainties in the ubiquitous infrastructure, most notably limited coverage of wireless communications and limited coverage and accuracy of positioning systems. Studies of Can You See Me Now? (CYSMN), a game of chase in which online participants were chased through a 3D virtual model of a city by performers who, equipped with PDAs and GPS units, had to run through the actual city streets in order
54
L. Oppermann et al.
to catch them, showed that limited coverage and accuracy had a major impact on the game [7]. Performers, game managers and designers had to understand and account for the local characteristics of the ubiquitous computing infrastructure at each new city that they visited, building up a common stock of knowledge as to coverage and performance blackspots. The extent of the problem can be seen on in figure 2 which visualizes the characteristics of GPS and Wi-Fi over the course of a two hour performance on a peninsula in Rotterdam. The light colored areas correspond to locations where performers acquired a GPS fix and were able to transmit it back to the game server over Wi-Fi (i.e., where both Wi-Fi and GPS were available). Light blue shows high estimated GPS accuracy and light green lower accuracy. The dark areas represent areas where either the performers never ventured or some combination of GPS and/or Wi-Fi were not available. Given that the performers ventured widely, there were clearly many areas that were not playable, especially the central thoroughfares between the main buildings (which are shown as back rectangles – the surrounding black area is water where the performers never went). Furthermore, the availability and performance of GPS and Wi-Fi vary over time as objects move into and out of the environment and GPS satellites move over the sky, so that such pictures can in fact be highly dynamic. This raises a fundamental question for location-aware authoring tools? What is the point of placing assets at physical locations where there is insufficient coverage or performance to reliably access them? Or put another way, how can we enable designers to taken account of the usually invisible ubiquitous infrastructure as well as the nature of the physical environment when authoring an experience. This is the key challenge addressed by this paper.
Fig. 2. A visualization of GPS and Wi-Fi coverage and accuracy from CYSMN
Extending Authoring Tools for Location-Aware Applications
55
3 Extending Authoring Tools to Reveal the Ubiquitous Infrastructure In our extended approach, we support the designer with knowledge of the physical world and the ubiquitous infrastructure at the same time. The authoring process now involves overlaying information characterizing communications and sensing systems on meaningful backgrounds that facilitate the orientation of the game designer. For geo-coded data this would be a map of the area, for time-stamped data this might be a timeline. For the scope of this paper we will limit the possibilities to interactively visualizing geo-coded data on maps. Figure 3 summarizes the overall concept behind our authoring tool for locationaware applications. The tool enables designers to work with three layers of information: the physical environment layer, with representations of the target physical environment; the infrastructure layer, with representations of the ubiquitous computing infrastructure across this environment, and the content layer, with representations of digital media. In practice, each layer may consist of a series of sublayers that reveal different types of information: maps, aerial photographs and GIS data for the physical environment; recorded and predicted information about the coverage and accuracy of communications and sensing systems for the infrastructure layer; and triggers, hotspots, assets and events and the content layer.
3. Content layer (regions, colour maps, events and assets)
2. Infrastructure layer (static and dynamic aspects of availability and accuracy)
1. Physical world layer (maps, aerial photographs, GIS)
Fig. 3. Authoring Content Triggers based on Infrastructure Visualization
The key innovation here in relation to previous tools is the insertion of the infrastructure layer. This layer helps the designer to understand the spatial spread of otherwise invisible infrastructure (e.g. GSM network, Wi-Fi, GPS) at authoring time.
56
L. Oppermann et al.
4 Designing the Authoring Tool Our new authoring tool is designed to work with a large number of samples that measure the characteristics of the underlying infrastructure. Due to potential screen space constraints we decided to implement the visualizations in an interactive way. We have adopted a ‘detail-on-demand’ approach to visualization following Ben Shneiderman’s Visual Information Seeking Mantra: “Overview first, zoom and filter, then details-on-demand” [8]. wxWidgets1, OpenGL2 and SDL3 are used as a platform independent base-system, providing the necessary user input and output channels for our interactive application as shown in the data-flow diagram in figure 4.
Map Data
Custom Parsers
DB
Interactive Application
Sensor Samples Fig. 4. Data-Flow Diagram
For our maps, we used closed source vector-data from Ordnance Survey4 and parsed it with FWTools5 before rendering it with Mapserver6 to a raster file-format. The sensor samples have been collected using a Place Lab compatible program and parsed from a flat-file storage using our custom parser. All input data for the physical world layer and infrastructure layer is currently accessible using abstract C++ interfaces. The overall architecture leans towards utilizing web services in the future, which would support the use of different programming languages across different computers for different interaction tasks. 4.1 Physical World Layer This layer provides the user with a meaningful background in regard to the target environment. For the current prototype we limited the background to 2D maps in Cartesian coordinates. Reading common file-formats, the tool also allows for using digitized maps or orthophotos. The loaded bitmaps can then be geo-coded by supplying two points, both in the image coordinate-space and the real world coordinates. Ideally this would be the lower-left and upper-right coordinates of the map. 1
http://www.wxwidgets.org/ http://www.opengl.org/ 3 http://www.libsdl.org/ 4 http://edina.ac.uk/digimap/description/ 5 http://fwtools.maptools.org/ 6 http://mapserver.gis.umn.edu/ 2
Extending Authoring Tools for Location-Aware Applications
57
4.2 Infrastructure Layer The infrastructure layer contains samples of network information tagged with time and/or position. It is projected in the same way as the background map and thus the samples are put into the right context when the two layers are stacked up. This enables the user to reason about the infrastructural data, e.g. decide about whether an area has been sampled densely enough. Figure 5 (middle) shows a generally well sampled area which could eventually need some more data on the northern and western side of the area. Each of the green dots in the image represents a sample (in this case of GSM cell data that has been geo-referenced using GPS).
Fig. 5. A map of Nottingham, UK (left), overlaid with GSM sensor-data (middle), view interactively filtered by Cell ID (right)
The samples are referenced by unique ID-strings. If the ID-strings themselves should be used to trigger content in the final experience they will have to be reproducible on the target device by accessing the infrastructure. Examples of sources for such possible ID-strings are given in the following table. Table 1. Reproducible ID-strings for different infrastructures
GSM Wi-Fi GPS
Full Cell ID (MCC_MNC_LAC_CI) BSSID (Mac Address of Access Point) Location (latitude/longitude)
4.3 Content Layer The interactive visualization itself only acts as a hint to the user. The actual content is defined and refined in the content layer. Here the author can make informed decisions about where to place assets, where the experience should take place or which observed sensors could serve as triggers for events. For example the right image in figure 5 shows the infrastructure filtered to show the possible spread of a single GSM cell across a region of space. This cell could arguably be used to trigger events which have a notion of location but do not require exact latitude/longitude coordinates, e.g. traffic info for the surrounding area or virtual locations in a mobile phone game. The following section presents a game design that works using this principle.
58
L. Oppermann et al.
5 Authoring Tycoon – A Game for Mobile Phones In order to demonstrate our proposed approach to authoring ubiquitous experiences we have created Tycoon, a location-based multiplayer trading-game for mobile phones. The gameplay in Tycoon is driven by players entering and leaving different mobile phone cells which are associated with different game resources. Choosing which resources to associate with specific cells is therefore a key aspect of designing the game and in turn, requires an understanding of how the cells are distributed across physical space. The basic idea behind Tycoon’s gameplay is a simple producer-consumer-cycle. Players have to roam a designated gaming area (e.g. a park or the centre of a city), collect resources and trade them for objects that earn them certain amounts of credits. The objective of the game is to collect as many credits as possible in order to win the game. Tycoon uses the different physical GSM-cells of a service-provider network within the gaming area and maps one or several of them to virtual locations in the game that represent producers, consumers or neutral areas (figure 6). The game uses the metaphor of a wild-west scenario to communicate its basic mechanisms of collecting resources from producer-locations which are called “mines” and using them to buy objects from consumer-locations which are called “brokers” and have the names of towns or counties in California, USA.
Fig. 6. Example of a Tycoon gaming-area with its mapping of physical locations to consumers (C) and producers (P)
During the game, players have to explore the gaming area and discover the locations of brokers and mines (figure 7, left). After their discovery, players can collect local resources from mines by staying in them for a certain period of time (figure 7, middle). Each mine provides an unlimited amount of one local resource in the game – gold, silver or copper – which differ in their value and the time it takes to collect them. They are called “local resources” since players collect them independently from each other and do not compete for them.
Extending Authoring Tools for Location-Aware Applications
59
Brokers on the other hand each keep a list of unique global objects, e.g. different buildings in towns or estates in counties (figure 7, right). Players can see these lists after having discovered the location of the corresponding broker. The number of global objects in the game is limited, each one is unique (e.g. there is only one hotel in the city of Bakersfield) and players compete against each other for claiming them. Each object has a certain price in different local resources and a corresponding value of credits. Players can claim global objects from a broker’s list after entering his area and earn the objects’ credits provided they can afford their price and the desired objects are still available.
Fig. 7. Different screen images from Tycoon’s interface: location discovery (left), collecting Resources (middle), claiming objects (right)
5.1 Seams in Mobile Applications Ubiquitous Computing systems and especially applications for mobile communication and navigation are susceptible to the effects of so-called seams in the underlying technical infrastructure. Seams can be seen as deviations in actual use from a notional ideal of technological continuity or uniformity, including discontinuities in technologies themselves and discontinuity between what actually happens and what the system observes [9]. They are mostly caused by technical limitations and constraints in the heterogeneous infrastructure of a system. They may reveal their effects on the user experience, showing themselves as uncertainties, ambiguities or inconsistencies when users interact with a supposedly seamless system. Common examples for seams in mobile applications are deviations in positioning, unavailability of network services or patchy network coverage. As such, they may come to the users’ attention as interruption and loss of services or uncertainty about current position. Previous research has identified a range of different strategies for dealing with these kinds of seams when designing ubiquitous experiences [10]. These include improving or deploying the underlying technologies so as to remove them; carefully designing the experience to hide their worst effects from users; carefully managing the experience from behind the scenes; revealing the presence of seams to users so that they can adapt to them; and even deliberately exploiting the seams as part of the design (the approach of ‘seamful design’ as described in [9]). All of these approaches
60
L. Oppermann et al.
require that the designer become aware of the presence of seams across their chosen physical environment, which of course is the purpose behind our proposal to add a visualization layer to current authoring tools. Tycoon follows the approach of exploiting seams that arise from the use of GSM cells as a basis for positioning. Users are normally unaware of their current GSM cell or location while using their mobile phones, as this information is mostly hidden by the system and the handover between adjoining cells is handled seamlessly [11]. While seamless design usually neglects this information, our seamful approach to mobile games reveals the presence of individual mobile phone cells and uses them to drive the overall user experience. Since location-awareness is crucial for Tycoon’s gameplay, we use information from GSM cells for positioning which is sufficient for our approach but also implies several constraints and seams. Figure 8 gives an example of GSM cell propagation and coverage which depends on many factors, is very dynamic and has irregular, fluctuating shapes. Cells do not have fixed boundaries and do not share exact borders with adjoining cells but often overlap with them. These properties can become a problem for mobile applications whose behaviour is dependent on exact positioning.
Fig. 8. GSM cell coverage and propagation indicated by samples of cell-IDs and their GPSposition in London
Being a mobile multiplayer game, Tycoon’s gameplay is driven by the competition between several players with mobile phones. Tycoon is implemented using a clientserver-architecture and players need continuous access to globally shared data on the game-server in order to compete successfully with each other. Therefore mobile Internet-access is most convenient and almost indispensable for accessing the global server and synchronising its shared game-state with mobile clients. A seamless approach would try to guarantee as little inconsistency between the global and all
Extending Authoring Tools for Location-Aware Applications
61
local game-states as possible by frequent updates between the mobile clients and the game-server. Whenever a mobile client locally changes the game-state, e.g. by claiming a globally shared object, the game-server would either have to route this change to all clients or they would periodically have to ask the server for updates on the latest changes. Either way, a considerable volume of expensive GPRS-traffic would be generated. Given the definition of seams in [9], these expenses might also be considered to be self-made, artificial seams, since they are neither caused by technical constraints nor show themselves as uncertainties or ambiguities. But as costly expenses for GPRS-traffic still constrain the uniformity and continuity of mobile applications, we want to treat them as a significant seam. The frequency of updates between server and mobile clients in order to maintain a globally correct game-state is tightly linked to the probability of data inconsistencies which occur as a consequence of insufficient synchronisation. Inconsistencies emerge when individual clients update data that is shared with other clients through a common server. When a Tycoon client synchronises its locally altered data with the shared server in order to update the globally shared game-state, e.g. after claiming an object, local copies of that data on other clients become inconsistent with the updated game-state and have to be updated accordingly. 5.2 Augmenting Mobile Gameplay with Seamful Design Tycoon’s game-logic and interface are designed to incorporate and exploit the seams of dynamic cell positioning, expensive internet-access and data-inconsistencies in order to enrich its gameplay. Tycoon’s gameplay is location-dependent and applies positioning through the recognition of unique GSM cell identifiers in a gaming area. As mentioned above, this method of positioning is not very accurate and can comprise considerable deviations. In order to improve navigation and orientation for the user, the main screen of Tycoon’s interface always displays the name of the current location, along with the amount of collected resources and earned credits (figure 9, left). When a player changes from one cell into another, an alert is triggered and a notification about them entering a new location is displayed (figure 9, middle). Afterwards the name of the new location is shown on the main screen until the next change (figure 9, right).
Fig. 9. Tycoon’s alert-mechanism for location-visualization
62
L. Oppermann et al.
This alert-mechanism is more flexible than a static map of the gaming area, reveals information about fluctuating coverage and propagation of GSM cells, and improves the visualization of locations and their boundaries. Thus the design of Tycoon is intended to decrease players’ uncertainties about their position, improve their interaction with the seam of dynamic cell propagation and use it as an important part of game logic. Instead of providing players with a complete map of the area showing them exactly where to find mines and brokers, we want them to start the game by having to explore the area, gather their own knowledge about it and discover mines, brokers and their locations themselves. This way, players are encouraged to use their spatial knowledge about the gaming-area in order to adopt their own strategies of moving between cells and finding the most efficient tactics of which resources are needed to buy which available objects and where to find them in nearby mines. During this process of appropriation, players can exploit positioning ambiguities caused by dynamic cell propagation more effectively when they find an area where adjoining cells overlap and easily flip after some time without moving. This issue is especially interesting for designing the gaming-area and assigning mines and brokers to different locations. 5.3 Considering Locations According to the Game-Design Since the game is running on a mobile phone, it is easy to assume the phone visits different locations given that people move over time. We wanted the game to be playable during the players’ daily life without neither interfering too much with what they are doing nor requiring to them to visit special places. We therefore decided that locations in the game should be mapped to places that people come across anyway. Our game-design defines that Tycoon is going to be played by the inhabitants of a city over a long summer weekend to facilitate discovering places. Assuming that people are doing their shopping at weekends we choose the city-center and shopping malls to be the brokers. Parks and recreational areas are defined to be the mines. The next section shows how the current prototype tool can be used to author Tycoon for these locations.
6 Authoring for Locations in Tycoon This section shows our prototype design tool can be used to author Tycoon. We show how the infrastructure visualization layer helps to make a GSM based location based game. Using the tool we define cell ID based regions that serve as triggers for virtual locations in the game. In the following, the presented GSM infrastructure data has been collected over a period of two days using one phone and a GPS receiver. Figure 10 above shows two screenshots of the authoring tool with a 6 x 6 km view of a city in England overlaid with infrastructure data from the Orange GSM network. Thousands of coloured dots represent where a combination of GSM and GPS data has been sampled. Each dot represents one sample and shows that a particular cell ID has
Extending Authoring Tools for Location-Aware Applications
63
Fig. 10. Initial graphical definition of a region
been seen at this specific position. Dots with the same colour represent the same IDstring (here: cell ID). By hovering the mouse over the visualization the user can see which ID is next to the mouse pointer and where else that ID has been observed. The samples with the current ID are highlighted by jumping yellow dots (bold dots in these screenshots). Having animated features allows the user to easily distinguish between samples that belong to the current selection and those that do not. The current selection of one or more IDs is shown in the listbox to the right of the window. The user can store this selection as a region by giving it a unique name (figure 10, right). The region then appears in the region listbox to the left (see figure 11, left).
Fig. 11. Close up (left) and refined definition (right)
By zooming the view, the user can look over the details of their selection. Figure 11 shows the same selection as before but with a more close-up view. In this example, the user wanted to select the area around the park in the left part of the image. Although the area seems to be well covered by just one cell, the visualization certainly shows samples with different colours nearby. The user can now modify his selection in the zoomed view and apply any modifications should they be needed.
64
L. Oppermann et al.
Fig. 12. Zoomed out for overview (left) and save refined region (right)
Figure 12 (left) shows the refined region definition that covers more of the park. There are now 4 Cell IDs in the selection. By zooming out, the selection can be visually verified once again before saving it as a region if satisfied (figure 12, right). The resulting region definitions (textual ID-strings) can then be post-processed and translated into an application specific format to serve as location triggers in the game. 6.1 Deploying the Game Content to the Clients The Tycoon game client runs on Symbian phones and uses a client-server architecture. The server uses a standard LAMP7 software installation. The MySQL database on the server is filled by a PHP script parsing the XML region definitions (see figure 13). The same XML-file is also used by another PHP script to configure the clients when they log into the server.
4 Sacramento broker 234_33_237_29678 234_33_37_34457 234_33_37_24949 Fig. 13. Example XML Content Definition (Excerpt)
Figure 13 shows an excerpt from the content definition. It defines the region number 4 called “Sacramento” to be a broker. The region will be triggered when a player’s phone is connected to one of the cells that construct the supplied ID-strings. Figure 14 shows the propagation of the regions from the XML file through the server to the clients. 7
Lamp = Linux, Apache, MySQL, PHP/Perl/Python.
Extending Authoring Tools for Location-Aware Applications
XML
PHP
Symbian
content
/MySQL
clients
65
Internet (GPRS)
Fig. 14. Propagation of content from server to client
7 Challenges for Further Work We have presented a prototype authoring tool that supports the designer of locationaware applications with interactive visualizations of ubiquitous infrastructure in addition to the typical map background. The prototype has been used to author a location aware game for mobile phones that does not require a GPS device for positioning. However, if we want to generalize our authoring approach and transform it into something that can be used anywhere in the world we need to address some key issues related to our three underlying layers: the maps, the infrastructure and the visualizations. 7.1 Issues with Maps (Physical World Layer) Maps form the background of the authoring view. They are not strictly required but do help give an understanding of the physical area in which an experience will take place and will therefore often be used. There are two important data-sources for this layer: printed maps/photos and online-services/GIS. Existing printed maps or aerial photos can be used if the coordinates are known or can be measured. The image just needs to be digitized in this case and afterwards geo-referenced. Online services can be used to acquire digital maps. The advantage of digital maps is that they are already precisely measured. This means that the overlay in the authoring tool will be as precise as possible. Unfortunately most of the services on the market are only available after the payment of a fee. Our visualization tool is based on Digimap data from Edina [12] due an existing subscription made by our university. Edina offers several products that could be added to the visualization as additional information layers, e.g. transport networks, boundaries, buildings, contours lines, etc. Another (free) service we used is Google Earth, especially its KML markuplanguage [13]. The KML-language allows us to quickly plot the surveyed data onto a virtual globe and thus visually check the data for validity. The real image of buildings, roads or cities for example helps us to understand the context in which the data was collected. However, one of the main issues for the future of our approach is scalability. This can bring some difficulties in the visualization process, especially in map acquisition across many different countries. We are conscious that our approach needs to be flexible to interoperate with web based services like Web Map Service and Web
66
L. Oppermann et al.
Feature Services as defined by the Open Geospatial Consortium [14]. Both services are Internet operated and return data and/or images of a specific area on request. Also Google Maps and Google Earth will continue to be helpful for what we are doing. Another important point is to make our tool is able to communicate with GIS packages like the open source GRASS [15] and PostGIS [16] or commercial packages like Erdas Imagine [17] or ArcGIS [18]. More generally, we suggest that authoring tools for location based games are getting closer to Geographical Information Systems (GIS) and will soon begin to merge. 7.2 Issues with Data Collection (Infrastructural Layer) Our main source of infrastructure data is self-collected. It has been initially obtained using the Placelab [19] stumbler software but we later moved to our own program called Pystumbler. The program takes a snapshot of the supported networks (i.e. GSM on the phone) about every second and also gets a GPS fix about every few seconds. The file-format is a well formed text-file which is compatible across platforms. We wrote a parser to support this file-format. Pre-mapping an area using a network stumbler is time-consuming. It can take several hours to map an area in a dense city centre and several days to get a rough picture of the observed infrastructure in wider parts of a city. The ultimate goal of the stumble method would be to know how the characteristics of the infrastructure on every street. This is not feasible with only few people doing the data collection. For this reason this approach is not feasible for games that are to be played on a much larger, national or planetary scale without modifications. It is therefore interesting to see how the data collection process can be interwoven with the game-play or how data-collection can be archived as a by-product of other applications, e.g. games. In addition to the more sophisticated stumbler setup using GPS we also created a less obtrusive long-term data collector that can run as a background task on a phone for a very long time without requiring any attention. It logs the cell ID once per minute and also every cell ID change up to a granularity of ten seconds. We have tested this software to track the movement of ten persons over the period of a month. The generated log-files contain universally valid cell adjacency information as well as subjective (i.e. only valid for the subject) patterns of movement. Knowing that other people do similar things, it is also interesting to interoperate with projects like Wigle [20], Place Lab [21] or Mobilife [22] to utilize the data which has already been collected. Infrastructural data collection (also known as ‘wardriving’), especially for Wi-Fi networks has been done for many years. Some of the data is shared through online-databases. The data presented there is usually the averaged position per network ID which means that the data shows less detail than the self collected trails of stumbler data but is potentially still very useful as it would allow authoring for unknown areas if it can be trusted. A verification and modification (cleaning) process will also be needed later on. 7.3 Issues with the Visualization Choosing an appropriate visualization style for the data is the key question for any information visualization application. For our current prototype we deliberately decided to not abstract the data in any way (e.g. creating bounded regions from collections of
Extending Authoring Tools for Location-Aware Applications
67
individual sample points) but rather have tried to do as much as we can with the full data in its pure form (i.e. each sample visible on screen). We choose an interactive visualization because it allows the user to make quick visual queries into the data. The current point visualization is however not enough. Many different views on the same data are possible and the tool should be flexible enough to allow those different views to be set up simultaneously. Not all visualizations will require a highperformance implementation in C++ and it should be possible to access the shared data using other programming languages like Java and Python as well. Also an advanced user should be able to interact with the data through a scripting interface for those occasions where the visualization alone is not sufficient. Beyond this, we identify two major challenges for visualization. First is representing uncertainty. The collected data will be subject to uncertainties that designers may need to bear in mind. Positioning technologies such as GPS (that underpins the Tycoon data) are subject to error. Furthermore, data sets will be incomplete, requiring designers to interpolate between particular samples. Visualizations may need to convey uncertainty or be able to suggest possible interpolations. A second challenge lies in the time-varying nature of the ubiquitous infrastructure. Visualizations may also need to account for the ways in which coverage and accuracy vary over time as well as over space.
8 Conclusion The idea that designers can author location-aware experiences by placing locales or hotspots on maps and other representations of physical spaces is already well established and forms the basis of several existing prototype tools. However, studies of previous experiences suggest that designers also need to be able to account for the characteristics of the usually invisible infrastructure of positioning and communication technologies, especially (but not only) their limited coverage. We are therefore proposing that authoring tools for location-aware experiences be extended to include visualizations of the likely state of the infrastructure across a selected environment so as to guide designers as to how their intended experience might actually be experienced by end users in practice. Essentially, we propose that future authoring tools should provide access to three broad layers of information (each of which might consist of several sub-layers): a physical world layer that describes the target physical environment; an infrastructure layer that represents the likely state of the infrastructure across this target environment; and a content layer that defines locales or hotspots that associate events or media assets with geographic locations. We have produced an initial demonstration of how this general approach can assist the design of a location-based game, and have discussed key challenges that need to be met at each layer in order to develop it further. Our future work will of course involve addressing these challenges. It will also involve refining our current prototype authoring tool and working with external designers in order to elicit their more detailed requirements and to evaluate the longer-term potential of our approach through practical experience.
68
L. Oppermann et al.
Acknowledgements We gratefully acknowledge the support of IPerG, the European Project on Pervasive Gaming, project FP6-004457 under the European Commission's IST Programme (www.pervasive-gaming.org). We are also grateful for additional support from the Equator Interdisciplinary Research Collaboration which has been funded by the UK’s Engineering and Physical Sciences Research Council (www.equator.ac.uk).
References 1. Hull, R., Clayton, B., Melamed, T.: Rapid Authoring of Mediascapes. In: Proc. Ubicomp (2004) 125-142 2. Mediascape. Scape the Hood.[Online]. Available: http://www.webtales.us/Scape1.html 3. Mediascape. Riot! - An Interactive Play for Voices.[Online]. Available: http://www.mobilebristol.com/QueenSq.html 4. Flintham, M.: Painting the Town Red: Configuring Location-Based Games by Colouring Maps. In: Proc. ACE (2005) 5. Benford, S., et al.: Life on the Edge: Supporting Collaboration in Location-Based Experiences. In: Proc. CHI (2005) 721-730 6. Naismith, L.: CAERUS Overview. The University of Birmingham (2004). [Online]. Available: http://portal.cetadl.bham.ac.uk/caerus/default.aspx 7. Crabtree, A., et al.: Orchestrating a Mixed Reality Game ‘On the Ground’. In: Proc. CHI (2004) 391-398 8. Shneiderman, B.: The Eyes Have It: A Task by Data Type Taxonomy for Information Visualization. In: Proc. IEEE Symposium on Visual Languages (1996) 336-343 9. Chalmers, M. And Galani, A. 2004. Seamful Interweaving: Heterogeneity In The Theory And Design Of Interactive Systems. In Proceedings Of The 2004 Acm Symposium On Designing Interactive Systems, August 2004, Cambridge, Massachusetts, Acm Press, 243-252. 10. Benford, S, Crabtree, A., Flintham, M., Drozd, A., Anastasi, R., Paxton, M., Can You See Me Now?, ACM Transactions on CHI, 13 (1), March 2006. 11. Willassen, S. Y.: A method for implementing mobile station location in GSM. Norwegian University (1998) 12. Edina. (2005) Digimap.[Online]. Available: http://edina.ac.uk/digimap/description/ 13. Google. Google API.[Online]. Available: http://code.google.com/apis.html 14. Open Geospatial Consortium Adopted Document Baseline.[Online]. Available: http://www.opengeospatial.org/specs/?page=baseline 15. GRASS Development Team. (2005) Welcome to GRASS GIS.[Online]. Available: http://grass.itc.it/ 16. Refractions Research. (2005) What is PostGIS?[Online]. Available: http://postgis.refractions.net/ 17. Leica Geosystems LLC. (2004) IMAGINE Virtual GIS. Leica [Online]. Available: http://gis.leica-geosystems.com/Products/files/vgis_ds.pdf 18. ESRI. (2006) ArcGIS Family of Products.[Online]. Available: http://www.esri.com/software/arcgis/index.html 19. LaMarca, A., et al.: Place Lab: Device Positioning Using Radio Beacons in the Wild. In: Proc. Pervasive Computing (2005) 116-133 20. Wigle. [Online]. Available: http://www.wigle.net/ 21. Placelab. [Online]. Available: http://www.placelab.org/ 22. MobiLife. [Online]. Available: http://www.ist-mobilife.org/
Automated Generation of Basic Custom Sensor-Based Embedded Computing Systems Guided by End-User Optimization Criteria Susan Lysecky1 and Frank Vahid2,* 1
Department of Electrical and Computer Engineering University of Arizona Tucson, AZ 85721
[email protected] 2 Department of Computer Science and Engineering University of California, Riverside Riverside, CA 92521
[email protected]
Abstract. We describe a set of fixed-function and programmable blocks, eBlocks, previously developed to provide non-programming, non-electronics experts the ability to construct and customize basic embedded computing systems. We present a novel and powerful tool that, combined with these building blocks, enables end-users to automatically generate an optimized physical implementation derived from a virtual system function description. Furthermore, the tool allows the end-user to specify optimization criteria and constraint libraries that guide the tool in generating a suitable physical implementation, without requiring the end-user to have prior programming or electronics experience. We summarize experiments illustrating the ability of the tool to generate physical implementations corresponding to various end-user defined goals. The tool enables end-users having little or no electronics or programming experience to build useful customized basic sensor-based computing systems from existing low-cost building blocks.
1 Introduction The cost, size, and power consumption of low-end microprocessors in the past decade has dropped tremendously as silicon technology continues to follow Moore’s Law. For example, an 8-bit microprocessor chip may cost less than $1, occupy just a few millimeters, and consume just microwatts of power. Such reductions enable integration of microprocessors into domains previously unthinkable, such as RFID tags, ingestible pills, and pen tips. Meanwhile, a problem in the design of basic sensor-based embedded computing systems is that end-users cannot setup basic custom embedded systems without the assistance of engineers. An end-user is an individual developing a sensor-based computing system who likely does not have programming or electronics expertise, *
Also with the Center for Embedded Computer Systems at UC Irvine.
P. Dourish and A. Friday (Eds.): Ubicomp 2006, LNCS 4206, pp. 69 – 86, 2006. © Springer-Verlag Berlin Heidelberg 2006
S. Lysecky and F. Vahid
(a)
In hallway
70
Light Sensor
Motion Sensor When A is
yes no
AND B is yes no OR then the output is yes
Combine
Wireless Rx Wireless Tx
LED
Parent’s Room LED (green)
When A is
(b)
Button
Splitter
Yes Prolonger 20 min
Splitter
Yes Prolonger 25 min
Splitter
yes no
AND B is OR
yes no
then the output is yes
Combine
LED (yellow)
LED NOT
(red)
Fig. 1. Example sensor based applications built with fixed-function blocks, (a) Sleepwalking Detector, (b) Presentation Timer
such as a homeowner, teacher, scientist, etc. For example, a homeowner may wish to setup a custom system to indicate that a garage door is open at night, that a child is sleepwalking, or that an ageing parent has yet to get out of bed late in the morning. A scientist may wish to setup an experiment that activates a video camera when an animal approaches a feeding hole, or activates a fan when a temperature exceeds a threshold. Countless other examples exist. Despite the fact that such systems could be built from computing and sensing components whose total cost is only a few dollars, end-users cannot build such systems without knowledge of electronics and programming. Just connecting a button to an LED (light-emitting diode) would require knowledge of voltages, grounding principles, power supplies, etc. Making such a connection wireless requires further knowledge of communications, microcontroller programming, wireless devices, etc. Hiring engineers to build the system immediately exceeds reasonable costs. Off-the-shelf solutions for specific applications are hard to find, costly due to low volumes, and difficult to customize. Our previous work addressed this problem through development of basic blocks, called eBlocks, that enable end-users with no electronics or programming experience to define customized sensor-based systems merely by connecting blocks and performing minor configuration of those blocks [5]. The blocks incorporate small inexpensive microprocessors into previously passive devices like buttons, motion sensors, and beepers. Each device has a fixed function, and can be easily connected to other devices merely by snapping together standard plugs, with the devices communicating using predefined basic networking protocols. Figure 1 illustrates several applications built using eBlocks. A sleepwalking detection application is shown in Figure 1(a), which consists of motion and light sensor outputs combined using a logic block (configured to compute motion and no light), whose output connects to a wireless transmit block. The wireless transmit block is matched with a wireless receiver block (through setting of switches to identical positions), which ultimately activates a beeper block when motion at night is detected. Figure 1(b) illustrates a second example, a presentation timer, which turns on a green light for 20 minutes, followed by a yellow light for 5 minutes, and lastly a red light indicating time has expired. The design splits a button press to two prolonger blocks, one that
Automated Generation of Basic Custom Sensor-Based Embedded Computing Systems
71
prolongs the button press for 20 minutes, another for 25 minutes. Additional logic turns on the appropriate lights depending on whether both, one, or neither prolonger block is outputting a true value. eBlocks represent one of several new research approaches that utilize physical (“tangible”) objects to enable end-users to program electronic devices [12]. Other examples include Media Cubes [1], Electronic Blocks [20], and Tangible Programming Bricks [16]. Commercial X10-based devices [18] communicate through household power lines and are complementary to our approach. This paper describes our recent efforts to develop computer-based tools that an end-user could use to optimize an eBlock system or to map a virtual eBlock system to a limited set of physical blocks. While the work in this paper describes a tool to help end-users to build eBlock systems, related work that we have done [15] describes a tool to help end-users to tune low-level eBlock parameters, such as microprocessor clock frequency and communication baud rate, in order to achieve goals like maximizing battery lifetime and/or minimizing system latency.
2 eBlocks Overview The key idea of eBlocks is to enable end-users to build useful customized sensorbased systems merely by connecting blocks, like buttons, motion sensors, logic, beepers, etc. eBlocks’ key feature is that they encapsulate previously passive components by an ultra-lightweight compute wrapper. The following sections briefly describe two types of eBlocks, fixed function blocks and programmable blocks. A section also describes the eBlocks simulator, a multifaceted graphical environment that can simulate system functionality, configure programmable blocks, and provide the interface for the technology mapping and optimization tool introduced in this paper. Further details on eBlocks are discussed in [6]. 2.1 Fixed-Function Blocks Fixed-function eBlocks have a specific predefined function. Two types of fixedfunction eBlocks are Boolean and integer. Boolean blocks send “yes” or “no” packets, while integer blocks send integer packets. While this paper focuses on Boolean blocks, the methods generally apply to integer blocks, and our future work will address such application. Four categories of Boolean blocks exist: sensor, compute, communication, and output blocks. Sensor blocks sense events, such as motion, light, sound, button presses, or temperature. When a sensor detects the presence of an event (i.e. a light sensor detects light), the sensor generates a yes output, and otherwise generates a no output. Compute blocks perform logic or state computations on inputs and generate new outputs. A 2-input “Combine” block (a.k.a. “Logic”) computes a 2-input logic function configured by the end-user (e.g., AB, or A’+B). A 3-input Combine block is also available. An inverter block inverts a yes input to no output, or a no input to yes output. A “Yes Prolonger” block prolongs a yes input over the block’s output for an end-user-configured duration. A “Toggle” block switches between yes and no outputs on successive yes inputs. A “Pulse Generator” block generates yes and no output
72
S. Lysecky and F. Vahid
Contact Switch
Light Sensor LED
To connect blocks snap connectors together Combine Wireless Tx
Wireless Rx
Fig. 2. Garage Door Open At Night Dector is built by snapping various fixed function eBlocks together
pulses for an end-user-configured duration. A “Once-Yes Stays-Yes” (a.k.a. “Tripper”) block trips to a yes output state when the main input receives a yes, and stays in that state until a yes appears on a reset input. Communication blocks include wireless transmit and wireless receive blocks, which must be configured to implement a point-to-point channel by setting the corresponding switches on each block to the same channel value. A splitter block splits a single input into multiple identical outputs. Output blocks beep, turn on LEDs, control electric relays, or provide data to a PC for logging or other processing. A yes input activates output blocks. For example, a beeper block beeps when its input is yes, and is silent when its input is no. Figure 2 shows our initial physical prototype versions of eBlocks. Each physical block contains a PIC microcontroller for local computation and inter-block communication. The connections among blocks (along with any configurations of each block) define a system’s functionality. A unidirectional, packet-based protocol provides the basis for block communication. Each block includes hardware specific to the block’s task (e.g., sensors, resistors, voltage regulators, etc.). An end-user connects blocks using wired connectors or can replace a wire by a wireless connection by utilizing wireless transmit and receiver blocks. We have built over 100 prototype physical blocks, successfully used in controlled experiments by over 500 people of various skills levels, mostly end-users with no programming background [5]. 2.2 eBlock Simulator The eBlock simulator, shown in Figure 3, is a Java-based graphical user interface (GUI) for eBlock system entry and simulation and is available online at [6]. End-users can connect, test, and optimize various eBlock systems before interacting with physical blocks. End-users drag a block from a catalog on the right edge of the simulator to the workspace on the left and connect the blocks by drawing lines between the blocks’ input and outputs. The user can choose between a “simple mode” in which commonly-blocks appear in the catalog, and an “advanced mode” containing more blocks. eBlocks that sense or interact with their environment include accompanying visual representation to simulate the corresponding environment. For
Automated Generation of Basic Custom Sensor-Based Embedded Computing Systems
73
Clear workspace button
Run tutorial button
eBlock system specified by the end-user Palette of available eBlocks Simulator/eBlock usage guide
Fig. 3. Capture and synthesis tool illustrating the cafeteria food alert system
example, a “day/night” icon accompanies the light sensor. End-users can alternate the icon between day and night by clicking on the icon, causing the light sensor’s output to change accordingly. The gray text box situated in the bottom-left of the simulator displays context-sensitive help such as a block’s description and interface when the mouse cursor hovers over the corresponding block. 2.3 Programmable Blocks In contrast to fixed-function blocks, a programmable block can be programmed to implement arbitrary behavior. Our current programmable block has two inputs and two outputs, as shown in Figure 4. An expert user could write C code that describes the block’s behavior, and our tools would then combine that with the eBlock protocol code into a binary, which the user could then download using our serial cable interface. However, most end-users will not have C-coding expertise. We thus provide a tool for automatically converting internal (non-sensor, non-output blocks) fixed-function blocks into a smaller network of programmable blocks that preserves the system’s functionality, automatically generating code for each programmable block. In this context, a programmable block is a means for slightly more advanced users to reduce the block count and hence cost of their systems.
3 Technology Mapping 3.1 Problem Description A problem with using physical fixed-function blocks is that an end-user may only have a subset of possible blocks readily available, and/or may have limited numbers
74
S. Lysecky and F. Vahid
Toggle Inputs
(a)
Outputs
PC Interface
LEDs Splitter Buttons
Combine
(c)
Buttons
Programmable Block
LEDs
(b)
Fig. 4. Programmable blocks: (a) An original system using fixed-function blocks, (b) programming a programmable block to replace the inner fixed-function blocks, (c) the new system using the programmable block
of particular blocks. For example, an end-user may have three types of blocks available – 2-input logic, tripper, and prolonger blocks – of which many instances of each exist. Defining the desired application behavior using just those blocks would be a significant challenge even for an expert end-user. Instead, we provide for the enduser a way to capture the desired behavior in the graphical simulator using any combination of standard fixed-function blocks. The end-user also lists the types and numbers of available physical fixed-function blocks, forming what is essentially a constrained block library. We then define an automation tool that creates a new block network with the same functionality as the desired network, but only using blocks from the constrained library. Sensor and output blocks have specialized circuitry (e.g., light sensors, LEDs, beepers), so the physical counterparts for those blocks must be available – i.e., we cannot build a light sensor out of motion sensors. The mapping problem thus only involves inner blocks, namely compute and communication blocks. The above-described problem is essentially a technology-mapping problem, common in chip design, with some differences from traditional problems. Technology mapping is a central part of the chip design process. Chip designers describe a desired circuit’s behavior using easy-to-work-with components, such AND, OR, and NOT logic gates, with any number of inputs on each gate. However, a chip’s underlying technology may support only 2-input and 3-input NAND gates, requiring that generic AND/OR/NOT circuits be mapped to a circuit consisting only of such NAND gates. Modern technologies consist of far more complex mappings of generic circuits to technology-specific components. Efficient technology mapping has been intensely researched for decades [2,3,4,8,9]. Our technology-mapping problem has some differences from previous technology-mapping problems. In the domain of field programmable gate arrays (FPGAs), technology mapping translates a digital circuit to
Automated Generation of Basic Custom Sensor-Based Embedded Computing Systems
75
a physical implementation on lookup tables (LUTs) [7]. A LUT is capable of specifying any logic function with a given number of inputs (defined by the size of the physical LUTs within the FPGA). Because LUTs are general programmable structures, the mapping methods correspond more to the problem of converting to programmable blocks than to that of fixed-function blocks. In application-specific integrated circuit (ASIC) design, technology-mapping implements a hardware circuit using a library of physical cells with fixed functions [7]. However, the final ASIC implementation can use an essentially unlimited number of any physical cells within the library. In contrast, our problem has a fixed numbers of each block, and furthermore does not necessarily have a balanced set of blocks. Nevertheless, our solution approach borrows from existing ASIC techniques. 3.2 Transformation Rule Base A straightforward but non-optimal ASIC technology mapping method converts every technology independent circuit element into a technology-dependent universal gate element. In circuits, a NAND gate or a NOR gate represents a universal gate. In our problem, we found that we could implement nearly every block using some network of 2-input logic and splitter blocks. Defining a universal mapping heuristic that would replace each unmapped block in a network by an equivalent network consisting of the universal blocks is one possible method to perform technology mapping. A better optimizing ASIC technology-mapping method involves graph-covering methods [13]. A library is built of mappings from technology independent subcircuits to technology dependent sub-circuits, and then directed acyclic graph covering methods cover the unmapped circuit. The methods are built on similar graph methods used for instruction coverage generation in compilers. While such methods could be applied to our problem, we found that the state-based functions associated with our fixed-function blocks might introduce significant complexity into the graph cover heuristics. In fact, such traditional methods typically focus on the combinational part of the circuit, whereas state-based (sequential) blocks are a key part of our problem. Another ASIC technology mapping method involves rule-based technology mapping [10,11]. Those techniques perform local optimizations on a circuit based on a set of transformation rules. We used this method as the basis for our first solution to the problem, which we call the transformation rule method. We developed a transformation rule base as follows. For each standard fixed-function block, we manually built alternative implementations of that block (the source block) using other various subsets of standard fixed-function blocks (target blocks). For example, for a 2-input logic block, we defined a transformation for implementing that block using a 3-input logic block, as shown in Figure 5(a) (Config. shows the truth table entries for the block). Figure 5(b) shows a transformation of a tripper block into an equivalent set of blocks involving a logic block and a splitter block, with the required logic configuration shown. Figure 5(c) shows multiple transformation rules for a inverter block. The invert block can be replaced utilizing either a 2-input logic block or a 3-input logic block, configured to implement the invert on the first input.
76
S. Lysecky and F. Vahid a Logic2 out b Config. : c0 c1 c2 c3 Block equivalency a b
Logic3
in rst
Tripper Block equivalency
in rst out
Config. :c0 c0 c1 c1 c2 c2 c3
in
Invert
in
Logic2 Config. : 0011
Block equivalency 2 in
out
(c)
out
Config. : 01001100
out
Block equivalency 1
(a)
Splitter
Logic3
Logic3
(b) out
Config. : 00110010
Fig. 5. Sample block equivalencies used in the technology mapping equivalncy library
We point out that we could have treated logic blocks using logic synthesis methods, wherein we would convert every sub-network of logic blocks (2- and 3input logic blocks and invert blocks) into a Boolean expression, optimize the expression, and then map the expression into logic blocks in the library using traditional circuit technology mapping. However, as this is a first work, we preferred a transformation rule approach for consistency with the other blocks, resulting in a simpler tool but a less optimized mapped network. Nevertheless, incorporating logic synthesis methods is an area of future improvement.
4 Optimization 4.1 Problem Description Given a network of fixed-function blocks, there may exist more blocks than necessary, arising from two situations. First, an end-user may have created a network of fixed-function blocks that is easy to comprehend, but has more blocks than necessary. Alternatively, technology mapping may have inserted two adjacent subnetworks with perimeter blocks that could be merged into fewer blocks. We thus developed a method to reduce block count while preserving network behavior. We considered different methods for reducing blocks. A model-based method would utilize a formal understanding of the underlying finite-state machine (FSM) (or combinational) behavior of each block. This method would compose the FSMs into a single network-level FSM, eliminate equivalent and redundant states, and remap the reduced FSM to physical blocks. This approach appeared overly complex and possessed the problem that the reduced FSM might not be mappable to existing physical blocks. Another method builds on peephole optimization, an optimization method commonly found in compilers. This method inspects a local area of code to identify and modify inefficient code [17]. We can similarly inspect sections of the block network to optimize inefficient or redundant sections. The peephole optimization method enables us to preserve the pre-defined block structure because we are operating on the block level and eliminates the need for additional mapping.
Automated Generation of Basic Custom Sensor-Based Embedded Computing Systems a b
Invert
Logic2
Invert
a
out
Yes Prolonger
Yes Prolonger Configuration: Y1time
Configuration: c0 c1 c2 c3
77
out
Configuration: Y2time If: Y1time + Y2time <=YMax
a b
Logic2
a
out
out
Configuration: Y1time + Y2time
Configuration: c2’ c3’ c0’ c1’
(a) a b
Yes Prolonger
(b) Splitter
c d
a b
Splitter
out
Splitter
out
d
(c) Fig. 6. Sample peephole optimization used in the optimization library
4.2 Peephole Optimizations We analyzed a variety of networks and identified commonly occurring inefficient block combinations. We added optimization templates to the library that reflect these inefficient block combinations, along with a corresponding optimized block network. The optimizer traverses the network specification searching for subsystems matching any of the corresponding optimization templates and replaces the inefficient block combination by the optimized block network defined by the template. Figure 6 illustrates several optimization templates defined within the optimization library. Figure 6(a) illustrates inverters located at the input or output of logic blocks that an end-user could have merged into the logic block. The optimizer eliminates the inverters and updates the logic block configuration accordingly. The optimization shown in Figure 6(b) merges chained prolonger blocks into a minimum number of prolonger blocks. If the combined yes time of the chained prolonger blocks is less than the maximum yes time of a single prolonger block, the optimizer can merge the chained prolonger blocks into a single prolonger block. If the combined yes time of the chained prolonger blocks exceeds the maximum yes time of a single prolonger block, then the minimum number of prolonger block are used. Figure 6(c) analyzes the number of unused inputs on chained splitters and attempts to combine splitters. Each peephole optimization is treated independent of others peephole optimizations as well as independent of the technology mapping transformations.
5 Programmable Block Operations Technology mapping transformation rules and peephole optimizations discussed in previous sections pertain to fixed function blocks. Inclusion of programmable blocks presents further opportunity for technology mapping and optimization. For example, if a fixed function block does not exist in the physical library, a programmable block can by configured to replace the missing fixed function block. Furthermore, multiple fixed-function blocks can be replaced by a single programmable block to reduce block count and/or cost of the system.
78
S. Lysecky and F. Vahid Technology Mapping Library logic2: { 1 logic3 c3c3c2c2c1c1c0c0 in1 in2 -1 out1 } logic3: { 1 splitter in1 6 7 -1 2 splitter in2 4 5 -1 3 splitter in3 4 5 -1 4 logic2 c3c2c1c0 2 3 5 logic2 c7c6c5c4 2 3 6 logic2 0010 1 4 8 7 logic2 1000 1 5 8
Optimization Library
Block Count/Cost
b b
-1 -1 -1 6 7
b d
...
$1.20
x7
$1.50
x2
$2.25
x3
$2.00
x 10
$1.00
x3
yes no
b d
A’ B’ A’ B A B’ A B
1
…
0
…
Physical/Virtual Block Weights Physical Fixed Function Virtual Fixed Function Physical Programmable Virtual Programmable
1 10 2 10
Objective Minimize Cost Minimize Block Count
System Specification
eBlock Simulator
Fig. 7. Technology mapping and optimization environment
Several options exist to deal with the existence of programmable blocks. One option is to develop a separate partitioning algorithm, utilized in a secondary stage, which aims to assign multiple fixed-function blocks to programmable blocks. A simpler method is to define low-level technology mapping transformation rules and peephole optimizations specific to programmable blocks, and to incorporate those rules and optimizations into the main technology mapping and optimization heuristic (discussed in Section 6). The second method follows closely what we have done with fixed-function blocks, thus we defined several programmable block operations and incorporated them into the appropriate libraries.
6 Technology Mapping and Optimization Methodology Figure 7 illustrates the overall technology mapping and optimization design methodology intended to aid end-users in generating an optimized physical sensorbased system based on end-user defined criteria. Two parties are responsible for the input specification, the node designer and the end user. The node designer is an expert who has an understanding of the underlying details of the various eBlocks and provides the pertinent block information prior to the release of the mapping and optimization tool. The end-user may have no expertise in programming or electronics but wants to construct a customized sensor-based system. The end-user provides input specific to their situation and the application being created. 6.1 Node Designer Input The node designer defines the technology mapping transformations discussed in Section 3 by creating a text-based technology-mapping library read in by the tool. Furthermore, the node designer defines various peephole optimizations discussed in
Automated Generation of Basic Custom Sensor-Based Embedded Computing Systems
79
Section 4 utilized by the tool. The optimization library is currently a C file that contains various functions to perform each of the peephole optimizations, alternatively these optimizations could be defined in a text file as the technology mapping transformation which is then translated by the technology mapping and optimization tool. These files are provided by the node designer and are independent of the various sensor-based systems constructed by the end-user. 6.2 End-User Input The end-user needs to specify which fixed-function and programmable blocks are physically available, the functionality of the specific application being built, and the optimization goals. The end-user first defines the “Block Count/Cost” input, i.e., how many of each type of block is physically available and the cost of each eBlock (regardless of whether they are physically available). The input specification can be done in a graphical environment in which an end-user can manually enter a number in a text box next to the graphical depiction of the block of interest or click on up/down arrows until the appropriate value is displayed (shown in Figure 8 under the “Block Count/Cost” heading). The end-user can similarly define block cost. The end-user’s next task is to define the “Physical/Virtual Block Weights” input. We define physical blocks as blocks that are physically available in the end-user’s block set; virtual blocks do not exist in the block set, meaning they would have to be purchased to create the physical system. The end-user specifies four weights: WP_FF = physical fixed function block weight WV_FF = virtual fixed function block weight WP_PROG = physical programmable block weight WV_PROG = virtual programmable block weight
These values are used within a cost equation to evaluate whether a given system configuration yields an improvement (further discussed in Section 6.3). By simply assigning various weights, with lower weights indicating preferred block types, an end-user can direct the technology mapping and optimization to use preferred block types when possible. For example, an end-user who is uncomfortable with programmable blocks and wants to only utilize fixed function blocks, whether virtual or physical, can set the blocks weights to WP_FF = 1, WV_FF = 1, WP_PROG = 10, and WV_PROG = 10. Selection of a programmable block by the tool yields ten times the cost of a fixed function block, guiding the tool to favor fixed-function blocks. Alternatively, an end-user who wants to utilize blocks already existing in their physical block set, whether fixed or programmable, can set the block weights to WP_FF = 1, WV_FF = 25, WP_PROG = 1, and WV_PROG = 25. Again, virtual blocks yield higher cost, thus the tool is biased to select physically available blocks before utilizing any virtual blocks. The end-user can adjust the four block weights to reflect a variety of situations and to guide the technology mapping and optimization tool in creating an appropriate physical sensor based system.
80
S. Lysecky and F. Vahid
Block Count/Cost
End-user
A’ A’ A A 1
Physical/Virtual Block Weights Physical Fixed Function 1 Virtual Fixed Function 10
$1.20
x7
$1.50
x2
$2.25
x3
Physical Programmable 2 Virtual Programmable 10
$2.00
x 10
Optimization Objective :
$1.00
x3
yes
Minimize Cost Minimize Block Count
Fig. 8. Using a graphical interface, the end-user specifies, (a) the block count/cost library and, (b) the physical/virtual block weights
Within the “Physical/Virtual Block Weight” input, the end-user must also choose the optimization criteria – either to minimize the number of blocks utilized or to minimize the monetary cost of the resulting system. The end-user selects the optimization goal by selecting the corresponding radio button. The last task required of the end-user is to create the eBlock system, thus defining the desired system functionality. The end-user creates the eBlock system within the block simulator (Section 2.2) by dragging and connecting blocks in the workspace. 6.3 Design Space Exploration Once all inputs to the technology mapping and optimization tool are defined, our tool uses simulated annealing to explore the design space and generate the finalized sensor based system. The simulated annealing algorithm [14] is a popular optimization approach modeled after annealing in metallurgy, wherein a material is continuously heated and cooled to increase the material’s strength. The algorithm randomly searches the design space by generating random changes, and accepts a change if an objective function value is decreased. Alternatively, a change that increases the objective function value can also be accepted based on a probability linked to a global “temperature” value. Early in the algorithm, changes that increase the objective function cost are more likely to be accepted, to avoid being trapped in a local minimum early on. As the algorithm continues to run, these higher cost changes are less likely to be accepted, thus settling into a minimum cost solution. The rate of decline in which higher cost solutions are accepted is based on a definable cooling schedule. The longer the algorithm runs, the higher a chance of a good solution, thus the key is to define a cooling schedule that balances the solution quality and runtime. We chose simulated annealing due to the heuristic’s generality – we can simply define a set of possible changes consisting of the various transformations and optimizations discussed in previous sections, and let the tool search the solution space. While computationally expensive, the power of modern computers coupled with the relatively small sizes of eBlock systems make the use of annealing effective. Depending on the end-user defined optimization criteria, we use one of two weighted cost functions to determine the system cost. If minimizing block count is the objective, the following cost equation is utilized:
Automated Generation of Basic Custom Sensor-Based Embedded Computing Systems
( (W
) ( * # of physical programmab le blocks ) + (W
81
)
block cost = W P _ FF * # of physical fixed function blocks + WV _ FF * # of virtual fixed function blocks + P _ PROG
V _ PROG
* # of virtual programmab le blocks
)
If minimizing total system cost is the objective, the following cost equation is utilized: system cost = W P _ FF * ∑ ( price of block i * # of physical fixed fucntion block i ) + i
WV _ FF *
∑ ( price i
W P _ PROG *
∑ ( price
of block i * # of physical programmab le block i ) +
∑ ( price
of block i * # of virtual programmab le block i )
i
WV _ PROG *
of block i * # of virtual fixed fucntion block i ) +
i
The end-user, as described in Section 5.2, assigns the various block weights. The technology mapping and optimization tool utilizes simulated annealing and performs random changes consisting of an optimization or a transformation from a randomly chosen single technology mapping transformation rule (Section 3, a peephole optimization (Section 4), or a programmable block operation (Section 5).
7 Technology Mapping and Optimization Results Utilizing the technology mapping and optimization methodology previously discussed, we now consider several eBlock systems and provide the corresponding physical eBlock system implementation determined by the technology mapping and optimization tool taking into consideration user-specified block availability and preferences. We considered six different scenarios in which end-users have varying types and quantities of physical eBlocks already available, as well as differing preferences as to the types of eBlocks the end-user wants to utilize to build the desired eBlock system. Section 7.1 looks at scenarios where the optimization criterion selected is block size reduction and Section 7.2 looks at scenarios where the optimization criterion selected is system cost reduction. For each scenario, we considered sixteen eBlock system specifications, ranging from a “Night Light Controller” consisting of two internal blocks to a “Digital Hourglass Timer” consisting of over 50 internal blocks. The “Technology Mapping” library and “Optimization” library input files specified by the node developer are consistent across each scenario considered. The “Block Cost” input contains the monetary cost of each block type, derived from [19], are also consistent across each scenario considered. Each scenario defines the “Block Counts” input specifying the number and type of physical blocks already available to the end-user, i.e., the eBlocks that end-user currently has on hand. Each scenario further defines the “Physical/Virtual Block Weight” input specifying the end-user’s preference towards available physical blocks or virtual blocks the end-user wants to utilize in constructing the final eBlock system implementation. 7.1 Minimizing Block Count We first consider a scenario, referred to as Scenario 1, in which an end-user may have just stumbled upon eBlocks online and wants to try to build various systems using the
82
S. Lysecky and F. Vahid Table 1. Breakdown of physical block counts and weights for each sceanrio
Scenario 1 & 7 Scenario 2 & 8 Scenario 3 & 9 Scenario 4 & 10 Scenario 5 & 11 Scenario 6 & 12
Block Count Assignment Block Set A Block Set A Block Set B Block Set B Block Set C Block Set C
WP_FF
WV_FF, 10 10 1 1 1 1
WP_PROG 10 10 10 10 10 10
WV_PROG
100 10 1 1 1 1
100 10 100 10 100 10
Table 2. Breakdown of physical block counts for each block set, all input and output blocks are assumed to contain unlimited corresponding physical block counterparts
0
0
0
0
0
0
0
0
0
0
2
2
Splitter
Pulse Generator Wireles s Transmitter Wireles s Receiver
Tripper 0
Prog_6 _6
0
Prog_4 _4
0
Prolon ger
Toggle
0
Inverter
0
Prog_2 _2
Block Set A
3-Input Logic
Block Set
2-Input Logic
Physical Block Count
Block Set B
10
2
2
10
2
2
2
10
10
10
0
Block Set C
10
2
2
10
2
2
2
10
10
10
2
eBlock simulator and has no physical eBlocks available. Block Set A, listed in Table 2, reflects that the end-user has no physically available blocks. Furthermore, the Block Set corresponds to the “Block Count” input of the technology mapping and optimization tool. To realize the eBlock system specified within the eBlock simulator, the end-user is willing to purchase fixed function blocks but is weary of utilizing programmable blocks. Thus, the end-user can specify this preference by adjusting the “Physical/Virtual Block Weights” input, setting the fixed function block weights to 10 and programmable block weights to 100, as listed in Table 1. Having no physical eBlocks (as specified by Block Set A) and a desire to utilize only fixed function block (as specified by the block weights listed in Table 1), we then utilized the technology mapping and optimization tool to implement each of the sixteen eBlock systems. Figure 9 illustrates a breakdown of block types for each system, indicating the number of physical fixed function, physical programmable, virtual fixed function, and virtual programmable blocks each eBlock system is composed of. Only a subset of systems is illustrated in Figure 9 due to space limitations. In Scenario 1, programmable blocks are penalized, thus all sixteen final eBlock systems consist solely of fixed function blocks with solutions yielding and average of 12.9 inner blocks. In the second scenario (Scenario 2), the end-user again has no physical blocks available, but the end-user in this scenario is willing to purchase fixed function and programmable blocks. Again, as the end-user has no available blocks, Block Set A is utilized to specify the “Block Count” input. Furthermore, as the end-user does not have a preference towards fixed function or programmable blocks, all block types are
Automated Generation of Basic Custom Sensor-Based Embedded Computing Systems
83
Fig. 9. Resulting number of physical fixed function (FF) blocks, physical programmable (Prog) blocks, virtual fixed function blocks, and virtual programmable blocks for several scenarios and systems
equally weighted in the “Physical/Virtual Block Weight” input. Figure 9 illustrates a breakdown of block types for a subset of the sixteen eBlock systems considered. Overall, five solutions can take advantage of programmable blocks, resulting in solutions that on average require 10.5 inner blocks. By allowing programmable blocks, the final eBlocks systems can be implemented using on average 2.4 fewer inner blocks compared to a system composed of solely fixed function blocks (Scenario 1). In the next two scenarios considered (Scenario 3 and 4), an end-user has access to some physical fixed-function blocks, perhaps having purchased a initial set eBlocks consisting of only fixed function blocks, with the number and type of physical blocks available listed in Block Set B. In addition, the end-user in Scenario 3 is willing to purchase fixed function blocks if needed but is apprehensive to purchase programmable blocks. Block weights are set to so physical blocks have lower weights, virtual fixed-function blocks are weighted slightly higher, and virtual programmable blocks are heavily weighted, as listed in Table 1. In Scenario 4, an end-user is willing to purchase fixed function and programmable blocks. Virtual blocks have slightly higher weights, but the block weights make no distinction between fixed-function and programmable blocks. In Scenarios 3 and 4, a limited number of physical fixed function blocks exist, thus the tool will bias solutions to utilize physically available blocks before choosing virtual blocks as shown in Figure 9. Scenario 3 further penalizes usage of virtual programmable blocks, thus no solutions include virtual programmable blocks. On average, solutions require 16.44 inner blocks but only require end-users to acquire an additional 3.44 inner blocks. While Scenario 3’s final inner block count is higher than in Scenarios 1 and 2, existing blocks are utilized minimizing the number of additional blocks required. Scenario 4 yields solutions with an average inner block count of 13.7, of which an average of 2.3 additional blocks are required. Again, this occurs because the programmable blocks are utilized, enabling reduction of fixed-function blocks.
84
S. Lysecky and F. Vahid
In Scenarios 5 and 6, an end-user has a slightly larger set of physical fixed function and programmable blocks available as listed in Block Set C. The end-user in Scenario 5 is willing to purchase fixed function blocks, thus the end-user sets WV_FF to 10 while WV_PROG is set to 100. Lastly, in Scenario 6, an end-user is willing to purchase fixed function and programmable blocks. While higher than the physical block weights, both virtual block weights (WV_FF and WV_PROG) are equally weighted. In scenario 5 and 6, a larger physical block library exists, including both fixed function and programmable blocks. Figure 9 shows Scenario 5 yields a further reduction of inner blocks of 14.1, and 2.3 additional blocks because an expanded physical block set is available and specifically because physical programmable blocks are available. Scenario 6 results in further decrease resulting in an inner block count of 12.9 and an additional block count of 2.9 because virtual programmable blocks are not penalized. 7.2 Minimizing Total System Cost In this section, we consider the same scenarios previously discussed but aim to minimizing the total system cost. Because the end-user is interested in cost reduction, the tool must consider the price of blocks utilized in the final solution. In block count reduction utilizing a 2-input logic block and 3-input logic block made no difference because both had a block count of 1. However, in the system price reduction a 2-input logic block is a better choice at $7.42 than the 3-input logic block at $9.05. Figure 10 provides a breakdown of system cost based on the block classification physical fixed-function, physical programmable, virtual fixed-function, or virtual programmable. Figure 11 indicates the cost of blocks not currently available within the physical set that an end-user needs to purchase to implement the physical system indicated by the tool. Scenarios 7 and 8 again consider block libraries in which no physical blocks are available, thus the tool tries to find the lowest cost system implementation. Although Scenario 8 does not penalize use of programmable blocks, there was no cost benefit in utilizing programmable blocks. Both Scenarios 7 and 8 resulted, on average, in an inner block cost of $71.29. Scenarios 9 and 10 include a library of fixed function blocks, where the end-user assigns larger weights to virtual blocks with virtual programmable block receiving an even higher weight.
Fig. 10. Cost of physical fixed function (FF) blocks, physical programmable (Prog) blocks, virtual fixed function blocks, and virtual programmable blocks
Buying Cost ($)
Automated Generation of Basic Custom Sensor-Based Embedded Computing Systems
85
125 100 75 50 25 0 7 8 9 101112 Garage Door Open At Night
7 8 9 101112 Podium Timer
7 8 9 101112 Alarm Clock
7 8 9 101112 Flood Detection
Scenario
7 8 9 101112 Motion On Property
7 8 9 101112 AVERAGE
Fig. 11. Additional inner block cost required to implement the corresponding system
Again, there was no cost benefit in utilizing programmable blocks with both scenarios resulting in a total system cost of $71.20, and a cost of $14.80 for virtual blocks. Scenarios 11 and 12 included a library of both fixed function and programmable blocks thus resulting in inner block costs of $58.62 and $64.13 and an additional block cost of $9.31 and $14.12 respectively. Overall, our technology mapping and optimization enables end-users to successfully design a system with existing blocks or with minimal additional blocks required. Additionally, our optimization tool is effective in reducing the size of enduser designed systems and reducing system cost. On average, our tool is extremely fast, requiring only 6 second per application, executing on a 2.8 GHz Xeon computer. When the end-user selected system cost reduction as the optimization criteria, the tool yielded a 23% reduction of system cost compared to the original implementation. When the end-user selected block count minimization as the optimization criteria, on average the tool yielded system implementations requiring six virtual blocks.
8 Conclusions and Future Work We described a technology mapping and optimization tool to aid end-users in transforming a virtual eBlock system into an optimized physical block system. The tool requires no programming or electronics experience on the end-user’s part, yet provides end-users with the ability to guide the tool in producing a system optimized for size or cost based on a constrained block library. The tool presented in paper is part of a larger framework. We plan to continue to add more blocks to the eBlock set as well as to expand the tools to support customization of the communication protocol and block parameters. The blocks, combined with the tool, help end-users setup useful basic sensor-based embedded computing systems to monitor and control the endusers’ environments.
Acknowledgments This work is supported in part by the National Science Foundation under grant CCR0311026. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
86
S. Lysecky and F. Vahid
References 1. Blackwell, A., R. Hague. AutoHAN: An Architecture for Programming the Home. IEEE Symposia on Human-Centric Computing Languages and Environments, 2001. 2. Chen, K., J. Cong, Y. Ding, A. Kahng, P. Trajmar. DAG-Map: graph-based FPGA technology mapping for delay optimization. IEEE Design & Test of Computers, Volume 9, Issue 3, Sept. 1992. 3. Cong, J., Y. Ding. On area/depth trade-off in LUT-based FPGA technology mapping. IEEE TVLSI, Volume 2, Issue 2, June 1994. 4