Inhabited Information Spaces: Living with your Data
David N. Snowdon Elizabeth F. Churchill Emmanuel Frécon, Editors
Springer
Computer Supported Cooperative Work
011
011
011
011
11
Springer London Berlin Heidelberg New York Hong Kong Milan Paris Tokyo
i
Also in this series Gerold Riempp Wide Area Workflow Management 3-540-7643-4
Ralph Schroeder (Ed.) The Social Life of Avatars 1-85233-461-4
Celia T. Romm and Fay Sudweeks (Eds) Doing Business Electronically 3-540-76159-4
J.H. Erik Andriessen Working with Groupware 1-85233-603-X
Fay Sudweeks and Celia T. Romm (Eds) Doing Business on the Internet 1-85233-030-9
Paul Kirschner, Chad Carr and Simon Buckingham Shum (Eds) Visualising Argumentation 1-85233-664-1
Elizabeth F. Churchill, David N. Snowdon and Alan J. Munro (Eds) Collaborative Virtual Environments 1-85233-244-1 Christine Steeples and Chris Jones (Eds) Networked Learning 1-85233-471-1 Barry Brown, Nicola Green and Richard Harper (Eds) Wireless World 1-85233-477-0 Reza Hazemi and Stephen Hailes (Eds) The Digital University – Building a Learning Community 1-85233-478-9
Christopher Lueg and Danyel Fisher (Eds) From Usenet to CoWebs 1-85233-532-7 Kristina Höök, David Benyon and Alan J. Munro (Eds) Designing Information Spaces: The Social Navigation Approach 1-85233-661-7 Bjørn Erik Munkvold Implementing Collaboration Technologies in Industry 1-85233-418-5
Elayne Coakes, Dianne Willis and Steve Clark (Eds) Knowledge Management in the SocioTechnical World 1-85233-441-X
Related Title Richard Harper (Ed.) Inside the Smart Home 1-85233-688-9 A list of out of print titles is available at the end of the book
ii
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
David N. Snowdon, Elizabeth F. Churchill and Emmanuel Frécon (Eds)
Inhabited Information Spaces Living with your Data
With 94 Figures
011
011
011
11
iii
David N. Snowdon, BSc (hons) MSc, PhD Xerox Research Centre Europe, 6 Chemin de Maupertius, 38240 Meylan, France. Elizabeth F. Churchill, BSc, MSc, PhD FX Palo Alto Laboratory Inc., 3400 Hillview Avenue, Building. 4 Paol Alto, CA94110, USA. Emmanuel Frécon, MSc Swedish Insitute for Computer Science, Interactive Collaborative Environments Laboratory, Platforms for Collaborative Environments Group, Box 1263, 164 29 Kista, Sweden. Series Editors Dan Diaper, PhD, MBCS Professor of Systems Science & Engineering, School of Design, Engineering & Computing, Bournemouth University, Talbot Campus, Fern Barrow, Poole, Dorset BH12 5BB, UK Colston Sanger School of Management, University of Surrey, Guildford, Surrey GU2 7XH, UK
British Library Cataloguing in Publication Data Inhabited information spaces : living with your data. – (Computer supported cooperative work) 1. Human-computer interaction 2. Interactive computer systems I. Snowdon, David N., 1968– II. Churchill, Elizabeth F., 1962– III. Frécon, Emmanuel 004′.019 ISBN 1852337281 Library of Congress Cataloging-in-Publication Data A catalog record of this book is available from the Library of Congress Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers. CSCW ISSN 1431-1496 ISBN 1-85233-728-1 Springer-Verlag London Berlin Heidelberg Springer-Verlag is a part of Springer Science+Business Media springeronline.com © Springer-Verlag London Limited 2004 The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use. The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made. Typeset by Florence Production, Stoodleigh, Devon, England Printed and bound in the United States of America 34/3830-543210 Printed on acid-free paper SPIN 10910238
iv
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Foreword The Human Touch: Reflections on i3
011
011
011
011
11
The Machine-centred Mind Set At the Chicago World Fair of 1933, the official motto was: “Science Finds – Industry Applies – Man Conforms”. To many of us today this seems quite shocking, yet it has been the driving force of much development in the last century. In particular, if you look at the rise of computing over the last 50 years, you will see that, on the whole, development has been extraordinary, but fairly straightforward: it can be characterised as trying to make “faster and faster machines fit into smaller and smaller boxes”. Starting from the time of the ENIAC, one of the colossal computers of the 1940s, most IT progress has been driven from the point of view of the machine. Since then things have changed – but perhaps not really that much. Even if computers can today calculate many times over what was possible a few years ago, and the machines have become somewhat less obtrusive, much of the “mind set” has stayed the same. It is the visions of huge calculating machines spanning massive rooms, trying to recreate an absolute artificial intelligence, that still haunt much of the thinking of today. Clearly, it is difficult to shake off old mind sets.
Alternatives Alternatives to the idea of fitting computing into ever smaller boxes can mainly be attributed to Mark Weiser. In his paper, “The Computer for the 21st Century”, he outlined notions of how computing could become integrated into the fabric of everyday life by becoming completely distributed into the environment. In this way computing would become “ubiquitous”. More recently, similarly inspired work on “tangible media”, by Hiroshi Ishii has emerged from the MIT Media Lab. Apart from this, the technological revolution of GSM and the mobile phone has also had its share of making information technology come out of its “traditional shell”. Alternatives to the machine-centred view to computing were also starting around the same time, such as the “anthropocentric” ideas proposed by Mike
v
Foreword
Dertouzos at the MIT Computer Science Lab; and in a similar vein, cognitive scientist, Don Norman has been pointing out the lack of well-designed information environments. Roughly at the same time, but from a different perspective, we started thinking about how to give technology more of a “human touch”. Now, in principle, this should not be that difficult, as technology is after all, made by humans. In practice, however, one has to go quite far to break down the machine-centred and box-centred ways of thinking. We decided that the only way to attack the problem with any significance was to try to invert the picture completely – that is, to start thinking from the human point of view and work outwards. Our idea of “human centredness” was that it should nurture technological innovation but within a broader context of human values and aspirations. This was not the same as “user” driven, or “defined by user needs”, all of which tend to become stuck in improving the status quo, but not growing beyond it. At the same time, we also wanted to make sure to break out of the box-centred ways of thinking as much as possible and avoid doing “traditional HCI”, which was mainly involved in improving computers as they were. Our ideas were designed to balance questions of technically “how”, with questions of “why?” and “what for?”. And the aim was to see if we could start restoring the balance between people’s inventiveness to make new machines, with the essence of being human. Our questions became rather: How can we reach a better and more fulfilling balance between technology and people? What could be new ways of thinking about the problems? What could be the new paradigms that could lay the paths for further research and development?
The i3 Research Programme It is along these lines that we launched our first call for proposals back in 1995. Our general aim was to look at the relationship between people and information technology in the future: how could people access and use information, and exchange things with others using information technology as a medium? A clear break was needed to get out of stale thinking. Therefore, we called for new paradigms of interaction and research on new interfaces between people and the world of information. We also asked how such work could intertwine human, societal and technological elements into one dynamic research activity. One of the main quotes from our call for proposals was: The goal of i3 is to research and develop new human-centred interfaces for interacting with information, aimed at the broad population (1996).
To help define a specific research agenda, we first had a competition for more specific visions of the future. “Connected Community” and “Inhabited Information Spaces” were selected as the two visionary themes on which we based a subsequent call for research projects. Even though it took some time
vi
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Foreword
● 1 ●●●●●●●●●●●●●● to have an extra layer of calls for proposals, in retrospect it was better to “reculer pour mieux sauter”. The two selected themes had similar yet contrasting underlying philosophies. The Connected Community theme, proposed by a team headed by Irene Mac William (Philips Design, Eindhoven) and Marco Susani (Domus Academy, Milan) asked: forget about virtual environments and trying to fit people into some artificial world – how can we help people in their everyday environment, and integrate technology into this? The idea is to understand how information and communication tools start making a difference when they are embedded in a real context, and start being more meaningful for actual people and communities. How can technology enhance these environments and activities, rather than replace them? The other schema, Inhabited Information, proposed by a team headed by Tom Rodden (University of Nottingham) took a slightly different perspective. It stated: the Internet and the Web already represent a suspended reality, and people want to participate more in these spaces. Given that this is a reality, how could it evolve in the future? How could we make it more accessible to the broadest possible public, and make it socially interactive for large groups of people, in meaningful ways? And in similar spirit to the first theme, how can such environments link to the physical everyday world rather than be removed from it? At a later stage, we decided to supplement the research with an emphasis on learning. We wanted to explore new relationships between learning and technology. The idea was that a lot could be learnt about designing new interfaces by looking at how children interact, play and learn. Similar ideas had been experimented in a Lego context by Seymore Papert of the MIT Media Lab. In 1997, we decided to have a call on experimental school environments (ese). This centred around learning for very young children, in fact, the 4–8-year-old age range. This age range struck us as being particularly challenging because at this stage children don’t have too many of the adult preconceptions of the world, and are still open to new things. Young children have a different kind of “language” – a form of communication and expression from which adults can learn a lot. From this we wanted to gain insights about how to design meaningful interaction tools for the population at large. The header of our call was: The aim of i3-ese, is to research new kinds of IT-based tools designed to enable new approaches to learning, focussing on the age range of 4 to 8 (1997).
From each of these programmes we selected a number of individual research projects. Together these spanned many universities, research centres and companies across Europe, and involved a mix of people from many walks of life – artists, designers, computer scientists, game companies, technology companies, experimental schools, teachers and children, people in communities, etc. At the same time all these different outlooks were united by the common vision: of exploring new relationships between people and technology.
vii
Foreword
Grains of the Future In this book you will find some examples of work in particular from the Inhabited Information Spaces Grouping. It is interesting to see how some of these ideas are still “futuristic” and others have started to become part of mainstream thinking and made their way into products. Some people say that you can find “grains of the future” in the present today – the only problem is, where do you start to look? One of the potential advantages of this book is that by looking at the research developments stretching out into recent past, one can identify how some grains developed into trends of the present, and other are still just emerging. For those still interested in seeking out “grains of the future”, this book will be a valuable source. Jakub Wejchert Information Society DG European Commission Jakub Wejchert grew up in Ireland, with a family background of artists and architects, of Polish origin. He studied natural science at Trinity College Dublin, specialising in physics, and holds a doctorate (modelling of non-linear networks) from the same institution. Later he worked in the USA with IBM research, working on computer graphics and interface design. He joined the European Commission in 1992. At the Future and Emerging Technologies unit, he set up and managed a number of research programmes such as i3 – intelligent information interfaces; i3 – experimental school environments; and the “disappearing computer”. He now works as an advisor on vision and strategy to one of the Directors in the Information Society Programme. Jakub lives in Waterloo, south of Brussels, with his wife and three sons. The opinions expressed here are those of the author and do not necessarily reflect the position of the European Commission.
viii
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Acknowledgements
011 The editors would like to acknowledge the European i3 initiative and all the authors of the chapters in this volume for their contributions. Much of the work described in this volume would not have taken place without funding from the European Commission. We would also like to thank SICS, XRCE and FX Palo Alto Laboratory for supporting our activities within this domain. Rosie Kemp and Melanie Jackson of Springer also deserve thanks for their help and support throughout the process of preparing this book for publication.
011
011
011
11
ix
This page intentionally left blank
Contents
011
011
011
List of Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Part 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1. Inhabited Information Spaces: An Introduction Elizabeth Churchill, David Snowdon and Emmanuel Frécon . . . . . . . . . . 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Chapters in this Volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Pure Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Mixed Reality Environments . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.3 Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.4 Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.5 Community . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 3 5 5 6 6 7 8 8
Part 2. Pure Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2. WWW3D and the Web Planetarium Mårten Stenius and David Snowdon . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Producing a 3D Representation of a Web Page . . . . . . . . . . . . . . 2.3 Browsing the Web Using WWW3D . . . . . . . . . . . . . . . . . . . . . . . 2.4 Improving Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 The Web Planetarium: Creating a Richer Visualisation . . . . . . . . 2.5.1 Visual Differentiation of Nodes . . . . . . . . . . . . . . . . . . . . . . 2.5.2 The Web as a Road Network . . . . . . . . . . . . . . . . . . . . . . . . 2.5.3 Hybrid Browsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11 11 12 13 16 20 20 22 22 24
3. PlaceWorld, and the Evolution of Electronic Landscapes Steve Pettifer, Jon Cook and James Marsh . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Background: The Physical and the Abstract . . . . . . . . . . . . . . . . . 3.2.1 Watching a Cityscape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 The Distributed Legible City . . . . . . . . . . . . . . . . . . . . . . . . .
25 25 27 28 29
011
11
xi
Contents
3.3 3.4
3.5
3.6
3.2.3 Finding “Something to Do” . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.4 Abstract Influences: Nuzzle Afar . . . . . . . . . . . . . . . . . . . . . PlaceWorld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 The Design of PlaceWorld . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 The User Interface and Presentation System . . . . . . . . . . . . Technological Challenges for Electronic Landscapes . . . . . . . . . . 3.4.1 Synchronising the Behaviour of Entities . . . . . . . . . . . . . . . 3.4.2 Distribution and Communications . . . . . . . . . . . . . . . . . . . . 3.4.3 Defining the Behaviour of Entities . . . . . . . . . . . . . . . . . . . . 3.4.4 Methods and Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.5 The Distribution Architecture . . . . . . . . . . . . . . . . . . . . . . . System Support for PlaceWorld . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.1 Menus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.2 Access Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.3 Exploiting Subjectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.4 Becoming a Place Where Places Meet . . . . . . . . . . . . . . . . . Conclusions
4. Using a Pond Metaphor for Information Visualisation and Exploration Olov Ståhl and Anders Wallberg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 The Pond . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 The Pond Ecosystem Metaphor . . . . . . . . . . . . . . . . . . . . . . 4.2.2 The Pond Example Application . . . . . . . . . . . . . . . . . . . . . . 4.2.3 The Hardware Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.4 The Software Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 The Pond Audio Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Observations from Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Summary and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
31 33 33 34 36 37 39 40 41 43 44 46 46 46 47 48
51 51 54 54 55 56 57 58 63 64 65 68
Part 3. Mixed Reality Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 5. City: A Mixture of Old and New Media Matthew Chalmers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Ongoing and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
xii
71 71 73 77 82 86 88
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Contents
● 1 ●●●●●●●●●●●●●● 6. Soundscapes Tony Brooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 The Soundscapes System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Therapeutic Uses of Soundscapes . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Artistic Performances Based on Soundscapes . . . . . . . . . . . . . . . . 6.4.1 Interactive Painting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.2 The Four Senses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
89 89 89 92 94 94 96 99
011
011
011
011
7. The Computational Interplay of Physical Space and Information Space Enric Plaza . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 7.2 The Interplay of Physical and Information Spaces . . . . . . . . . . . 102 7.3 A Framework for Context-aware Agents . . . . . . . . . . . . . . . . . . . 104 7.3.1 Awareness and Delivery Services . . . . . . . . . . . . . . . . . . . . 105 7.3.2 Agents Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 7.4 The COMRIS Conference Centre . . . . . . . . . . . . . . . . . . . . . . . . . 107 7.4.1 Delivery Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 7.4.2 Awareness Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 7.4.3 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 7.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Part 4. Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 8. Communicating in an IIS: Virtual Conferencing Adrian Bullock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Virtual Conferencing – a Historical Perspective: Past, Present and Future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.1 What Do We Mean by Virtual Conferencing? . . . . . . . . . . 8.3 Approaches to Virtual Conferencing . . . . . . . . . . . . . . . . . . . . . . 8.3.1 Early Videoconferencing . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.2 MUDs and MOOs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.3 The Arrival of Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.4 Video Comes of Age . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.5 Graphics Come of Age . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 Using Virtual Conferencing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.1 Understanding Collaboration . . . . . . . . . . . . . . . . . . . . . . . 8.4.2 The Importance of First Impressions . . . . . . . . . . . . . . . . . 8.4.3 Sharing Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.4 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.5 Real Versus Abstract: The Role of Video? . . . . . . . . . . . . .
115 115 116 117 117 117 118 118 120 122 124 124 125 125 125 126
11
xiii
Contents
8.5 8.6
Virtual Conferencing Versus Telephony . . . . . . . . . . . . . . . . . . . Guidelines for Using Virtual Conferencing Effectively . . . . . . . . 8.6.1 What Is the Task at Hand? . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.2 Communication Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.3 Infrastructural Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . Final Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
127 129 129 130 130 131
9. Getting the Picture: Enhancing Avatar Representations in Collaborative Virtual Environments Mike Fraser, Jon Hindmarsh, Steve Benford and Christian Heath . . . . 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.1 Awareness and Co-ordination . . . . . . . . . . . . . . . . . . . . . . 9.3.2 Anticipation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.3 Occlusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 Reflections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.1 Scaleability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.2 Reciprocity of Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.3 Unrealism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
133 133 135 137 137 140 142 145 145 146 147 149 150
8.7
10. New Ideas on Navigation and View Control Inspired by Cultural Applications Kai-Mikael Jää-Aro and John Bowers . . . . . . . . . . . . . . . . . . . . . . . . . . 151 10.1 Introduction and Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 10.1.1 Challenges for Interaction Design . . . . . . . . . . . . . . . . . . . 152 10.2 Interactive Performances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 10.2.1 Lightwork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 10.2.2 Blink . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 10.3 Inhabited Television . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 10.3.1 Heaven and Hell – Live . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 10.3.2 Out of This World . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 10.4 Production Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 10.4.1 Finding and Framing the Action . . . . . . . . . . . . . . . . . . . . 170 10.4.2 The Round Table: A Physical Interface . . . . . . . . . . . . . . . 172 10.4.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 10.5 Discussion: Navigation, Presence and Avatars . . . . . . . . . . . . . . 176 10.5.1 Avatar-centred Navigation . . . . . . . . . . . . . . . . . . . . . . . . 176 10.5.2 Object-centred Navigation . . . . . . . . . . . . . . . . . . . . . . . . 177 10.5.3 Activity-oriented Navigation . . . . . . . . . . . . . . . . . . . . . . . .178 10.5.4 Navigation as Montage, Dispersed Avatars . . . . . . . . . . . . 178 10.5.5 Accomplishing Presence and Intelligibility . . . . . . . . . . . . 179
xiv
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
Contents
● 1 ●●●●●●●●●●●●●● 11. Presenting Activity Information in an Inhabited Information Space Wolfgang Prinz, Uta Pankoke-Babatz, Wolfgang Gräther, Tom Gross, Sabine Kolvenbach and Leonie Schäfer . . . . . . . . . . . . . . . 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Related Work and Requirements . . . . . . . . . . . . . . . . . . . . . . . . 11.3 User Involvement and Studies . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.1 Partner Settings and Evaluation Methods . . . . . . . . . . . . . 11.3.2 Do Users Meet at all in a Shared Workspace? . . . . . . . . . 11.4 The Tower Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5 Personalised Overview of Activities: The Tower Portal . . . . . . . 11.6 Awareness in a Working Context: Smartmaps . . . . . . . . . . . . . . 11.7 Symbolic Actions in a Context-based 3D Environment . . . . . . . 11.7.1 The Tower World . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.7.2 User Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.8 DocuDrama . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.9 Ambient Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.10 Lessons Learned About Awareness . . . . . . . . . . . . . . . . . . . . . . . 11.10.1 Awareness Is Something One Is Not Aware of . . . . . . . . 11.10.2 Synchronicity of Awareness . . . . . . . . . . . . . . . . . . . . . . 11.10.3 Walking and Talking Are Means to Achieve Awareness . 11.10.4 Peripheral Awareness in Electronic Settings . . . . . . . . . . 11.10.5 Awareness Is Double-situated: The Workspace’s and the Observer’s Situation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.11 Summary and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
181 181 182 184 185 186 188 189 191 194 194 196 198 201 203 203 204 205 205 206 207
Part 5. Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 12. DIVE: A Programming Architecture for the Prototyping of IIS Emmanuel Frécon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 The Virtual World as a Common Interaction Medium . . . . . . . 12.3 Partial, Active Database Replication . . . . . . . . . . . . . . . . . . . . . . 12.4 Programming the System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4.1 The DIVE Programming Model . . . . . . . . . . . . . . . . . . . . 12.4.2 Programming Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4.3 Building your Application . . . . . . . . . . . . . . . . . . . . . . . . . 12.5 DIVE as a Component-based Architecture . . . . . . . . . . . . . . . . . 12.5.1 System Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.2 User-oriented Components . . . . . . . . . . . . . . . . . . . . . . . . 12.5.3 The DIVE Run-time Architecture . . . . . . . . . . . . . . . . . . . 12.6 The London Demonstrator: An Example Application in More Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
211 211 212 213 215 216 216 218 223 223 224 225 226
11
xv
Contents
12.6.1 Centre of London . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.6.2 Collaboration Services for Use by Groups . . . . . . . . . . . . 12.6.3 Tourist Information Data Visualisation Service . . . . . . . . 12.6.4 Real-time Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.7 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . .
228 229 229 230 231
13. Communication Infrastructures for Inhabited Information Spaces David Roberts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1.2 Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1.3 Avatars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1.4 Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1.5 Communication Requirements . . . . . . . . . . . . . . . . . . . . . 13.1.6 Resources: Computers and Networks . . . . . . . . . . . . . . . . 13.2 Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.1 Localisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.2 Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.3 Persistence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.4 Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.1 The DIVE Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.2 PING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4 Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4.1 Point-to-point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4.2 Tunnelled Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4.3 Hybrid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
233 233 234 235 236 237 238 240 240 241 246 251 252 256 256 260 263 264 265 266 266
Part 6. Community . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 14. Peer-to-peer Networks and Communities Mike Robinson . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2 Early Inhabited Information Spaces in CSCW . . . . . . . . . . . . . . 14.2.1 Rendering the Invisible Visible . . . . . . . . . . . . . . . . . . . . . 14.2.2 ClearBoard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2.3 Feather, Scent and Shaker: Supporting Simple Intimacy . 14.2.4 Gesture Cam: The Nodding Robot . . . . . . . . . . . . . . . . . . 14.3 P2P Themes and Overall Direction . . . . . . . . . . . . . . . . . . . . . . . 14.4 Design for Community: Inhabited Information Spaces . . . . . . . 14.4.1 Communities: An Aside on Definitions . . . . . . . . . . . . . . 14.4.2 Communities: An Aside on Use . . . . . . . . . . . . . . . . . . . . 14.4.3 Communities: An Aside on Philosophy . . . . . . . . . . . . . .
xvi
271 271 274 274 275 276 277 278 281 281 282 284
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
Contents
● 1 ●●●●●●●●●●●●●● 14.5 P2P, Community and the Design of Inhabited Information Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 14.6 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 15. Inhabitant’s Uses and Reactions to Usenet Social Accounting Data Byron Burkhalter and Marc Smith . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.3 Netscan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.4 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.4.1 Social Accounting Data and Author-assessment Threads 15.4.2 Social Accounting Data and Newsgroup-assessment Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
291 291 293 294 295 295 301 305
References
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
Index
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
011
011
011
11
xvii
This page intentionally left blank
List of Contributors
011
011
011
011
11
Steve Benford Mixed Reality Laboratory, School of Computer Science, University of Nottingham, Jubilee Campus, Nottingham, NG8 1BB, UK
[email protected] John Bowers Department of Numerical Analysis and Computer Science, Royal Institute of Technology, SE-100 44, Stockholm, Sweden
[email protected] Tony Brooks Aalborg University, Niels Bohrs Vej 8, DK 6700 Esbjerg, Denmark
[email protected]
[email protected] Matthew Chalmers Computing Science, University of Glasgow, 17 Lilybank Gardens, Glasgow, G12 BQQ, UK
[email protected] Elizabeth Churchill FX Palo Alto Laboratory, 3400 Hillview Avenue, Building 4, Palo Alto 94304, USA
[email protected] Jon Cook Department of Computer Science, The University of Manchester, Oxford Road, Manchester, M13 9PL, UK
[email protected]
Adrian Bullock Swedish Institute of Computer Science, Box 1263, SE-164 29, Kista, Sweden
[email protected]
Mike Fraser Mixed Reality Laboratory, School of Computer Science, University of Nottingham, Jubilee Campus, Nottingham, NG8 1BB, UK
[email protected]
Byron Burkhalter Department of Sociology, University of California, Los Angeles, 264 Haines Hall, 375 Portola Plaza, Los Angeles, CA 90095–1551, USA
Emmanuel Frécon Swedish Institute of Computer Science, Box 1263, SE-164 29, Kista, Sweden
[email protected]
xix
List of Contributors
Wolfgang Gräther Fraunhofer Institute for Applied Information Technology, Schloss Birlinghoven, 53754 Sankt Augustin, Germany wolfgang.graether@fit.fraunhofer.de
James Marsh Department of Computer Science, The University of Manchester, Oxford Road, Manchester, M13 9PL, UK
[email protected]
Tom Gross Fraunhofer Institute for Applied Information Technology, Schloss Birlinghoven, 53754 Sankt Augustin, Germany tom.gross@fit.fraunhofer.de
Uta Pankoke-Babatz Fraunhofer Institute for Applied Information Technology, Schloss Birlinghoven, 53754 Sankt Augustin, Germany uta.pankoke@fit.fraunhofer.de
Christian Heath Work, Interaction and Technology Research Group, The Management Centre, King’s College London, Franklin-Wilkins Building, London, SE1 8WA, UK
[email protected] Jon Hindmarsh Work, Interaction and Technology Research Group, King’s College London, UK
[email protected] Kai-Mikael Jää-Aro Department of Numerical Analysis and Computer Science, Royal Institute of Technology, SE-100 44, Stockholm, Sweden
[email protected] Sabine Kolvenbach Fraunhofer Institute for Applied Information Technology, Schloss Birlinghoven, 53754 Sankt Augustin, Germany sabine.kolvenbach@fit.fraunhofer.de
xx
Steve Pettifer Department of Computer Science, The University of Manchester, Oxford Road, Manchester, M13 9PL, UK
[email protected] Enric Plaza IIIA, Artificial Intelligence Research Institute, CSIC, Spanish Council for Scientific Research, Campus UAB, 08193 Bellaterra, Catalonia, Spain
[email protected] Wolfgang Prinz Fraunhofer Institute for Applied Information Technology, Schloss Birlinghoven, 53754 Sankt Augustin, Germany wolfgang.prinz@fit.fraunhofer.de David Roberts Department of Computer Science, University of Reading, Whiteknights, Reading, RG6 6AY, UK
[email protected]
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
List of Contributors
● 1 ●●●●●●●●●●●●●● Mike Robinson Sageforce Ltd, 61 Kings Road, Kingston-on-Thames, Surrey, KT2 5JA, UK
[email protected] Leonie Schäfer Fraunhofer Institute for Applied Information Technology, Schloss Birlinghoven, 53754 Sankt Augustin, Germany leonie.schaefer@fit.fraunhofer.de Marc Smith Microsoft Research, Microsoft Corporation, One Microsoft Way, Redmond, WA 98052, USA
[email protected] David Snowdon Xerox Research Centre Europe, 6, chemin de Maupertuis, 38240 Meylan, France
[email protected]
Olov Ståhl Swedish Institute of Computer Science, Box 1263, SE-164 29, Kista, Sweden
[email protected] Mårten Stenius Alkit Communications AB, Aurorum 2, SE-977 75, Luleå, Sweden
[email protected] Anders Wallberg Swedish Institute of Computer Science, Box 1263, SE-164 29, Kista, Sweden
[email protected]
011
011
11
xxi
This page intentionally left blank
011
011
011
011
11
Part 1 Introduction
This page intentionally left blank
011
011
011
011
Inhabited Information Spaces: An Introduction
1
Elizabeth Churchill, David Snowdon and Emmanuel Frécon
1.1 Introduction Studies of co-operative work have shown that, even when co-operation is not explicit, a surprisingly large amount of work relies on the knowledge of what other people are doing (or have done) so that work can be co-ordinated. Further, people collaborate over objects that are manipulated and exchanged: reports, diagrams, maps, books, models and drawings are all examples. People’s conversations and actions are part and parcel of producing the social frame within which work takes place (Giddens, 1984; Lave and Wenger, 1991). However, the technological circumstances within which that work takes place has a profound impact upon how it is achieved – technologies afford resources and constraints that affect practice. This volume offers readers an introduction to the field of Inhabited Information Spaces (IIS). We attempt to shed some light on the most important issues, including examples of representing information, how people interact in such systems, how IIS systems are constructed, and emerging notions of communities. There are already many books dedicated to “pure” information visualisation (techniques for visually representing information) and so in this volume we concentrate mainly on the value added by inhabited spaces rather than visualisation techniques per se. For a review of the field of “pure” information visualisation we recommend Card et al. (1999), who give a good overview of the field. The question addressed by IIS design is how best to design spaces and places where people and digital data can meet in fruitful exchange – that is, how to create effective social workspaces where digital information can be created, explored, manipulated and exchanged. IIS are the confluence of research into distributed, augmented and virtual reality (VR) spaces, information visualisation and computer-supported co-operative work (CSCW).
11
3
Inhabited Information Spaces
The term “Inhabited Information Spaces” derived from work being carried out within a number of European research initiatives (e.g. INSCAPE, COMIC) and laid out an agenda whereby virtual reality and information visualisation techniques were explicitly combined in support of collocated and remote collaborative work. Thus, in Inhabited Information Spaces (IIS) both information and people who are using that information (viewing it, manipulating it) are represented. This supports collaborative action on objects, provides awareness of others’ ongoing activities and offers a view of information in the context of its use. Thus, while information visualisation systems are useful tools in themselves, a representation of others who are also using the information and what they are doing with it could potentially add considerably to the value of such systems for co-ordination of collaborative work. Just knowing that many people are accessing a particular piece of information could be almost as useful as the piece of information itself. The specific representations can vary but in all cases they are manipulable: data can be interrogated, representations of people are mobile and interactive. Further, being shared, navigable, “live-in” spaces, information can be explicitly sought, can be “discovered by chance” (cf. Williamson, 1998), or “encountered” (cf. Erdelez, 1999) or tacitly consumed as part and parcel of navigating an “information neighbourhood” (cf. Savolainen, 1995). In Europe, the European Commission (EC) funds large programmes of research covering most fields of human endeavour (science, technology, medicine, culture etc.); one of the domains supported by the EC is Information Society Technologies (IST, http://www.cordis.lu/ist/). In 1996 the i3 (Intelligent Information Interfaces) network (http://www.i3net.org) was formed as a mechanism to create a community of people working on a number of research projects. One of the aims of this was to encourage researchers participating in European projects to exchange ideas and information and to allow collaboration on a larger scale. From 1997–2000 the i3 supported a programme called Inhabited Information Spaces and much of the work that appears in this book resulted from this programme. The field of IIS overlaps that of Collaborative Virtual Environments (CVEs) (see Churchill et al., 2001), as CVEs are one of the preferred implementation techniques for visualisation information in a collaborative way. However, IIS does not necessarily imply the use of online virtual environments technology – for example, it is possible to imagine a system that enables co-located groups to co-operatively work with information by using a display projected onto physical artefacts. Research into IIS also overlaps with work carried out in Social Navigation, which explicitly addresses social aspects of information seeking, searching and use (see Munro et al., 1999). The chapters in this volume cover all variants on IIS, the technology required to make it work and the social and psychological issues raised by such work. We also present recent innovations in “hybrid
4
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
An Introduction
● 11 ● ● ● ● ● ● ● ● ● ● ● ● ● ● environments” and augmented real-world environments. Our aim with this broad coverage is to offer readers the opportunity to reflect on the intersection of technology design, communication, representation, and collaborative work practices. Technological, psychological and sociological issues in the design and use of Inhabited Information Spaces are considered, including: design issues in the development of technology for human–human and human–system collaboration around information visualisations; applications demonstrating uses of the technology; and psychological/sociological analyses of the way such systems are used in practice. Our aim was to provide a broader perspective than solely graphical aspects of visualising information, systems aspects underlying the distribution and sharing of graphical and textual workscapes, or the communication and work practice aspects of the use of such systems.
1.2 Chapters in this Volume We have divided the chapters in this volume into a number of broad areas, although observations made by the authors often span these areas. The areas are pure virtual environments; mixed reality; communicationoriented systems and applications; construction; and community.
1.2.1 Pure Virtual Environments Stenius and Snowdon, in their chapter ‘WWW3D and the Web Planetarium’ (Chapter 2) describe the WWW3D 3D web browser, and how it turns browsing the web into exploration of a 3D space. The chapter describes the initial version of WWW3D and describes how it evolved into the Web Planetarium. Not only is the Web Planetarium more aesthetically pleasing and more scalable than WWW3D, but in this incarnation it has also been re-purposed to serve as a gateway between 3D environments. In Chapter 3, Pettifer, Cook and Marsh describe the Placeworld system, and its implementation in the Deva virtual environment. Placeworld was inspired by Jeffrey Shaw’s artwork “PLACE – A User’s Manual” and, like the Web Planetarium, aims to provide a connection between virtual spaces. The chapter describes the series of user trials that led to the final Placeworld design and how the Deva system is used to create a high performance virtual environment that implements the Placeworld design. This chapter also gives some insight into some of the issues that must be tackled to implement large scale IIS and CVE systems efficiently and serves as an appetiser for topics covered in more depth in Part 5 of this book. The final chapter in Part 2, Chapter 4, is by Wallberg and Ståhl, who use a pond metaphor for information visualisation and exploration. The Pond is a system that allows people to browse collections of multimedia data, such as music albums. The interface is presented in the form of a
5
Inhabited Information Spaces
large back-projected display on a table surface allowing several people to gather around it and use it collaboratively.
1.2.2 Mixed Reality Environments Chalmers, in his chapter entitled “City: A Mixture of Old and New Media” (Chapter 5), argues that one of the problems with “traditional” CVE systems is that they are disconnected from the physical environment and from other media (for a review of CVE systems see Churchill et al., 2001). He calls for an approach in which there is an explicit linking between different media, where people are considered to inhabit the “real” (physical) world, not the virtual, and have a number of media both “old” and “new” available to them. The chapter presents an experiment where visitors experience a gallery via different media – one by physically experiencing it, one via the web and one via an immersive 3D VR environment. All three are able to communicate via an audio link and are given awareness of the locations and actions of the others. The chapter presents the results of this experiment and the ways in which the participants used the features of the technology to interact and share their experiences. In Chapter 6, Brooks describes the Soundscapes system, which allows unencumbered interaction with visual and auditory systems projected into the physical world. This work illustrates a different form of information system, an auditory one, and offers an example of a mixed reality system, and example applications are covered including therapeutic use and public artistic performances. In the next chapter, Plaza illustrates the computational interplay of physical space and information space. Drives within mobile computing push computers further “into” the physical world (see also Mark Weiser’s vision for ubiquitous computing, Weiser, 1991). In order to design tools that are context sensitive and not inappropriately intrusive, it has been argued that such devices need to be “aware” of the activities of their users. This chapter describes the approach used in the COMRIS project in which wearable computers were linked with an information space composed of agents who attempt to find information useful to a person at a given moment.
1.2.3 Communication In Chapter 8, Bullock describes human–human communication via the medium of an IIS, considering design issues in the development of virtual conferencing. He makes the points that, for an IIS to function effectively, all technological elements need to work in concert. With video conferencing as a backdrop, Bullock explores opportunities and pitfalls of using IIS for mediated communication.
6
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
An Introduction
● 11 ● ● ● ● ● ● ● ● ● ● ● ● ● ● Fraser, Hindmarsh, Benford and Heath in their chapter “Getting the Picture: Enhancing Avatar Representations in Collaborative Virtual Environments” discuss how humanoid avatars are the most widely used type of avatar in VR systems. They note that although the avatars have humanoid forms they don’t have human-like perceptual abilities within the virtual worlds – the data that are relayed back to people from their avatar as prosthesis-in-the-virtual-world is in fact often misleading. This chapter examines some of the problems inherent in using different forms of avatar, and the problems this poses for collaboration in IIS. The authors describe some extensions the authors have made in an attempt to rectify some of the problems they have encountered. Jää-Aro and Bowers describe some new ideas on navigation and view control that have been inspired by cultural applications. This chapter describes the lessons learnt from a number of public VR and mixedreality performances. The authors discuss what they learnt in terms of the content of performances, the pacing and the means given to participants to navigate within the space. An important issue addressed is how such performances can be made accessible to non-interactive audiences who can only see a TV-like rendering of the event. In Chapter 11, Prinz et al. consider how to present awareness of the activities of others for better support of collaborative work. This chapter describes the TOWER (Theatre of Work Enabling Relationships) system, which provides a number of mechanisms to communicate awareness information to members of a work group both via 3D displays and also via Smartmaps integrated with the Basic Support for Co-operative Work (BSCW) document management system. Smartmaps are 2D displays based upon the tree-map visualisation technique. The TOWER world is an automatically constructed 3D environment that represents both users and documents and indicates the actions that users are taking with respect to the documents via symbolic actions and gestures performed by the avatars. DocuDrama allows 3D presentations of the past actions taken by members of a project team as a sort of 3D virtual theatre in which avatars look and turn towards one another to enhance the impression of an ongoing conversation, and camera navigation is carefully controlled in order to generate an interesting presentation. Finally, small simple robots were employed in order to give a tangible presentation of the activities of other users.
1.2.4 Construction Frecon’s chapter in Part 5 introduces readers to DIVE, a programming environment for prototyping IIS. The DIVE CVE system is described, with a focus on the mechanisms it provides to allow developers to rapidly develop VR applications, including IIS. DIVE provides several different APIs (application programmer’s interfaces) and mechanisms for
7
Inhabited Information Spaces
creating dynamic 3D content thereby allowing developers to choose the combination that works best for them. The chapter concludes with a number of examples that show how significant applications have been built using DIVE. DIVE is one of the oldest and most mature VR systems and the current version is the result of many years of experience; both the Web Planetarium and Pond systems are built in DIVE. Readers are urged to read these three chapters (2, 4 and 12) to fully understand what is possible with a mature VR system. In Chapter 13, Roberts considers communication architectures for IIS, describing the most important networking issues that need to be faced when trying to construct distributed IIS systems. He also details some of the techniques that can be used to create an illusion of a shared space in the face of delays caused by communications technology. A number of different CVE systems are described in order to give concrete examples of the techniques described in the earlier part of the chapter.
1.2.5 Community Robinson’s chapter deals with peer-to-peer networks and communities. Robinson argues that the concept of peer-to-peer and notions of community are heavily interdependent, and that the design of IIS would benefit by focusing more closely on community as the organising principle of peer-to-peer. The author first considers the early days of research into CSCW, analyses the metaphor of community and how it relates to electronically mediated communication, and draws parallels with the current state of peer-to-peer systems today. Burkhalter and Smith move away from what might be considered a true IIS in Chapter 15 to explore how the availability of social accounting data can help the auto-regulation of online communities. The authors consider Usenet news but the principles could just as well be applied to other systems. The intent of this chapter is to show how similar information might be used to help CVE-based communities such as those being created by there.com (http://www.there.com), if they ever reach the scale of communities such as Usenet.
1.3 Summary As noted above, the chapters in this volume cover a broad range of technical and social issues. All are focused on creating useful and “habitable” environments for information representations, seeking, searching and manipulation. There are many challenges to be faced at technical and at social levels, and far more research needs to be done on the long-term use of Inhabited Information Spaces, and how they co-evolve as a result of being regularly inhabited. We hope you enjoy reading about the work as much we have.
8
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
Part 2 Pure Virtual Environments
011
011
011
11
9
This page intentionally left blank
011
011
011
011
11
WWW3D and the Web Planetarium
2
Mårten Stenius and David Snowdon
2.1 Introduction This chapter will describe an Inhabited Information Space based around a 3D visualisation of a portion of the WWW. The system was originally called WWW3D (Snowdon et al., 1996), and evolved from an experiment in immersive 3D web browsing into a richer inhabited information space called the Web Planetarium during the i3 eSCAPE project (Stenius et al., 1998). We include it here as an example of an inhabited space supporting a common modern activity – web browsing – and also as an example of how a basic visualisation was extended to correct deficiencies in the earlier system. WWW3D started life as an experiment in 3D web browsing. We were thinking about some of the original ideas behind VRML (Virtual Reality Modelling Language, www.vrml.org), namely that it would be a sort of 3D web. In fact many people working on VRML were inspired by the Cyberspace detailed in William Gibson ’s novels: Cyberspace. A consensual hallucination experienced daily by billions of legitimate operators, in every nation, by children being taught mathematical concepts . . . A graphic representation of data abstracted from the banks of every computer in the human system. Unthinkable complexity. Lines of light ranged in the nonspace of the mind, clusters and constellations of data. Like city lights, receding (Gibson, 1986).
However, VRML became more a means to represent 3D models with hyperlinks between them, than a richer way of experiencing the web. It so happened that as we were thinking about this during the summer of 1996 we were asked to provide a demonstration for SICS’s (the Swedish Institute of Computer Science’s) new large-screen immersive VR system. This system, while having impressive graphical performance, did not support interaction via a standard 2D GUI (Graphical User Interface); we therefore decided to create a web browser that was entirely 3D. While
11
Inhabited Information Spaces
the result was not a web browser as usable as Netscapes ’s or Microsoft’s for standard web browsing, the prototype had several interesting features and evolved into a more capable system during the i3 eSCAPE project. In this chapter we will describe the original system, WWW3D, and its evolution into the Web Planetarium. Just as a normal web browser, WWW3D allowed users to follow links and view the web pages associated with those links. However, rather than simply showing the current web page the user was exploring, WWW3D also visualised the structure of the portion of the web that the users had explored, historical information showing when pages had been last viewed and (thanks to the DIVE environment for which WWW3D was written) other users browsing the web at the same time. Note that WWW3D was never intended to visualise the web – the web is far too large for this to be meaningful, instead we simply visualise the portion of the web that users have explored and are exploring. We assume that existing search engines are sufficient to locate new pages of interest. The two key features of WWW3D were its method for representing web pages in 3D and the method for organising multiple web pages in 3D space. We will describe how this was done before continuing with a description of how the basic prototype was improved.
2.2 Producing a 3D Representation of a Web Page Many HTML tags provide semantic information about the marked-up text – that is, what the text is rather than how it should be displayed on the page. 2D web browsers will typically use this information to determine what style (i.e. font, font style, font size, etc.) to use when drawing a particular piece of text. WWW3D uses the information contained in HTML tags to produce a representation of the document in 3D space. A web document is represented as a sphere that is labelled with the document’s title. The contents of the document is placed around the inside surface of the sphere. Displaying large amounts of text in a satisfactory way is difficult in current VR systems, so textual information is currently represented by icons that can be unfolded to reveal the entire text. The first few words of the piece of text are displayed under the icon to give some indication of the contents. Images are displayed by texture mapping them onto polygons on the inside surface of the sphere. Finally, links to other documents are represented as icons labelled with their destination. To reduce the visual complexity of the virtual environment WWW3D makes extensive use of level of detail (LOD) operations. When viewed from outside, a document is represented as an opaque sphere and the actual document contents are not displayed. When a user enters a document to view it, the sphere is drawn in wire-frame so that the rest of the virtual environment is still visible.
12
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
WWW3D and the Web Planetarium 2 ●●●●●●●●●●●●● ● 1 ●
011
011
011
Figure 2.1 A WWW3D representation of a web document seen from the inside. The spheres in the background represent other web pages.
Figure 2.1 shows the contents of a web document as displayed by WWW3D. Figure 2.2 shows the display generated when a user reads some of the text comprising a web document.
2.3 Browsing the Web Using WWW3D
011
11
When a user selects a link icon, WWW3D creates a new sphere representing the target document and places it near the document from which the user selected the link. In order to indicate the structure of the portion of the Web that the users have explored, WWW3D draws arrows between the spheres representing linked documents. If the documents are resident on the same Web server then the arrow is drawn in blue, otherwise
13
Inhabited Information Spaces
Figure 2.2 Reading part of the text of a web document.
it is drawn in green, thereby helping to provide additional information on the structure of the documents that the user has explored. In addition to this, the brightness of the arrow is dependent on the time since a user last followed that link thereby providing users with a visual representation of their browsing history. If WWW3D fails to fetch a document then a small red arrow is attached to the source document to represent the “broken” link . WWW3D is implemented using multiple lightweight threads so users do not have to wait for a document to be retrieved before selecting another link. This behaviour is essential if multiple users are to be able to browse independently. Users are also free to navigate through the space and browse other documents while waiting for a document to be retrieved.
14
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
WWW3D and the Web Planetarium 2 ●●●●●●●●●●●●● ● 1 ●
As WWW3D parses a newly retrieved document, it checks for links to documents that the user has already explored and draws arrows to represent them. This means that at any given moment the complete set of links between documents is displayed without users having to follow every link. This is intended to aid the user by indicating links between documents that the user might have been unaware of. This also has the result that several users can be browsing different parts of the web and yet any links between the sets of documents they are exploring will be displayed. This might be useful since users will then have a visual representation of possible common interests. To produce an acceptable layout of the set of linked documents an incremental version of the Force Directed Placement (FDP) (Fruchterman and Reingold, 1991) algorithm is used. Links between documents act like spring forces that result in linked documents being moved closer together. Documents exert repulsive forces on one another which prevents documents being placed closer together than a userspecified minimum separation. At regular (user-specified) intervals WWW3D applies the FDP algorithm to refine the inter-document layout for a specified number of iterations. The more links a document has, the greater “inertia” it is considered to have when the FDP algorithm is applied. This has the result that heavily referenced documents are less likely to move and provides some stability to the visualisation. In addition damping is applied to try and prevent large changes to the visualisation for a given iteration of t he FDP algorithm – this helps to prevent the user from becoming disorientated. The result of this is that the inter-document layout gradually evolves over time to produce clusters of inter-linked documents. Instead of running the FDP algorithm until it converges (which could take a long time) only a specified number of iterations are executed. The rational for this is that the space will change anyway as users browse new web pages so it is not necessary to force users to wait until the layout stabilises before allowing them to continue browsing. If there is no change to the space (no new web pages) then the system will gradually converge to a stable state. However, the FDP algorithm suffers from the disadvantage that it can take a great many iterations to converge to a stable state. Figure 2.3 shows the display produced by WWW3D after the user has browsed a number of documents. The FDP algorithm has resulted in the formation of clusters of closely linked documents. The colours of the documents provide some indication of how long ago the user last visited them. Between invocations WWW3D stores information on the current set of documents, the links between them and the current 3D layout to a file. When a new instance of WWW3D is started, it reads this history file and displays the structure as it appeared in the last session. The contents of the documents found from the history file are not retrieved until the user
15
Inhabited Information Spaces
Figure 2.3 A collection of web documents, showing the links drawn between them.
enters a particular document. Since the document representations are opaque this process is invisible to the user except for an occasional delay in seeing a document’s contents.
2.4 Improving Scalability One problem with the WWW3D prototype is that it suffers from scalability problems both in terms of display and number of users. Since a single DIVE application is responsible for management of the visualisation, it is vulnerable to becoming overloaded if several users simultaneously place demands on it. In addition to this the complexity of the display increases as documents are added, lessening the advantage of
16
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
WWW3D and the Web Planetarium 2 ●●●●●●●●●●●●● ● 1 ●
the displayed web structure information and making it harder to browse. Even though WWW3D makes extensive use of level-of-detail (LOD) operations for individual nodes, there will still be a point where the world becomes too complex to be rendered on even the most powerful hardware. For this reason we created a new version of WWW3D running on the MASSIVE-2 CVE (Benford et al., 1997a) that attempted to solve the display problem by making more extensive use of LOD effects combined with clustering of web pages. We decided to extend the use of LOD beyond the contents of a single web page to encompass groups of web pages (e.g. all pages on the same server, same domain name, owned by the same user etc.). Doing this meant that instead of showing all the pages in a website, or all the personal pages (home page and related pages) for a user, we could display an object that represented the whole group. The user would only see the contents of the group when a certain criteria was met – in this case getting close enough to the group. This meant that when a user was outside a cluster their client had no need to know anything about the web pages in the cluster or the users that were currently browsing web pages in the cluster. This would have the following advantages: ● ● ● ●
A reduction in the visual complexity of the world. A consistent metaphor since clusters looked and behaved similarly to individual web pages when users entered them. Increased interactivity since the computational load on the client machine was reduced. Reduced network bandwidth since MASSIVE-2 assigned separate multicast communication channels to each cluster. This meant that a user who could not see the interior of a cluster had no need to receive network updates concerning objects or users contained in that cluster.
We used a simple scheme based on URLs to perform our clustering. A more advanced alternative would be to explicitly consider the legibility of the CVE and cluster accordingly as is done by LEADS (Ingram and Benford, 1995) which is also capable of adding additional objects such as landmarks to aid users navigating through a CVE. Our new implementation was based on two fundamental building blocks, containers and links. Containers have the ability to enclose other objects in a spatial sub-region of the virtual space. Links connect the containers. A special form of container is the page, which represents a web page and its contents. A cluster is a container that is able to treat its contents as a sub-visualisation complete with layout, other containers and links. Figure 2.4 shows a cluster both from a distance and as the user gets sufficiently close to see the contents. In order to simplify the representation of links between clusters, we introduced compound links. A compound (or aggregate) link between two clusters is formed whenever there exists one or more links between the
17
Inhabited Information Spaces
a
b Figure 2.4 A cluster abstraction dissolves to display its contents when the user approaches: (a) the exterior of the cluster; and (b) the view seen by a user as they get closer to the cluster.
18
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
WWW3D and the Web Planetarium 2 ●●●●●●●●●●●●● ● 1 ●
A
B
c
f d AB g e
011
011
Figure 2.5 A compound link (AB) between two clusters (A, B).
contained objects of the two clusters. Thus, a compound link both serves as a link structure between two clusters and as an abstraction for one or more lower-level links between the contents of these clusters, this is illustrated in Figure 2.5. Figure 2.6 shows a collection of linked clusters. When refined, the current application structure approach should lend itself to parallelisation: letting different processes handle different clusters. This in turn would enable a distribution of calculations on different hosts to allow more complex visualisations.
011
011
11
Figure 2.6 A set of interconnected cluster abstractions. Their sizes roughly reflect the number of objects (here pages) contained within.
19
Inhabited Information Spaces
2.5 The Web Planetarium: Creating a Richer Visualisation The Web Planetarium extends the WWW3D concept with the idea of the system becoming a gateway between different virtual spaces. The metaphor of a planetarium was used to create a new, more informative and more visually appealing look and feel, since the system was also going to be used in public settings using a large, dome-like display (Schwabe and Stenius, 2000). Nodes in the Web Planetarium display can represent either web pages, 3D models (in VRML or DIVE format) or 3D DIVE worlds that can be jumped to. The Web Planetarium can therefore also act as a space of portals to other spaces and provide a means of navigation between different virtual worlds. Compared to WWW3D the Web Planetarium provides a more visually interesting virtual space and different layout options.
2.5.1 Visual Differentiation of Nodes One problem with the WWW3D visualisation is that all nodes look alike. The only distinguishing feature is colour and that colour represents the time since the last access and not a feature of the node itself. When creating the Web Planetarium, it was felt more important for users to be able to quickly scan the space for potentially interesting pages than to show the usage information. The Web Planetarium replaces the use of colour by texture maps extracted from the web pages. Alternatively, the sphere representing a node can be replaced by a 3D model. Not only does this result in a more interesting environment but it allows the users to dinstinguish more easily one node from another and gain some idea of what the node represents. Figure 2.7 shows an example Web Planetarium view in which the texture maps clearly distinguish nodes from one another. A simple algorithm is used to select the image to texture onto a sphere – the first image found in a web page is used as the texture. In the case of a user’s home page this image is typically a photo of the user – this is easily visible in Figure 2.8 in which several of the spheres clearly show people’s faces. Obviously, a simple algorithm such as this will sometimes fail to produce an interesting image for one of several reasons. ● ● ●
20
There is no image on the web page. A banner advert is selected instead of an image representative of the content of the page. The image selected is a piece of web decoration (a line, a bullet point) instead of something really representative of the subject of the web page.
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
WWW3D and the Web Planetarium 2 ●●●●●●●●●●●●● ● 1 ●
011
011 Figure 2.7 The 3D layout alternative emphasises the concept of a planetarium.
011
011
11
Figure 2.8 A screenshot of the Web Planetarium. Compared to the original WWW3D, one of the obvious differences is the use of texture mapping to give more information about the contents of web pages instead of a simple colour indicating the time since last access.
21
Inhabited Information Spaces
It would be hard, probably impossible, to produce an algorithm guaranteed to work, but there are a number of simple filtering operations that could be used to ensure that in many cases unsuitable images are rejected. ●
●
●
Reject images that are too small (according to predefined parameters) or are long and thin. This extremely simple approach was used successfully in the CWall electronic notice board (Snowdon and Grasso, 2002) and rejected most web decoration . Use a blacklist of URLs and URL components to reject banner advertisements or configure a proxy server that performs this function (Hood, 2000). There are websites that maintain such blacklists (@Man, 2000) so each user would not have to shoulder this burden. Reject images that do not come from the same server as the web page. This is a very simple approach that will work in many cases, but it may sometimes reject legitimate images.
An alternative approach could be to use a conventional web browser – embedded within the application or running in parallel – to generate a snapshot of the web page, including background colour, text layout, images and so on. The snapshot would then be used as an icon (or “thumbnail”) to be pasted on the outside of the 3D page representation. This method would have the advantage of creating a stronger connection to the original design of the web page as a whole.
2.5.2 The Web as a Road Network WWW3D and the Web Planetarium can generate very tangled and convoluted visualisations of nodes. Although users can manually reposition nodes this is not a solution that can be applied to large spaces. The Web Planetarium offers users the options of restricting the FDP layout algorithm to a horizontal plane, thereby giving the layout shown in Figure 2.9. A disadvantage of this approach is that in certain cases the FDP algorithm may take longer to converge.
2.5.3 Hybrid Browsing The original aim of WWW3D was to provide an acceptable and novel web-browsing experience using an entirely 3D interface. This made an interesting demo and made sense on a large screen or head-set based system but is less useful on the more common desktop-VR systems in which the VR system is just another window on the user’s 2D desktop. The Web Planetarium therefore allows users to combine the 3D view of the web structure with a traditional web browser view of the contents of a web page. This is illustrated in Figure 2.10 which shows a Web Planetarium view and a web browser (Netscape) view of the same web page side by side.
22
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
WWW3D and the Web Planetarium 2 ●●●●●●●●●●●●● ● 1 ●
011
011
Figure 2.9 The 2D layout alternative results in a road-like network of sites and links.
011
011
11
Figure 2.10 Using a conventional web browser as a side-viewer for HTML documents encountered in the Web Planetarium. Whereas the 3D view of the HTML document (to the right) gives an overview of the content and interaction points for links, Netscape provides a detailed close-up of the current page (to the left).
23
Inhabited Information Spaces
2.6 Conclusion This chapter has described WWW3D, a simple 3D web browser and its evolution into the Web Planetarium. Starting from a humble beginning WWW3D has been changed to address the issues of scalability and producing a more pleasing and informative visualisation. In addition it can now be seen as providing a means for navigating between disjoint 3D worlds as well as between web pages. However, the current version of the Web Planetarium is not without problems. One major issue is related to the viewing of 2D standard web content. Neither the current 3D view nor the separate browser and 3D views side by side really provide a convenient browsing experience. There is still work to be done to find the optimal way to provide information about web structure and a view on web page contents in an integrated way. The second problem with the Web Planetarum is that while it is perfectly possible for multiple users to share a single instance of the Web Planetarium there is no support given to the interrelation of different planetaria nor to different views on the same planetarium. The use of subjective visualisations (Jää-Aro and Snowdon, 2001) may provide a technique for merging planetaria since the same web page could be presented differently to different users, thus preserving individual layouts while allowing a degree of interaction between users.
24
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
PlaceWorld, and the Evolution of Electronic Landscapes
3
Steve Pettifer, Jon Cook and James Marsh
3.1 Introduction The eSCAPE project (ESPRIT Long Term Research Project 25377) set out to investigate the idea of an “electronic landscape” (e-scape): large-scale inhabited information spaces represented as a three-dimensional (3D) virtual environment (VE) in which users could interact with applications, find routes to other virtual worlds, access sources of data, and interact with one another at a social level. The project brought together computer scientists, artists, social scientists and graphics designers from Lancaster University, the University of Manchester, the Swedish Institute of Computer Science and the ZKM (Zentrum für Kunst und Medientechnologie – Centre for Art and Media Technology) in Germany. This chapter describes the evolution of PlaceWorld, an example e-scape. It is organised in four sections. We begin by chronicling the developments of the eSCAPE project that led to the implementation of PlaceWorld as its final demonstrator. We then consider in more detail the nature of PlaceWorld itself, and discuss how its development has influenced the underlying system technology. We conclude by tying together some of the technological issues with concrete examples of their use in the PlaceWorld landscape. The vision of an electronic landscape is easy to articulate, but difficult to make real. Distributed 3D virtual environments of any kind push the limits of today’s graphics, networking and processing technology; some of these issues will be considered in detail much later in the chapter. Add to these problems the social factors involved in making a virtual environment actually usable by the general public, and the issues become much more complex. The eSCAPE project’s approach to these difficulties is described in this chapter.
11
25
Inhabited Information Spaces
The project set out to address two problems. First there was the technological difficulty of integrating diverse environments and their applications. There is as yet no accepted standard technology for building a VE, and though the majority of implementers end up using OpenGL to render the visual part of the world at some point in their implementation, the sources of semantics and behaviour vary wildly from bespoke handcrafted pieces of code to those using higher level more generic systems such as MASSIVE (Greenhalgh, 1999), DIVE (Chapter 12) or DEVA (Pettifer et al., 2000). We will return to consider the way in which the DEVA VR system evolved to support electronic landscapes later in this chapter. The second, and much more open-ended, issue is that of how to make an environment really inhabitable and of practical utility to a user, and it is to this that we first turn. The methodology used in the eSCAPE project to develop effective electronic landscapes was iterative. Inspiration for the environments was drawn from two sources: artistic vision and ethnographic study of suitable real-world situations. The environments were then implemented and put in front of the general public for testing. Further ethnographic studies were then carried out on the installations, and the environments were refined and redeveloped in the light of the results from these studies. PlaceWorld combines lessons and technological perspectives discovered during the three-year period of this process.
Figure 3.1 Snapshots of PlaceWorld.
26
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
Figure 3.1 shows snapshots from the final implementation of PlaceWorld, and will serve for now, with the brief description that follows, to give an overall flavour of the environment. Users entering PlaceWorld find themselves in an exotic landscape containing many artefacts. Situated here and there are cylindrical “buildings” covered with images from other worlds: these are portals that lead elsewhere. A network of pathways, hovering above the mist-covered landscape lead off into the distance in all directions. The scene is populated with towers, billboards, flying craft and other miscellaneous structures, and in the sky ethereal floating ribbons of light drift back and forth. Other users move around the landscape, represented by animated walking stick figure-like characters, and leave behind hazy coloured trails. Interaction with any of these entities brings up on screen a selection of possible actions. Touching a user’s trail, for example, even when that user is a long way away, reveals that it can be used as a shortcut to find its owner. Touching any other artefact allows it to be “picked up” and carried around, to be deposited later at another location. The remainder of this chapter explains why the esoteric nature of PlaceWorld is as it is, beginning with the project’s initial experiments with electronic landscape .
011
011
011
11
3.2 Background: The Physical and the Abstract eSCAPE examined two extreme styles of virtual environment, to extract from them appropriate features that would make an environment inhabitable. At one extreme, there was the physical or concrete “cityscape”: familiar, constrained, slow to change, easily understood. At the other extreme, the “abstract landscape”: unfamiliar, dynamic, unrestricted. Aspects from both these styles of environment were finally integrated into PlaceWorld. Virtual environments have the potential to include aspects of both these extremes: there is no technological reason why one should be confined to a groundplane in the virtual world just because gravity works in a particular way in the real world; there is no technological reason why one should be forced to walk from one part of a virtual world to another rather than simply “teleporting” to one’s destination, much like clicking on a link in a web page. On the other hand, it is easy to imagine getting completely disorientated in a space where there are no constraints on navigation, or of missing out on chance meetings and opportunistic discoveries if one’s only means of transport is by instantaneous hyperlink. Thus some balance needs to be achieved between these two extremes of physical and abstract. The evolution of the project is easier to tell from the perspective of the concrete cityscape metaphor rather than the abstract, simply because of familiarity with its nature and terminology. The following vignettes chronicle various attempts at building electronic landscapes, and lead towards the final design of PlaceWorld.
27
Inhabited Information Spaces
3.2.1 Watching a Cityscape The first environment built during the project was a straightforward cityscape (Figure 3.2), consisting of buildings, roadways, parks and street furniture generated by algorithms based on Hillier and Hanson’s (1984) studies of urban evolution. A set of ethnographic studies were completed to see how users behaved in this environment, setting the participants the task of finding a number of landmarks in the city such as fountains, statues or pavilions. Studies were carried out at a desktop computer, using a standard mouse as an interaction device. The results were in some ways surprising. In spite of the user being a lone inhabitant of the cityscape, and without prompting from the scientist, wayfinding in the city was almost invariably carried out on the pedestrian areas (pavements, parks etc.) in spite of there being not a single car in sight. Though there was no collision detection preventing passage through the buildings, users mostly followed the road system. Even though the mouse navigation included an ability to fly up in the air (a feature that the participants were told about, and which clearly would give them an advantage in terms of spotting land marks from on high), this was rarely used (indeed, one participant who did briefly levitate to a suitable
Figure 3.2 A virtual cityscape.
28
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
vantage point rapidly came back down to ground level having felt uneasy up in the air). Although this evidence was anecdotal rather than definitive (and there were a small number of exceptions in terms of behaviour), the studies suggested that even with little effort by way of interaction device to induce the sense of presence, that the general affordances of a real cityscape, transferred rather strongly to the virtual world. A more detailed description of the study can be found in Murray et al., 2001. On this basis, a more ambitious city-like environment, the Distributed Legible City, was built.
011
011
011
011
11
3.2.2 The Distributed Legible City The artist Jeffrey Shaw, a partner in eSCAPE, had once developed an artistic installation piece called the Legible City. In this work, a solitary cyclist, seated on a real bicycle and situated in a darkened room with a large projected wall, could cycle their way around interpretations of the cities of Amsterdam, Manhattan and Karlsruhe. Rather than ordinary buildings, the cities were populated with towering, solid letters, forming extracts from texts associated with the areas of the cities. For example, a virtual tourist in the Legible City could follow the ramblings of a taxi driver around Manhattan, or find sections of poetry in Amsterdam. Informal observation of the piece, which is still in situ in the ZKM in the real Karlsruhe, showed that, even with now outdated rendering technology and simple interaction style, the piece would capture people’s attention, and they would happily tour around the cities for some time, often trying to find out which piece of text was associated with their favourite part of the world. For entirely opportunistic reasons (much of the technology already existed and the piece was known to be “engaging”), the project decided to extend this installation to form a distributed environment, and so the Distributed Legible City (DLC) was constructed (Figure 3.3). Using consumer-level graphical accelerators, the original database of city structures was reused, and modified exercise cycles were pressed into action as interface devices. Rather than the large screen projection system used in the original work, stands for monitors were built in front of the cycles, and where the first Legible City used an LCD panel attached to the handlebars to display maps of the cities, in this new version the map was displayed virtually as an overlay on the monitor screens showing now the layout of the world as well as the positions of other inhabitants. For the purposes of communication, the users were provided with headphones and a microphone. Other users were represented by animated cycling avatars. Two versions of the DLC interface bike were situated around the ZKM, and another across a wide area network in Vienna, and the installation was made available to the public during a number of exhibitions.
29
Inhabited Information Spaces
Figure 3.3 Images from the Distributed Legible City.
For several days, the technology behaved perfectly; however, the overall result of the piece was extremely disappointing. No longer did users engage with the environment and there was no interest in exploring the world for its own sake. Quickly, users realised that the buildings offered no resistance to them cycling through them, and the overlaid map was discarded, with users making a beeline towards one another and completely ignoring their surroundings. Once users had found each other, another disappointing interaction took place. Unexpectedly, users were resolute that they should be facing one another in order to carry out any conversation (this was not a requirement on the audio layer, which had no directional bias). The positioning of the monitor in front of the cycle, and the inability to pedal backwards meant that a significant amount of effort was put into achieving “conversational orientation”, cycling round in ever decreasing circles until this was achieved. With this effort expended, the users exchanged little more than a few perfunctory greetings, and then left the installation. Our expectations of multi-user cycling tours of the cityscape were far from met. With hindsight, the problem was one of a mismatch between the affordances of the environment, and the expectations of the users. First, the main novelty of the environment to its user turned out to be the potential to find another inhabitant, and perhaps then to tour the world.
30
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
Flashing arrows on the overlaid map made it clear that other users were around somewhere, and once users realised that the “buildings” did not block their attempt to cycle through them, the map became a meaningless diversion, which was not returned to as a useful navigation aid later (in spite of its apparent successful application in the original piece). Similarly, on a real bike one would expect to be able to cycle side by side with a friend, or at least to look over one’s shoulder to carry out a conversation with a cyclist behind. The fixed position of the screen in front of the cyclist, and the difficulty of arranging conversational orientation with these limitations, made casual touring and chatting unrealistic. To solve these problems, two simple modifications were made to the installation. First, the map was discarded (the alternative of forcing the utility of the map by making the buildings impenetrable seemed unproductive since the cities were large and users simply wanted to find one another in the first instance). As a replacement, a bird was positioned in the sky above each participant, which would fly in the direction of the nearest other user, meaning that now finding one another was straightforward: simply follow the bird. The most significant alteration was the introduction of a tracked head mounted display. The original fixed monitor was retained so that bystanders could see interactions within the environment; however, the cyclist was now able to look around in the environment, and to see the world in a different direction from that in which the cycle was facing. The improvement to social interaction with the new version of the DLC, this time tested out at ESPRIT IST98 in Vienna, was dramatic. With the ease of finding other users and of achieving a convenient orientation for comfortable communication, participants were much happier to spend extended periods of time exploring the cities together. The conclusion at this stage was that cityscapes provide easily understood metaphors for interaction, but these metaphors must be matched with the actual affordances of the environment and its input devices with great care.
3.2.3 Finding “Something to Do” For all the lessons learned from the Distributed Legible City, it was far from being an e-scape: it lacked interaction with other environments, and had no “information” as such beyond the possible interest in the cities’ texts. A more concrete cityscape was built to address these limitations. An ethnographic study of a real-world tourist information centre, in the northern England seaside town of Morecambe was carried out, concluding that it was an interesting enough location of social interaction and information browsing to warrant further attention. In particular the question of “What can we do here?” seemed as relevant to a tourist turning up in a new holiday location as it did to an inhabitant of a new electronic landscape. A virtual Tourist Information Centre (TIC)
31
Inhabited Information Spaces
was constructed to see in what way the behaviours observed in the real world TIC were transferred to the virtual landscape, and studies of this new environment were carried out in situ. The virtual TIC (Figure 3.4) was based on a map of Morecambe, and included 3D representations of the various tourist attractions. The map and its contents distorted dynamically to bring objects or areas of interest to a particular user into view, according to search criteria. Once more, users tended to follow city-like conventions in their exploration of the virtual world, and there was evidence that to some extent the environment supported social interaction and exploration. Its limitation, however, was clearly one of content. The hand crafted nature of the virtual Morecambe and the relatively small number of attractions represented within the world meant that its real potential as a source of holiday ideas was nowhere near reached. More importantly, it was also obvious that it was unrealistically costly for any individual to attempt to populate such a world. From this it was concluded that an important aspect of any worthwhile e-scape was the ability for its inhabitants and stakeholders to straightforwardly introduce their own content and to modify existing artefacts to reflect their own experience.
Figure 3.4 A virtual Tourist Information Centre, showing the deformable map and buildings.
32
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
3.2.4 Abstract Influences: Nuzzle Afar In parallel with the thread that has been described so far, informed very much by the “concrete” and “physical” end of the e-scape spectrum, a number of other studies and sub-projects were being explored under the eSCAPE banner. One of these in particular: a multimedia installation of an abstract nature called Nuzzle Afar had a significant influence of the nature of navigation and wayfinding in PlaceWorld. Nuzzle Afar, a multimedia installation by Masaki Fujihata, consisted of an enclosed room, with left and right side entrances. Within the room were two podia, in front of which were projection screens. Trackballs embedded on top of the podia allowed users to control movement through the computer-generated environment displayed on the projection screens, and microphones similarly located allowed distributed users to communicate. As users moved through the electronic environment, a string-like trace is left upon the virtual landscape, which may be locked onto and followed by another user. The enclosed space of the virtual world consisted of four walls, ground and sky plane. Upon the walls were images of “sense organs” (e.g. a hand, an eye, an ear, etc.). In addition to spherical “avatars” of unique colour, a sphere and a cylinder are placed within the virtual room. These latter objects were the means by which users could enter or depart a series of three rooms. Within each of the spaces in Nuzzle Afar navigation needs to be learnt anew: effort, space and travel have different relationships within each. When two or more users are in close approximation “within” any of the spaces, they are able to see a video image of one another mapped and wrapped, visor-like, around the middle of the sphere. This allows for recognition of the others’ identity. In meeting each other, any two users are able to enter a new virtual space which encompasses them, while locking out the previous environment and any other inhabitants. This new space is, however, visible to other users as a spherical object inside of which the colours of the two users “inside” merge. Once inside this new space, users are represented via their video images on a 2D square. When one or both of the users leave this space, a video still of the two users remains, along with details of the time and physical locations of the encounter. In spite of the very abstract nature of the Nuzzle Afar world, its means of wayfinding and of locating other users proved much more intuitive than our initial experiments with maps and other metaphors borrowed from the cityscape metaphor. The influence of this installation on the final implementation of PlaceWorld is described in the following sections.
3.3 PlaceWorld PlaceWorld became the final eSCAPE demonstrator and assimilated much of the work done in the earlier parts of the project. It aimed to
33
Inhabited Information Spaces
become an effective “place where places meet” building on the lessons learnt from studies of the smaller scale landscapes. Another multimedia installation piece by Jeffrey Shaw formed the basis for its artistic design, and a novel user interface device was commissioned for the purpose of public display of the new environment. The next sections describe the vision and rationale that underpinned the world.
3.3.1 The Design of PlaceWorld The design of PlaceWorld took as its point of departure many features of Jeffrey Shaw’s 1995 artistic work “PLACE – A User’s Manual” (Shaw, 1997). The paradigm that is embodied in both implementations is that of a navigable virtual environment that is populated by cylindrical artefacts that form portals into other virtual environments. The world is also characterised by pathways that provide identifiable routes between these places. As boundary architecture, cylindrical space is quite different from more typical rectangular spaces: the latter is perceived as an enclosure, whereas the former constitutes what is in effect a scaleable panoramic horizon. This aspect of the virtual cylinder is extended into the real world by the panoramic navigator interaction device described later. Cylindrical spaces are especially appropriate within PlaceWorld because they allow all the idiosyncratic virtual environments to define their own unique enclosures within a neutral and scaleable circumference. Despite the apparent exterior uniformity of these cylinders, attaching images to their surfaces offers an expressive way to signal their interior contents. In this manner, rather than by architectural variation, they constitute a characteristic and very expressive visual syntax for PlaceWorld. This cylindrical uniformity does not preclude the possibility to scale them variously in height and width, or to work with the relative transparency of their surface, and this allows further expression of spatial as well as temporal meanings as will be described later. From this initial vision, a graphic designer developed a design specification that took the earlier PLACE work, and extended it with new features that addressed the particular needs set by the objectives of an electronic landscape. One important aspect was the fact that an everyday urban landscape is characterised by the presence or absence of its population of citizens. PlaceWorld also socialises its unusual landscape with representations of its visitors and occupants, but in accord with the inherent properties of this information space, these avatars have unique design characteristics that enable them to effectively function as true citizens of PlaceWorld. A visitor in PlaceWorld would be expected to orient themselves to a large extent by the visual identities of the cylinders themselves – that is, by the individual images that would be textured on the outer surfaces of
34
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
these cylinders (a form of advertising their interior contents) and the general stability of the layout. Much as in the real world, it seemed useful to also establish some method of signposting that could offer and guide visitors to the various locations. Such signposts could also be set up by persons who had built a place in PlaceWorld as pointers to their location. The issue of the relative permanence of the overall geography of PlaceWorld and its cylindrically bounded environments was felt to be an important issue if regular visitors to PlaceWorld are to be expected to readily gain familiarity with it. On the other hand, the ability to introduce new content to an environment means that some controlled modification of the world must be possible, as was made clear by our experiments with the virtual TIC. A description of the access model – a set of malleable guidelines and policies that could develop over time under the influence of the inhabitants of the world – was developed to support coherent and manageable changes to the underlying structure. A brief description of the access model is given later in this chapter. The issue of balancing familiar structure with the desire for modifiable content formed a fundamental feature of the temporal nature of PlaceWorld’s landscape, and reflects once more the desire to combine appropriate aspects of the concrete and the abstract. Masaki Fujihata’s project Nuzzle Afar provided an exemplary demonstration of the value and effectiveness of attaching traces to users’ movements through a virtual environment. Colour coded for each visitor, these lines gave temporal evidence of people’s presence and their paths of interest. And the added feature that a visitor in such an environment could “hook on” to someone else’s trace and then get a roller coaster ride along that convoluted line that brought him face to face with that other person was also very effective and enjoyable. However, further discussion of this methodology revealed a basic weakness – as soon as there would be a large number of visitors in the virtual environment these lines would become an unintelligible tangle. Because we had decided to restrict movement to ground level in PlaceWorld, such traces could then only be drawn in one plane, thus adding to the confusion if there were many visitors. On the other hand, the restriction of these markings to one plane revealed the possibility of another very interesting design strategy. We realised that these traces could in fact be merged with the geographical and physical articulation of roadways in PlaceWorld. These roadways could be programmed to dynamically and temporally express the volume of their usage, being a virtual reflection of “desire lines” in the real world that are created, say, when pedestrians create their own shortcuts across grassy areas in parks. The hierarchy could be say trail, path, road, highway, which would be defined by the density of traffic experienced along these routes, and would change according to the varying density. In this way a new feature of informative comprehensibility could be added to PlaceWorld: the nature of the pathways would themselves express a history of visitor interests, so one could, for instance, choose
35
Inhabited Information Spaces
the well-trodden paths or venture along trails into relatively unfrequented regions, or forge a new trail into previously unexplored territory. Following this line of thought about the informative value of temporal signifiers in PlaceWorld, the notion was extended to the cylinders themselves, which expressed the temporal condition and/or the popularity of their contents. As with the tourist information centre, cylinders that are of interest or frequently visited were designed to increase in scale over time, with those of little interest shrinking into obscurity. A point of debate was the notion of hyperlinking the individual cylindrically bounded environments so that a visitor could “teleport” instantly from one related place to another without having to negotiate the PlaceWorld landscape. A more radical extension of this facility would be that a visitor could at will completely reorganise the distribution of cylinders in PlaceWorld to suit a specified personal need or desire (e.g. he only wants to visit the popular places, so these all become clustered around him and the others disappear). However, going back to the premise of PlaceWorld as a physical electronic landscape that should be strongly anchored in real world physics and experiences, it was felt that such hyperlinks and hyper-rearrangements should be kept to a minimum, since they were not entirely consistent with the “physical landscape” objectives set for this environment and shown to be effective in the previous e-scapes. As a compromise, the ability for a user to generate a hyperlink between places that they had already visited was made available, with the link being visible only to that user as a glowing trail in the sky, anchored at both ends on the groundplane.
3.3.2 The User Interface and Presentation System The Panoramic Navigator (PN) is a patented technology developed by Jeffrey Shaw and the ZKM whose initial function was to provide a panoramic method of allowing visitors to the ZKM to preview and explore the contents of the ZKM building. Using a 360-degree rotatable touch screen coupled to a video camera, this augmented reality approach enables visitors to the ZKM to look around the building (via the live camera image) and use the touch screen to access additional multimedia information that is attached to specific locations in that building. While these past implementations of the PN have used the real environment as its frame of reference for interactive information delivery, it was realised that this technology could also be an ideal interface to explore and interact with wholly virtual environments, and that this would be eminently suitable for PlaceWorld. What was needed was a method of generating a representation of that virtual environment around the PN, so that the touchscreen could then be used to interact with it. The design solution for this was to attach a video projector to the back of the PN (behind the touchscreen) and to put a circular projection screen
36
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
011
011
011
011
11
Figure 3.5 The Panoramic Navigator – Augmented Virtuality version, in use with PlaceWorld.
around the PN. In this way a new presentation and interaction method which embodies the innovative concept of augmented virtuality (PN-AV) was developed. This PN-AV technology is used in PlaceWorld to embody two distinct but interrelated functions: ●
●
to display the 360-degree immersive representation of virtual environment, wherein one also see the effects of user interaction with that environment. to give the user a touchable interface that allows him/her to explore and interact with that surrounding virtual environment by showing a parallel representation of that environment which is augmented by a dynamic set of user interaction tools.
The final version of the PN-AV, in use during a public demonstration, is shown in Figure 3.5. Having described the evolution of the PlaceWorld environment, we now turn our attention to the problem of its actual implementation, and describe the development of the DEVA VR system, which grew in parallel with the e-scape demonstrators. We will return to how aspects of PlaceWorld were supported by this technology at the end of this chapter.
3.4 Technological Challenges for Electronic Landscapes Until recently, technological limitations were the clear brake on progress for VEs (the graphics challenge is perhaps the clearest example). Beyond the technology the key task is that of writing software. The scale and complexity of this task is often underestimated, and it is here where we believe that the major problems lie. We do not at this time have adequate frameworks to simplify the task of implementing virtual worlds. Today, a person wishing to implement a challenging VE application has two broad options, as evidenced in current demonstrations. The most
37
Inhabited Information Spaces
sophisticated VEs are usually bespoke applications constructed from the graphics layer upwards; this is a substantial undertaking. The alternative is to use the VE system. These fall into different categories, (for example, VRML browsers at the lower end, up to VE systems such as DIVE and MASSIVE at the more ambitious extreme). The software support challenge to facilitate large-scale VE applications is twofold. First, to find techniques and algorithms to address specific needs in VEs. Collision detection, parallelism, distribution, synchronisation, navigation and so forth all require work of this kind. Secondly, and perhaps rather harder, it is to find frameworks that allow all the parts to be put together in “flexible yet powerful” ways. The rather trite nature of such a statement belies the difficulty of quantifying that task. Finding the “right” framework is particularly difficult in the case of virtual reality (VR), since it brings together a number of complex technical issues, and binds them with real-time constraints at the social/ perceptual interface. A desirable approach for the necessary flexibility to experiment is to build as little into the system as possible, so the system provides a set of mechanisms and policies and default behaviours that can be unplugged and tailored at each level. The issue of scale is an important one; simple small-scale VEs that do not challenge today’s hardware can be constructed by any number of means. Building large-scale complex applications raises a number of challenges. For shared VEs these relate, broadly speaking, to the following areas: number of entities; complexity of behaviour required of these entities; complexity of individual rendering techniques; number and geographical distribution of simultaneous users; and number of co-existing and interacting applications. Matters of scalability and synchronisation, and some architectural and network topologies for achieving this are discussed in Chapter 13. Here we present the approach implemented in the DEVA VR system, and arising from the observation that for a shared VE there is a natural distinction between the users’ perceptions of the world and what is actually “going on” within it. That is, between the distribution and simulation of application behaviours on the one hand, and the task of presenting a coherent perception of the VE to each user on the other. This is similar to the traditional philosophical distinction between subjective perception and the underlying objective reality. The reason that this is useful is that in VR it separates out the challenging graphics/interaction tasks from the semantics of the underlying world simulation. The distinction also legitimises efforts to make a perceptually smooth presentation of the world in the light of fundamental networking limitations. Architecturally we have addressed these two aspects of the VE (perception/reality) separately. First, rendering and spatial management seem to need special treatment that is different from current approaches to graphics toolkits. The MAVERIK system aims to address these issues. The unusual approach taken by the MAVERIK system is to avoid having any internal
38
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
“structure” for the representation of VEs and their contents. Instead, an object-orientated framework is provided that supports an application builder in implementing rendering, interaction and spatial management routines that are tailor made and appropriate to their particular purpose. This is as much a performance issue as programming elegance, for key graphics optimisations are highly application specific, and are generally unavailable when the application must export its representation into the VR system. The system is described in detail in Hubbold et al. (2001). The more difficult problem is that of defining the underlying behaviour of entities in a way that can be distributed to multiple users. It is to this problem that we now turn our attention.
3.4.1 Synchronising the Behaviour of Entities The distribution of semantics or behaviour in shared VEs is a particularly difficult issue. The ideal solution would be to describe the required behaviour in a single location and to have it instantaneously sent to all participants (a pure “client/server” approach). Current network technology is far too limited to make this solution feasible. An alternative is to replicate behaviour locally for each participant (a pure “peer-to-peer” approach); this introduces extreme synchronisation problems. The approach taken in DEVA is a hybrid solution, part way between client/server and peer-to-peer. The model is one of an objective and subjective reality; the former being located (logically) on the server, the latter being represented on each client. Each user interacts with the objective reality via their own subjective view. This introduces the idea that each entity in the virtual environment has two definitions, which may differ significantly in their semantics. Our usage of the term “subjective” is not intended to imply that each user experiences an entirely different VE (this would hardly count as a shared experience). Rather, we argue that an amount of the user’s experience may be decoupled from the “objective” behaviour of the world without disturbing consequences. Given that absolute synchronisation is in any case impossible, it is our contention that so long as the length of the delay is not too large, and causal events occur in the correct order, it is possible to accept a degree of subjectivity without it affecting the users’ understanding of the semantics of the application. Under these circumstances, given no other frame of reference the users are unlikely to even realise they are not receiving the same view of the world as each other (as will be described later, PlaceWorld demonstrated this phenomenon in extreme ways). In general it would therefore be useful to differentiate between what is “actually” happening in a VE and what the users perceive. In this way it is possible to optimise the manner in which information is transferred to minimise causal discrepancies. In the DEVA system this notion is implemented by describing an entity as comprising of a single “object” and a number of “subjects”, the former
39
Inhabited Information Spaces
part being the “objective non perceived” aspect of the entity and the latter being a collection of the “subjective perceivable” characteristics (e.g. its visible, audible, haptic parts). An alternative view of this distinction is that the “object” represents what an entity does, while its “subjects” represent what the entity looks/sounds/feels like when it is doing it. Communication between an object and its subjects can be implemented using whatever high-level “vocabulary” is appropriate. In this way the system minimises the need for strict synchronisation while maximising the accuracy of causally important events. One example of this is the use of twines (Marsh et al., 1999). Twines are lightweight parametric curves that are used to “smooth out” spatially and temporally the discrete updates to the visible aspects (e.g. position and rotation) of appropriate entities by interpolation thus increasing their “plausibility” by eliminating disconcerting jumps. We are also currently investigating an infrastructure for providing “quality of service” based on balancing such smoothing against frequency of updates depending on the perceived importance of various events. The flexible nature of the communications is also advantageous for subject-to-object communications; for example when a user “grabs” and moves an entity. Such manipulation with a classic client–server architecture would involve a round trip: the event is sent from the client to the server to be processed and then the effects of the event are then transmitted back to the client to be visualised. With the commonly available network technologies we are targeting for the client–server connections such a round trip would introduce an unacceptable lag. It is perceptually important for the cause and effect of the manipulation to be as tightly coupled as possible. With the separation of subject and object, changes caused by the manipulation can be immediately perceived by the subject, with a userdefinable policy function being used to determine when the object is updated. For example, an arbitrary fraction of the changes (say, one every fifth of a second) are transmitted back from the subject to the object and thus onto other visualisers connected to the server. Alternatively, a change in position is only updated to the object when the subject has moved a certain distance from its previously synchronised location. The goal is to ensure that the entity behaves “correctly/optimally” on the visualiser performing the manipulation and also that it behaves “plausibly/acceptably” on any other visualiser connected to the cluster (while accepting that latencies and bandwidth issues rule out strict and absolute synchronisation in such a distributed system).
3.4.2 Distribution and Communications The programming model employed by the DEVA system is one of communicating “entities” which can represent objects in the virtual
40
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
environment, the properties of the environment itself, or alternatively abstract programming concepts that have no direct representation to the inhabitant of the environment. These entities are coarse-grained programming objects, exporting a number of methods that can be called by other entities, and implementing these internally using optimised imperative code (currently written in C++). The DEVA programming model makes entity behaviour explicit, allowing entities to query one another’s features, those of the environment in which they exist, and to reason about these, rather than having behaviour emerge as an implicit side effect of a piece of code being executed. In this section we describe the makeup of an entity, and the mechanisms provided for enabling transparent and lightweight communication between entities distributed around the system.
3.4.3 Defining the Behaviour of Entities We can identify at least four conceptual sources of behaviour for each individual entity within a VE: 1. Behaviour unique to an entity. Entities each have their own particular role within a VE, for example the ability of a stopwatch to record elapsed time. 2. Behaviour common to a range of entities. Often many entities share some aspects of their behaviour through being of a similar type. For example, each pawn on a chessboard is subject to the same detailed rules governing movement; these rules do not apply in the same way to other pieces which have their own restrictions. However, all the pieces have in common the rules determining whether they may be “captured”; a notion that is specific to chess pieces but not necessarily to other entities in the same environment. Thus a hierarchy of common groups can be determined. 3. Behaviour common to all entities in a particular world. We generally consider gravity to be a phenomenon associated with everything found in the world around us rather than as a property of each individual entity. This also applies to social constructs such as “monetary value”. 4. Behaviour that is dynamically required at runtime. If an entity becomes inhabited by a user for example, it will behave differently – being controlled by a navigation device and so forth – to when uninhabited. Similarly an entity that is “set alight” will suddenly have properties previously unavailable to it (for example, being able to set fire to neighbouring entities or raising the temperature of an environment). Commonly, VR systems define the behaviour of an entity by attaching pieces of code, often written in Java, Tcl or some other scripting language. With little specific architectural support, however, it is often laborious to code and difficult to ensure consistency. The DEVA
41
Inhabited Information Spaces
programming model attempts to improve the situation by taking an object-orientated approach to the definition of behaviour and by providing features such as introspection (the ability to ask an object what methods it has, what parameters they take etc.) and run-time combination of environmental influences with entity-specific behaviour. This essentially allows the traditional inheritance graph to be modified while an entity is “live” in an environment, adding or removing properties and behavioural traits as necessary. Our solution to the problem of merging the various sources of behaviour that comprise an entity is to use “characteristics”. In DEVA a characteristic is a collection of methods and attributes relating to a single concept that can be attached or detached from an entity at runtime. DEVA supports three types of characteristic: 1. Innate: behaviours that define an entity and make it different to those around it, for example its physical shape. 2. Imbued: behaviours that are offered to an entity when it joins an environment, but can be over-ridden by that entity by a more specific innate behaviour (for example, an entity lacking mass in an environment that requires such an attribute, may inherit an imbued characteristic that approximates a default mass from its bounding volume; an entity that has a better idea of what its mass is can provide the functionality itself). 3. Enforced: behaviours that must be processed in order for an entity to conform to an environment’s requirements. The methods and attributes that comprise the characteristic are divided into a single “object” and multiple “subjects” as outlined in the previous section. While typically a characteristic will contain both parts, some characteristics are entirely abstract and have no directly perceivable representation in the virtual environment, that is, they have no “subject” part. The researcher trying out new low-level ideas is free to write characteristics directly in C or C++ that interface to the VR kernel at whatever level is appropriate. More general users are free to use the library of existing characteristics to construct entities without concerning themselves with their implementation. We speculate also that this characteristic system has the added benefit of being usable from an immersive “graphical user interface” allowing the user to “mix and match” characteristics at runtime. In order to facilitate efficiently moving entities between worlds with different behaviours, they contain two lists of characteristics, one inherited from the environment, and the other containing its own innate behaviour. Traditional object-oriented inheritance is supported through characteristics being able to load other characteristics when they are initialised. The order in which the characteristics are searched enforces the correct precedence.
42
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
Income method call
Deva Entity
011
011
Characteristic 1
method 1 method 2 method 3
method 1 method 2 method 3
Characteristic 2
Characteristic 2
Characteristic 2
method 1 method 2 method 3
method 1 method 2 method 3
method 1 method 2 method 3
Characteristic 3
Characteristic 3
Characteristic 3
method 1 method 2 method 3
method 1 method 2 method 3
Imbued
Characteristic 1
method 1 method 2 method 3
Innate
Enforced
011
Characteristic 1
method 1 method 2 method 3
Optional return valve
Figure 3.6 Structure of a DEVA entity, showing the ordering of method processing.
When a method is called upon an entity, the two lists are searched in strict order. First of all the list of characteristics given to the entity by the environment is scanned for methods marked as being “enforced”. If one with the correct name is found, then this is called. Enforcing methods allows an environment to ensure all entities contain a particular method that cannot be overridden. Next, the entity’s innate characteristics are searched. If the method is still not found then the environment characteristics are searched again for methods marked as “imbued”. These are methods given to the entity by the environment but which can then be overridden by the entity itself. For example, an environment may enforce the notion of “solidity” upon all its entities (“you can’t pass through walls”). It may also “imbue” all entities with a concept of mass with a default value estimated based on its volume, but which the entity is free to override should it prefer. This process is illustrated in Figure 3.6.
011
11
3.4.4 Methods and Filters The strict order in which characteristics get searched for methods leads to another useful concept, that of the filter. As well as methods just returning results, they can be declared to be filters. These are allowed to return a new method call as their result, which then continues to
43
Inhabited Information Spaces
propagate along the characteristic lists. This is useful in a number of situations. Say, for example, it was necessary to constrain an entity to a plane. The entity inherits the imbued method setXYZ from its ThreeDSpace environment. In this case the entity would define a characteristic itself that defines setXYZ as a filter. Since this would be found before the imbued method, the entity’s filter is able to constrain the coordinates to a given plane before returning a new setXYZ message with the new co-ordinates that then propagates through to the environment imbued setXYZ method. Another example would be storing an entity’s state prior to migrating the entity to another server node process or prior to a complete shutdown of the system. Each characteristic defines a snapshot filter that adds any state variables it possesses to the input message and returns a new snapshot message. The final characteristic then returns a restore message containing all the variables that is sent to the new entity. The new entity contains a restore filter in each characteristic that originally stored variables, which then unpacks the message. Continuous behaviour is supported by permitting characteristics to define a special method called “activity” which is polled intermittently. Activities are not subject to the usual calling precedence, but always polled if defined.
3.4.5 The Distribution Architecture DEVA is logically a client–server architecture, which to a first approximation provides a single definitive locus of control for the VE using its server component, with “mirrors” of the entities being maintained in each client process. Behind the scenes, however, DEVA pragmatically manages the delegation of control dynamically to the most appropriate parts of the system, thereby achieving the highest fidelity of perceptual and causal coherency attainable for the application at hand. The “server” is in fact a cluster of processors running identical processes called “server nodes” that together form a single multithreaded parallel virtual machine capable of processing large numbers of entities. The intention is that the server provides a computing resource for multiple virtual environments, and maintains a far heavier processing load than any one user’s client could manage at any one time. A networking layer provides lightweight position independent messaging between entities. Entities are created in and managed by the server node processes, and client processes – such as a visualiser or user application – connect to the server to interact with and obtain state information about the entities. The server is persistent: it remains alive, processing any entities regardless of whether or not any clients are connected. Administrative tools exist to simplify the start-up, monitoring and shut down of the parallel server.
44
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
Creating and Addressing Entities Each server consists of a (large) fixed number, M, of virtual servers. The M virtual servers are trivially mapped using a lookup table to the N server node processes that comprise the server. This conversion takes place at a low level within the system and essentially hides the configuration of the server. An arbitrary virtual server is chosen to create and manage each entity (currently a random virtual server is chosen but this selection process could take into account loading factors). Each server node potentially manages multiple entities. Each entity is assigned a unique “pool ID” – an offset into the list of entities managed by a given server node. The location of an entity in the server is uniquely defined by the virtual server and pool IDs. This pool ID is not strictly necessary and is provided for efficiency only, since each entity has a unique name that can be searched for in the list of entities managed by the server node. When an entity is created a hash function is applied to the entity’s name to obtain a second virtual server. This virtual server manages the name of that entity, that is, it definitively knows the virtual server of where the entity is actually located and its pool ID. The name of the entity and the entity itself are managed by separate virtual servers. The same hash function is used throughout the system allowing any DEVA process to obtain the location of a named entity. This data is cached for future use. Entity name to location lookup, while a lightweight process, is a central and frequent task in a distributed system. A scheme where this load is spread equally across all server nodes is therefore advantageous. The main advantage of the addressing mechanism is that it allows entities to dynamically migrate across server nodes to help balance the processing load. When an entity moves it only needs to inform its name manager of its new location. DEVA processes – which now contain out of date cached data – will receive an error the next time they try to communicate with the entity at its old location. This error is trapped internally and the new location of the entity is obtained from the name manager; the originator of the communications with the entity is oblivious to the migration. The name manager can always be relied upon to know the correct information and its location is trivially obtained.
Server Reconfiguration It is also possible to migrate the names managed by a given virtual server onto a different server node by updating the virtual server-to-server node lookup tables in every client and server process. The migration of both entities and name management allows server nodes to be dynamically added and removed from a running server.
45
Inhabited Information Spaces
Networking Protocol Currently, standard TCP/IP point-to-point socket communication is employed although some work has been undertaken investigating multicast, since for local area networks at least it promises improvements in performance for our application. Although not a strict requirement, the communications strategy is based upon the assumption that inter-server node communication is fast compared to server–client communication. For example, the server nodes are connected via a dedicated network or protected to some extent from superfluous traffic by a bridge; while clients connect to the servers via a high-traffic shared LAN (local area network) or modem connection.
3.5 System Support for PlaceWorld In this final section, we describe how a number of the artefacts and features of PlaceWorld, informed by the previous multimedia installations and ethnographic studies, were supported by the technology of the DEVA system.
3.5.1 Menus Every entity in PlaceWorld is able to respond to the user in one way or another. When the user touches the entity on the navigator screen, a menu appears showing the available features. Most entities, for example, can be picked up and put in the user’s “pocket”, and in this way moved to another part of the same world, or indeed somewhere completely different. Even entities that were coded before PlaceWorld was conceived have this feature “enforced” upon them by PlaceWorld’s master environment giving a degree of coherence to the interface. Other entities hand-crafted for the world, such as the generators or noticeboards offer more sophisticated behaviour via the “innate” mechanism. Filters are used to cascade through the possible methods provided by each entity, combining the imbued, innate and enforced facilities available to the user and presenting these graphically at the interface.
3.5.2 Access Model PlaceWorld’s access model demonstrates a more sophisticated use of the characteristic architecture. In order to maintain a degree of order in the environment, users are able to specify rules that govern the use of entities they have placed in the world. The mechanism for describing these rules is beyond the scope of this chapter, however the method of
46
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
enforcing them can easily be explained. The PlaceWorld master environment enforces on its contents (which includes all its sub-environments and their contents) an “authenticate” filter that responds to all possible method calls. This effectively intercepts communication of any kind arriving at any entity in the world from any source. The authenticate filter examines the incoming method call, and matches the credentials of the caller against whatever rules are currently being requested by the individual entity via an innate method higher up the call chain. If the attempted invocation is valid according to these rules, the filter “releases” the original method call to cascade through the remainder of that entity’s methods, with the caller effectively unaware of the validation process that has occurred. Methods that are not authenticated are returned to the caller, which in turn has a method imbued by the environment to cope with such “return to sender” results. As before, entities that were coded without any knowledge of the access model automatically pick up a minimum amount of functionality to enable them to interact with the access model in a suitable manner; no extra effort on the part of the implementer is necessary, while those entities aware of their surroundings are able to easily extend the rule base in bespoke ways.
011
011
011
11
3.5.3 Exploiting Subjectivity PlaceWorld has comparatively relaxed synchronisation requirements. Interaction takes place mostly at a social level, and there is currently no detailed manipulation of shared entities in real-time such as might be necessary in a collaborative CAD package. The subject/object distinction embodied by DEVA allows us to take advantage of this by programming quite relaxed synchronisation routines for these entities. Though it is necessary to synchronise, say, two users attempting to pick up the same entity at the same time, or to reflect the change of text on a noticeboard to all users in a timely fashion, it is not necessary to synchronise the gentle floating effect of the PlaceWorld artefacts as they hover above the groundplane, nor, as another example, the swaying of the glowing hyperlinks in the sky. The public exhibition of PlaceWorld at the I3Net conference in Sweden and at the Sixth Doors of Perception conference in Amsterdam demonstrated that the relaxation of synchronisation and the use of subjectivity could be pushed further than we originally expected. Due to a technical problem with the graphics drivers at the time, it became necessary to use an extra rendering client to generate the display on the touch sensitive panel of the PN-AV device, rather than using the camera as was originally intended: the projected display on the external cylinder showed the user’s view of the world, and the display on the touch screen showed the same view, but augmented with the controls and menus etc. The idea of relaxed subjective behaviour had always been intended to overcome
47
Inhabited Information Spaces
network lag where users were geographically distant, and unable to see each other’s views into the environment: here, unusually, the same user was able to see one view on the touch screen in front of them, and “the same” view on the projected cylinder. The potential power of the subjective set-up was demonstrated when not a single user of the installation noticed that there were significant variations between the two displays that they could now see simultaneously: where their attention was focused (changes of text, avatars moving around the environment, entities coming and going), all appeared to behave normally. The aesthetics of the floating entities, which were in fact moving independently of one another with no attempt to synchronise the views, went unnoticed. More dramatically, one of the artefacts, a version of ‘QSPACE’ (Pettifer et al., 2001), is a graphical network much in the same vein as the Web Planetarium described in Chapter 2. To preserve bandwidth the distributed version of this artefact in DEVA was implemented with the force-placement routines as subjective behaviour, and the weights and connectivities that define the structure as objective and synchronised behaviour. What was overlooked during the implementation was that in the force placement routines, a small random factor was introduced to the positions of the nodes prior to application of the algorithm, as this seemed to produce more aesthetically pleasing results. The implication of this, however, was that although the logical and gross structure of the resulting 3D network was the same for each run of the algorithm, these small perturbations in the initial state of the network resulted in slightly different placement solutions. For the several weeks of implementation, and during the public demonstration of PlaceWorld, this went entirely unnoticed, even though, with hindsight, it was clear that the various visual representations of the QSPACE being manipulated by the collaborating users were very different. The fact that the logical structure of the artefact was coherent across views appeared to mask the visual differences in orientation and absolute positioning of the nodes.
3.5.4 Becoming a Place Where Places Meet PlaceWorld aimed to bring together many diverse virtual environments, not just trivially by allowing a user to “switch” from one application to another, but by making it possible for artefacts created in one place to be used elsewhere, analogous to cutting and pasting a spreadsheet into a word processor document. It was decided to integrate a number of existing virtual environments, including other multimedia art installations, into the PlaceWorld system. Artefacts and applications that are written with the DEVA framework in mind have this facility by their very nature; more challenging, however, was the problem of making legacy applications, such as the Legible City, or the Memory Theatre VR (an installation by Agnes Hegedüs) work within this
48
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
framework (this last application, in particular, is interesting since it was written using Silicon Graphics Performer, rather than raw OpenGL, and therefore was already part of its own “system”). Figure 3.7 shows this and other environments embedded within PlaceWorld. MAVERIK’s (Hubbold et al., 2001) agnostic approach to data structures made this integration at the graphical level relatively straightforward, even to the extent of enabling the low-level graphics context to be shared between Performer and native MAVERIK objects, and DEVA’ s ability to combine behaviour at runtime based on enforced and imbued characteristics allowed legacy applications such as the Memory Theatre VR to pick up appropriate interaction features such as the menus and the ability to be carried around by a user.
011
011 a
b
c
d
011
11
Figure 3.7 Images from PlaceWorld and its embedded environments: (a) the QSPACE; (b) the Legible City; (c) Memory Theatre VR; and (d) the Advanced Interfaces Group’s laboratory.
49
Inhabited Information Spaces
3.6 Conclusions The vision of an electronic landscape remains an exciting one, and as graphics technology and higher performance networking become more affordable and available, there are increasing opportunities for the kind of research and development described in this chapter. The iterative approach of the eSCAPE project has been an effective way of understanding the social, aesthetic and technological problems associated with making these worlds meaningful and useful to the general public.
Acknowledgements PlaceWorld was influenced in many ways by all the partners and participants in the eSCAPE project: too numerous to mention, this chapter draws upon much of their work and our thanks go to them all. Special thanks are due to Paul Arnold, Annika Blunck, John Bowers, Andy Colebourne, Andy Crabtree, Timo Fleish, Simon Gibson, John Hughes, Andreas Kratky, John Mariani, Gideon May, Craig Murray, Andreas Schiffler, Jeffrey Shaw, Adrian West, and the project’s co-ordinator, Tom Rodden. The work was supported by a grant from the European Commission.
50
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Using a Pond Metaphor for Information Visualisation and Exploration
4
Olov Ståhl and Anders Wallberg
4.1 Introduction The constantly increasing amount of information and media available in electronic form leads to a growing demand for new methods for searching and browsing. Traditional text-based database queries can be limiting, often requiring a user to know exactly what it is she is looking for and to express this interest using predicates in a query language such as SQL. Furthermore, to be successful in locating the right information, a user will often have to be familiar with the standard interface metaphor of desktop computers and know how to use a mouse and keyboard efficiently. Approaches to improve access to online information and to visualise it in an intuitive manner have been under development for a long time. Examples of systems that display information graphically in three dimensions include VR-VIBE (Benford et al., 1995a), QPIT (Colebourne et al., 1996) and BEAD (Chalmers, 1993). All of these systems require (at least to some extent) the users to use navigation to access particular information objects, because objects may obscure each other or may be out of view. However, navigation within 3D spaces is known to be difficult (Ingram and Benford, 1995), especially if the navigation needs to be precise. In this chapter we describe The Pond, a system used to search for and visualise data elements on an engaging tabletop display. The Pond uses methods of unencumbered interaction and audio feedback to allow users to investigate data elements, and supports shoulder-to-shoulder collaboration with the physical Pond artefact mediating the collaboration between those people gathered around it. The user interface is based on an ecosystem metaphor, presenting data elements in the form of shoals of aquatic creatures inside a virtual 3D pond. The design makes use of a
51
Inhabited Information Spaces
static view of the information space, making viewpoint navigation unnecessary. Instead the information creatures move and form groups, allowing the user to easily identify related information and to distinguish results from different queries. The work draws heavily on our experiences in developing two previous systems, and concepts and approaches can be traced back to these systems. The Web Planetarium, a 3D interpretation of HTML structures in the form of a graph (Chapter 2), and the Library Demonstrator, an information landscape constructed from the contents of an online library database (Mariani and Rodden, 1999), were both interfaces to active online information. Common features of these systems were the spatial arrangement of data elements, navigation around these data elements and the introduction of new data elements into the display. While these systems were successful in presenting information to end users, they were not necessarily easy to use. When using the Web Planetarium, new information was only introduced as a result of explicit interaction (clicking) by the user, and novice users could be too shy to discover this fact and not load new information. The positioning of this new data could also be problematic. In the Planetarium the user must be observant otherwise she may miss the introduction of the new data element. The Library Demonstrator addressed this problem using animation and self-organising models to show the emerging relationships between information, and it is this approach we build upon in The Pond with its shoal metaphor. Navigation around the data used a point and click method, automatically transporting the user to (or close to) the selected object. The core of this navigation technique was to adopt an object centric philosophy where users were explicitly freed from the overhead of having to manage their navigation and movement in three dimensions. This restriction of the overall freedom of movement meant that users were able to focus on the exploration of the information space. However, users still had problems navigating the structures. It could range from getting completely lost to not being able to look upon the data in the way they wanted. These systems were also subject to single user interactions and did not encourage a social atmosphere for the exchange of gathered information. When we started the work on The Pond, the objective was to design a multimedia user interface where users without any prior knowledge or acquired competence should be able to easily handle both single as well as groups of objects in an affordable and easy way. The objects in turn should be able to represent any type of information or media. They should be able to present themselves in a natural fashion at the user’s convenience. Also, they should disappear in non-obtrusive fashion when no longer needed. The user should be able to select, move, sort and explore objects of interest without having to confront the rigid hierarchy that is the hallmark of traditional file handling and database applications. Instead of using colour, form or position to indicate group or class
52
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
membership we wanted to use motion dynamics of objects to indicate these properties. If new information is requested, objects should just float to the surface and present itself to the user. Objects no longer relevant should slowly sink to the bottom and quietly disappear. The viewpoint should stay fixed above the virtual pond surface, thus making it unnecessary for the users to engage in navigation within the information space. An observation made early in the design process was the difference in the behaviour of users when confronted with a horizontally placed display vis-à-vis a vertically placed one. When a group of people gathers in front of a vertically placed display – be it a big screen TV or a wall projection – immediately a somewhat authorial situation tends to develop. The person in control of the display content is perceived as a teacher or lecturer. The rest of the group will play the role of an audience or school class. When, on the other hand, the display or projection area is positioned horizontally, people will gather on a more equal basis. With inspiration from as disparate sources as roulette tables, billiard tables and the military’s “classical” tactical war gaming board, it was decided that The Pond should use a horizontal display and that none of the sides of the physical artefact should be more important than any of the others. Tabletop displays have been in use for a number of years now, for example in visualising information (Kreuger and Froehlich, 1994) for command and control scenarios (Chase et al., 1998) and augmenting physical objects (Ishii et al., 1999). The table provides a natural centre for interaction to take place around and encourages collaboration between the users while they are interacting with the table (Benford et al., 2000). The developments in plasma display technology coupled with touch-sensitive surfaces now make it possible to dispense with potentially clumsy projected displays in favour of a neat, compact display. Interaction with The Pond is object based, using physical tags to load information into the application by placing them around the edge of the table (Ishii and Ullmer, 1997) (Ullmer et al., 1998). Unlike other approaches where the physical objects are placed directly on the table and manipulated (Underkoffler and Ishii, 1999; Fjeld et al., 1999; Schäfer et al., 1997), the tags used in The Pond are passive. They make it possible to load and store information, but do not manipulate this information any further. Interacting with the contents of The Pond is supported through direct manipulation of the virtual pond objects (Shneiderman, 1983) using the touch screen display. Typically, interactions with the tabletop systems discussed above make use of stereoscopic glasses, data gloves, and magnetic position trackers. These techniques are not used in The Pond as our aim was to create as easy and direct an interface as possible and not to encumber the user with devices needed to experience the material presented. In the remainder of this chapter we will describe The Pond as it stands today, details of its implementation, and observations from a study of system in use. We conclude with a discussion on some design choices and their implications on the potential utility of The Pond in different settings.
53
Inhabited Information Spaces
4.2 The Pond We now describe The Pond system in detail, examining its construction, both in terms of the physical artefact and the underlying software, and the way in which it is used.
4.2.1 The Pond Ecosystem Metaphor The Pond uses a user interface based on an ecosystem metaphor. The objective was to present an aesthetic that would hide the work chore aspect and act as a complementary backdrop for dialogue. When interacting with The Pond, the users see a 3D presentation of a virtual pool or pond (see Figure 4.1), an aquatic environment in which the information objects resulting from queries are presented as shoals of marine creatures. The visual presentation is complemented by a sound environment, consisting of a number of bubbling, splashing and whirling sounds, indicating various active processes inside the pond virtual environment. Each information creature has a simple navigational behaviour that governs its movements within the virtual pond. The behaviour is inspired by the boid flock model described in Reynolds (1987). The basic steering rules are: ● ● ●
Avoid collisions with nearby creatures. Match velocity with nearby shoal mates. Stay close to nearby shoal mates.
Figure 4.1 The Pond virtual environment.
54
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
These simple rules are then combined with additional influences like curiosity (a creature might temporarily break away from its shoal to “examine” its surroundings) and competitiveness (an urge to be in the lead), resulting in a very dynamic and fascinating pond environment, where shoals of information creatures move around in tight groups, avoiding each other as well as the pond walls, and reacting to interactions from the users. The movement of a shoal is entirely dependent on the combined navigational behaviour of its creatures, that is, there is no shoal “intelligence” that determines where the shoal is going or how its creatures should behave in certain situations. The argument for using a flocking algorithm for positioning the information creatures is not based on aesthetics alone. Our earlier experiences in information visualisation mixed with an ambition to explore a somewhat alternative user interface made us choose a self-animating system for the data elements. Not only animating during the creation/insertion stage but during the whole time the elements were available. In addition, the human visual system is quite good at separating objects with common velocity vectors making it easier to identify group belongings in a crowded environment. The Pond does automatic garbage collection, which means that one or several of the shoals that exists in the virtual pond may be removed. This may happen, for instance, if the environment is too crowded or because a particular shoal has not been interacted with for a long time. A shoal that is selected for removal by the system will sink down towards the bottom of the virtual pond, where it will disappear. However, should a user interact with a sinking shoal, for example, select one of its creatures, the shoal will return to the surface and another shoal might be removed instead.
4.2.2 The Pond Example Application
011
011
11
To investigate the concepts behind The Pond we developed an application that allows users to search for and browse music content on the Amazon.com web site. The music theme was chosen to maximise The Pond’s visual and sonic impact, being a theme common to most potential end-users. However, The Pond is not limited to this domain and is adaptable to other database resources. A search task is initiated by a user providing a keyword string, for instance the name of an artist, group, or a musical genre. The Pond communicates the keyword to Amazon.com and presents the resulting album hits as a shoal of creatures, each representing one specific album. The virtual creatures are represented by simple geometric shapes, which are texture mapped with the albums’ cover images. By interacting with a creature, users are able to access the album data (e.g., artist name, album and track titles, etc.) and play samples from some of the tracks. The virtual pond itself consists of a 3D model of a deep, narrow shaft that extends down from a watery surface.
55
Inhabited Information Spaces
The music-oriented Pond application was selected since we wanted an application that would allow us to explore and evaluate the ideas behind The Pond without having to implement a lot of advanced search functionality, and which would also be interesting and enjoyable to use (e.g., looking for and listening to your favourite music). Furthermore, the size of a “typical” query result was expected to be less than 50 items, which would allow us to present a number of simultaneous shoals in the virtual pond environment without making it appear too crowded.
4.2.3 The Hardware Platform The physical Pond artefact has the form of a desk, on top of which a large touch-sensitive plasma display is horizontally placed. On top of the display surface is a wide wooden frame with an irregular curved outline, representing the bank of the virtual water pond rendered on the display. The frame is covered by pieces of thick carpet so that users standing around it can comfortably lean over the display when interacting with the virtual pond environment (see Figure 4.2). Built into the frame are a number of speakers that are used to output various sounds and music samples. The use of audio is an important
Figure 4.2 The Pond desk.
56
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
feature in interacting with The Pond, and the sound system consists of several devices including a sampler, sub-woofer and amplifier. The frame also encloses several RFID tag readers. Each reader is entirely embedded into the frame carpet and uses three light-emitting diodes to indicate its position and state to the users. Users may initiate queries by placing a RFID tag on such a reader. The tag’s identifier, sensed by the reader, will identify a query keyword or phrase and the query will be initiated.
4.2.4 The Software Platform The Pond software platform consists of two different components (see Figure 4.3): ● ●
The visualiser application renders the view from the virtual pond environment on the plasma display. The pond manager application accepts query keywords from the users and communicates these to the Amazon web server. The resulting information is used to introduce and control shoals of creatures in the virtual pond.
The visualiser and the pond manager are built using the DIVE (Chapter 12) distributed VR system from the Swedish Institute of Computer Science (SICS). The 3D virtual pond is in fact a DIVE virtual world, shared by the visualiser and the pond manager. When the pond manager Tag readers
Speakers
011 Dive pond virtual world
Pond Manager
Query and result via web services API
011
11
Amazon Web server
Visualiser
Figure 4.3 Overview of The Pond software components.
57
Inhabited Information Spaces
application is started it loads a file containing a model of the pond graphical environment. Initially the virtual pond is empty, that is, it contains no information creatures since these are only created as the result of user queries. When the visualiser is started it joins the world created by the pond manager and will receive a state transfer containing the graphical environment. From this point on the two applications share the pond environment, exchanging DIVE messages to notify each other of any world changes (e.g., the introduction of information creatures or creature movements). The visualiser is responsible for detecting user interactions on the plasma display’s touch sensitive surface. If a user clicks on a creature, the visualiser distributes an interaction event that will be received by the pond manager. After examining the event, the manager determines the appropriate action to take (if any), which typically involves some change to the virtual creature (e.g., a change of appearance or position). The change will generate a new DIVE message, which will be received and handled by the visualiser, thus making the world change visible to the users. The visualiser uses a number of rendering plug-ins to enhance the visual appearance of the pond environment. A ripple plug-in creates ripples and waves that graphically deform the perceived environment when fingers are pressed on and moved over the touch screen of The Pond. A second plug-in is responsible for generating a caustic lighting effect on all objects in the water. These plug-ins use hooks in the DIVE renderer and operates on the image it generates. User input in the form of query keywords is handled by the pond manager. Queries are initiated either by direct request, that is, a user placing a tag on a reader, or more indirectly through inferred user interest. The manager continuously monitors the tag readers embedded in The Pond table and immediately senses any change in their status (i.e., RFID tags being added or removed). To get data requested by these queries the manager uses the Amazon.com web services (Amazon.com, 2002). With its web services program, Amazon.com offers third party web sites and applications the ability to search or browse Amazon’s product database. The manager retrieves information like album titles, artist names and URLs to cover images through the web services using XML over HTTP. This information is then used in the process of creating representations of individual albums and shoals representing more than one hit on a query subject. The pond manager also handles the audio output to The Pond table speakers, which is described further in Section 4.4.
4.3 Interaction Users standing around The Pond table are able to interact with it in several ways to perform various tasks. In the absence of keyboard or mouse devices, the users perform most interactions by tapping or
58
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
stroking the touch-sensitive display surface. Furthermore, prepared tags that are spread out on top of The Pond frame allow users to input information about queries without having to type on a keyboard. Queries are initiated by placing tags on the tag readers in table frame. Since each available query tag has a sticker attached to it showing the keyword text or an image of an artist or group, users are able to determine what keyword a certain tag represents (see Figure 4.4). When a tag is placed on a reader, the reader senses the tag’s unique identifier, which is pre-mapped to the keyword. As soon as a query is initiated, an empty shoal appears inside the virtual pond, representing the ongoing query. The shoal is indicated by a circle and a text string specifying the query keyword, as seen in Figure 4.5a. The circle will commence to float inside the pond environment, bouncing off the walls and avoiding other shoals. When the query results are initially delivered from the Amazon.com web server, creatures start to appear inside the empty query shoal. Each such creature represents an information element from the query result; in this case a CD album. As soon as a particular result creature has been created, it will begin to move around inside the virtual pond. However, since all creatures resulting from a particular query stay close together, different query shoals are easily identifiable, even with a vastly populated pond environment.
011
011
11
Figure 4.4 Query tags representing the keywords “Frank Sinatra”, “Billie Holiday”, “ABBA” and “Kraftwerk”.
59
Inhabited Information Spaces
a
b
c
d
e
f
Figure 4.5 Actions and events in The Pond: (a) To the right is an empty shoal, representing an ongoing query using the keyword “Nitin Sawhney”. To the left is a shoal of creatures representing a finished query. (b) Two shoals representing the results of queries using the keywords “Zeppelin” and “Queen”. (c) To the right a selected creature dragged a bit from its flock. (d) Playing sample one (out of five available) on the album “Siamese Dream” made by Smashing Pumpkins. (e) Three creatures dragged into a creel, limited by the pond walls and two buoys. (f) Getting related albums to an album called “Lamb” made by the group Lamb.
60
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
When all the results have been delivered and the corresponding virtual creatures created, the shoal circle label will change to only display the query keyword (see Figure 4.5b). Shortly thereafter the shoal circle and label will disappear, leaving behind only the creatures visible to the users. However, it is possible for users to make the circle and label visible again by tapping on any of the creatures belonging to the shoal. Each shoal member represents an information element that is part of the result of the corresponding query. The information includes name of group or artist, album title, and URL to the album cover image, as well as URLs to a number of short Real Audio® samples of some of the album tracks. Users are able to access this information by manipulating the creatures in different ways. A user selects a creature by tapping on it on the display, and is then able to see the information identifying the artist and title of the corresponding CD. This information is presented as a virtual text string, encircling the 3D creature and moving alongside it (see Figure 4.5c). The text will only be visible for a brief period (around five seconds) and will then disappear. The frame of the creature will become green to indicate the selection, and will remain so until changed back to white again when another creature is selected. Tapping once more on an already selected creature makes it float up to the surface and initiates playback of the corresponding album’s first Real Audio sample (see Figure 4.5d). By using the Real Audio player and the sample URL, the audio data is streamed over the network from the Amazon.com web site and output to the speakers embedded in table frame. By tapping repeatedly on the creature being played, users are able to step through the sample tracks available for that particular album. The text encircling the creature displays the number of the sample being played as well as the total number of available samples. By default, the viewpoint in the virtual pond environment is placed above The Pond surface, looking down, and at a distance from where the view is always guaranteed to include all the existing creatures. In this way the users are able to get a good overview of all the activity within the environment. However, this also constrains the creature representations to be rather small (as seen on the plasma display), which might present a problem when trying to identify the creature graphics of the CD albums they represent. In order to allow users to get a closer view of one of the shoals, a zooming mechanism allows for the translation of the virtual viewpoint to a position close to a shoal centre. The viewport is just large enough to encompass the whole shoal, with the benefit of making the creatures in that shoal together with their associated text strings and images appear larger. As a result, other shoals may end up out of view, not visible from the new viewpoint position. Another feature of the zooming mechanism is that, while zoomed, the viewpoint is attached to the shoal, which means that it will move as the creatures within the shoal moves. In this way the
61
Inhabited Information Spaces
viewpoint will always stay centred on the chosen shoal, even as this shoal changes its position within the virtual pond. While zoomed, the shoal creatures may be interacted with in the same way as before, for example, selected to initiate replay of the music samples. The zooming mechanism is triggered when a tag is placed on a reader while the corresponding shoal already exists in the pond environment. Removing the tag from the reader will reset the viewpoint to the default overview position. Thus, if a user places a tag associated with the keyword “Dylan” on a reader while a “Dylan” shoal exists, the viewpoint will change to a position close to this shoal, and stay there as long as the tag is on the reader. Whenever the tag is removed, the viewpoint is reset. Only one user at a time can use the zoom mechanism. If a zoom is active while a user initiates another zoom the viewpoint state will not be overridden and the second request is ignored. As users initiate more and more queries, older shoals may have to be removed in order to incorporate newer ones. To prevent a particular creature from being removed from the environment, it is possible for users to move individual creatures to safe areas, called creels. This is done by touching the creature with the finger, and then dragging the finger (and thus the creature) along the surface and releasing it over the creel area (see Figure 4.5e). Creels exist in several places in the virtual pond environment, close to the tag readers. Once inside a creel, a creature is constrained to move only within the creel boundaries. By moving several creatures, possibly from different shoals, into a creel, a selection shoal is formed consisting of creatures that a user finds interesting for some reason. Since this particular action shows an interest from the user in a specific data element, this instantiates an extra functionality. A more focused query is automatically launched pertaining to the corresponding album (usually resulting in fewer hits and consequently a smaller shoal), thus further populating the environment (see Figure 4.5f). In the case of Amazon.com the system issues a related albums query resulting in items that have a high chance of being interesting to the user as well. The creatures in the creels may be interacted with in the same way as other creatures (e.g., tapped on to play music samples), the only difference being that they won’t be removed from the environment as long as they stay inside the creel. Creatures that are dragged out of a creel will return to their native shoal, or form a new shoal if their native shoal no longer exists. By using a recordable tag, it is possible for a user to save the contents (i.e., creatures) of a creel selection shoal. A recordable tag is a RFID tag which is not pre-mapped to a search keyword. Placing such a tag on a tag reader next to a creel creates an association between the tag and the creatures within the creel area. When the tag is then removed from the reader, the creel’s selection shoal will disappear from the environment and may be regarded as being stored on the tag. By placing the same tag on a reader at a later stage, the creatures of the “saved” selection will
62
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
reappear, added to the corresponding creel shoal. Thus, creels and recordable tags allow users to save references to one or several albums of particular interest, and to gain quick access to these on a later occasion, as well as to share references with each other.
4.4 The Pond Audio Environment The sonic environment of The Pond consists of two different parts, the soundscape and the interface sounds. In accordance with the ecosystem metaphor, the nature of the soundscape is founded on a family of aquatic whirling sounds and a deep, obscure mumbling giving the impression of the data elements ascending from an abyss of ooze or mud. These ambient sounds fade out when samples from a selected creature starts to play, and fades smoothly in when the music stops. The interface sounds acts as a feedback mechanism to indicate user interactions like selecting/unselecting, clicking, dragging etc. This collection also originates from a number of concrete water sounds. Samples from different types of bubbles, a dripping tap, whirlpools etc. are heavily processed to suit their particular purposes. Examples include: ● ●
●
●
●
When a user initiates a query, the appearance of the query shoal is accompanied by the sound of sluggish bubbles rising from the bottom. When a sound file is retrieved over the network, the waiting time is masked with a bubble vortex that after a few seconds is smoothly merged with the music sample. When the user removes a tag from a reader, the action is accompanied by sounds of a cluster of bubbles being rapidly inhaled by The Pond itself. When the user draws his finger across the touch screen to drag an object, a glass organ sound reminiscent of drawing a finger along the damp edge of a crystal wineglass is heard. The visual zooming in of a shoal is illustrated by the familiar bubble sound gradually magnified through lowering of the pitch. When zooming out, the process is reversed.
Every time an interface sound needs to be played, the system will randomly choose a sound from a collection of sounds available for the particular interaction. There are ten query sounds, ten RFID sounds, ten dragging sounds, etc. The idea is to give the impression of the sonic interface being somewhat organic and unpredictable. The precise spatial placement of every sound is achieved through a builtin high quality four-channel sound system. A subwoofer in the table foot produces a deep and suggestive bass. The computer-controlled software mixer makes it possible to physically move sounds around and to create expressive musical gestures.
63
Inhabited Information Spaces
4.5 Observations from Use To test out The Pond a number of sessions were held with external participants in the expectation that some qualitative assessment of the effectiveness of system could be made. After being presented with an introduction to The Pond and the example application by one of the developers, the participants were told to explore the system, using the available tags on the table frame. The focus of the studies was on the technology-in-use (Button, 1992), that is, the sequences of interaction within which The Pond came to be used in real time. The developers stayed close the whole time to answer questions and to help out when problems occurred. Most sessions involved two or three participants and lasted for about an hour. Video was used to record each session and from this material a number of observations were drawn. The general impression is that the participants had very few problems in understanding the basic principles of The Pond design and its operation, for example, how to use the tags to initiate queries or how to interact with the creatures in order to play sample music. On very few occasions did the developers have to repeat the instructions given at the start of the session. The Pond is engineered to support hands-on experience and ease of use, and most participants did engage in various activities, together and alone, such as selecting CDs, listening to samples, transferring CDs to tags, and transferring CDs from tags after only a short time. During a discussion after one of the sessions, a participant stated: “The main good thing about The Pond is that the interface doesn’t require any special level of computer skills for using it.” Not only did the participants get the grasp of The Pond fairly quickly, but were also able to envisage its use for practical purposes in every day settings. For example, one of the participants suggested: “Place it in a Virgin Megastore, it can be used as a jukebox. Perhaps you should specialise it more so it fits into a category like film, books, articles, and radio. And you should be able to tell The Pond which CD you want and then be able to put it into your shopping basket.”
In some sequences of interaction it is noticeable that the sound worked as functional feedback. After the participants had recognised the connection between the function and the sound, the sound worked as a support for the user in interacting with The Pond. This can be seen in a sequence when a participant drags a creature out from a creel. In the sequence the sound that is heard as the dragged creature passes the creel boundary makes it clear to the participant that she has succeeded and the creature is released. The problems that participants had during sessions were mostly related to the hardware, and especially the touch screen. A serious drawback with the large area touch screen technology we currently use is a limitation when it comes to detecting multiple simultaneous touches.
64
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
This sometimes caused conflicts when more than one participant was trying to interact with creatures at the same time. For instance, it was impossible for two users to drag creatures at the same time. Also, if one user was dragging a creature while another user tried to select a different creature, the selection would most often fail. The reason for this is that when two positions on the touch screen are clicked simultaneously, a position in between the two will be returned by the driver software, most likely causing the second user to miss the intended creature. Future multi-user touch screens will alleviate this shortcoming. In the present set-up we try to work around the problem by making most interactions single click based, dragging being the exception. Another problem with the touch screen was that of sensitivity. Sometimes when the participants where dragging creatures they seemed to unintentionally lose contact with the touch surface, causing the creature to stop following the finger and return to its shoal. What happened was that even though their finger was actually in contact with the surface, they didn’t put enough pressure onto it, which made the driver report that the interaction had ended. A number of “phantom clicks” (i.e., the driver reported a click when there was none) were also experienced which had a similar effect on the interaction. We expect most of these hardware problems to disappear in the future as the technology matures. One of the software mechanisms that caused some problems was zooming. It was evident from the video material that zooming could cause confusion if being performed by one participant when others where trying to interact with creatures in shoals different from the one being zoomed (as these shoals would disappear from the view). Zooming probably requires synchronisation of the users’ activities in order to be useful. As it turned out, the navigational behaviour of the information creatures became a source of fascination for many of the participants when encountering The Pond, and it was not uncommon to see them standing silently at the side of the display for some time, simply watching the creatures swim back and forth.
4.6 Discussion One of the most frequently expressed comments from users was the desire to be able to search The Pond in a more dynamic fashion, that is, to issue custom search queries, which is not possible through the use of the tags at the moment. The seriousness of this limitation depends of course on the application and its intended use. The Pond is not primarily designed to be used for the task of quickly locating and browsing arbitrary known information items on the web, in which case the use of the pre-configured query tags would probably be unacceptable. Instead, the focus is more on applications where the size of the data set is fairly small (i.e., a product catalogue) so that the query tags more or
65
Inhabited Information Spaces
less cover the whole search space. In addition, if the database is hierarchical or allows items to somehow contain references to each other, the users can traverse the data not only by issuing direct queries using the tags, but also by exploring the creatures already existing within The Pond (following the inter-creature links to discover other “parts” of the data space). In such settings the tags can be seen as a way to introduce information objects that probably won’t be the end result of a user’s query, but merely a starting point for further exploration. Examples include using The Pond in a record store setting where the top 100 albums are to be displayed and played, or in a hotel lobby where menus from local restaurants, museum information, etc., can be examined in a collaborative fashion. In the Amazon.com data source set-up, a way of search refinement and database traversal was available through the use of related album queries. This introduces items that have a high chance of being interesting to the user. Combined with the fact that items considered uninteresting will sink and disappear, this has the effect of gradually refining the ecosystem to contain more and more interesting items. So even though very specific selections are being used as “seed” material, the contents of the environment might develop through time to contain a mix of several general tastes, specific favourites and complete wildcards. Our aim when designing the interface was towards a non-intrusive form of interaction, avoiding introducing keyboard type input. The RFID tags do serve well in cases where a hierarchical, finite and discrete database structure exists, as the music database in our case. Notice that the environment is in the most part populated by relations and not exact matches, so we force exploration on the users. It proved at times frustrating for users not to be able to directly summon the artist or track of their choice, and at times gratifying in discovering alternative music of their liking. A solution that would in fact allow users to input exact search keywords or query phrases and which wouldn’t entail the use of some sort of keyboard function would be to use voice as input. An issue that came up during the user studies was the possibility of modifying or even replacing the example application to add new functionality or to support different types of media. For instance, instead of saving references to albums onto recordable tags, one participant wanted to be able to save the actual sample data onto some kind of portable device, for example, a PDA, which could then be used for playback. This could be achieved by using some kind of point-to-point (e.g., infra-red or Bluetooth) connection to transfer the audio data from The Pond to the PDA. Also, since DIVE runs under PocketPC™ it would be possible to develop a PDA application which would join the pond virtual world to be able to access the information (e.g., music samples) represented by the virtual creatures. Modifying The Pond to support a different type of content media, or even combinations of different media, would require changes to the pond manager application (see Section 4.2.4 above) in two different ways. First, it would require the pond manager to be able to
66
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
communicate and issue queries to a database service different from the one provided by Amazon.com (assuming that Amazon.com does not support access to data elements of the “new” media type). This would be fairly straightforward since the code responsible for handling queries is isolated in one specific module, which can be easily replaced. It is also possible to add a second (or more) module to exist in parallel with the first if there is a need to connect to more than one database service during a session. Secondly, to support new types of content media would require new rendering mechanisms to present the creatures to the users when there is interaction with them. As is the case with the current Amazon.com web service version and the RealAudio player, special information rendering that goes beyond small pieces of text, images and texture-mapped videos will probably need external renderers. These can be in the form of DIVE plug-ins or external applications (e.g., Windows Media Player to render video material). The pond manager application defines a number of events (e.g., OnClick) that are fired when creatures are interacted with and needs to display details of their associated information (e.g., album title in the Amazon.com application). By modifying or replacing the current handling of these events in the pond manager code, it is possible to change the way the creatures present themselves to the users. One of the consequences of using a static view into the pond environment is that the display space is restricted, thus limiting the number of creatures that can be presented simultaneously. As described earlier, garbage collection can alleviate the problem to some extent by removing old shoals when new ones need to be introduced, but it does not help if the new query result is too large to fit into the virtual pond by itself. Making creatures smaller in size will make more of them fit before the pond becomes to cluttered, but also makes it harder to distinguish the album images, read the text strings, etc. One possible extension of the shoal concept that could help in presenting large data sets is hierarchical shoals. Assume that a shoal could be made up of sub-shoals as well as information creatures, and that a sub-shoal was initially represented by a single graphical object, just like an ordinary creature. When clicking on a sub-shoal object it would expand to display all of its elements, some of which might themselves be sub-shoals, and so on. An expanded sub-shoal might collapse into the single sub-shoal object after a certain amount of time, or possibly as the result of an explicit user interaction (e.g., clicking once more on the sub-shoal object). In this way it would be possible to make available a large number of information creatures to The Pond users without having to display them all at the same time. The rules governing the creation of sub-shoals are probably application specific, that is, the criteria used to determine when to create a sub-shoal and which creatures to add to it. Another possible solution to handle large query results would be to make each shoal present only a limited set of the resulting creatures at one time, and then allow the users to somehow (via some shoal interaction) make the shoal present
67
Inhabited Information Spaces
the next set of creatures, and so on. This is the method used by most Internet search engines where the user is presented with a list of about ten hits, and is then expected to use the “next” or “previous” links to move forward or backward in the result material.
4.7 Summary and Future Work We have presented The Pond, a multi-user system for browsing information (currently CD record data) on the Web using an engaging tabletop display. Users input musical search keywords (typically names of artists or groups) using RFID tags and are presented with matching Web information in the form of shoals of aquatic creatures inside a virtual 3D pond. The virtual pond environment is presented on a big touch-sensitive plasma display, which is placed horizontally to better support shoulder-to-shoulder collaboration between those people gathered around it. By touching the surface of the display users can interact with the creatures to access the information they represent, for example, play music samples from the corresponding CDs. A series of user studies have provided initial insights into the utility of The Pond. The results indicate that the device and metaphor are easy to understand and use, but also identify a number of problem areas. For instance, the touch sensitive display doesn’t currently support multiple simultaneous interactions, which sometimes caused the users’ actions to interfere with each other. The Pond has been demonstrated to members of the research community as well as to the public on numerous occasions. The feedback and observations from these sessions have been used to refine The Pond under an ongoing evolution process. Future ideas include support for multiple simultaneous interactions on the display surface, using a voice input system for entering search keywords, and using PDAs to extract and input information to and from The Pond environment.
Acknowledgements We would like to thank Lennart E. Fahlén, Jan Humble, Jenny Lundberg and Jonas Söderberg for their participation in the development of The Pond system. We would also like to thank Adrian Bullock for his contribution in documenting The Pond work as well as Andy Colebourne for his Pond-related efforts within the eSCAPE project. This work was undertaken as part of the Swedish Research Institute for Information Technology (SITI) funded Electronic Landscapes Project (ELP). We would like to thank all those who have experienced The Pond and provided us with invaluable feedback. The Pond work has also been described in (Ståhl et al., 2002).
68
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
Part 3 Mixed Reality Environments
011
011
011
11
69
This page intentionally left blank
011
011
011
011
11
City: A Mixture of Old and New Media
5
Matthew Chalmers
5.1 Introduction The majority of the chapters in this book describe what might nowadays be called “traditional” inhabited information spaces: collaborative virtual environments (CVEs) or virtual worlds. Although not all CVEs centre on computer-rendered 3D graphics, the paradigmatic CVE does so. Shared 3D virtual environments are emblematic of CVE research, but have only gained public acceptance in the form of computer games. The focused engagement in such games is designed to fit with the closed world of the virtual environment. A player can become immersed in a game – closed off from the “real” world – by attention as much as by apparatus. A personal computer (PC) at home can be as engaging as the head-mounted displays and immersive projection technologies in research labs. However, even a single-player non-networked game may be a resource for social interaction, for example played by one person while friends and family shout advice from the sofa, order pizza by phone and slip into the kitchen to get more drinks. A computer game is a resource for far more social interaction than the software’s architecture may suggest. In general, the wider context of use is hardly modelled or represented in the system. Games’ internal data structures are designed to be decoupled – that is, closed off – from the other media people use in everyday life, and this decontextualised design approach has worked well in this domain. Although many households, schools and workplaces have computers that could support 2D or 3D virtual environments, such CVEs are rarely used as a medium of family interaction, education or work. One reason for this may be the decoupling from the more traditional interactions of family members, the overall educational activities of the school and the business of the workplace. The information within the CVE would have to correspond with each user’s wider context, and this would require sensing and tracking of users’ activity in media beyond that of the computer. To paraphrase part of this book’s introduction, a large amount of
71
Inhabited Information Spaces
our activity relies on the knowledge of what other people do, and what people do in the home, street, school and workplace involves many noncomputational media. However, CVEs are designed on the basis of narrowly focused engagement, decoupled from their users’ wider context of family, friends, learning and work. Other than in games, CVEs’ decontextualised design approach has not led to popularity or widespread use. As the introduction to the book also points out, a number of researchers have begun to work on inhabited information spaces that are more “out in the world” than traditional CVEs. A rhetorical example the editors give is “a system that enables co-located groups to co-operatively work with information by using a display projected onto physical artifacts”. An IIS may include tangible artefacts in more traditional media, such as urban models and interaction devices made from wood, wire and plastic (Underkoffler and Ishii 1999), or tiles and pages made from toner and paper in augmented reality systems such as MagicBook (Billinghurst et al., 2001). The work discussed in this chapter is intended to go further in this direction. As part of the Equator interdisciplinary research collaboration (www.equator.ac.uk), the City project explores digital spaces that are peers with others, rather than a digital space that is the primary focus or locus of activity. For example, in our system, one person’s use of a 3D VR model of a museum exhibition is coupled with another person’s use of hypermedia describing the exhibition as well as a third person’s use of the traditional “bricks and mortar” exhibition. No one of the three is primary; each is part of the context of the other two. We combined the media of traditional exhibitions, mobile computers, hypermedia and virtual environments in one design, and support interaction between people using different subsets of this heterogeneous set of media. The project was initially theory led. A number of theoretical issues were outlined in a discussion document, and then exemplified by scenarios of technology use. As the project grew and developed, our theoretical issues, design scenarios, system development, evaluation of pilot trials, and observational studies all affected each other. Although I sometimes campaign for the rule of theory, not one of these areas is primary. Instead, each is part of the context of the others. The project has always aimed to get out into the streets of the city, but we decided to begin our work in a more controlled setting: the Mackintosh Interpretation Centre, a permanent exhibition devoted to the life and work of Charles Rennie Mackintosh (1868–1928). Mackintosh was a Glasgow architect, designer and artist, and several of his buildings and other institutions related to his work stand in the city. Often simply called the “Mack Room”, the Centre is comprised of textual and graphical displays with some original artefacts, as well as over 20 screens presenting video and interactive digital material. The Mack Room is in The Lighthouse, Scotland’s Centre for Design, Architecture and the City (www.thelighthouse.co.uk).
72
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
More generally, we are exploring the way that digital information is just another part of an interwoven set of media, the use of which constitutes inhabiting the city. Unlike traditional CVEs, we aim for systems coupled with and contextualised in everyday activity, and hence in accord with contemporary theory of the use of language and space. A basic theoretical premise is that we can only use digital media because of such interweaving and interdependence, and we are looking for ways to increase and take advantage of the interdependence of traditional and new media. This theoretical standpoint is set out in the next section of this chapter. The subsequent section outlines a system design based on this standpoint, followed by a discussion of user experience in trials of an implemented research prototype. Some of the details of our ongoing and future work are then outlined before a concluding section that offers more general reflections on the project.
5.2 Theory This section focuses on conceptions of space and the media associated with work, and how we often conceive of space as a medium that stands above or apart from others. It is this usually implicit assumption that lets us talk of information spaces as being “inhabited”. I would like to present an opposing view that treats information spaces as merely one medium among the many used in everyday life. My approach is based on experience with information visualisation and virtual environments, as well as some borrowing of structuralist semiotics (Saussure, 1983; Nöth, 1995) and philosophical hermeneutics (Grondin, 1994). Part of this latter work was set out in a recent paper in the Journal of Computer Supported Cooperative Work (Chalmers, 2003). When discussing work, and designing systems for remote collaboration, we all too often concentrate on emulating the spatial aspects of the workplace, for example modelling spatial forms and supporting remote communication that appears to be like face-to-face interaction. All design has to be biased in some way, and the bias towards space in CVE research may be due to it being technologically led more than sociologically, semiologically or philosophically led. The arrival of cheap graphics hardware and the eye-catching novelty of 3D images gave rise to a good deal of work that focused rather narrowly on the construction of rather decoupled and decontextualised information spaces. This is true in some of my own work over the past twelve years, ranging from Chalmers (1991) to Morrison et al. (2002), and there are strengths, weaknesses and alternatives to such a bias (Chalmers, 1999). Newer technologically driven research is weakening or revealing CVE research’s implicit assumptions of space’s primacy and independence. Many of the characteristic design principles and assumptions were established before the current fashion for mobile computers and ubiquitous
73
Inhabited Information Spaces
computing. Nowadays, it is possible to obtain tolerable frame rates for 3D graphics on a wirelessly net-connected handheld computer. A person can thus be walking down a city street with a friend, chatting about a museum they intend to visit, while simultaneously watching the avatar of another friend moving through a CVE – with that “remote” friend also taking part in the conversation. In this case, it would seem difficult to claim that the person “inhabits” the information space. One might ask whether the person is in digital or virtual space, or in real or physical space, but the question is based on two false dichotomies: digital media are no more or less real than older media, and computers are just as physical as buildings and books. The workplace has always been affected by communication with people in other locations. Many traditional, everyday and non-digital media support remote interaction, for example letters, books, maps and the landline telephone. There are already digital media in the contemporary workplace that support remote interaction, such as email and mobile telephones. Nowadays, why do we not speak of “entering cyberspace” when we use email, as people did a decade ago? Why do we not inhabit telephone space, or speech space, or MacDonald’s employee name badge space? I suggest that a principle from philosophical hermeneutics is useful here: we don’t talk about these technologies in such marked ways because we have appropriated them into our everyday life and language. We no longer “enter cyberspace” because email is so interwoven in our everyday life and familiar in our experience that we don’t need to mark it out in such a way. We don’t inhabit telephone space because we understand telephones, in particular how to present ourselves through them and how to present ourselves to “spectators” nearby who can perceive our use of them. We only “inhabit” virtual worlds because their designs are so new and decoupled from other media. Experience and understanding of such coupling lets us focus on the task of communication, not on the tool for communication, just as a carpenter engaged in his work focuses on hammering and not the hammer. We continually mix and couple media in our everyday communication: walking, gesturing and pointing while talking, and referring to places and what people did in them as one writes. Space is an essential part of this mix. It has its unique characteristics that differentiate it from other media, but it has no privileged position above or apart from them. It does not stand alone as a paragon for computational media to emulate. More generally, a medium cannot be fully used or understood in an isolated or “singular” way. People’s activity continually combines and cuts across different media, interweaving those media and building up the patterns of association and use that constitute experience and understanding. A person’s work or activity may be influenced by the configuration of space around them and the interactions that space affords, but also by books, telephones, hypermedia, 3D computer graphics and so forth. People act and work through the full range of media they have
74
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
ready to hand. A narrow emphasis on space as the paramount resource for activity underrates the influence of other media. Recent technological developments, such as mobile phones and email, heighten or highlight a phenomenon already familiar in the use of older media such as written text, maps and cinema, and well explored in older disciplines than computer science. For example, a city’s meaning is not just in its bricks and mortar, but also in our understanding and use of the information about it. At any time, one is likely to have symbols in a number of heterogeneous media available for interpretation and use. As I walk through a train station towards a city square, the map in my hand, the voice of a friend on my mobile phone, the signs informing me of exit routes and the posters advertising exciting shopping opportunities are all open for my interpretation and action. Temporally, symbols in an even broader range of media influence me, as my activity is influenced by my past experience and my expectations of the future. Past experience may include my previous visits to that city, my browsing of a web site with good maps to print out, and my experience of magazines, books and films about urban life, and so forth. My language and culture, spanning media old and new, affect me as much as the immediate perception of spatial form. Since Heidegger and Saussure, a fundamental tenet of philosophy and linguistics has been that language is constituted by all the symbols and all the media one uses, with each symbol interpreted through immediate individual perception as well as past social experience. Contemporary neurophysiology is in strong accord with this view (Churchland and Churchland, 1998; Edelman and Tononi, 2000), as is the field most obviously related to the design of space, architectural and urban design (Leach, 1997). The differences between media are usually very obvious. We can characterise media and treat each one as if it were an isolated individuated entity because of the senses we use in perceiving each one, and also because of our understanding of how to relate and to distinguish examples of each one. For example, it is easy to distinguish the spoken word “red” from the written word red because of the senses one uses in each case. Despite having the same letters, it is easy to distinguish tar from rat by looking at the order of letters within each written word. Simple rules about what one can immediately see, hear, etc. within a word begin to strain and then break when one considers, for example, how we distinguish homonyms such as rose. The written word rose can mean many things, including a flower and having risen. When spoken, the same syllables can also mean linear structures (rows), about or belonging to fish eggs (roe’s), moving in a boat (rows), small deer (roes) and multiple occurrences of the Greek letter (rhos). The word’s usage is understood through its context – one’s understanding of the other symbols co-occurring with its use – rather than perception of the word’s pattern of syllables or letters.
75
Inhabited Information Spaces
Context becomes progressively more important as we turn from thinking about the differences between media, and distinguishing symbols, to considering the similarities of media and the relatedness of symbols. For example, the spoken word “red” and the written word red are related because we can use either of them in the context of rose blooms, fresh blood, the former USSR and so forth. We understand, relate and differentiate symbols through experience of contexts of use within a culture. As shown in the early twentieth century by Saussure, this understanding is not solely dependent on the form or medium of each symbol, but also on how we use each symbol in the context of other symbols – and this context includes symbols in other media. Taking fuller account of the interdependence of media enriches our understanding of space and of work. Consideration of how to make systems that are consistent with this standpoint opens up new possibilities for technology design and for computer-mediated social interaction. More particularly, it opens up a wealth of approaches based on coupling and contextualisation. For example, a museum exhibition might be associated with a set of web pages so that walking into a room of a particular architect triggers the display of text describing the life and work of that architect. Similarly, reading the text might trigger display of a map or visualisation of the room, affording access to a structured collection of blueprints, design sketches and building models. The space of the room would be coupled with the text of the page, with each becoming part of the context of the other. In terms of social interaction, a person walking into the room might be made aware of a friend’s reading of the web page, and hence open to conversation about the exhibition despite the two people being geographically remote from each other. Our intention is to support social interaction, as is familiar in traditional museums where co-visitors use awareness of each other’s interaction with exhibits as a resource for their interaction with each other, and use interaction with each other in interpreting the exhibits (Falk and Dierking, 1992; Galani and Chalmers, 2002). The City project explores the coupling of new and traditional media, weaving them together to form resources for social interaction and interpretation. In particular, our 2002 experiment explored social interaction between people in different locations and contexts where, by definition, they have different resources at hand. As they discuss and refer to contextual information, heterogeneity of media is inevitable: one person can use the nondigital resources of his or her location while others have only digital representations of that location. A case that is more easily handled is audio: each person will hear his or her own voice and sounds from other nearby sources differently to others, because of the digitisation and transmission of audio, but we have become relatively accustomed to handling this. A much more challenging heterogeneity is that of people’s position, orientation and gesture within rooms and buildings. For example, the Mack Room presents much greater visual and tactile richness than
76
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
the room’s digital representations, e.g. maps and VR models. Unlike most earlier CSCW research, the City project addresses this inevitable heterogeneity by coupling media together, tracking activity in each medium and representing it in others, and so letting participants interweave these media in their social interaction.
5.3 System This section outlines our prototype system, beginning with its infrastructure: the EQUIP platform, the Auld Leaky contextual link server, the VR Juggler framework, and the Bristol ultrasonic positioning system. More detail of this system can be found in MacColl et al. (2002). The EQUIP platform is being developed within Equator to support information sharing between heterogeneous devices. It provides a runtime infrastructure to support interoperation between Java and C++, and supports extensibility, for example dynamic loading of C++ classes. The University of Nottingham is leading the development of EQUIP, with contributions from the various Equator projects. City uses it as a blackboard architecture through which VR Juggler, Auld Leaky and the Bristol ultrasonics interoperate. Data items representing user context, an underlying spatial model and context-dependent content are stored for manipulation by City clients and services. Additional EQUIP facilities support real-time 3D graphics and mathematical operations, and abstract, renderable scene graph nodes. In addition, interfaces between EQUIP and a number of other systems have been developed, including the University of Iowa’s VR Juggler. VR Juggler is used as the renderer for 3D graphics. It is described as a virtual platform for virtual reality (VR) application development. It is a high-level, cross-platform architecture supporting immersive and nonimmersive presentations. Both UCL and Nottingham have immersive projection facilities, and the UCL facility has been used for development and pilot trials in the City project. The 3D graphics rendering is used to provide an analogue to the traditional exhibition space visited by traditional visitors. For World Wide Web visitors, the space is represented as a 2D map. We also require a presentation of the information in the exhibition displays, and this is provided by Auld Leaky. Auld Leaky is a lightweight contextual link server being developed within Equator to store and serve hypermedia structures, using context to filter query results. The model used to define the structures is capable of representing a variety of hypermedia domains: link-based, spatial and taxonomic. Auld Leaky is being developed by the University of Southampton, and is written in Perl although it has a Java API. Information is encoded as an XML linkbase, loaded into Auld Leaky and queried using HTTP. The text and images of the hypermedia were taken from the Mack Room’s catalogue. Contextual queries are used to
77
Inhabited Information Spaces
generate location- and device-specific content to be delivered by the Apache World Wide Web server and servlet engine. For 2D or 3D renderings, a location can easily be derived from the position of a visitor’s avatar or map marker. For the mobile computer, an ultrasonic system is used to provide position information. The University of Bristol is developing a low cost indoor positioning system using a combination of radio frequency (RF) and ultrasonics (Randell and Muller, 2001) as part of its contribution to Equator. The system uses a single RF transmitter for synchronisation, with ultrasonic transmitters for positioning. The ultrasonics transmit at known intervals after the RF, and are received by a handheld or wearable receiver. Each second, the variations in flight time of the ultrasonic transmissions are used to calculate the spatial position of the receiver. The receiver incorporates a magnetic compass to provide orientation information. The City project installation involves eight ultrasonic transmitters covering the approximately 10 m by 20 m area of the Mack Room. The room is a challenging environment for ultrasonics, as it is split into two large areas by a partial “time line” wall and has some areas set up as cubicles within which ultrasonic reception is virtually impossible. For aesthetic and coverage reasons, the transmitters are set on top of walls, displays and cubicles so that ultrasonic transmissions are reflected off the ceiling. The system supports a shared visiting experience, with one visitor using a handheld or wearable computer in the Mack Room, a second visitor using the World Wide Web on a laptop or PC in another room, and a third using 3D graphics on a similar machine in a third room. These computers communicate through 802.11 wireless Ethernet. A separate audio subsystem, that we will not detail here, handles visitors’ speaking to and hearing each other. In discussion and development scenarios we name the visitors Vee, Dub and Ana respectively, and these will also be used in the remainder of this chapter. The names do have roots: Vee is for ‘visitor’, and Vee was the user in our first design scenario; Dub is from the first syllable of “double-U”, as in WWW; and Ana stems from ‘analogue’, playing with the way that the digital space of the VR is an analogue of the Mack Room. Some people have suggested that Vee should be the ‘virtual visitor’ but we decided to keep the name, so as to irritate those who think that Vee’s experience is not strongly influenced by digital media. Spatial awareness is supported by tracking activity in each of the 2D, 3D and handheld systems, sending position and orientation information for each one into EQUIP, and then rendering the information about all visitors to each individual visitor. The components of the prototype system operate similarly for each visitor, broadly as follows: 1. store spatial position and orientation in EQUIP; 2. retrieve and render positions of other visitors;
78
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
3. store named location in EQUIP in response to position change; 4. store content from Auld Leaky in EQUIP in response to location change; 5. format content for presentation and advise client program of availability in response to content change. For Ana, position and orientation information is automatically published in the EQUIP data space by the VR Juggler client, and the positions of all visitors stored in the data space are automatically rendered as 3D avatars. Figure 5.1 shows a non-immersive spatial awareness display for Ana, with avatars representing Vee and Dub (displaying only heads rather than complete avatars). Vee uses a Hewlett-Packard Jornada that polls position and orientation sensors, and sends the results via a proxy into EQUIP. The proxy is also responsible retrieving the information about the other visitors, and the positions and orientations of all visitors are presented to Vee on a 2D map. Figure 5.2 shows a visitor the Mack Room, with a handheld and an ultrasonics receiver. The figure also shows a close-up view of the handheld as an inset.
011
011
011
11
Figure 5.1 A non-immersive VR display of the Mack Room for Ana, with avatars representing Vee and Dub (displaying only heads rather than complete avatars).
79
Inhabited Information Spaces
Figure 5.2 A visitor in the Mack Room, “Vee”, with a handheld computer and ultrasonics receiver. The figure also shows a close-up view of the handheld as an inset.
Dub interacts with a Java applet in a World Wide Web browser frame. The applet communicates via a proxy that converts mouse clicks on a 2D map of the Mack Room to position and orientation information. The applet also displays representations of all visitors. An example of Dub’s map is shown in Figure 5.3, corresponding to Ana’s 3D display in Figure 5.1. The red boxes on Dub’s map are trigger zones, discussed in the next section. Vee’s map is similar to, but simpler than, Dub’s. Shared visiting requires a sense of shared context, and hence some comparability of the information available to each visitor, but we also
80
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
011
011
011
011
11
Figure 5.3 An example of Dub’s map. For paper publication, the image has been annotated with the names of the visitors.
wanted to maintain and explore a degree of the heterogeneity that is inevitable in remote collaboration. Vee has the rich environment of the Mack Room’s displays and artefacts, as shown in Figure 5.2. Providing hypermedia to Dub and Ana involves converting positions to named locations, querying Auld Leaky with the visitor’s device and location in order to generate informational content, and then formatting and presenting the content. In initial trials of our prototype system we did not deliver hypermedia content to Vee, encouraging her to use the rich content of the existing room when interacting with her friends. Dub and Ana have rich access to the web that Vee lacks, and they can move and jump between Mack Room locations in ways that Vee cannot. Ana’s
81
Inhabited Information Spaces
3D view of the Mack Room VRML model has greater visual richness than the 2D maps of Dub and Vee, but she also has visual occlusions to deal with. Positions are converted to locations by an EQUIP-connected service. These locations represent semantically significant volumes or extents within the spatial model (shown as red outlines on Dub’s map, as in Figure 5.3). Also, for each visitor a target is inserted into the EQUIP data space, currently equivalent to a 10 cm cube held in the hand. Detection of a collision between a target and a sensor invokes code that inserts a new (user, location) item into the data space. Adding or updating a userlocation item in the data space triggers a query to Auld Leaky and the results – a set of hypermedia fragments – are stored in the EQUIP data space. Adding or updating such a set of hypermedia fragments in the data space triggers formatting for delivery to the visitor. The content fragments are retrieved and combined into an HTML page. Dub’s applet displays the HTML page in a separate browser frame set aside for this purpose. Ana also runs a browser and applet that, when advised, displays HTML pages, i.e. Ana has no map display but does have a textual display. The overall effect, then, is that all three visitors have location information about their co-visitors’ shared awareness of location. Dub and Ana each have spatial and textual information about the exhibition, with the text updated as they use the 2D map and the 3D graphics. Vee has the traditional information of the exhibition room. Many of the artefacts and exhibits have corresponding representations for each of the three visitors. We sometimes refer to such artefacts and exhibits as “hybrid objects” because of the visitors’ tight interaction around corresponding heterogeneous representations. Space, text and audio afforded sufficient interaction, context and reference to support a shared visiting experience, as the next section discusses.
5.4 Use In the summer of 2002, the City project carried out a set of system trials in the Mackintosh Interpretation Centre. Rather than trying to make the best possible system for the Mack Room in particular, our focus was on general lessons we could learn for the design of systems involving heterogeneous representations and interactive media. We aimed to increase our experience and understanding of how these systems serve as constraints and resources for users’ interaction. We had already studied the use of a number of cultural institutions, including the Mack Room, without our technology (Galani and Chalmers, 2002), and were interested in comparing the Mack Room with and without our technological intervention. Fuller discussion of the trial can be found in (Brown et al., 2003). The trials involved 34 participants: ten groups of three and two groups of two. The groups of three consisted of a Dub (using the web), an Ana
82
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
(using VR) and a Vee (using a mobile computer). Dub was in the Interpretation Centre, while Ana and Dub were in separate rooms on a different floor of The Lighthouse. The pairs explored different combinations: Dub and Ana visiting without the physical visitor (one trial), and Vee and Ana (one trial). For the first half of the trial, participants were asked to explore the Centre together, to familiarise themselves with the technology and how they could co-visit. Since we were specifically interested in how the system supported social interaction, we introduced an artificial task for the second half of the trial. Each participant was given three questions, and the group was asked to answer these questions together. Some questions were designed to provoke open-ended discussion and interaction between the participants. For example, participants were asked “What is the group’s favourite Mackintosh painting?” and “What contribution has Mackintosh made to Glasgow?” as well as more factual questions such as “What was Mackintosh’s birthday?” This combination of open and task-centred behaviour allowed us to study activity that was typical of a museum visit, such as finding exhibits, and to observe how the system supported the shared aspects of visiting a museum. During the trial, use of the system was heavily logged and each Dub was video taped. After the trial, the participants were interviewed as a group in a recorded semi-structured debriefing. For analysis, we combined the map view used by participants, with the video and audio recordings. We analysed transcripts of the post-trial debriefings, and the logs of the visitors’ use of the system. We paid close attention to the details of how users interact with each other and with technology, especially through video analysis. With particular interest in the use of location and of exhibits, we looked for “critical moments” where the system was used in a way that would let us reveal design lessons, consistencies and inconsistencies with theory, and comparisons with earlier studies. The participants engaged in rich social interaction around the hybrid or coupled exhibits. When participants found that objects corresponded in this way they quickly were able to move on to using them in their shared tasks and activities, for example discussing the qualities of the object and comparing it to the other exhibits. In the following extract the participants discuss a set of Mackintosh pictures to decide which one they like the most. Square brackets (e.g. [pet]unias) show overlapping talk and italics shows a speaker’s emphasis. Vee: Petunias is errm better for me than Rosemaries Ana: Ok [pet]unias Dub: [hhh] Petunias it is Vee: Early work Ana: Hey guys see: this other one it’s really nice. It’s called Fort Mailly hhh Fort Mailly in nineteen twenty seven Vee: Nineteen twenty seven Ana: Yeah, it’s got the light
83
Inhabited Information Spaces
Vee: Yeah I know but I like [Roses] Ana: [Can you] see it? Dub: Fort Mailly? Ana: Hmmmm Dub: Yeah that’s quite nice Vee: I still prefer Roses
The visitors do not focus on the system or the media involved, such as the differences between the digital and the printed reproductions of the Fort Mailly painting. Instead they focus on its aesthetic qualities and the task of deciding which picture they like most. They don’t focus on the tool but on the task. However, interacting around these hybrid exhibits was not without its problems. In ordinary face-to-face interaction, we assume a degree of commonality in the objects that we can see, hear, touch and so forth. Each user of our system had to build an understanding of the perspectives the other users had of the Centre. They could often see similar things, or related things, but not the same things from the same perspective. Dub and Vee frequently guided Ana to specific exhibits verbally. We designed the system so that Ana did not have an overview map, and therefore might find that occlusion of objects in the exhibit was a problem. In turn, since Dub and Ana could shift attention between artefacts placed far apart in the room much more quickly than Vee, they frequently found information and then guided Vee to the corresponding location. Again this was often done verbally, but more spatial or graphical guiding is discussed later. Participants also made use of shared location and orientation, using icons on an outline map for Vee and Dub, and avatars in Ana’s 3D display. Shared awareness of location also allowed users to quickly move to their friends, and to quickly find or confirm the exhibits being discussed, that is, to quickly find what their friends were looking at and then move so as to look at the same or a closely related thing. They developed simple means to gesture. In one case, Ana moved her avatar back and forth in a “wiggle” so as to confirm which icon represented her and to show her location to Dub. This gesture was something like the wave of a hand used by someone to show he or she is in a crowd. Global location could be seen “at a glance” on the map, without the need for the visitors to use talk, but examples such as Ana’s wiggle show participants’ awareness of how they would be perceived by each other in different media. Participants learned about each other’s perspectives through questions and observations, building understanding of what they shared and what they did not, and thus how to more smoothly interact with each other through the resources at hand. Indeed, participants put considerable effort into designing their interactions to take into account the characteristics and limitations of their varying views of the Centre. For example, the above extract shows Vee emphasising and confirming the year in
84
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
which “Fort Mailly” was painted (“nineteen twenty-seven”). One of the largest exhibits in the Mack Room is a long wall of panels, with each of the chronologically ordered panels showing a number of images associated with a year of Mackintosh’s life. Mentioning a year to Vee would let her move quickly to a corresponding part of the exhibit. The information presented to Ana and Dub about this wall was broken up into a page for each year, so any one of the visitors could help guide the others to a particular image in that exhibit by specifying the year. The use of our system involved more talk, and louder talk, between the co-visitors than we observed in conventional museum settings. For example, during a post-visit discussion two participants were asked: Q: Is it different to a museum visit? A: Yeah, it’s really talkative. B: You kind of go “Mmmm, that’s nice” [. . .] If you find something interesting, you go “Look”, and “That’s over here”
Another commented on being able to talk without disturbing others: I quite enjoyed the social engagement . . . being able to talk about everything more and not feeling that you are disturbing. Not thinking about other users in the gallery, you know it’s kind of liberating . . .
I offer here a few of the many possible reasons for this, stemming from the use of multiple rooms, the audio hardware used and, perhaps more interestingly, the set of media available for interaction. Two of the participants were in rooms other than the traditional exhibition room, and so were less influenced by the normal hushed reverence adopted inside museums. All three participants were engaged in a trial of an unfamiliar technology, without established social norms for speech levels. Vee often talked more loudly and paid less attention to other members of the public in the Mack Room. Much like a telephone, the loudness of speech needed for audibility through the audio hardware may not be the same as in faceto-face interaction. Lastly, the participants often used shared audio because the coarse-grained representations of position and orientation did not afford familiar use of gesture and posture. However, participants were not solely using talk in order to circumvent the tools given; they were often talking while engaged in the tasks. Also, they developed gestures for the media at hand, for example Ana’s wiggle: a gesture made in a 3D VR to be seen on a 2D map. In everyday face-to-face situations, interaction between two people can be impeded for a number of reasons such as one or both of the participants choosing to interact with others (e.g. on the telephone) or to interact with objects, or by participants being forced to interact with other people or objects because of interruption, breakdown, occlusion and so forth. In the trial, interaction would often pause when participants found a difference between the visitors’ representations of the Centre. For
85
Inhabited Information Spaces
example, the room’s interactive video displays were only available to Vee. When a visitor started to use and talk about a display or exhibit that was not shared, the other participants would refrain from interacting and move on to other exhibits. Similarly, Dub and Ana generally used movement on the map or VR in order to access information, in preference to the more conventional hyperlink navigation. This seems to have been partly due to the fact that such spatial movement was more a part of the shared experience, in terms of conversational references to locations but also as a way of avoiding future confusion: following links to a web page about a new exhibit did not move the participant’s icon or avatar to the corresponding new position. (This capability was implemented during the trial period but, to maintain consistency, was not deployed.) This meant that “web movement” could leave a visitor’s icon in a potentially confusing place. While there were occasional interactional breakdowns, they were not fatal for the sense of a shared visit or for interaction. Overall, participants showed skill in finding ways to handle the differences between the different representations, and exploiting corresponding and coupled features. The system successfully supported a shared experience by enabling users to talk about and interact around the exhibition, offering a socially engaging experience beyond that available to a conventional web site visitor.
5.5 Ongoing and Future Work We continue to explore remote collaboration in cultural information and cultural institutions, in particular collaboration involving heterogeneous media. We support social context as a resource for the interpretation of information, and contextual information as a resource for social interaction. We are extending our system to be used in more of the city than The Lighthouse, adding GPS (Global Positioning System), dead reckoning, GPRS (General Packet Radio Service), and further 802.11 aerials. We are using the publicly available VRML model of central Glasgow from the University of Strathclyde, and 2D maps from services such as EDINA (www.edina.ac.uk). We have been undertaking field studies of visitors in a range of locations in Glasgow, seeing how their visits include far more than traditional cultural institutions, and how they use resources such as tourist information centres, maps, guidebooks and signage. We aim to run another field trial that involves a number of participants visiting the city, each of whom has a wearable computer on which tightly coupled 3D VR, 2D maps and hypermedia can be used. Each participant’s activity will be available to the others synchronously, much as in the Mack Room, but also asynchronously. One reason for this latter development is to move beyond the traditional objectifying or “scientific” systems of classification and retrieval
86
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
that too often are the only means of access to digital information. Influenced by the theoretical standpoint outlined in Section 5.2 and the Recer recommender system (Chalmers et al., 1998), we have built a central information resource, connected to EQUIP and thus accessible via a variety of media and devices, and which stores a growing and evolving body of individuals’ paths or narratives through a range of symbols: our own images and fragments of hypertext, annotations made by users, locations in the city, and other locations and web pages worldwide. The system uses this resource to make contextually specific recommendations of people, places and things by comparing each person’s recent activity with similar sections of the past activity of selected others. Paths can also be more directly shown on maps, in VR models and woven into web pages. We will allow this body of information to grow as people use it, making new associations between symbols and adding in new ones. Information access based on evolving inter-subjective patterns of contextual association and use will complement access based on more static and objective interpretation. We are also experimenting with bridging between the two, using pre-written “official” explanations of the connections between symbols as a means to enrich the dynamically created recommendations. Remote collaboration brings abstraction and approximation as a system monitors and senses activity such as a person walking across a city. Issues such as sampling, resolution, delay, disconnection and uncertainty have to be faced as one decides how to represent the activity inside a system, even before one considers how to represent the activity to a remote collaborator. There is no getting away from the fact that activity is going to be interpreted through sensors and transducers such as cameras, GPS and ultrasonic systems, and any digital medium has characteristic losses and uncertainties. While we want to make new resources for interaction available to people, combining new media with old in a perfectly seamless way is not going to happen. Uncertainties and inaccuracies (“seams”) are an inherent part of any communicative medium, and people often learn to use these characteristics for their own ends. For example, mobile phones can be set to display the current cell, if the service provider permits, and some people choose to enable this facility. This is an elegant ambient or peripheral presentation of potentially useful information: users can choose what use to make of it, for example seeking a stronger signal by moving to a location that forces handover to another cell. Cell boundaries and signal strengths are interactional resources of the medium. Similarly, long-term use of videomediated communication was reported by Dourish et al. (1996) to lead to “complex patterns of behaviour built up around the interactional details of the video medium . . . When the medium changes, the mechanisms change too; but the communicative achievements remain.” Recalling a term used by Mark Weiser (1994), we see seamfulness as an important design goal for our future work. We plan to design in
87
Inhabited Information Spaces
explicit representations of the errors and uncertainties in our systems, letting people take account of the characteristic heterogeneity, errors and limitations of the systems we offer them. For example, we are starting to develop explicit presentations to accommodate uncertainty due to ultrasonic and GPS-based positioning, showing a person’s sensed position as a spatial extent, rather than as a point, and showing estimates of sensing accuracy and communications bandwidth on our city maps and models.
5.6 Conclusion The City project emphasises the interdependence of media, such as computer graphics and audio, with others. We have explored the combination of CVE technology with hypermedia and mobile computers, and also with the architecture and exhibits of the Mackintosh Interpretation Centre. Supporting broad social context in remote collaboration involves heterogeneity, and our project aimed to address this through coupling and correspondences between media. Users of a mixed collection of interactive media were able to enjoy a shared visit experience, engaging in collaboration through awareness of each other’s activity and through more focused talk and interaction around “hybrid” objects. By presenting the theoretical issues underlying much of this work, as well as system design and experiences of use, this chapter may serve to complement many other chapters of this book. Rather than considering that users inhabit our information space, we see people as inhabiting cities and towns, and using new technologies and older media to interact with friends, relatives and colleagues. Looking to the near future, CVE technologies will be widely available via phones and mobile computers. I suggest that taking fuller account of their use among a wider set of technologies and media, and designing for contextuality, heterogeneity and seamfulness, will greatly enrich our work.
Acknowledgements Special thinks go to all the City project members past and present, especially Barry Brown, Areti Galani, Chris Greenhalgh, Ian MacColl, Dave Millard, Cliff Randell and Anthony Steed. Also, we are all grateful for the generosity of our hosts at The Lighthouse, especially Lynn Bennett and Stuart MacDonald. Thanks also to Ziggy Stardust, Fugazi and Low. Equator is an Interdisciplinary Research Collaboration (IRC), supported by the UK Engineering and Physical Sciences Research Council, and the City project was supported by a donation from the HewlettPackard Art and Science programme.
88
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Soundscapes
6
Tony Brooks
011
011
011
011
11
6.1 Introduction Although purely virtual information spaces receive much attention and are indeed useful in certain situations, it can be difficult to integrate them with real-world activities and awkward to interact with them. As pointed out in Chapter 5, even notionally single-user activities often happen together with other people and a purely virtual environment tends to cut people off from their surroundings making it difficult for several colocated people to share an experience. Fraser et al. (Chapter 9) also show that collaborative interaction within a CVE is not quite as intuitive as it might first appear. For these reasons purely virtual environments may not be desirable in cases where the participants have a disability or where the environment is intended to be shared with other people, as in a public performance . The creative process is something that is inherent in most people, even the severely disabled, yet it is an often uncharted channel within which to explore their expressive human potential. “Productive creativity” is becoming recognised as a beneficial therapeutic treatment for people with disabilities. In this chapter the Soundscapes system, which is built to allow unencumbered creative expression for both able-bodied and handicapped people, will be discussed. The next section will describe the system itself and then further sections will show how it has been used therapeutically and also to stage public performances.
6.2 The Soundscapes System The origins of the Soundscapes system lie in a MIDI bass (a modified bass guitar that can generate MIDI – Musical Instrument Digital Interface – signals allowing the control of electronic synthesisers) and an expression pedal (a foot-controlled sound effects processor that can change the sound of an audio signal). From interactions with his severely
89
Inhabited Information Spaces
handicapped uncle in which the author played the bass and his uncle manipulated the expression pedal the author realised that even a simple means of expression could open up ways for people to communicate and could provide immense satisfaction. The author started searching for a richer means of expression and it soon became clear that the MIDI bass itself was too intimidating and required too much physical skill to be satisfying for people like his uncle. This led to a search for a rich expression medium that required no training to use. Although immersive reality hardware could track peoples’ movements in a natural way it was expensive and uncomfortable to use to the equipment since it had to be physically worn by the user. After much experimentation the author selected infrared movement sensors. Within the range of the sensor, movement from the flicker of an eye to a full body movement can be detected. The Soundscapes system uses three such sensors to allow triangulation of movement or triggering of three separate operations. Each sensor head is mounted on a flexible “gooseneck” support to allow for different configurations. The
Figure 6.1 Photograph of the first three-headed infrared movement sensor used by the Soundscapes system.
90
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
Soundscapes 6 ●●●●●●●●● ● 1 ●●●●●
sensors output MIDI information that have been used to control movement and navigate through 3D space as well as controlling filters applied to computer-generated images. The Soundscapes system is formed from a library of tools for body function capture and a number of programs for creating the results that each user response determined. The library of input tools ranges from biofeedback sensors for brain activity, muscle tension, GSR (galvanic skin resistance), heart rate, to infrared (see Figure 6.1), ultrasound and video tracking (the EyesWeb http://www.eyesweb.org/ is used to perform the video capture) technologies. Most often the captured body movement data is transmitted using the MIDI protocol and the MAX program (http://www.cycling74.com/products/maxmsp.html) is used as a central hub directing and manipulating the data as needed. In conjunction with a physiotherapist the Soundscapes operator calibrates the system such that the range of velocities measured by the sensors matches the movement capability of the individual who will be participating in the session. The system is configured so that movement triggers audio and visual feedback with the exact nature of the feedback depending on the individual – we have found that some individuals respond better to audio stimulus and others to visual stimulus. System settings as maintained between sessions and movement data are logged so that the therapist can measure progress from session to session.
011
011
11
Figure 6.2 Version two of the three-headed infrared sensor system in use. The three sensor heads are visible at the centre of the image in between and a little below the performer’s hand and the image projected on the screen.
91
Inhabited Information Spaces
6.3 Therapeutic Uses of Soundscapes One of the uses of the Soundscapes system is to act as an “expression amplifier” for people with physical disabilities – it provides them with the possibility to generate sounds and images from whatever movement they are able to control and this can be a powerful therapeutic experience. The therapeutic use of Soundscapes is illustrated by the i3 funded future probe project called The World Is As You See It (TWI-AYSI, http:// www.bris.ac.uk/Twi-aysi/). TWI-AYSI was a successor to the CARESS project, which was also funded by i3 as part of the Experimental School Environments programme. The TWI-AYSI project members consisted of members of the Bristol University Electrical Engineering Department, and Stefan Hasselblad both of whom were involved in the EU CARESS project, and Tony Brooks. The CARESS project successfully motivated and empowered children to develop creativity, imagination and expression through interactive acoustic environments. The objective of TWI-AYSI was to answer the question “Can immersion in a visual environment hold similar potential for such children in terms of the aesthetic resonance they might derive from movement within such a visual space?” During the project we brought young children from a school for multi-handicapped people into the Centre for Advanced Visualisation Projection screen Video camera
Full scene camera
Infrared motion sensor
Video projector Disabled child Infrared motion sensor Video camera
Computer
Face shot camera
Camera feed quad splitter monitor & VCR
Figure 6.3 Typical set up as used in TWIAYSI (Sweden 2001) with Multiple Camera Analysis (MCA) utilising six video cameras and three infrared motion sensors.
92
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Soundscapes 6 ●●●●●●●●● ● 1 ●●●●●
and Interactivity (CAVI, http://www.cavi.dk) in Denmark. The children ranged in age from three-and-a-half to five-and-a-half. A common experience for all of the children involved was the manipulation of sounds, robotic lights and coloured images. However, in CAVI we also wanted to explore the possibilities for using the movement sensors to enable 3D navigation. MIDI signals from the movement sensors were received by a Linux workstation and translated to movement information that was used to control the rendering of the 3D world on a SGI Reality Monster (Silicon Graphics Inc.). In this experiment, the movement sensors allowed a person to control the movement of a rocket ship that was projected onto a large screen in front of them. It was possible to view the image in stereo using LCD shutter glasses . However, the children did not want to wear the necessary head sets and shutter glasses that were used to view the virtual objects and so we made the decision to project the ship in mono and the youngsters once placed with their head inside of the active sensor space, were able to control the various degrees of movement of the spaceship by small head gestures (a video of this is available online at http://media.nis.sdu.dk/video/ twi-aysi.html). A small gesture to the right and the rocket ship moved to the right, a movement left and the rocket ship moved to the left, a head movement down and the ship’s nose tipped down as if to dive, and a head movement backwards and the ship’s nose tipped up as if to climb. As an example of the results of this experiment we shall describe a session with a multi-handicapped five-year-old boy. At first we tried to place the 3D shutter glasses on him so he could view the scene in stereo but he shook his head to remove them. We removed the glasses and changed to a mono projection. This was a big success as the young boy was totally immersed for around six-and-a-half minutes. The young disabled boy was only able to express through a single verbal expression (a sort of “AYYE” phrase). While in the session and immersed in the interaction he could clearly be heard making his sound, which was translated by
Figure 6.4 Freja, a severely disabled girl with her helper Maggie. Freja is painting via her movement and can see herself in the window on the right. The colour painted depends on the velocity of the movement, with each individual having his/her own settings to match their ability. Freja’s face tells the story. Reproduced with permission from Stefan Hasselblad.
93
Inhabited Information Spaces
his helper as an expression of a joyful experience, and this was probably due to the fact that he quickly understood the functioning of the interactive space and was able to control the image on screen – an experience of controlling his environment that was probably completely new for him. As a result of the CARESS and TWI-AYSI projects we learned that when audio only or visual only feedback was present some people reacted better to audio and some better to video. When both audio and video were present in the environment a greater degree of immersion was observed. This work resulted in another European project to take these ideas further, called CARE HERE (http://www.bris.ac.uk/carehere), whose results are due to become available as this book goes to press.
6.4 Artistic Performances Based on Soundscapes 6.4.1 Interactive Painting The inspiration for “The Interactive Painter” came from a discussion at a party in which it was stated that some people believed that painting on canvas was “dead” since computers enabled us to do so much more with Video camera Projection screen Infrared motion sensors Back projected image
Mirro
Painter
Video camera Computer
Video camera
Video projector
Figure 6.5 Diagram of the equipment used in the interactive painting performances. The three sensors capture motion via infrared beams. In this diagram the video image is back projected but front projection is also an option to allow working by casting shadows and to overlap the video on a canvas.
94
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
Soundscapes 6 ●●●●●●●●● ● 1 ●●●●●
images and colours. We decided to try to investigate a way for a traditional painter to utilise new technology while painting on canvas in the traditional way. The resulting performance involved the painter Manu Rich from Paris, France and within Tony Brook’s COIL (Circle of Interactive Light) interactive installation, it toured a number of museums of modern art in Scandinavia in 1998 and 1999 culminating at the Danish NeWave festival performed at the Gershwin Hotel, in Manhattan, New York 1999. More recently we have experimented with a therapeutic version of the installation for elderly and handicapped people. Interactive painting is designed to be a live experience with an audience and takes place in a darkened room, which provides a challenge to the painter since painting normally requires a well-lit room. The event makes use of projected computer-generated imagery and synthesised sounds that change according to the movement of the painter. In turn the painter may be influenced by the sound and images thereby creating a feedback loop. In order to give maximum freedom of movement, infrared and ultrasonic movement sensors were used to capture the painter’s movement, thereby allowing him to move freely without trailing wires. Owing to the size of the canvas and the limited range of the infrared sensors a jacket with reflective strips attached to it was worn by the painter, thereby allowing movement to be detected from a greater distance. The sensors were strategically set up around the canvas and translated light reflected from the jacket into movement information that a computer translated into sounds, colours and images. One of the primary aims of the performance was to see how the painter would be influenced by images projected onto the canvas. The light from the projected image would alter the appearance of the painted image and change how both the artist and audience perceive it. In order to do this a video projector is set up to project images onto the canvas. The movement sensors control filters applied to the image such that when there is no movement there is no image. As the painter moves then portions of the image are projected onto the canvas.
011
11
Figure 6.6 Photograph of an interactive painting session Gershwin Hotel, Manhattan, New York 1999.
95
Inhabited Information Spaces
In addition to the projected image, the painter’s movement also triggers sounds from a sound synthesiser. The author changed the synthesiser’s patch data in real-time to match the “mood” of the composition and also to try and influence the mood of the painter. The audience of an interactive painting performance therefore has many elements to focus on: ● ● ●
The painting that the painter is actually producing on the canvas. The movements of the painter, which the painter may change in order to change the sound and visuals as well as the painting. The sound and images caused by the movement of the painter.
The artist naturally tends to focus on the creation of the painting and not the multimedia experience. However, it was apparent that the sound and images had an effect on the painter and the painting. From experience we noted that smooth “pad” sounds tended to result in a corresponding smooth motion of the artist’s brush and passive choice of colour and similarly a hard-edged “Hendrix” guitar lick resulted in an aggressive motion and severe choice of colour.
6.4.2 The Four Senses In April 2002 the first collaboration between artist/researcher Tony Brooks and artist Raewyn Turner resulted in a series of multisensory performances called The Four Senses in Auckland, New Zealand, with the Aotea Youth Symphony, Touch Compass, (a mixed able-bodied/ handicapped dance company), and HANDSS (Hearing ANd Deaf Sign Singers, a deaf signing choir). The performances were an improvisation in light, sound and olfactory information. Brooks and Turner created a real-time translation of sound and the gestures of making that sound, into light and colour and multiple layers of smell. The concerts aimed to engage and reframe perception of music and to play with subjective experiences and simulated synesthesia. Each sensory element was constructed from information relating to the other elements. The associations and correspondences of the elements made by the audience were according to their own individual and personal experiences. Tony Brooks utilised movement sensors and video cameras to capture body part movement and translate it into painting with coloured light. In this way the orchestra conductor was able to “paint” the scene through his gestures. Similarly, orchestra members, dancers and a special signing choir for the deaf images were blended into the backdrop in real-time such that their velocity of movement affected the colour of image generation and collage composition.
96
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Soundscapes 6 ●●●●●●●●● ● 1 ●●●●●
Projection screens Percussion / Timpani
Primary camera capture areas
011
011
011
011
Lead cello
Lead violin
Deaf signing choir Basses
Singer Conductor Dancers
3 × roving video cameras
Figure 6.7 Diagram showing the locations of performers and projection screens for the Four Senses performance.
Raewyn Turner translated the music through drawing and proportional relationships into coloured light and olfactory stimulation. This was done using computer-controlled lighting rigs and by releasing aerosols into the ventilation system at key moments during the performance. The light collage thus created was a play of interaction between live video feeds and sensors, and coloured light pre-programmed to an interpretation of sound, each affecting the other in a dynamic visual loop. The fourth sense employed was touch. In order to give hearing impaired people a feel (pun intended!) for the concert experience vibrating cushions were used. These cushions were developed as an accessory for the Sony Playstation® and can transmit vibration at various levels of intensity. These cushions were coupled to the PA system so that they would vibrate according to the music played by the orchestra and a number of them were distributed throughout the auditorium. Hearing impaired people could sit on the cushions or hug them to their chests in order to get a “feel” for the music. In addition a deaf signing choir participated in the performance to counterbalance the conventional (but sight impaired) singer also performing on stage.
11
97
Inhabited Information Spaces
a
b Figure 6.8 Two photos of the Four Senses performance: (a) dancers in front of the orchestra; (b) an example of the projected video generated during the Four Senses performance. Reproduced with permission from Milan Radojevic annd Mirjana Devetakovic Radojevic.
98
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Soundscapes 6 ●●●●●●●●● ● 1 ●●●●●
6.5 Conclusion We believe that there will one day be a programmable ion layer surrounding our body which will be able to stimulate our individual senses so that we are enabled towards a truly augmented reality without wearables and special hardware such as screens. This is not pure fantasy since a number of people in the Nanotechnology community such as Storrs Hall (http://discuss.foresight.org/˜josh/UFog.html) and Drexler (1992) are exploring methods that would allow the creation of a “Utility Fog” (Storrs Hall) that would, among other things, allow our senses to be stimulated and our movement sensed. Our human-centred work with people with special needs, who are disabled, elderly or in rehabilitation allows us, in certain circumstances and with certain individuals, to get closer to higher nuances of the senses and what the senses mean. By exploring that research and working in crossdisciplinary teams including neuropsychologists, computer scientists, human–computer interaction (HCI) researchers and others, we hope to work slowly towards that goal. We believe that the immersive “play” within an interactive environment has much potential to do good. Specific to the current work is the belief that everyone is so individual that a system is required that can be tailored to each desire, facility and requirement – this entails adaptability with a capital A. Libraries of input HCI devices, together with libraries of mapping devices and libraries of output software is the optimal way forward. Through non-fixation on interactive spaces through invisible system components – hardware (disappearing computers, sensors embedded in environments etc.) – we will obtain the mapping of body function subconsciousness that can help people. The subliminal pervasive aspects – or really more so terms such as “proactive computing” (that which focuses on improving performance and user experience through speculative or anticipatory actions) and “autonomic computing” (which focuses on improving user experience through the system’s self regulation) in relation to this work are preferred as both relate to the user experience rather than the artefacts often referred to in pervasive computing – are obvious. We also believe that responsive audio/visual/haptic feedback may have much more to offer in the future over and above what we now utilise, in fact we believe that correspondences between synchronised feedback especially sonic and visual (in the first instance) are only just scratching the surface and that there are many new discoveries waiting to be made.
Acknowledgements The CARESS and TWI-AYSI projects were funded by the European Union via the i3 network.
99
This page intentionally left blank
011
011
011
011
11
The Computational Interplay of Physical Space and Information Space
7
Enric Plaza
7.1 Introduction There is a current trend in computer science to develop new devices and applications that make the transition from desktop (and laptop) computing to computing devices that are embedded in the physical and social environment in which people live. Several approaches have been proposed in this direction and they have different names and focus on related but distinct issues. The first, known as ubiquitous computing, pervasive computing or the disappearing computer, focuses on embedding computing devices into the physical objects and surroundings where people work and live. A second related trend is that of wearable computers, that focuses on embedding personal computing services in devices that people can carry or wear while moving around in their everyday activities. Next, augmented reality focuses on enriching people’s perception of physical surroundings with computer-generated information. And finally, to be brief, there is a trend for developing autonomous agents that take on people’s goals and try to achieve them on their own. However different these approaches are, a common issue they all have to deal with is the awareness of the physical (and social) surroundings in which people interact with computing devices. Traditionally, computers (ranging from mainframes to personal computers) operate in a purely informational world – typically, screens and printers, plus a customised connection to manufacturing machines or task-specific sensors. The advent of the Internet and the World Wide Web links these computers into a common (or rather shared) information world. The relation of the Internet, as an information space, with physical space is a research issue that essentially deals with the issues of context awareness: who is where and when, with whom, doing what. This chapter focuses on the interplay of physical space where people act and live and
101
Inhabited Information Spaces
an information space where software programs reside, interact among them, perceive some properties of the physical world, and perform some tasks and actions on both the informational and physical spaces.
7.2 The Interplay of Physical and Information Spaces Since all computers, and the software they run, are potentially connected over the Internet, we can consider this as an “information space”. Then, the computing devices that populate the physical world, from personal digital assistant (PDAs) to the emerging “ubiquitous computing” devices, can be considered as the interface between the physical spaces people inhabit and the “information space” inhabited by software programs. The most critical issue to improve this interface is that nowadays software programs have little or no awareness of the physical space and of the activities in which people engage in that physical space. In this chapter we will first discuss the general issues that need to be addressed to improve the interfacing of physical spaces by the “information space” inhabitants, in particular awareness of the physical and social context of people. Later, we will present the COMRIS project and explain how these general issues are addressed and solved for a specific physical space (a conference centre) and a series of informational tasks useful in that space. Awareness of physical space involves more than merely spatial or geographic reasoning. People perform activities and interact with other people while moving in physical space, and the more aware they are of those activities the better the interface with the information space. Two levels of context awareness are required: 1. Physical sensors, determining the granularity of the perception of physical space activity. The physical sensors can be a Global Positioning System (GPS – giving a co-ordinate point); wireless tags that people wear to detect who is close by or in which room; or even more sophisticated speech capture and analysis systems (e.g. trying to determine topic, mood, etc., of ongoing activities). 2. Common sense knowledge, determining the inferences a system can make on the “world situation” given the physical sensor information. For instance, if a wireless tag indicates that a person is in a room of type Meeting Room, there is a series of inferences that can be made from a knowledge base that models business activities (e.g. the person is in a business meeting with other people and should not be interrupted unless there is something urgent). Perception using physical sensors establishes the baseline for awareness capabilities. For instance, GPS can be used together with personal devices like PDAs to yield information customised to the person (using the personal profile in the PDA) in that spatial situation (by a server that
102
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
The Computational Interplay of Physical Space and Information Space 7 ●●●●●●●● ● 1 ●●●●●●
has a database of services available near that location). However, this approach is very centralised, depending on a service provider holding geographical information. For some tasks it is better to add more sensors that allow the person’s behaviour to be determined: a microphone can be used to determine if the person is busy (talking with somebody) or idle. Currently, project Esprit “IT for Mobility” 26900 is developing a sensor board that could be integrated with mobile phones or PDAs. The sensors on this board include two microphones, a dual axis accelerometer, a digital temperature sensor and a touch sensor. With them, a computing device can locally infer different contexts where this device is situated – such as “sitting in a pocket”, “lying on the desk”, “in user’s hand” – allowing the device to adapt its behaviour to each context. Even discussing the physical sensors used we have had to include the idea of a computational model that transforms the raw input data into some interpretation of the state of affairs in the world. This is because context-awareness is essentially an interpretation of the world situation and, as such, what is needed is plenty of knowledge about what the world is like, in other words, common sense knowledge is needed. There has been a lot of research in artificial intelligence (AI) in the last 10–12 years, concerning the issue of common sense, with the Cyc project (http://www.cyc.com) being the most well-known endeavour. The current understanding of common sense can be summarised as: 1. an ontology, defining the objects existing in the world that we want to talk about, and 2. an inference engine, capable of using a model of the state of affairs in the world expressed in that ontology to conclude new facts or statements about that state; new facts that are “obvious” or “implicitly” known by people, by what we are calling “common sense”. Moreover, context aware applications need to have some properties that differ from current applications: they need to be persistent, responsive and autonomous. We will call this collection of properties continuative computing1 because they set apart context-aware applications from the usual applications oriented to input–output. First, a context-aware application needs to be persistent, that is, persistently in runtime state, non-terminating. Commonly, an application accepts input and produces output (in fact, the same definition of algorithm is based on the idea of transforming an input to an output) – an exception is programs that are operating system services (that are difficult to model in the algorithm paradigm based on termination). A common application is a file that, when a user needs it, becomes runtime, receives an input and after some processing time yields a result and then goes offline. 1
Continuative: tending or serving to continue.
103
Inhabited Information Spaces
A context-aware application needs to be non-terminating, awake and running persistently, much like an operating system or a PDA. Moreover, it needs to be persistent in order to be responsive: able to adapt and produce adequate responses when something changes in a context or the context changes to become a new context. Finally, context-aware applications need to be autonomous in the sense of having an identity persistent in time and a memory (or internal state) that is individual. Since changing context is one of the most important pieces of information a context aware application can deal with, it makes no sense that each particular physical space has a context-aware application that is independent from other locations. Since people move around it is better to think of a context-aware application as centred on users, like a PDA that a user carries around. In this way the context-aware application can know the past history of contexts of the user, and even learn to anticipate the most likely future contexts of the user and prepare for it. There is a current paradigm and associated technology that fits these requirements, which we have outlined as continuative computing: intelligent agents. Agents and multi-agent systems being developed in artificial intelligence are conceived of as autonomous, permanent entities capable of using ontologies to perform inferences for solving problems, and for co-operating and/or competing with other agents or people. Intelligent agents are classified as reactive or deliberative, depending on some design properties, but we will show later in the chapter that designing agents with a particular architecture like the one proposed, means they can also be responsive and thus exploit awareness of physical and social context to improve their performance on behalf of users.
7.3 A Framework for Context-aware Agents Our framework is composed of a collection of context-aware personal information agents (CAPIAs) working in an information space and a collection of human users interacting in the same physical space. A useful way to visualise this distinction is the dual space schema depicted in Figure 7.1. Human users, on the right-hand side of Figure 7.1, are in a location, interacting with other persons (who may or may not be users) in the context of social activities. Information agents, on the left-hand side of Figure 7.1, inhabit an information space where they interact with other agents and gather information in the interest of the users. Moreover, we have mediation services connecting the information space of agents and the physical space of human users. Specifically, we are currently using two mediation services, namely an awareness service and a delivery service (see Figure 7.1).
104
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
The Computational Interplay of Physical Space and Information Space 7 ●●●●●●●● ● 1 ●●●●●●
CAPIA
CAPIA
CAPIA
CAPIA CAPIA
Delivery services
Awareness services
Information space
Physical space Social activity
Figure 7.1 A schema of the dual space of information agents and human users with the mediation services between them.
7.3.1 Awareness and Delivery Services The awareness service takes charge of pushing information from the physical space to the information space. Specifically, the awareness service provides real-time information about the physical location and movements of users to CAPIAs. The specific data provided depends on the particular sensors available in the awareness service for a particular application. For instance, in the conference centre application the awareness service provides a real-time tracking of attendees’ location as well as the group of other attendees nearby a given attendee – see Section 7.4 for the features of the awareness service in the COMRIS Conference Centre application. The delivery service offers mediation and brokerage capabilities (subscribed by the human users) for delivering information from the information space to the physical space. Specifically, the delivery service provides the channels for delivering the information gathered by the CAPIAs to their corresponding users. For instance, in the conference centre application the delivery service allows information such as audio output to be sent by means of a wearable computer and HTML pages by means of screen terminals scattered through the conference building.
011
11
7.3.2 Agents Requirements The society of agents has to be able to communicate using a common ontology for a specific application, and they have to share a collection of interaction protocols appropriate for that application. Our approach is to use the notion of agent-mediated institution (Noriega and Sierra, 1999)
105
Inhabited Information Spaces
to specify the ontology and interaction protocols to be used by a society of agents for a particular application. In addition, to support the ontology and interaction protocols of an agent-mediated institution the agents should be able to handle contextawareness information. That is to say, a context-aware agent should be able to react dynamically when new physical context information is received from the awareness service. Moreover, since the future physical and social context of the user is not known, a desired feature of CAPIAs is the capability of gathering information that may become relevant in a future context. For instance, in the conference centre application, when an attendee is at a specific exhibition zone, the CAPIAs use the knowledge provided by the conference about the physical distribution of booths for trying to anticipate the next movement of the attendee. In our framework, CAPIAs are based on the distinction between two kinds of information valuation, namely interestingness and relevance. Information interestingness measures the intersection of a given piece of information with the user model a CAPIA has for the tasks with which it is charged. That is, interestingness: Info × UM |→ eI where Info is a given piece of information; UM is the user model; and eI is the estimation of the interest that the user has in Info. For instance, in the conference application a preliminary criterion for determining the interestingness of a given paper presentation is performed comparing the user interests (described as a collection of topics with different weights) and the keywords associated with the presentation (also described as a collection of topics with different weights). Then, other criteria such as the knowledge about the speaker or the user’s agenda increase or decrease the initial assessment. Depending on the physical and social context of the user and on the time, however, some information may be more or less relevant for the user at each particular point of time. Information relevance measures this intersection of a given information with the time and the context of the user. That is, relevance: Info × Time × UC |→ eR where Info is a given information; UC is the user context; and eR is the estimation of the relevance of Info in UC. For instance, in the conference application when an attendee is near an exhibition booth, the information related to the booth is estimated as more relevant. Another example of increase of relevance is when a conference event is close to start: a CAPIA has a time constraint for deciding if that event is useful for the user interests. We can say that, basically, the CAPIAs go on their tasks, interacting with other agents (and other accessible information resources), to gather information that is interesting for their users. Concurrently, each CAPIA uses an awareness service (see below) to keep track of the whereabouts of its user and decides which information is relevant for the user in a particular physical and social context. Clearly, interestingness and relevance are not completely independent, and the information gathering is correlated with the information the agent expects to deliver to the user, but for exposition purposes it is handy to talk about them separately.
106
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
The Computational Interplay of Physical Space and Information Space 7 ●●●●●●●● ● 1 ●●●●●●
At this point it is useful to put this framework into a concrete application to illustrate the dual space and the personal information agents’ exploitation of context awareness.
7.4 The COMRIS Conference Centre This section applies the framework of context-aware information agents to a particular application, that is to say in the physical location and the social activity context of a conference centre in the COMRIS project.2 We view the conference centre (CC) as an agent-mediated institution where a society of information agents work for, and are aware of, the attendees of a conference (Plaza et al., 1998). The ontology of the CC institution defines the conference activities that take place in the CC. Examples of conference activities are exhibition booths and demo events, plenary and panel sessions, and so on. The ontology also defines the roles that a person takes in different locations while performing different activities, such as speaker, session chair, attendee, organisation staff, etc. Other important elements defined by the CC ontology are the different locations of the conference such as the exhibition areas, the conference rooms, and the public areas – i.e. halls, cafeterias and restaurants. This information is used by the agents for reasoning about the movements of users in the conference. The schedule of conference events is also defined in the CC ontology. Finally, the CC ontology supports the definition by each user of the “instruction set” that their CAPIA should follow. The instruction set is entered by the conference attendee using a WWW browser while registering, and basically includes (1) an interest profile (specifying the topics, with weights, in which the attendee is interested); (2) those tasks the user commissions the PIA to do on their behalf (e.g. if they are interested in making appointments); and (3) the delivery modes that the CAPIA will use to communicate with the user. We implemented two types of CAPIAs in the CC application: CAPIAs representing interests of attendees and CAPIA advertisers. There is a CAPIA for each attendee, a CAPIA advertiser for each exhibition booth, and a CAPIA advertiser for each paper session. The goal of CAPIA advertisers is to convince people to attend the conference event they are representing.
7.4.1 Delivery Service The delivery service in COMRIS allows the users to receive information in two ways: by means of a wearable computer with text and audio output 2
COMRIS stands for Co-habited Mixed-Reality Information Spaces. More information is available at: http://arti.vub.ac.be/~comris/
107
Inhabited Information Spaces
and by screen terminals scattered throughout the Conference Centre. The wearable computer is used to convey short messages that are relevant for the user with respect to their current physical and social surroundings. The user can walk to a terminal if they wish to have more information about this message or other recent messages they have received. When the user approaches a screen the wearable computer detects this terminal’s identifier, and then sends this identifier to the user’s CAPIA. Once the CAPIA is aware of this situation, the agent sends to that screen the report of the performed tasks and the report of ongoing tasks. The delivery service comprises several components. The first component is the natural language generation (NLG) component. The NLG component receives the message sent by a CAPIA and generates an English sentence explaining the message content and taking into account the current attendee context and the sentences previously generated. Then, when the message has to be delivered as audio, the sentence structure is sent to a speech synthesis component that produces the actual audio heard by the user. Similarly, there are components that transform CAPIA’s messages into HTML or VRML in order to be delivered to the screen terminals.
7.4.2 Awareness Service The awareness service keeps track of the whereabouts of the attendees in the Conference Centre. In the COMRIS CC the detection devices are a network of infrared beacons (marking the different rooms, places and locations in the CC) and the wearable computers carried by the attendees. The COMRIS wearable computer (shown in Figure 7.2 and commonly called parrot) detects the infrared beacons and thus informs the
Figure 7.2 The wearable computer also known as “the parrot”. The CPU is on the front unit while the back one hosts sensors and batteries.
108
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
The Computational Interplay of Physical Space and Information Space 7 ●●●●●●●● ● 1 ●●●●●●
awareness service of the location of its user. Moreover, the wearable device possesses an infrared beacon, allowing the detection of other persons, wearing a parrot, located nearby. In order to have access to this information, each CAPIA in the information space “subscribes” its user to the awareness service. As a result, the CAPIA receives messages about the changes in location of that person and a list of other people close to that person. When the CAPIA interacts with other CAPIAs (representing other conference attendees), and decides that those CAPIAs are interesting persons, it subscribes those persons to the awareness service. Consequently, the CAPIA is aware of the location of the most interesting persons for its user and detects, for instance, when one of these persons is in the same location as the user – a most relevant situation to push to its user the information concerning that person who is interesting and nearby.
7.4.3 Tasks The tasks that the COMRIS Conference Centre supports are at the core of the activity in the CAPIAs. It is important to remark here that, in order to perform these tasks, the information agents use both the CC ontology and the awareness service to infer the situation of the user. That is to say, knowing that the user is in a particular place, the current time and the activity scheduled by the Conference for that place at that time, the information agent can infer the social activity in which the user is involved. The tasks performed by COMRIS CAPIAs and the scenes in which they are involved are summarised below. ●
011
011
11
●
Information gathering: this task is responsible for establishing initial conversations with other CAPIAs for estimating the interestingness of the attendees or conference events they represent. We say that the information gathering task constructs the interest landscape of a given attendee. The interest landscape holds all the information considered to be useful for the interest of the attendee and is used and refined in the other tasks. When the information-gathering task assesses a conference event with a high interestingness valuation, the information is directly delivered to the attendee. This delivery strategy was adopted for biasing the future decisions of the attendee. In CAPIA advertisers, this task has been specialised for attracting persons who might be interested into the conference events (exhibition booths or conference sessions) they represent. Appointment proposal: in this task, using the interest landscape, the CAPIAs try to arrange an appointment between two attendees. First, CAPIAs negotiate a set of common topics for discussion (the meeting content). When they reach an agreement, CAPIAs negotiate on the appropriate meeting schedule.
109
Inhabited Information Spaces
●
●
Proximity alert: in this task an attendee is informed that she is physically near to another person with similar interests – or near an exhibition booth or a thematic session with similar topics. Commitment reminder: this task is responsible for checking if attendees are aware of their commitments. The CAPIA uses context to determine that the user may be unaware of a commitment, e.g. if she is not near the location of an appointment (or a bus scheduled to leave) a few minutes before. Commitments of attendees are only noticed when the context information available to CAPIAs indicates that the attendee is not aware of the commitment (e.g. it is five minutes before the starting of a session chaired by the attendee and the attendee is physically in another place).
For each task several activities are launched in the CAPIA. For instance, when an agent in COMRIS is discussing appointments with several CAPIAs, each thread of interaction is managed by a distinct activity. The activities can start either by an internal decision of a CAPIA or because a CAPIA has received a request from another CAPIA.
7.5 Conclusions We have shown a specific context-aware application (the COMRIS Conference Centre) developed in the framework of the COMRIS project that illustrates the interplay of physical space and information space. We have seen that the physical infrastructure, consisting of an individual wearable computer and localisation beacons, was used as a “awareness service” by a society of agents inhabiting the information space. We have focused on the kind of software required to develop a context-aware application, showing that using an agent-based approach we can fulfil the properties we required in what we called continuative computing. We also discussed the kind of agent architecture that can exploit context awareness. The approach we presented was based on the idea of having a personal agent per person. This allows the continuative dimension of processing to be user-centred. As the user changed context, the personal agent received the corresponding perception data from the awareness service and followed the user to a new physical context. In addition to this, services that need to be provided to the users (services provided by the Conference organisation in our example) are agentified, that is, they are also provided by agents (and they also are aware of the contexts in which they are interested). Moreover, this approach is scalable: the agent could use another awareness service at a different conference centre. Moving from one application context to another (from one conference to another) requires a standardisation effort of the awareness services, but this effort is reasonable since it can improve the performance and lower the costs.
110
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
The Computational Interplay of Physical Space and Information Space 7 ●●●●●●●● ● 1 ●●●●●●
We see thus that context awareness can be integrated into an agentbased paradigm in a well-understood way. A better infrastructure on context perception, as we can expect to be developed in the next decade, can be integrated in the agent-based paradigm without major problems. The main reason for this is the AI approach to agents that employ ontologies describing the world. Clearly, with a better perception infrastructure the agents could perform better inferences about the state of the world. For instance, in the COMRIS conference centre the agents knew when two people were in the same room (using beacons) and when two people were in front of each other (using the wearable computer beacons), but since there was no microphone on the wearable computer there was no way to know if the user was already busy talking with someone or not. In addition to improved awareness services and perception infrastructures, a second issue that was considered in the COMRIS project but never tried was that of learning. Agent learning is developing into an active area of research, initially focused on reinforcement learning but it is rapidly broadening. Agents in context-aware applications should be able to adapt to new contexts but also to learn from the user satisfaction (or not) of the agent’s action. However, learning from examples requires a sufficient number of examples to be worthwhile, and the experiments conducted in the framework of the COMRIS project assumed that the Conference only lasted one day – and the amount of data was too sparse to allow significant learning. It turned out that learning would be interesting if personal agents were carried over by the user to different conferences, allowing the collection of examples significant enough in number and variability. Only when awareness infrastructures are more proficient and more readily available (and awareness services are more standardised, allowing agents to continue from one application context to the next) the AI and agent technologies will be able to respond ubiquitously and intelligently to the requests and the needs of people.
Acknowledgements The research reported in this paper is partly supported by the ESPRIT LTR 25500-COMRIS (Co-habited Mixed-Reality Information Spaces) project.
011
11
111
This page intentionally left blank
011
Part 4 Communication
011
011
011
11
113
This page intentionally left blank
011
011
011
011
11
Communicating in an IIS: Virtual Conferencing
8
Adrian Bullock
8.1 Introduction This chapter considers the role that virtual conferencing has to play in realising a successful Inhabited Information Space (IIS). For any IIS to be successful it needs to weave together many different constituent elements and present these in a coherent and seamless manner. For example, Maher et al. (2000) describe how many different components are used together to create a virtual design studio for architectural collaboration. For the IIS to function, all the elements must work both individually and collectively. Communication is one of the basic building blocks for an IIS, and can be in many modes across many media. Approaches to virtual conferencing offer support for communication across a number of media and can be utilised in an IIS. These approaches are also starting to offer support for collaboration. By providing an introduction to, and overview of, various possibilities for virtual conferencing this chapter aims to show how these solutions can provide the required and appropriate support for communication and collaboration between inhabitants in a shared information space. Of course virtual conferencing solutions exist at many levels of sophistication and fidelity. Communication media can range from text through 3D graphics to video representations. The aim of this chapter is to present these many and varied possibilities, drawing on the experience of the author as well as insights into the past, present and future. In this way it is possible to see how diverse a range of IISs can make use of virtual conferencing functionality. We talk about virtual conferencing as an activity that takes place within an IIS, the aim being to examine a range of communication and collaboration possibilities available for use in an IIS. Many of the approaches could well be termed IISs in their own right, whereas others provide only part of the functionality required by an IIS. Our aim is not to simply overview virtual conference approaches, but to demonstrate
115
Inhabited Information Spaces
how these approaches can be incorporated, borrowed or used to complement other applications and so aid in the construction of an IIS. We begin by considering just what we mean by the phrase “virtual conferencing”, and then we look at a number of approaches, demonstrating the range of areas that virtual conferencing covers. Next, we examine how virtual conferencing can be used, and identify some issues that have a bearing on this use. We also consider the roles of video and graphics in realised solutions for virtual conferencing. These approaches are then compared to telephony, arguably the most pervasive virtual conferencing solution in use today in global terms, and we suggest areas that need to be addressed if conferencing is to approach these success levels in the future. We end by presenting general guidelines for making use of virtual conferencing approaches effectively.
8.2 Virtual Conferencing – a Historical Perspective: Past, Present and Future We have come a long way since the first telephone call in 1876 in Boston (Farley, 2001), the first videoconference meeting in 1930 in New York (Rosen, 1996) and the first TV broadcasts in March 1935 by the German Post Office. The subsequent developments in these technologies have concentrated on reproducing lifelike representations of shared audio or/and video between participants. The introduction of computers in the 1960s saw the development of more abstract forms of communication: email, text chat and MUDs / MOOs (see Section 8.3.2) are early examples. As computer graphics developed so did the possibility of creating simple shared environments, something picked up by authors at that time. W. Gibson (1986) coined the term ‘cyberspace’ in his writing in the 1980s to describe the virtual world in which we will be as comfortable occupying in the future as we are the physical world now. Stephenson’s (1992) Snowcrash further describes how physical and virtual worlds may become intertwined, undertaking actions in both worlds at the same time. In Star Trek we see a “Holodeck”, where the ship’s crew walk into an empty room, speak to the computer and suddenly it is as if they are in a completely different physical location, able to experience all that it has to offer – though we are still some way from achieving this today (Raja, 1998, Leigh et al., 1999). While early shared environments such as AlphaWorld (http://www.activeworlds.com/) encouraged thousands of users to use them and indeed have been used to host international conferences (Jones, 2000), it is perhaps with online gaming that we are starting to see the potential of IISs with hundreds of thousands of users interacting together in shared gaming environments, such as EverQuest (2003). By examining past and current approaches to virtual conferencing, providing examples and experiences of use, we aim to present virtual
116
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Communicating in an IIS: Virtual Conferencing 8 ●●●●●●● ● 1 ●●●●●●●
conferencing as an abstract area, fulfilling the requirements of many different environments and scenarios, including those listed above. It is an activity that takes place in the IIS to support the work that is the primary purpose of the system.
8.2.1 What Do We Mean by Virtual Conferencing? We define a virtual conference as a meeting between two or more participants, where the participants are physically located at different places. Typically participants communicate and collaborate through some form of visual and audio communication channel, along with the sharing of task-related information. Videoconferencing systems such as the H.323based Polycom ViaVideo (Polycom, 2003) and the IP–based Marratech (2003), telepresence systems such as MASSIVE (Greenhalgh and Benford, 1995) and DIVE (see Chapter 12) and text-based systems such as MUDs and MOOs (Burka, 1995) are common forms for supporting this communication. An important aspect of virtual conferencing is that there is a shared space that supports appropriate communication across a number of media in as natural a manner as possible; an IIS is such a space.
8.3 Approaches to Virtual Conferencing We now examine how virtual conferencing has developed, considering its origins in video in the late 1960s and multi-user dungeons in the late 1970s, the effects of developments in graphical, computational and network capabilities, and application areas that benefit from and drive forward the development of virtual conferencing systems.
8.3.1 Early Videoconferencing Early videoconference developments dating from the 1960s placed the emphasis on communication, enabling two people in different geographical locations to talk to and see each other. The Picturephone system (Massey, 2003; Schindler, 1969) developed by AT&T was an early example of an attempt to enhance the experience of talking on the telephone. A telephone was augmented with an accompanying video display unit with a built-in camera, enabling the user to see the person to whom they were talking. Functionally the system was sound, but people felt that the unit was too bulky, the controls too unfriendly, the cost of use too expensive and the picture too small. Another difficulty was the high bandwidth requirements for a video call, 333 times higher than a standard phone call. Clearly this would not scale very well with the telephone
117
Inhabited Information Spaces
infrastructure of the early 1970s and the Picturephone disappeared very quickly. However, with advances in compression techniques, improved networks (both Internet and Telecommunications) and increasing availability of low-cost bandwidth we are seeing conditions where videophones are becoming viable. Indeed, we increasingly see their use by the media when reporting from countries with little infrastructure (e.g. satellite video phones from Afghanistan in 2002).
8.3.2 MUDs and MOOs The roots of more abstract forms of virtual conferencing can be traced back to MUDs (Multi User Dungeons) and MOOs (Object Oriented MUDs) in the 1980s (Bartle, 1990). At this time computational resources were limited and expensive (mainframes), so people had to rely on simple text message passing and, perhaps more interestingly, the use of text to construct some of the first IISs (even if they were not thought of as such then). Despite the impoverished appearance of such environments (only a screen of text after all), they were extremely effective at engendering a sense of community and presence – a very effective IIS indeed! Emoticons were used to enrich communication (Rivera et al., 1996), perhaps the most famous being the smiley :-) (Fahlman, 2003). These textual environments showed that effective collaboration and community building was possible. Embodiment, information representation and technological issues were addressed in the minds and imaginations of the end users, and textual descriptions aided in this process (a textual description of how and where someone appears). It is not surprising that this works well, given the way in which it is possible to immerse oneself while, for example, reading a book. These environments had a range of uses, some being for games and role-playing, some a place to simply hang out and meet friends (Evard et al., 2001), while others were support environments for work (Evard, 1993, Churchill and Bly, 1999).
8.3.3 The Arrival of Graphics In the early 1990s graphical add-ons to MUDs and MOOs started to appear, as graphics technology became something that was available on the desktop and was no longer the preserve of expensive, high-powered servers. However, these early approaches to bolt on graphics (e.g. BSXMUD, 1994) did not really work (see Figure 8.1). The interesting and useful interaction continued to take place in the simple text exchanges, while the graphics were simply a series of updated still images and scenes that bore little resemblance to the ongoing dynamic activity. More successful approaches to graphical MUDs and MOOs were ActiveWorld’s
118
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
Communicating in an IIS: Virtual Conferencing 8 ●●●●●●● ● 1 ●●●●●●●
AlphaWorld (http://www.activeworlds.com/) and Blaxxun’s Contact (http://www.blaxxun.com/), realisations of the kinds of online communities described in Neal Stephenson’s (1992) Snowcrash (indeed Blaxxun takes its name from the virtual club from the book). Here there was a more direct correlation between the user and their embodiment (Cuddihy and Walters, 2000), albeit in a very constrained and rather unnatural way. Typed text was still the main communication channel, but now gestures could be performed by the embodiment as well as emoticons such as :-). These environments appealed to teenagers in particular, offering a customisable place to hang out with friends and meet new people. Creativity could also be expressed, with parallels between the early settlers in the USA and the early settlers in AlphaWorld, arguably the world’s largest virtual reality world, being easily made through the creation of homes and communities. Activeworlds Maps (2003) charts the development of this world from December 1996 to August 2001 in pictorial form. Currently the world is approximately the same size as California, i.e. 429,025 square kilometres. Today these companies have grown from their MUD roots to offer a 3D home on the Internet with sophisticated possibilities for interaction .
8.3.4 Video Comes of Age Away from MUDs and MOOs, work in the late 1980s was also being undertaken on shared media spaces, where video technology was used to create a real shared space inhabited by real representations of people. Xerox EuroPARC pioneered this work with their RAVE media space (Gaver et al., 1992), in tandem with research into media spaces at PARC (Stults, 1986, Bly et al., 1993). ISDN-based videoconferencing was the
011
11
Figure 8.1 A screenshot from BSXMUD. Reproduced with permission from Henrik Rindlow.
119
Inhabited Information Spaces
standard for business and commercial use, and a new suite of software tools that ran over the Internet were developed in the academic community at this time: the MBone tools (Macedonia and Brutzman, 1994). However, workstations and peripherals to support video were expensive and not in widespread use, even by the mid-1990s. All this was to change with the arrival of inexpensive USB devices and dedicated video hardware in the latter part of the 1990s. Video became accessible on the desktop to all users, software became more widely available (e.g. Microsoft bundled NetMeeting with their operating system), plug and play all-in-one hardware devices such as the Polycom ViaVideo removed the complexity from local peripheral configuration, and network infrastructures made it possible to support the bandwidths required for video communication. We give two examples of different types of videoconferencing solutions that are most commonly in use today. Figure 8.2 shows a roombased solution, where dedicated hardware is installed in a conference room, and a large screen displays the other members of the videoconference. This is purely videoconferencing, and not really an IIS in the way it is being used. However, if the other screens were used to display shared workspaces, e.g. using Virtual Network Computing (VNC) (Richardson et al., 1998) to share a computer desktop between all the sites then this simple communication space is quickly transformed into an IIS.
Figure 8.2 A typical room-based videoconference session
120
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
Communicating in an IIS: Virtual Conferencing 8 ●●●●●●● ● 1 ●●●●●●●
Our second example shows how a desktop-based videoconference solution can be thought of as an IIS. Figure 8.3 shows what a typical desktop looks like when holding a meeting using the desktop videoconference system Marratech Pro (http://www.marratech.com). Four windows make up the session and we describe them moving from left to right. The viewer enables the user to load HTML web pages. By default these pages are local to the viewer, though users can choose to transmit their content to others in the same session. In this case the project web site, being discussed in the meeting, is displayed. The next window is a shared whiteboard that supports the import of images, Microsoft Word and PowerPoint documents as well as standard whiteboard functionality. This holds the meeting agenda and is updated with annotations as the meeting progresses (who is responsible for developing which sections of the web site in our example). The video window shows who is currently talking, and the video image follows the audio source, changing dynamically. Finally, the participant window shows thumbnail video images of everyone who is in the meeting. Participants join the meeting by contacting a portal and joining a session. Sessions can be encrypted and/or password protected for privacy purposes. It is also possible to communicate privately with other participants in the meeting while still being present in the main meeting room.
011
011
11
Figure 8.3 A meeting in a desktop videoconference system.
121
Inhabited Information Spaces
8.3.5 Graphics Come of Age From the mid-1990s onwards, technological advances were such that 3D graphical environments that supported audio communication between participants were possible on desktop workstations, opening the door to potentially rich interactions between standard users. Early examples of these types of systems included DIVE (see Chapter 12), a fully programmable distributed interactive virtual environment, and MASSIVE-1 (Greenhalgh and Benford, 1995), a 3D graphical teleconferencing system. Another important driving force in the development of CVEs at this point was the gaming market. Doom (1993, Id Software) was a revelation, demonstrating that engaging 3D environments into which users could immerse themselves for literally hours on end were possible, and on very lowly specified PCs. OK, so these were hand-crafted, specific and optimised solutions, but if such levels of realism and involvement were possible in a game then surely it would only be a matter of time before such environments were possible for business and leisure use? The main drawback in the development of realistic environments in the early 1990s was the available computational power. Despite pushing
Figure 8.4 A virtual conference in DIVE (c. 1995).
122
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
Communicating in an IIS: Virtual Conferencing 8 ●●●●●●● ● 1 ●●●●●●●
the technological capabilities to the limits, the early environments supported by systems such as MASSIVE and DIVE appeared primitive and impoverished (see Figures 8.4 and 8.5). Textures helped increase realism, but at the time this had a major effect on performance. Unlike the predefined scenes from games, these new systems were programmable on the fly and very much dynamic in nature. The collaborative nature also had a major impact on networking, especially with the bandwidth requirements of audio communication between participants. Initial experiences showed that the configuration of such distributed environments was most definitely non-trivial, even for those who simply wanted to run a client. However, once sufficient experience and training has been acquired it has been observed that people could collaborate well in such environments, could build personal and social relationships with others, and adapt their behaviour quite flexibly (Greenhalgh et al., 1997, Bullock, 1997). It was possible to examine collaboration and usability issues, and not just be content with the technical “success” of merely giving everyone the possibility to communicate, though this was a result in its own right.
011
011
011
11
Figure 8.5 Participants examine a representation of network connectivity in MASSIVE-1. Reproduced with permission from The University of Nottingham.
123
Inhabited Information Spaces
8.4 Using Virtual Conferencing In the previous section we examined the development of systems that have overcome the technical challenge of simply enabling people to meet together in a shared environment. This stable base allows the investigation into a number of interesting research questions and challenges for supporting virtual conferencing and meeting in an IIS. Some of these are quite technical in nature – such as how can scalable systems that support many thousands of simultaneous users be constructed and how complex can the environments and users be? – while others are concerned with social factors – such as how do we convey subtle, involuntary communications inside an IIS, and how can we assess the successfulness of the IIS as a whole? Perhaps an easy way to summarise is to say that we need to understand the process of collaboration and identify the key aspects and challenges to be addressed.
8.4.1 Understanding Collaboration When we meet together with other people we are naturally immersed in a shared environment; it is the environment in which we spend our entire life and we spend many years learning how to understand it and interact in it. Through the use of computing and telecommunication technology for supporting virtual meetings we introduce an explicit obstacle between each of the users, in that each person must first interact with the artificial shared environment and then interact with the occupants and contents of that environment (Churchill and Snowdon, 1998). There have been many studies of how people collaborate together in both virtual (Fraser et al., 2000) and real (Heath et al., 2001) environments. One of the major challenges of teleconferencing, and one that is still far from being solved, is the ability to interpret the many and complex interactions that take place in real life when two or more people meet together. Information is exchanged in what is said, how it is said, gestures, facial expressions, body orientations as well as people’s natural ability to somehow sense how the other person is feeling (possibly from the sources listed above but maybe from other factors as well). This information is very difficult to capture and process, though there are attempts to build dedicated installations to do this such as in the EU IST-funded Virtue project (Schreer and Kauff, 2002) and the Office of the Future project (Wei-Chao Chen et al., 2000). There are also continuing developments in the area of image recognition and tracking, so that systems can be developed that automatically capture not just the voice and appearance of a user, but also their gestures and other subtle interactional components and automatically translate these into input for the shared meeting system (e.g. Fuchs, 1998 and Intelligent Spaces in MIT’s
124
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Communicating in an IIS: Virtual Conferencing 8 ●●●●●●● ● 1 ●●●●●●●
Project Oxygen (2003)). Another possibility to capture user data would be to expect someone to explicitly control a virtual embodiment to represent their range of interactional possibilities. However, the complexity and cognitive load in using such an interface would most likely make it impossible to use – users would spend all their time trying to control their avatar to the extent that they do not concentrate on the actual interaction itself, the whole reason for being in the environment in the first place!
011
8.4.2 The Importance of First Impressions While meeting someone for the first time should be a real-life experience, repeat meetings can benefit from virtual conferencing approaches. There is already a shared experience on which to build (the first meeting), and people can use that to bridge the gap between the real and the virtual (be it graphical or video) world. These techniques are equally valid for graphical environments and videoconference systems.
011
011
011
11
8.4.3 Sharing Context Another important aspect of collaboration is the ability to share the context or setting in which each user is situated. The communication itself is important, but so is the work that is going on (Luff et al., 2000). We need to share work as well as communication, and tools that let us do this are being continually developed, e.g. shared CAD design software, whiteboards and application sharing systems such as NetMeeting and VNC. We also need to be able to share experiences between local and remote participants, an example being the TELEP system from Microsoft (Jancke et al., 2000) that allows remote viewers to attend seminars, having a presence in the room and an ability to interact.
8.4.4 Scalability When it comes to scale there are big problems to be overcome. What does it mean to be in a virtual audience of a thousand users listening to a presenter give a speech? Hard choices have to be made as to how much awareness of the rest of the audience we have while we naturally concentrate on the speaker. There should be the potential to interact with any individual in the room (e.g. in a plenary session at the ACM CHI conference anyone is able to ask a question, and that person is heard by everyone in the room, but only at the instant the question is asked. At the same time we are aware of what the people immediately around us in the audience are saying.). Techniques to scope and limit
125
Inhabited Information Spaces
the interactions are necessary (Chapter 13, Benford et al., 1997a), and these techniques must be flexible enough for easy switching between the different scenarios, and ideally done without the awareness of the user. Here the technical issues have to be dealt with while considering the social implications of any given choice.
8.4.5 Real Versus Abstract: The Role of Video? Approaches to teleconferencing using 3D graphical systems (Greenhalgh et al., 1997, Lloyd et al., 2001) offer a “real” shared environment, graphically rendered with polygons and textures, where users are embodied in the same shared space and it is possible to observe the interactions between the users. This offers potential for developing and supporting the more involuntary aspects of communication that we touched on previously. As the embodiments can be quite crude, however, and even when they are very realistic as in computer games or state of the art research (VR Lab, 2003; Mira Lab, 2003) (see Figure 8.6), it is still not natural to immediately associate the embodiments with actual people. There are also very real performance issues when using more realistic embodiments, as significant computational resources are needed as the levels of realism increase, especially if the embodiments are to react dynamically in real time. And how should such embodiments be animated and controlled? One danger with a lifelike embodiment is that there is an expectation of lifelike behaviour from the avatar. If an avatar has legs is it not unreasonable to expect them to move when the avatar moves? Often this is not the case. With video the association between people and embodiment is automatic and we immediately get a sense of the person we are talking to. However a simple video image is quite poor in the depth and quality it can offer and much depends on the local user configuration (i.e. where the camera is placed, lighting, the behaviour of the person themselves).
Figure 8.6 Realistic human embodiments. Reproduced with permission from Daniel Thalmann.
126
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Communicating in an IIS: Virtual Conferencing 8 ●●●●●●● ● 1 ●●●●●●●
So while we recognise and relate to the person on one level, much of the subtle interactional methods are lost with a purely video approach. Therefore the immediate way forward would seem to be a hybrid approach where video-based avatars are placed in a shared 3D space and can interact with each other (examples being Benford et al., 1996; Insley et al., 1997), or mixing the real and virtual worlds together (Koleva et al., 2001).
8.5 Virtual Conferencing Versus Telephony In spite of the advances in support of virtual conferencing at a number of levels across a number of media, perhaps the most used form of virtual meeting today remains the humble telephone and conference calls, where there is no user embodiment nor support for shared information. In this section we examine some of the reasons why the telephone enjoys widespread success while the seemingly superior virtual conferencing services are lagging some way behind. At first it appears strange that more use is not made of virtual conferencing, given the potential savings in time and money if people were not to travel so often (Townsend et al., 2002). However, a Scandinavian survey (SAS, 2002) undertaken by Gallup for SAS, the region’s major airline, has found that for business, personal meetings are preferred to virtual ones (maybe not too surprising given who the survey was commissioned by). It is interesting that the same survey also found that women were more positive towards virtual meetings than their male counterparts. With the introduction of broadband (0.5Mb and above) to homes across Europe and the United States, one might think that the days of regular telephony were numbered (if everyone has a computer and it is always connected then why use the telephone?). This is definitely not the case. Let us consider why the telephone is so successful. People are familiar and comfortable with the telephone, they know the limitations and know what to expect when using it. They simply lift the phone, dial a number and talk to the person at the other end. Technical glitches are rare and not the fault of the individual. Also, the interface has changed little over the past 50 years – indeed if a 50-year-old phone is connected to today’s system it is more likely to work than not. Contrast this to computers, relatively new and complex devices that serve many and varied purposes, that are constantly being improved and developed, offering little backwards compatibility in terms of both software and hardware (things quickly become obsolete). With virtual conferencing we are often not dealing with a single device (the telephone), but a collection of devices: microphone, camera, screen, computer and the connections between all these devices. The software that holds everything together is often not as stable as the telephone systems and much configuration and fine-tuning is necessary to have
127
Inhabited Information Spaces
everything perform optimally. Also, much of this configuration is a local issue that the users have to deal with themselves. Unfortunately many people have poor experiences of trying to use virtual conferencing applications, and these experiences make them shy away from using such approaches in future, be it consciously or unconsciously. However, many of the problems of the past concerning interoperability and configuration are being solved. USB devices, such as cameras, are standard and easily used across platforms, and software wizards make the configuration of applications a relatively easy task. Also, instant messaging systems such as ICQ and Microsoft Messenger are educating people to the potential advantages for virtual conferencing, even if it does not explicitly seem that way. These tools offer facilities much like the standard telephone where people can select an icon and then start a conversation with another user. While this is not much different to a standard telephone call, it is providing the user with experience of using his or her PC for communication, and perhaps more importantly some confidence and benefit from the process. The next step of experimenting with conferencing applications should hopefully be easier with such a user, as they are aware of potential benefits and have some confidence in the technology in use (i.e. they have used it successfully and know the benefits to be gained at first hand). Long-term usage of room-based videoconferencing (Bullock and Gustafson, 2001) identified a number of potential problems that need to be addressed if people are to concentrate on their meeting and on each other, and not on the infrastructure supporting the meeting. A consistent experience is necessary, as with the telephone. Participants rated their experiences very highly the first time they experienced the system, but for subsequent meetings they rated their experience less highly, even though there were no perceivable differences between configurations. We suggested that people were excited and taken in by the set-up the first time they used it, and subconsciously compensated for small glitches in audio and video. Once familiar with the set-up, however, they were more susceptible to minor imperfections and noticed these, and these made people feel more tired. So, while systems might appear to be good in the short term, they need to offer consistency and reliability like the telephony system if regular long-term use is to occur. Perhaps the most significant problem in maintaining a sense of presence between the participants concerned audio, and feedback in particular. Audio headsets help eliminate this, but in a room setting these are not practical. While microphone/speaker combinations do exist that help eliminate feedback, these can be expensive, or else the quality they offer is relatively poor. In order to take advantage of the increased quality that computational approaches offer (compared to the telephone), then audio solutions offering feedback-free communication need to be developed.
128
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
Communicating in an IIS: Virtual Conferencing 8 ●●●●●●● ● 1 ●●●●●●●
8.6 Guidelines for Using Virtual Conferencing Effectively When deciding on what type of virtual conferencing solution to use, there are a number of factors that need to be taken into consideration. Perhaps most important is the task that is being undertaken. It is also important to think about what the most relevant media are for communication, and what types of communication need to be supported. Other factors to be aware of are the supporting infrastructures that will be used (e.g. networking, display possibilities) and issues of heterogeneity. We bring this chapter to a close by drawing up a set of guidelines to help in selecting appropriate approaches to adopt for communication in an IIS.
8.6.1 What Is the Task at Hand? The most important factor to consider is what activity is being undertaken in the IIS. A scheme for classifying different types of activity involves the activity itself and two parameters describing the activity. The first parameter describes whether the activity naturally occurs in real, physical space or in a virtual, constructed space, and can take the value real or virtual. The second parameter concerns information that will be shared between participants in the activity, and again takes the values real and virtual depending on whether the information is real (e.g. word documents) or virtual (e.g. a CAD model of a physical object). We give examples for each of the four resulting application areas and show how virtual conferencing can best be used to support these areas.
Activity(Real, Real) An example of this kind of activity is a project meeting using desktop conferencing that supports a shared workspace. Returning once more to the system shown in Figure 8.3, video and audio provide the necessary communication between participants, and shared editing and browsing tools allow web pages and office documents to be shared between participants. There is no abstraction in the virtual conference, with real life meeting characteristics and properties mapped directly onto the virtual conference.
011
11
Activity(Real, Virtual) An example of this kind of activity would be a system similar to the one described above, but where support is provided for sharing more abstract information between participants, rather than simply documents. The DING project (Törlind et al., 1999) is a good example of such
129
Inhabited Information Spaces
an environment, where a videoconference system is augmented with a shared 3D virtual world where participants are embodied and can act on shared 3D CAD models.
Activity(Virtual, Real) This kind of activity would take place in a shared virtual environment, but the information shared in the environment would be real. The Web Planetarium (Chapter 2) is a good example of an IIS that supports this kind of activity. The shared 3D environment is provided by DIVE but the information inside is real web pages.
Activity(Virtual, Virtual) Finally, truly virtual activities inside an IIS would involve things like shared browsing of information visualisations or similar activities.
8.6.2 Communication Media Above we gave examples of different types of real and virtual activities. What we didn’t explicitly mention was the possibilities for communication inside these IISs. Audio is arguably the most useful form of realtime communication in an IIS, and is equally applicable to any of the scenarios listed above. The one drawback with audio communication is the way it suffers from interference through dropped audio packets or poor availability of resources. Simple text messages are a very useful and very effective form of communication, supporting both synchronous and asynchronous communication, and providing an explicit history mechanism in the process. Video imagery is useful, and the more realistic the image the better the sense of presence will be, but video can normally be given a lower priority than the other media unless there is an explicit need for it in the activity.
8.6.3 Infrastructural Support In order to support high quality audio or/and video conferencing, high network bandwidths are required. If such networks are not available then simpler forms of communication need to be used, such as typing text messages or using lower quality encodings for the media streams. However, given the increasing availability of network infrastructures and supported bandwidths, it is better in the long run to make design choices based on infrastructures existing in the future (e.g. Smile! – Johansson,
130
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
Communicating in an IIS: Virtual Conferencing 8 ●●●●●●● ● 1 ●●●●●●●
1998)) rather than work around problems that are not really problems and not have chance to examine what are the interesting issues.
8.7 Final Remarks There is no doubt that virtual conferencing has a big part to play in future business and leisure scenarios and applications. As interfaces to the computer improve, and the computer as an artefact disappears to be replaced by intelligent devices with built-in computational and communication facilities, we will no longer think of running virtual conferencing applications or configuring infrastructures for communication – we will simply take part in virtual meetings as if they were happening in reality. Of course it will take some time before an experience of actually “being there” with the other person is possible, but the use of conferencing techniques in much the same way that the telephone is used today would represent a significant step forward.
011
011
011
11
131
This page intentionally left blank
011
011
011
011
11
9
Getting the Picture: Enhancing Avatar Representations in Collaborative Virtual Environments Mike Fraser, Jon Hindmarsh, Steve Benford and Christian Heath
9.1 Introduction Inhabited Information Spaces (IISs) can be either realistic or abstract. Realistic IIS systems presenting 3D graphics are usually termed “Collaborative Virtual Environments” (CVEs). However, difficulties encountered by designers of abstract spaces such as Populated Information Terrains (PITs) still arise in “realistic” CVEs. Notably, both abstract and quasi-realistic information spaces suffer the problem of supporting “natural” interaction in an unnatural world. Users of abstract spaces encounter difficulties because interactions in an unfamiliar world must be learnt. Users of CVEs encounter difficulties because interactions in a familiar world are assumed where many remain unsupported. In this chapter, we focus on the realism of inhabitant representations. The use of “avatars” is a fundamental part of CVE design. Users are represented in the 3D environment so that they can see their own representation and also see one another (Fraser et al., 1999). CVE systems usually take the approach of providing “realistic” avatars (e.g. GuyeVuillème et al., 1999; Salem and Earle, 2000). Designers mirror human appearance and behaviour as closely as possible, even though there is some debate about how close designers are to creating a virtual experience that is “truly realistic” (e.g. Brooks, 1999). Visual representation and animation of human-like 3D figures has made great advances over the past few years. Avatars can look and move like realistic humanoid figures more than ever (e.g. Badler et al., 1999; Faloutsos et al., 2001). This approach has pervaded the representation of people in CVEs, and has commonly meant that CVE avatars are, at the least, pseudo-humanoid. Actions that can be performed at the interface are represented through
133
Inhabited Information Spaces
the use of a human-like metaphor or model. However, problems with the use of realistic representations remain. Rendering realistic 3D graphics must take into account constraints of the system such as graphics processing speed, network bandwidth and so on. Another often-overlooked factor, however, is that the user’s control and perception is not completely realistic. For example, a human-like figure can suggest realistic perceptual and movement capabilities to other users. These capabilities are simply not supported by current, or even prototype, display and tracking technologies. Nor is it likely that many problems, particularly the speed-of-light limitation on data transmission, will be removed over time (Brooks, 1999; Fraser et al., 2000). Recent studies (Hindmarsh et al., 1998; Fraser et al., 1999; Valin et al., 2000) have shown how traditional approaches to representation in CVEs can cause problems for users. Communications technologies have frequently ignored the importance of the contents of the world to users. Designers have tended to focus on supporting face-to-face communication rather than providing the ability to refer to the environment that users share. This does not take into account all manner of interactions that we regularly rely on in co-present communication. In the case of CVEs, designers have tended to work around these kinds of communication by focusing on support for simple meetings or coarse informal interactions, rather than attempting to allow users to collaborate around objects or features of interest within the world. The limitations of fieldof-view, speed of movement, and ease of gaze changes, mean that other users’ avatars are often off-screen, a key problem for coherently working together around shared objects. This makes it difficult to use pointing gestures and references, not only because movement can be slow, but also because it is hard to show (parts of) an object to someone whose position and/or view are unclear. The focus on representational realism of 3D shared spaces contrasts directly with that of 2D shared spaces, for which usability of the interface has been the key design focus. Research in 2D groupware has produced a number of concepts to describe how users’ views are represented to each other, which do not rely on the realism of representation. In particular, approaches to handling multiple views often involve “relaxed WYSIWIS” (What You See Is What I See) (Steffik et al., 1987). In this approach, each user’s view is outlined with a rectangle to indicate that view to the other user(s). As with CVEs, the use of independent viewpoints in relaxed WYSIWIS systems offers greater flexibility for individuals to act by providing adaptable views of the shared space (Gutwin and Greenberg, 1998). This chapter presents an investigation of an alternative approach to that of realistic simulation. Removing a focus on realism, we aim to reconsider representation of people in CVEs using an analogous approach to 2D groupware systems, by investigating how people use and interact with particular representational forms. We have begun to experiment
134
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Getting the Picture: Enhancing Avatar Representations in Collaborative Virtual Environments 9 ●●●●●● ● 1 ●●●●●●●●
with different representations of people in CVEs. Instead of simply providing a human-like avatar, we provide a representation that embodies the capabilities of the particular user’s interface. We have adapted the use of an outlined field-of-view, as provided in relaxed WYWIWIS systems, for 3D environments. A user’s view is graphically embodied in the virtual world as a semi-transparent frustum. The extent of the frustum matches the size of the user’s horizontal and vertical field of view. We then present an observational analysis of the use of this technique. Our data is drawn from recordings of pairs of users performing a design task. Analysis reveals instances of detailed co-ordination of talk and gestures, anticipation of problems, and examples of difficulties with occlusion of the environment. On the basis of this investigation, we outline considerations for design, as well as reflecting upon our approach more generally.
9.2 Method There are a number of ways of representing views in virtual environments. The traditional method has been to provide an avatar; a humanlike representation whose head/eyes presumably represent the view that the corresponding user sees. Approaches that are less realistic and more explicit might include representation on the virtual environment itself, perhaps through subtle highlights or shadows on objects in view. However, the increased graphical complexity of rendering lighting or shadows on a moment-by-moment basis, as the avatar moves around the environment, would severely compromise system performance. Additionally, while lights or shadows would be rendered on the graphical features of the world, they would not be visible in the intervening environment. We have extended the MASSIVE-2 CVE system (Greenhalgh, 1999) to support a different form of representation. The field of view is revealed in the virtual environment by outlining the user’s view frustum. This method provides information on the extent of the field of view, and also embeds this information onto and within the environment, connecting the avatar to the objects in view. Our approach can therefore make a viewpoint visible to others who might not be able to see the source of the view (i.e. the avatar itself). The outline consists of a semi-transparent frustum. This “view frustum” technique is shown in Figure 9.1. Note how the black hi-fi object is bisected by the frustum, allowing observers to determine that it borders the other’s view. The use of semi-transparency explicitly highlights the viewing area while showing the objects in the field of view. There are two further aspects to the interface used in our study. Distorted windows called “peripheral lenses” (Robertson et al., 1997; Fraser et al., 1999) are included at the sides of the main view. An example
135
Inhabited Information Spaces
Figure 9.1 User’s “traditional” avatar (circled) with field of view made visible as a semi-transparent frustum.
is shown in Figure 9.2. Our design of the view frustum does not display the bounds of the peripheral lenses, only the main view. Bounding the distorted areas could mislead others to believe that objects in the peripheral view are easily seen, when in fact objects are fairly difficult to see in distorted conditions. Additionally, a complicated representation would be required to bound both the distorted and undistorted views. Our impression was that this representation might prove too complex and have the effect of confusing an observer. Instead, the representation displays only the undistorted view. Arms that can be stretched out into the environment are provided on the avatar – a conceptually similar approach to stretching the representation of the field of view. Participants could point by stretching a single arm out into the world. Picking up and moving objects was represented by both arms stretching to touch an object, and that object turning into a wire-frame for the duration of the grasp. This design helps pointing and grasping actions to be visible across the environment. Using these representations, we asked pairs of volunteers to perform a design task. They were told to collaboratively arrange furniture in a virtual room (pictured in Figure 9.1). This task was designed to investigate geographically distributed interaction and has been used to study CVEs and other communications technologies (Gaver et al., 1993; Hindmarsh et al., 1998; Heath et al., 2001). Participants were given conflicting opinions on the ultimate layout of the room to encourage
136
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Getting the Picture: Enhancing Avatar Representations in Collaborative Virtual Environments 9 ●●●●●● ● 1 ●●●●●●●●
The left-hand edge of Fred’s view frustum is in Harry’s view
The right-hand edge of Fred’s view frustum appears in the right of Harry’s view
011
011
011
011
11
Figure 9.2 Screenshot from Harry’s screen in Example A as he says “you’re behind me”.
discussions. However, we were not interested in the final outcome or success of the participants’ design. Rather, our data selection focused on elements of the data in which participants specifically used or discussed the view frustum as part of their interaction. Our method of data collection closely mirrored that reported by Hindmarsh et al. (1998). We collected audio-video data of each participant’s screen and of their physical environment, including what was being said through their microphone and heard through their headphones. We asked 16 participants to perform the task. Each pair took about an hour to finish designing the room. Our method of analysis has been to account for particular examples of interaction that are recurrent within the data we have collected.
9.3 Analysis Our study reveals three key issues with regard to the use of the frustum representation, which are discussed in the following sections. The first section describes how users are able to intricately co-ordinate their talk and gestures, implicitly using the frustum to understand views of the virtual environment. The second section describes how users are able to anticipate and circumnavigate potential problems that others may have with the visibility of the shared environment. Finally, the third section describes how view frusta may occlude actions without that occlusion being obvious to the frustum’s “owner”.
9.3.1 Awareness and Co-ordination The depiction of the user’s field of view extends into the virtual environment to make it visible to other users. As a result, it is possible that
137
Inhabited Information Spaces
another user’s position and orientation can be understood without recourse to fast and intricate gaze movement, and indeed without recourse to the other’s avatar. Consider Example A, which follows a long period of silence in which Fred and Harry1 have been moving objects in separate parts of the virtual room. Fred’s avatar is located behind Harry. Fred rotates so that they are both facing in approximately the same direction (i.e. Fred can see the back of Harry’s avatar).
Example A2 (Audio Perspective3: Harry) Harry: you’re behind me Fred: yeh (.) I’m gonna grab the white chairs and put em round the table Harry: oh go (on then)
At the initiation of the talk, Fred’s avatar has not been in Harry’s view for some time. However, Fred’s frustum has intermittently appeared in Harry’s view as it extends across the virtual environment. At the start of the example, the left edge of Fred’s frustum is in Harry’s main view. As Fred rotates his view to the left, the right edge of his frustum appears in Harry’s right peripheral lens (see Figure 9.2). At this point Harry says, “you’re behind me”. Harry uses Fred’s frustum as a successful resource in locating Fred and understanding where he is looking, without having to re-orient to find Fred’s avatar. Fred and Harry can easily proceed with their course of action regarding the design task – grabbing and placing the white chairs around the table. This example shows how the view frustum is used as a resource in locating others, even when their avatar is completely out of view. Two key aspects of the example are worth noting. First, direct visual contact does not need to be explicitly established. The frustum can be used to maintain awareness of the other participant’s movements over time. At key moments such as in Example A, this resource can be used 1 All participants in these trials have been called Fred and Harry for purposes of anonymisation. Actual participants are different pairs of users. 2 Examples show the turns at talk that participants have. Numbers in parentheses are the length of pauses in talk. A single period in parentheses means a pause of 0.2 seconds or less. Letters in parentheses show uncertainty by the transcriber in what is said at that point. Square brackets show overlaps in conversation. Italics shows emphasis of those syllables with volume. Colons show elongation of the previous sound, the number of colons being proportional to the elongation. 3 Small delays mean that the time at which words are said and heard differs slightly, as audio is transmitted across the local network. This, in turn, means that timings may differ if talk is transcribed from one participant’s perspective compared to the other’s perspective. In examples presented in this chapter, there is minimal discernible difference.
138
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Getting the Picture: Enhancing Avatar Representations in Collaborative Virtual Environments 9 ●●●●●● ● 1 ●●●●●●●●
in interaction. Secondly, the resources provided by the frustum often feature very plainly in conversations between participants. Thus, it is perhaps what is not said that is the most interesting feature of this example. An implicit recognition of viewpoint is displayed with “you’re behind me”. There is no need to explicitly discuss each other’s position and orientation for the purposes at hand. This implicit recognition shows that talk can be co-ordinated with the use of the frustum without detracting from the task of designing the virtual environment. Example A illustrates how easily interaction with the other participant occurs without the requirement of seeing their traditional avatar representation. The extension of the view into the environment means that users have a resource to collaborate without having to explicitly talk and find out about each others’ viewpoint, even when the avatar itself is not visible. However, the example is one in which understanding the other’s view to pinpoint accuracy is not demanding for the activity at hand. The frustum simply provides an understanding of the location from which the other is seeing, and therefore for co-ordinating conversation between one another. In addition, however, the frustum representation also gives the opportunity to accurately co-ordinate around “visual” features of interaction, such as pointing gestures. In Example B, Fred and Harry are discussing where to put the table and chairs. Harry has stretched an arm to point towards an area of the room (Figures 9.3a and 9.3b). Fred has rotated his view from looking directly at Harry’s avatar, following Harry’s pointing arm (Figures 9.3c and 9.3d).
Example B (Audio Perspective: Harry) Fred: yeh i- i- i see where you mean (0.4) right in the corner yeh? Harry: well somewhere over here cos they’re not really (.) that much use for much else but some people could sit (and eat out) Fred: yeh (1.0) Harry: play a game or something (0.8) cos the chairs and the table go together
As Fred and Harry discuss the placement of the furniture, Fred rotates his view left, following Harry’s arm to look at the table and chairs, and the proposed location. Figures 9.3c and 9.3d show Fred and Harry’s respective views at this point. As Harry says “go together”, Fred’s frustum passes the end of Harry’s pointing arm. Harry seems to align to the frustum moving past his pointing gesture. He releases his mouse button and drops his arm. Compare their views again in Figures 9.3e and 9.3f. The most noticeable aspect from Harry’s perspective is the movement of Fred’s frustum along his pointing arm. The edge of Fred’s frustum progresses along the outstretched arm. The arm is slowly “revealed”, as it is progressively less obscured by the frustum. As soon as his arm is
139
Inhabited Information Spaces
a
b
c
d
e
f
Figure 9.3 Screenshots from Example B: (a) Fred’s view: begins to follow Harry’s gesture, rotating left; (b) Harry’s view: points to “somewhere over here”; (c) Fred’s view: rotates left, following Harry’s arm; (d) Harry’s view: his arm is still just inside Fred’s frustum; (e) Fred’s view: Harry’s pointing arm has dropped; (f) Harry’s view: Frustum passes Harry’s arm and he drops the point.
wholly revealed, Harry drops his pointing gesture. He is able to co-ordinate the production of his gesture with whether Fred can see it. The detailed timing of holding and dropping the gesture relies on the frustum representation to indicate how, whether and when Fred can see the point. This allows the intricate co-ordination of their talk with the gesture. Again, as with Example A, the participants’ talk remains focused on the design task. Co-ordination of talk and gesture occurs without the need to explicitly talk about how to achieve those activities with the system. In other words, the use of a frustum representation can allow the participants to focus on design work rather than the technology itself.
9.3.2 Anticipation So far, we have seen how collaboration can be achieved by virtue of “knowing” what the other can see. The frustum allows gestures to be produced in conjunction with understanding how those gestures are seen. Participants use the representation to design their actions for their
140
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Getting the Picture: Enhancing Avatar Representations in Collaborative Virtual Environments 9 ●●●●●● ● 1 ●●●●●●●●
visibility by others. However, as the CVE interface does not provide rapid gaze movement, there are also many cases where the frustum makes it obvious that things will not be seen. In this case, the frustum allows a course of action that might be called “anticipation”. Participants can circumnavigate potential situations in which others are not able to see relevant features of the environment. This aspect of frustum use is shown in example C. Harry begins by asking which of two large chairs can be disposed of. Fred is busy trying to place the television in the corner (he is finding this a tricky operation).
011
011
011
011
11
Example C (Audio Perspective: Harry) Harry: which do you think is a- best of these two chairs the big three seater or the (0.3) big one comfy one (0.6) Fred: (errl- l- let me) wait there (trying to) put the TV there Harry: cos I don’t think we want this umm (0.3) this big seat that I’m moving now (.) I’ll move it into yer view so you can see it (0.7) oh: (.) sort of (1.0) Fred: hhh Harry: do you think we need this? (0.5) Fred: need what Harry: this- this one I’m selecting (0.4) its in yer view hhh Fred: well just leave it out (of) the way for the time being (0.5) Harry: well no do you think we need it at all (0.3) Fred: we could do (0.3)
This example shows how participants can use the frustum to circumnavigate potential problems in which a referent might not be seen. Harry can see the chair is not in Fred’s main view, as it is not within the bounds of his frustum (Figure 9.4a). Therefore, in order to show the chair to Fred, he moves it into his frustum, so that they can discuss whether it can be disposed of (Figure 9.4c). The frustum allows Harry to anticipate problems with pointing the chair out to Fred, and instead follows a course of action to ensure Fred can see which chair the “comfy one” is, before discussing its use. Harry has some difficulties with moving the chair (displayed to Fred by saying “so you can see it (0.7) oh: (.) sort of (1.0)”). However, once the chair is visibly within Fred’s frustum, Harry re-states his question “do you think we need this?” He proceeds to select and deselect the chair by repeatedly grasping and letting go of it. This changes the appearance of the chair from filled-in polygons to wire-framed polygons. This representation is supposed to show that an object is being moved. Here, however, it seems as if Harry is making the chair “flash” in order to make it more visible to Fred. As Fred continues to ignore the question (and object) Harry’s voice becomes irritated, as he “knows” that the chair is now visible. Harry’s talk in this example seems to show how the frustum provides a responsibility for its “owner” to attend to the object at hand. It is not
141
Inhabited Information Spaces
a
b
c
d
Figure 9.4 Screenshots from Example C: (a) Harry’s view: at the start of the example; (b) Fred’s view: at the start of the example; (c) Harry’s view: holding the wire-framed “comfy one” inside Fred’s view frustum; (d) Fred’s view: Harry’s arms and the wire-framed chair (outlined) are obscured from Fred by his view frustum.
that the object must be viewed, but rather that there is a social, rather than technical, explanation for non-attendance. The possibility of technical reasons for not being able to see the object are reduced, because the frustum implies that what is contained within it can be seen. Harry seems to assume that technical problems with seeing things are no longer a potential “get-out clause” for ignoring the chair. The frustum renders its “owner” accountable to attend to the reference, once work has been carried out to reconcile that the potential referent can be seen. When we see this example from Fred’s view (Figure 9.4d), however, the object and arm (circled) are indeed relatively difficult to see. There are representational issues which make the visibility of the object very different between Fred and Harry’s views. Specifically, the edge of Harry’s view extends between the chair and Fred’s location, and thus Fred’s view of the chair is obscured to some extent by Harry’s frustum. This is the issue to which we turn in the next section.
9.3.3 Occlusion The previous examples have shown that the co-ordination and course of activities can be supported by the frustum. In effect, the frustum beneficially transforms the ways in which gestures and object grasps are accomplished. The last example, however, showed that the effect of rendering a representation of one’s own view on a co-participant’s view could not be seen. Despite this problem, participants are often wary of
142
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Getting the Picture: Enhancing Avatar Representations in Collaborative Virtual Environments 9 ●●●●●● ● 1 ●●●●●●●●
blocking each other’s line of sight. In this particular CVE, sight blocking is especially relevant for a number of reasons: ● ●
●
there is no “solidity” programmed into the environment and thus participants can walk through each other’s avatars; there is no haptic feedback that an avatar collision has occurred. For example, one participant can stand in front of another and reverse into them without realising; and it can be difficult to quickly glance around to check another’s line of sight when their frustum is completely or partially out of view. For instance, having one side of the frustum in view may not indicate which side it is, leaving vastly different alternatives about where the other participant may be looking.
Example D shows how, as a matter of course, Fred and Harry organise sight-blocking movements. Harry anticipates blocking Fred’s line of sight to the object he is moving (the “desk thing”), and notifies him of the possibility.
Example D (Audio Perspective: Fred) Fred: I’m gonna move that desk thing (0.3) [out the way Harry: [I’m just walking across your (0.4) line of sight (I’ll) be out the way in a minute
Harry is walking towards the edge of Fred’s frustum, near to his avatar. At this moment, he interrupts Fred before he has finished speaking. He talks in an apologetic tone, and in a way that implies his turn is an “aside” from the main business of talking about the desk thing. He says “I’m just walking across your” and then pauses for 0.4 seconds as he crosses Fred’s frustum, continuing with “line of sight”. Harry is able to provide an apology for moving across Fred’s line of sight, and obscuring the “desk thing” by virtue of the frustum representation. Blocking Fred’s line of sight onto the “desk thing” is seen by Harry as something to warn him about. However, despite the accountability of obscuring another user’s line of sight, sight blocking occurs more than one might expect within the video data. This is because the frustum represents the exact edges of the participant’s view, and thus cannot be seen by that participant (the frustum is not even rendered on the local user’s view, as it would be invisible and slow their speed of movement through unnecessary rendering). It is simply a representation to other users, and only the frusta representing the views of others can be seen. As a result, it can be difficult to determine the effect of moving your own view on someone else’s view. Example E illustrates how Fred and Harry encounter this problem. Fred rotates his view until it centres on Harry’s avatar. He then rotates to centre the desk in his view.
143
Inhabited Information Spaces
Example E (Audio Perspective: Fred) Fred: what you d- d- what you doin with that desk (0.7) Harry: puttin it near a pink chair again (0.4) will you stop blinding me with that (.) flashlight! Fred: what flashlight? Harry: your [vision Fred: [oh
Fred’s perspective is that he moves so that his view shows Harry’s avatar and then the desk. His reply “what flashlight?” seems perfectly reasonable. On the other hand, when the example is viewed from Harry’s perspective, Fred’s frustum appears across his entire view just prior to his exclamation (compare Figures 9.5a and 9.5b). Perhaps more importantly, it bisects Harry’s view from the object that he is grasping. The effect of his movement on Harry’s world is unavailable to Fred. However, it makes definite changes to Harry’s view, to the extent that he asks Fred to “stop blinding” him. Participants can be unaware that their own movements are causing significant changes to what another sees. The occlusion effect that a participant is having on another’s view may be particularly hard to understand. Fred and Harry are standing side on to each other, a common configuration for discussing the same object. It may be hard to imagine that one’s representation would affect another who is not directly in one’s line of sight. In order to blind a person with a flashlight, it makes sense that they would be facing you, not standing to one side. Example D showed how obscuring with the avatar can be anticipated and accounted for. However, the view frustum is entirely invisible to its owner. Thus, it causes problems where apparently insignificant movements of a participant’s view can cause significant changes to a co-participant’s view. If we now return to Example C, it can be seen that, as Harry is selecting the chair (by grasping and releasing it to make it flash between wire-frame and solid rendering), the chair arguably becomes less visible to Fred behind Harry’s frustum (Figure 9.4d). Whether Fred is not seeing the chair on purpose is not an issue here.
a
b
Figure 9.5 Screenshots from Example E: (a) Harry’s view: “Putting it near a pink chair again”; (b) Harry’s view: “stop blinding me with that flashlight”.
144
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Getting the Picture: Enhancing Avatar Representations in Collaborative Virtual Environments 9 ●●●●●● ● 1 ●●●●●●●●
Rather, it is clear that one participant sees an object as highly visible when, in fact, it is barely visible to the other participant.
9.4 Summary Our study suggests three key issues to consider when representing users’ views in CVEs. 1. Establishing locations and views. Talk about design between participants can be made possible by subtly co-ordinating references to objects or spaces. The explicit frustum representation enables this co-ordination of talk and gesture. Frusta, unlike traditional avatars, can give precise details about what can and cannot be seen by others. This allows simple cases to be assumed, such as whether an object is visible to another user. However, it also allows intricate co-ordination that is not possible with avatars; it offers the ability to design gestures for their visibility by others; and to observe gestures in such a way that the gesturing user knows their movements are potentially visible. 2. Anticipation of problems with visibility. The frustum representation does not comprehensively fix all problems with the visibility of the virtual world; it simply transforms the problem space encountered by users. However, one key feature of the frustum is that it can allow anticipation of such problems. In turn, it provides the necessary resources with which users can work around their issues. Thus, it is possible to insert simple sequences of action (e.g. moving an object into someone else’s view) to make features of the world visible to others. 3. Orientation to occlusion by traditional avatar, but not by frustum. The traditional human-like avatar representation is made relevant in interaction as a potential obscuring device. For example, users are very aware of blocking the line of sight on which another user may be acting or planning to act. Similar alertness is not generally apparent with the frustum representation, which often partially obscures the others’ views.
9.5 Reflections In this section, we review the implications of our study for the broader design of representations in shared 3D systems. First, some suggestions are derived for addressing problems of sharing viewpoints and other forms of 3D representation. Secondly, we discuss the principle of reciprocal perspectives, including a treatment of “out-of-body” camera approaches to shared applications. Thirdly, we consider the ways in which views of information spaces can be shared in “unrealistic” ways.
145
Inhabited Information Spaces
In particular, we describe some problems with the pervasive notion of “real-world metaphor’.
9.5.1 Scaleability Although our study clearly shows some benefits of explicit view representations, we need to consider the scaleability of enhanced representations. We have provided explicit information about the user’s perception and actions by graphically embedding their properties within a representation. Such an explicit approach, however, can provide challenges for scalability. The more users we represent, the more visual information we must provide. We have observed problems with occlusion with two users. It is most likely that multiple representations to support multiple users will compound difficulties. Providing so much information about so many activities could blind a user from the activities themselves. Additionally, consider our use of peripheral lenses (Robertson et al., 1997; Fraser et al., 1999) on the interface. Simply providing extra information about the extent of these distorted views would treble the number of graphical representations required. Alternative representations of viewpoints are also likely to suffer from scaleability problems. For example, spatialised audio representations might be used to indicate locations or boundaries without occluding visual information. These sounds will soon compound, however, to “occlude” conversations between users. Another approach might be to allow users themselves to define their interest in explicit representations (Dyck and Gutwin, 2002). In this case, however, we need to ensure that the user controls themselves do not distract from the task at hand. Thus, we propose that representations of users (and, indeed, user controls) need to consider key contextual factors in order to address scaleability problems. Designers need to decide just what representations are appropriate, and when. These decisions about the context in which information should be presented must include the use of semantics and subtlety. Semantically, we need to consider whether the choice of representation matches the application or task. For example, in the case presented in our study, the task involves decisions about objects and places. Perhaps our system can show which objects are being viewed at any particular moment in a way that fits the task itself? We might derive examples from studies of interior designers experimenting with scale models, observing how they look at and show objects to one another. In terms of subtlety, we would like the message conveyed to appear as part of the virtual task; to appear as a natural metaphor for conveying that kind of information. However, it seems clear that our definitions of appearance of the representation, and of the context in which that representation is relevant transform the ways in which that representation is used.
146
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Getting the Picture: Enhancing Avatar Representations in Collaborative Virtual Environments 9 ●●●●●● ● 1 ●●●●●●●●
9.5.2 Reciprocity of Perspective Schutz (1970) describes how in everyday life we tacitly assume that our individual perspectives are irrelevant for the collaborative task at hand; that everyone encounters objects in the same way, and that if you were me, this is what you would see. The assumption holds until evidence to the contrary appears. This evidence arises frequently in cases involving occlusion with the view frustum. The assumptions that are made by participants regarding reciprocal perspectives are suddenly made problematic, and require work to overcome. This is not to say that participants are not extraordinary in their abilities to “interact their way out of troubles”. Rather it is to say that the reality of CVE interaction is one where reciprocal perspectives are difficult to tacitly maintain, especially with regard to the mutual availability of features of the virtual environment. In the case of our study, the frustum technique was provided to overcome problems of location and viewpoint confusion. However, rather than completely solve these problems, the frustum transforms the problem space that participants encounter in assuming the perspective of the other. The provision of the frustum makes it more difficult for participants to see the effect of moving their view on the world of the other. In Gutwin and Greenberg’s (1998) terms, “power” has been exchanged for “workspace awareness”. Providing awareness information of what you can see can make it harder for you to understand what others can see. This raises important points about the provision of techniques to provide awareness in CVE systems. If we consider providing reciprocal visual awareness, we can broadly categorise CVE techniques into three kinds: camera viewpoints; exchanged viewpoints; and augmented viewpoints.
Camera Viewpoints A common technique used in commercial and research CVEs is the use of out-of-body camera views. These generally take the form of a bird’seye view over the avatar, or an “over-the-shoulder” view to give a wider perspective on the environment, framing the scene or activity at hand. These techniques are commonly known as tethered viewpoints. Some computer games have implemented this technique with automated viewpoints in relation to the focus of activities within the “task”. For example, a martial arts game can be framed with both avatars “in shot”, or racing cars’ relative positioning shown. However, the success of tethered and automated views may be tied to the activity being supported. Racing in a driving simulation is quite a different pursuit from discussing objects with a colleague and making one’s actions intelligible. The problems that may be caused in such situations can, in any case, be incorporated into
147
Inhabited Information Spaces
a game as an additional challenge in winning the fight or race; participant problems can be useful additions for gaming. In a similar vein, activityoriented techniques are suggested in this book for working with virtual cameras in real-time production activities (Chapter 10). Unlike the situation presented in our study, however, the viewpoint of the camera is also the locus of work in this case. In other words, viewing activities from different angles is part of the work of camera operators, and therefore the separation between bodies, objects and views is integral to the production process.4 The problems in “seeing what others can see” in our study suggest that difficulties could occur when using these camera techniques for coordinating real-time work. Whether participants are provided with the ability to change views with respect to their avatar, or whether the interface uses algorithms to change those perspectives, problems in interaction are likely to arise. Consider how difficulties in discerning the extent of another’s view through their representation can disrupt participants’ interaction. If camera views relative to the avatar are used to improve the user’s perceptual experience of the world, then the avatar becomes misleading for others not just with respect to their field of view, but also to the very location from which the represented user is viewing the environment.
Exchanged Viewpoints Techniques in CVEs have been outlined in which different views of participants can be chosen at any moment, in order to support exchange of reciprocal perspectives; in other words, the ability to actually adopt another user’s view on the 3D information space, to see what they are seeing (e.g. Valin et al., 2000). However, the changes in metaphor between individually controlled views and shared views cause the representation of users again to become a critical issue. Just what does an avatar represent in cases where the individual perspective metaphor is abandoned?
Augmented Viewpoints The frustum technique we have presented here comes under an alternative category, “augmentation”, in which reciprocal perspectives are supported by augmenting the realistic appearance of the environment
4 Nonetheless, and for the same reasons we discuss in this chapter, the camera operators themselves may still need to be aware of each other, cf. Drozd et al. (2001).
148
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Getting the Picture: Enhancing Avatar Representations in Collaborative Virtual Environments 9 ●●●●●● ● 1 ●●●●●●●●
with additional information about the “reality” of the user interface (cf. Fraser et al., 1999). Our study indicates that these kinds of technique are feasible, in the sense that participants are able to use the augmented information as part and parcel of their strategies for collaboration. Nonetheless, there are also ways in which the augmentation itself becomes part of the task collaboration. All three techniques to support reciprocal perspectives may be useful in one or another situation. However, our study at least tentatively indicates that techniques of augmentation can support reciprocal perspectives well in situations of “closely coupled” collaborative work.
9.5.3 Unrealism Although technologies to support virtual reality (VR) will continue to develop, it is likely that a range of obstacles will prevent an allencompassing realistic experience. There remain severe difficulties in developing environments that satisfactorily support interaction between individuals within a convincing virtual world. Current haptic interfaces are crude and expensive enough to be uncommon. Even though the realism of 3D graphics is rapidly advancing, interfaces to realistically capture movement and real-time expression for these illustrations lag behind. Network bandwidth (and the speed of light) means that different versions of events and causality are presented to different users. Yet, fundamentally, the success of the virtual reality programme as it stands depends on the success of the realism metaphor; it depends on the willing belief that one is sensing and acting in a computer-generated environment. It may be worth reconsidering whether the successful design of VR applications rests upon our ability to completely simulate ordinary realities and physical worlds. This is particularly true of CVEs, where the nirvana of a “truly real” virtual world may cloud the fact that a communications technology exists which, however rudimentarily, has the potential to be deployed on currently available hardware. While the overriding interest in achieving a realistic sense of virtual presence continues, the shorter-term potential of CVEs to provide effective tools to support distributed collaborative work is being undermined. Note that here we are not suggesting that shared spaces should necessarily present less realistic representations in favour of abstract information. Some might see the frustum representation as more appropriate for an abstract data space in which coping with a realistic world is less of an issue. In some cases this may be true; however, we see this as an application-specific issue, to be considered in those terms alone. As outlined in the introduction to this chapter, whether abstract visualisations or real-world simulations are portrayed in CVEs, success will always depend on participants’ ability to collaboratively make sense of, and act
149
Inhabited Information Spaces
in, the environment. We would like to emphasise that our studies simply question the commonly assumed correlation between realism and usability. Perhaps it is time to directly investigate the usability of CVEs rather than measuring their realism through sense-of-presence and co-presence metrics. We have shown that representations that might be termed unrealistic (in the sense of simulating human embodiments) can still, in fact, be used. These representations actually turn out to be realistic (or, rather, practical) in the sense that, unlike humanoid forms, they are a viable proposition for accomplishing collaborative tasks in virtual spaces.
9.6 Conclusions The implication of the approach we have outlined here is to become far more pragmatic than trying to achieve a perfect representation of reality in information spaces. CVEs provide a technology that will currently support interaction between distributed participants and enable them to engage and co-operate in particular activities. Our study has shown that users are able to accomplish collaborative tasks in this way, given the right resources. Rather than assume that we need to simulate conventional realities, we have examined one way in which users can accomplish certain forms of collaborative work through CVEs. Instead of concealing the limitations of the technology through photorealistic portrayals, we have tried to provide individuals with a sense of the constraints of the CVE system. Our approach has further implications for designers of abstract inhabited information spaces. The conclusion of our study must be that realistic portrayals of inhabitants must be tempered with representations that convey the in vivo characteristic properties of an interaction, rather than its “natural” properties. Unlike quasi-realistic approaches to inhabitant representation, therefore, designers might apply our strategy to both abstract and quasi-realistic spaces alike. Our study has shown some benefits and considerations of constructing representations that allow users to identify and deal with the characteristics of the system. We hope to have shown an alternative approach to designing information spaces, by shifting focus away from the general trend of hiding their properties and towards providing an enabling technology for remote collaboration.
150
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
New Ideas on Navigation and View Control Inspired by Cultural Applications
10
Kai-Mikael Jää-Aro and John Bowers
10.1 Introduction and Overview In this chapter we describe some of the work which was conducted within the eRENA project of the EU’s Inhabited Information Spaces research activity. This project combined the expertise of partners from computing, social scientific, artistic and television backgrounds to investigate the application of advanced media technology in cultural settings. Our “inhabited information spaces” were “electronic arenas” for participation in cultural events of an entertaining or artistic nature. In this chapter we focus on how, in these application areas, we have rethought some familiar interaction issues in human–computer interaction (HCI) to offer the beginnings of an approach which may be useful in such complex settings. Let us give a little more of the flavour of the application areas which concern us and the approach we have adopted. We organise this around five key themes. 1. Mixed reality technologies. All the demonstrators, applications and other developments within eRENA hovered on the border between the virtual and the real. While we were particularly interested in designing inhabited information spaces which were rendered with 3D graphical techniques familiar from virtual reality (VR) research, we were concerned to do so in ways which were sensitive to our final applications settings (galleries, performance spaces, television studios). This required us to be concerned about how interaction with the system might be impacted by the real world with all its interruptions, non-computer-mediated communication channels and physical limitations. Furthermore, as we shall see, some of our demonstrators directly explored physical, embedded interaction techniques in relation to VR.
151
Inhabited Information Spaces
2. Real-time applications. We were not developing offline methods. All the rendering, interaction, sound manipulation and other techniques we were working with had to give real-time results. To set ourselves tough test cases, we often worked with formats that allowed for a degree of improvisation on the part of performers or production crew. 3. Large scale participation. eRENA applications went beyond the “one person-one computer” paradigm of classical human–computer interaction into interaction between crowds, over large virtual spaces and physically separated nodes. We were particularly interested in formats which enabled public participation, potentially on a large scale. This in turn set us challenges for formulating techniques that could be used with minimal training by a heterogeneous set of users. 4. Media rich environments. Nor was interaction restricted to graphics presented on a computer screen and input through a keyboard, but rather whole-body interaction with physical props, display on multiple screens (and alternative projection technologies) and multichannel audio were more commonly our concern. 5. Cultural context. The applications were intended for culture, art, performance and entertainment. We made a practice of regularly displaying our work to the public through relevant cultural outlets (television programmes, events at arts festivals, exhibitions, public performances). Again this set us challenges for delivering high-quality content appropriate to those settings and the expectations of audiences within them. It also gave us the opportunity to flexibly explore the roles of spectators, performers and producers – while obtaining direct feedback on the quality of our work.
10.1.1 Challenges for Interaction Design As we have already noted, such settings and applications set core challenges for interaction design. In this chapter we describe our contributions to addressing these challenges in three areas: 1. Navigation and view control. How does a user choose what to see of an interactive artistic installation or of a performance? How, within an interactive digital television show, does a participant’s view relate to the television audience’s view, and both of these to the views available to production and direction personnel? What ordering is given to how views are selected and scenes navigated between? As we shall see, in eRENA, we experimented with a number of techniques for supporting navigation and view control – in particular, ones which extend the conventional notion of an avatar as indicating a locus of unitary view control. 2. World and experience design. How are “electronic arenas”, the “inhabited information spaces” of art and entertainment, to be designed?
152
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
What are their constituents? How are they assembled? In particular, how does the structure and organisation of such environments serve to make them arenas for experience, that is, places where certain incidents might be encountered or effects enjoyed? 3. Production management. How does one produce an experience, and yet again, if it is a large-scale experience, how does one co-ordinate the possibly very many people and computer processes involved in the production? What are the “behind the scenes” activities like and how in turn might we offer support for them? Over the course of our work on eRENA, we came to interrelate our answers to such questions rather closely. We began to develop, for example, world design techniques in tandem with navigation methods for such worlds and do so in such a way as to facilitate real-time production work. We unfold some specific examples of this approach in this chapter. This is done most effectively through presentation of a series of examples of cultural events realised in the project.
10.2 Interactive Performances Many of the cultural events within the eRENA project were realised as interactive performances in one sense or another (see, e.g. Carion et al., 1998; Bowers et al., 1998a, 1998b; Benford et al., 1999c). The underlying ideas and themes of these spanned a wide spectrum, both aesthetically and technically. We will first look at the scope of work in general with regards to interactive performances, and then look in detail at some specific examples which involved the improvisation of intermedia environments – that is, where the construction of graphical (and, in some cases, sonic) environments took place live within the performance itself. Interactive performances can be classified along several dimensions: How tightly scripted are the performances? What can be affected by the interactor(s)? What is the number of performers? What is the relation between “performer” and “audience”? What is the overall physical setting of the performance (staged, promenade, public space . . .)? Some examples of interactive performances within eRENA demonstrate these factors. ●
CyberDance (Carion et al., 1998) consisted of a human dancer equipped with motion capture sensors and a set of realistically rendered computer-animated dancers. The virtual dancers would move in synchrony with the human dancer. The choreography was carefully planned beforehand, yet the behaviour of the computer-generated parts was created in real time, in response to the movements of the human dancer. The setting was one of a “traditional” staged dance performance, but where some of the performers were computer-generated, the audience were off-stage and not involved in the interaction.
153
Inhabited Information Spaces
●
●
To the Unborn Gods (for a description, see Norman et al., 1998) was a “virtual reality opera”, where a human singer interacted with computer-generated figures. Their behaviour was indirectly affected by the singer’s actions, by an off-stage operator choosing suitable paths. Here too, the setting was very much in the way of a traditional staged music performance, but where some of the actors were screenprojected. Lightwork (Bowers et al., 1998c) was an intermedia performance where two human musicians interacted with abstract computergenerated forms which were generated in the real time of the performance. The graphics were created through the actions of one player and responded to by the other. An audience faced a performance area in a conventional manner. We will describe Lightwork in more depth below.
While the above three examples involve interactivity between performers and computer-realised material, they involve quite conventional stagings. In each example, an audience has a space associated with them, the performers have theirs (in the first two examples, this was provided by a theatre stage). Other eRENA performances involved rather more flexible relations between the performance space, audience and performers. ●
●
154
Desert Rain (Benford et al., 1999c; Shaw et al., 2000; Rinman, 2002) featured a number of discrete spaces through which participants moved as they engaged with a game-like activity based upon events in the Gulf War. At a number of points, actors directly engaged with the participants, giving them instructions or “debriefing” them. At one striking moment in the production, each participant interacts with a virtual environment projected onto a “rain curtain” – a fine water spray which holds a back-projected image. At a key moment, an actor appears between the projector and the rain curtain and slowly moves towards a participant, casting a silhouette in the projected image. The actor passes through the rain curtain and gives a swipe card (used later in the performance) to the participant. Desert Rain manifests a complex set of relationships between participants, actors and production crew. Performances have not been realised in conventional theatre settings (the debut of Desert Rain was in a disused factory). The crowd behaviour simulation and animation system ViCrowd (Hoch et al., 1999b; Lee et al., 2000) was used in a number of performances/installations for interaction by groups of people with computer-animated figures. People entering the interaction space were tracked and their combined behaviour caused various scripted behaviours in the computer-generated figures. As the tracking was only position based, movements in the space could be made arbitrarily complex for the benefit of the other humans present, but the virtual humans would react the same way regardless of how the triggering conditions were achieved.
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
Let us give a little more detail of one aspect of all this endeavour, where we explore interactive performances with an eye to opening up the scope for improvisation by the performers.
10.2.1 Lightwork While being an artistic performance combining electro-acoustic music and the real time construction of graphical environments, Lightwork also served as a “live” occasion to test many of the ideas for navigation and interaction in virtual environments proposed within the eRENA project, all the more so as the format of an improvised performance would sharply bring to the fore any problems with these ideas and their implementation. The design goals, intermingling technical and aesthetic issues, included the following: ●
●
● ●
●
The use of a minimum of encumbering apparatus – rather the adaptation of the standard tools that our performers would most typically use (i.e. electronic music control devices). A large element of improvisation in the performance – we wished to avoid the use of prepared world models and instead generate material algorithmically on the fly. The gestures of interaction with the environment being made public and a legible, meaningful part of the performance. “Compressing” and “decompressing” interactive gestures, so that small causes may get large effects and vice versa – the challenge being to allow experimentation with this, while still ensuring the legibility of performer activity. The creation of an “infinite collage”, where sound, graphics and text are recombined and perceivable from a variety of viewpoints – any particular realisation of Lightwork is just one of an indefinitely large number of possible ones.
This being a real-time situation where any interaction would have immediate and publicly visible effects, a classic direct manipulation approach to interaction was felt to be insufficient in that it both would require the following of similar steps to achieve similar results, and small excursions from the intended path may lead to unintended and unrecoverable effects (in the sense of requiring explicit, publicly visible and time-consuming actions to undo). Rather, we followed an indirect principle of “algorithmically mediated interaction”. The idea was to interpret the actions of the performers as input parameter values to content generation algorithms that would create visually rich, animated graphics. At the same time, the interaction methods were chosen so that all actions would result in meaningful effects – thus movement, while being adjustable, would happen along paths that were guaranteed to always
155
Inhabited Information Spaces
have something interesting in view. In a sense the performers would be shaping material independently flowing by. The initial sequence of the performance consisted of a 90-second flythrough through large squares displaying textures representing the various themes of the performance. Finally, the viewpoint entered the large sphere that was the container for the graphics during the rest of the performance. This sphere was texture-mapped with maps and satellite images that were automatically switched every 80 seconds. The viewpoint orbited around the centre of this sphere in either of two types of paths: a. A cloverleaf pattern according to the function r = m + n × sin t, where r is the radius modulated by the sinusoidal function, n determines the maximum deviation from the base radius m and gives the period of oscillation. See Figure 10.1 for some samples of this function. b. A back-and-forth motion through the centre of the sphere.
a
b
c Figure 10.1 Three parameter settings for the “clover leaf” path: (a) m = 10, n = 10, = 3; (b) m = 20, n = 10, = 4; (c) m = 30, n = 10, = 5.
156
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
The plane of motion was slowly tilted back and forth, in order to give as many viewpoints as possible – yet the angle of tilt was kept within ±/4 radians in order to retain a sense of up and down. For the type a orbits, the view direction could be chosen to point either in the direction of motion or towards the centre; for the type b orbits the view could point either towards or away from the centre. The position of the viewpoint would also determine the spatialisation of the sounds. While this still was a geometrical navigation method, we had started moving towards a more target-centred approach, where the path was not under the direct control of a performer, but rather the viewpoint control was mediated by analyses of performer input. Within the sphere various objects could be created. One of the performers (V) could select between five form generation algorithms with the parameters of the algorithms being set by means of analyses of performer input. Specifically: ● ●
011
011
011
11
● ● ●
scaffold, a grid of “pipes” – cylinders or parallelepipeds. The colour of the pipes as well as their density and lengths could be varied. formModulator, a tessellated sphere, the vertices of which were displaced according to a frequency modulation algorithm. The same fundamental algorithm was used to create an immersiveForm, sculptureForm or cave, using different parameter ranges. These forms could then be animated, giving the impression of an organic, “breathing” shape. chamber, a cube with rotated and scaled cubes inserted in its surfaces. plenumbulator, simple rectangles with text or images, set at random positions and orientations. orbiting forms, groups of textures and text strings orbiting around a common centre.
All shapes except scaffold were texture-mapped with images selected from one of 13 themes. Each new object would step to the next image theme. Only one of each kind of object could be present at any time, but any combination of them could be displayed simultaneously. V could create and destroy objects, as well as determine the type of path (a or b orbits, above) by pressing foot switches. V’s activity was analysed by a program we called “The Interactive Narrative Machine”, which computed magnitude and irregularity parameters over various time-scales. These (indirectly) determined m and n, the view control parameters, while also setting parameter values for the form generation algorithms and affecting the shape and motion of the created objects. For a full overview of the set-up, see Figure 10.2. At the end of the 15-minute performance, the viewpoint automatically retraced its path through the initial textured screens.
157
Inhabited Information Spaces
V
S
T
Figure 10.2 The connections between the units. The data from the wind instrument and foot switches are passed to the Interactive Narrative Machine, a Max program on an Apple Macintosh, making analyses of the behaviour of V over three different time periods, compressing these into two values, fed to T’s SGI O2, which runs the visualisation software. The foot switches determine what objects are to be created at the centre of the sphere as well as which type of orbit is to be used, while the data values generated from the wind instrument set the parameters for object creation as well as for the orbits. The spatial position data are forwarded to the Mac of S, where SO-2, yet another Max program, uses these to spatialise the music improvised by S.
The sound was also created and mixed using algorithmically mediated interaction methods. There was a large set of prepared sounds, both physically modelled and sampled, that could be mixed together live – see (Bowers et al., 1998c) for more details.
Performance Experience The first performance was realised in December 1997 at the Royal Institute of Technology in Stockholm and revealed a number of difficulties with Lightwork as initially conceived. While the quality of the graphical images (cf. Figure 10.3) and the electro-acoustic sounds was admired,
158
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
011
011
011
011
11
Figure 10.3 A chamber in Lightwork.
the indirect connections between elements often made it difficult to understand what was going on. Our techniques of “algorithmically mediated interaction” involved the analysis of performer activity in a number of time windows. Effectively, this introduces a smoothing and a delay to relations that V’s activity might have to graphical consequence. As the worlds were quite complex and the analyses of performer activity had several different consequences, this made it hard for the audience (and performers for that matter) to pick up on the interactive relationships in the work. V found himself exaggerating his pressing of the foot switches which introduced new world contents or removed objects which were present as this gesture had a more immediate and legible effect. In short, while our methods of analysing performer activity and indirectly using this to parameterise world building and navigation algorithms worked in a technical sense, the degree of indirection in interactivity was excessive for enabling the audience and performers to track what was happening and why. In Lightwork there were multiple mappings requiring calibration. Performance data needed to be measured in different time windows. The results of these analyses needed to be mapped to parameter values for
159
Inhabited Information Spaces
the world building and navigation algorithms. In principle, we could vary how the performance data were measured and scaled, the number and size of the time windows, the functions which mapped analysis results to parameter values and so forth. Ideally, one would wish to explore many different alternative mapping functions under different performance conditions. We were careful to set the ranges of values so that (for example) generated graphical objects would all be placed within the enveloping sphere and that the navigation paths would never be out of an intended range. That is, a number of calibration issues could be settled a priori. However, this alone does not guarantee that the range of worlds that will be generated will be appropriate and that the views obtained of them will be interesting. This requires an empirical approach to calibration and, before the performance deadline, we were not able to deal with all these matters fully. Accordingly, it was left to performers to troubleshoot calibration problems in performance itself. For example, V felt forced to adopt a playing style that would be guaranteed to keep a reasonable path, but which was less expressive than desired. However, our indirect, algorithmically mediated approach to interaction did not always give performers adequate resources to troubleshoot such problems. For example, rather more often than one would wish, the projected view was uninteresting and could not be swiftly adjusted.
10.2.2 Blink Continuing the ideas from Lightwork, the next performance was Blink (Bowers and Jää-Aro, 1999). Many of the elements of Lightwork were retained, but expanded with new graphical themes and new texts (including randomly shuffled three-line poems which could appear superimposed upon 3D image material). Importantly, we introduced some new interaction and navigation ideas to avoid some of the problems encountered earlier. In particular, the viewpoint in virtual space could be attached to any of three cameras moving within the space. In such a way the risk that nothing interesting would be visible was minimised. Cutting between views was supported with a direct interaction technique – so that “degenerate” views could be cut away from there and then. In short, Blink complemented our ideas of indirect, algorithmic interaction (e.g. to generate world materials and to control some features of camera movement) with direct interaction (e.g. to cut between views and in other ways override the algorithms if necessary). This notion of navigation by cutting reveals an influence of a cinematic or television metaphor for view control and indeed, for us, was a direct result of our participation in the television-oriented applications of virtual environments which we will describe later. In Blink we defined three cameras which would tend to produce related but different views (again by analogy to a cinema–television convention of giving different
160
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
camera operators different responsibilities). Two of the cameras circled the centre of the virtual environment, one near, the other further out, while an “explorer” would move radially from the centre to random points on the periphery and back, giving a further variety of views. The images from one of these cameras could be selected for “transmission” or “TX” in the vocabulary of television. Another virtual camera could be selected for “preview” without altering the TX view. In this way, shots could be prepared and meaningful edits made from one camera to another. (Ideally, perhaps, all three cameras should have their images continually available with TX selection being a matter of just choosing the camera required. With the graphics hardware available to us, however, we found that computing four views – the cameras plus TX – was not feasible at the resolution and frame rate required.) Furthermore, based on experiences both from Lightwork and the Real Gestures, Virtual Environments workshop (Hirtes et al., 1999), it was decided to make the physical environment of Blink visually richer by using a large number of screens surrounding the audience, rather than having them as a backdrop for the performers. Perhaps the biggest change was that instead of relying only on computed data for the input, the software was now equipped with a user interface with manual overrides for all functions, so that technical problems, bad parameter settings, and so forth could be bypassed during the performance. See Figure 10.4 for a view of the interface. Blink was performed as part of the Digital Clubbing event at the Now98 arts festival in Nottingham, England in October 1998. The venue was The Bomb, a nightclub in the city. On a small stage at The Bomb, 4hero performed in collaboration with Carl Craig, who was located in his studio in Detroit, through a video and data link. Two graphics operators were seated at separate SGI O2 workstations, each running the Blink software, though unsynchronised with each other. Twenty-one monitors were distributed over the premises in seven groups of three. In each group, one monitor showed the live transatlantic video link into Carl Craig’s studio while the others displayed images from the two workstations. In addition a large screen was placed adjacent to 4hero’s performance area, showing a video mix of these three sources. The groups were positioned so that the audience would typically have a line of sight of at least one set of monitors no matter where in the somewhat cavernous environment of The Bomb they were located. The initial intention was that Carl Craig and 4hero were to exchange MIDI data with each other, and thus interact with each other’s music during the performance. The combined MIDI stream was to be analysed to yield input parameters to the graphics generation algorithms. The analysis was simpler than the one used for Lightwork in order to make the relation between music and objects more perceivable. In the event, the MIDI connection turned out to be unreliable and it was necessary to use the manual overrides throughout to set the graphics parameters. The
161
Inhabited Information Spaces
Figure 10.4 The manual interface to Blink. At the top are a number of buttons, sliders and selection lists that define all the available functions and their parameters and in addition the buttons that determine what views will be shown. The buttons marked “Near orbit”, “Far orbit” and “Explorerorbit” determine which camera view will be shown, “Left”, “Front”, “Right” and “Back” determine which direction the camera will look towards. “Set TX” will then actually put this image on the public screens. In this manner the view space can be explored to find an optimal shot without the audience being subject to the browsing. Texts and images can also be pasted onto the “visor”, i.e. placed right in front of the camera and moving along with it. These can be tested separately (using the buttons at the lower right) and when a suitable combination has been found, it can be sent to the TX screen.
162
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
graphics operators thus worked out a schedule in which they would gradually exercise the abilities of the software and independently, but in a generally co-ordinated way, increase the tempo of cutting and the complexity of objects during the progress of the performance. In this way, the dynamics of the performance of the Blink software approximately followed the dynamical shape of 4hero and Carl Craig’s musical performance.
Performance Experience The audience reception to the event was extremely enthusiastic, though this of course is due in large part to the participation of (in their field) popular and well-known musicians, yet it seemed that the graphical work (see Figure 10.5) was considered to add to the event and make it even more exciting.
011
011
011
11
Figure 10.5 The TX window in Blink. We see a scaffolding and within it an immersive form. We can also see a box-like object, half hidden behind the scaffolding; this is one of the other cameras in the environment.
163
Inhabited Information Spaces
There were, however, problems with the system, mainly concerned with the pace of interaction and cutting. A mouse-based interface with lots of on-screen buttons, of which many had to be pressed in a particular sequence, was not conducive for the rapid cutting usually associated with (for example) the music videos which typically accompany the kind of music performed. The synchronisation of the “preview” window and the TX view was only accurate to within a second or so, so the actual transmitted view would tend to be slightly different from the one intended. This made it hard to perform closely co-ordinated edits (e.g. where there’s a cut from one view to another at the moment when the camera collides with an object). Overall, we felt our concept of multiple virtual cameras and supporting live editing between them was validated in Blink. The software operators rarely found that they had nothing interesting to cut to. The camera paths had been selected to contrast with each other while yielding occasional overlaps of material to permit meaningful edits to suggest themselves. The interface to the Blink graphics software combined direct and indirect algorithmic techniques, enabling the operator to override inappropriate algorithmically generated material, while making edits that could be synchronised with the ongoing music. Furthermore, the combination of directly actioned cuts with machine-controlled camera paths allowed operators to “buy time” for themselves: a camera could be left to explore the environment while the operator thought about what to do next.
10.3 Inhabited Television A different interpretation of the concept of interactive performances is that of inhabited television (Benford et al., 1998, 1999b; Greenhalgh et al., 1999; Craven et al., 2000). Here the concern is (in some way) to hybridise some of the traditional concerns of television (e.g. broadcast to a potentially mass audience) with the interactive possibilities of online virtual environments. The sources we have just cited describe a number of experiments in inhabited television. In its fullest form, inhabited television manifests the following features: ● ●
● ●
The set for the show is a shared virtual environment. The actors-performers-presenters as well as (at least some) audience members access the shared environment through a computer interface. The audience can take an active role in the unfolding events. Views from the virtual set are broadcast to those audience members who cannot be present in the virtual environment.
The persons involved in such a production can thus be divided into the following groups: performers, those who have been engaged to create the show – this can be taken to also include staff not visible in the
164
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
environment but controlling it, such as directors, camera operators and support staff of various kinds; inhabitants, members of the public present in the virtual environment and able to interact with it and viewers, who receive images and sound from the virtual environment, corresponding to the traditional television audience. Inhabited television, then, can be regarded as a contribution to research on the possibilities of digital television and allied new media – one which particularly emphasises public participation in broadcast events through (characteristically) activity in an “inhabitable” virtual environment. In what follows we develop the theme of this chapter by particularly highlighting the interaction and navigation innovations that have been explored in this application of VR technology. We do this by contrasting two inhabited television shows: “Heaven and Hell – Live” and “Out of this World”.
10.3.1 Heaven and Hell – Live Heaven and Hell – Live was a game show with a theme based loosely on Dante’s Inferno. It was broadcast live on British television’s Channel 4 on 19 August 1997. The show was realised as a collaboration between three eRENA project partners: the University of Nottingham, Illuminations Television and British Telecom. Two celebrities and the programme host were the performers, and 135 members of the public who had received copies of the software client for the virtual environments (Sony’s Community Place) were the inhabitants. The two performers did a treasure hunt, an avatar stacking game, a quiz and a gambling game, the intention being that the inhabitants would help the contestants with their tasks. The interest level by the public can be gauged by noting that not only the 135 inhabitants but an estimated 200,000 others stayed up in the middle of the night to watch the show. While it was a great technical achievement, Heaven and Hell – Live was problematic as a television show. There were insufficient resources to do a full-scale test beforehand. Thus both performers and inhabitants were working through the game environments, controlling their avatars and performing their tasks as best they could live. This resulted in the ostensible game content – rule enforcement, point counting etc. – often not being taken very seriously, as the task of working in the environment while trying to make adequate improvised television of one’s efforts became the performers’ primary concern. The loose relevance of the game format to the show was also noted by the inhabitants, many of whom tended to not take the game very seriously either. Furthermore, as the software only supported text-based communication (the performers in the studio could of course speak directly to each other), interaction between performers and inhabitants was slow. As a consequence the inhabitants tended to ignore the game and instead drifted away to chat with each other.
165
Inhabited Information Spaces
Figure 10.6 A scene from Heaven and Hell – Live. Inhabitants and actors in a graveyard setting. Reproduced courtesy of Illuminations.
The properties of the interface also made for a strange mix of paces – while the actors were working hard at their respective machines, typing text and moving through the environment, the view for the television audience was quite a lot slower than normal for an entertainment programme. Furthermore the Community Place software placed constraints on the number of other avatars a user could be aware of at any one time. Thus, inhabitants did not have the impression they were part of a massparticipatory event and, in some cases, became suspicious about the authenticity of the occasion (e.g. if they failed to see themselves on television, in spite of being in the camera field). Similarly, for the television audience, the environment often looked quite desolate, in spite of there actually being well over a hundred people “on set” (see Figure 10.6). Whatever else Heaven and Hell – Live was, it was scarcely recognisable as the participatory game show it was intended to be.
10.3.2 Out of This World Out of this World was a determined attempt to avoid the problems encountered with Heaven and Hell – Live and bring off an event which
166
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
011
011
011
011
11
Figure 10.7 The opening briefing of Out of this World. The robot team stands in the foreground with the alien team in the distance. On the video screen is the game show host and producer, John Wyver of Illuminations. Reproduced courtesy of Illuminations.
was more recognisable in television terms. Again the game show format was followed but various production and navigation facilities were added into the show’s software to encourage (or even, if necessary, mandate) appropriate participation. Out of this World was not a television broadcast but was shown to a theatre audience as part of the International Symposium on Electronic Art in Manchester, England in 1998. (Full descriptions of the technologies in the show can be found in Benford et al., 1999b, while Bowers, 2001 recounts the story of the behind-the-scene production work.) The game scenario was of two teams, one composed of robots, one of aliens, being pitted against each other in a series of games on a doomed space station, the winning team being allowed to escape (see Figure 10.7). There were still two performers that were the main contestants, but in this case they used an immersive interface with motion tracking and were placed on stage either side of a projection screen showing a live edited TX view of the show. The inhabitants were eight volunteer members of the audience who were seated by workstations in a separate room, using
167
Inhabited Information Spaces
a joystick interface to move in the environment. MASSIVE-2, the system used, allowed all inhabitants and performers to be connected by audio and be visible to each other. Two specific applications had been built in the MASSIVE-2 system to help address earlier difficulties. A production support application allowed the show to be represented as a series of phases with varied capabilities assigned to avatars and objects on a per phase basis. In this way, the movement capabilities of the avatars could be constrained so as to aid their participation in a game depending on, for example, whether groundplane movement only needed to be supported or whether the avatars could climb upwards. This allowed a single familiar interaction device to be used (a joystick) without the user pressing modifier buttons. At key moments also control could be removed from the team members so as to enable the smooth movement of their avatars to a new location. For example, travellators took the avatars from one game environment to the next. The transition from one phase to another was actioned by a member of the production crew. This enabled reasoned decisions to be made about the pacing of the action in discussion with the show’s director. In addition, some special purpose phases were defined to speed things along or cover technical failures without suspending the show as a live event. In addition, a virtual camera interface was developed to assist members of the production crew in obtaining meaningful views of the action. Shots could be targeted on specific avatars or objects. The distance from the target and movement around it could be manually controlled to permit various zooming and panning effects. Free navigation could be engaged to enable “roaming” shots but the cameras could always “snap” to a particular target if the operator received an instruction from the director or picked up on a cue from the action itself. Out of this World was much more successful in realising an event which was recognisably a game show than was Heaven and Hell – Live. The pace of the action was more consistent with what one would expect from such a piece of entertainment. The inhabitants were much more focused on the games within the show and their role in competing against the other team in playing them. In large part this was due to furnishing the production crew of the show with applications which were specifically designed to assist in production management and camera control. It is also important to emphasise, though, the role of the production crew’s professional competence in working with these technologies. For example, there are, in principle, many different ways in which a game show in a virtual environment could be shot and edited from the views offered by the four virtual camera operators. Not every way of working with the cameras or of selecting shots would have led to a recognisable production of a game show. Indeed, the director experimented notably with different sets of instructions for the camera operators and different principles for making edits over the course of her work on Out of this
168
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
World (Bowers, 2001) before settling on a style and pace of shot composition and editing which seemed appropriate. The main difficulties with Out of this World as identified by audience members centred on three problems: ●
●
011
011
011
011
11
●
A lack of empathy with the show and its characters. It was hard for audience members to engage with the concerns of the robots and aliens. The cover story for the show was too thin. A lack of a sense of history and context. In particular, no effort was given to elaborating any sense of history for the characters, why they came to be on the space station, what their past relationships with each other might have been, and so forth. Objections to the game show format. Many audience members thought that the choice of a game show format was unambitious.
In more recent work on inhabited television, these problems have been directly addressed by embedding the broadcast within a whole series of preparatory activities which help elaborate a participatory narrative. Avatar Farm/Ages of Avatar prepared for a live Internet broadcast show by providing a series of online worlds and allowing a sense of “community” to develop over the course of several months. Characters were introduced into these online worlds and narrative fragments were suggested before four inhabitants were invited to participate in the live event itself (for details of this work, see Craven et al., 2000).
10.4 Production Management Whether we are concerned with cultural events in an artistic or entertainment tradition, it is clear that some major challenges are presented to virtual and mixed reality research if those technologies are to be used to realise the events. In the events described so far, we have addressed particular issues concerned with world design, and navigation and view control. In Lightwork and Blink, we were designing virtual environments of a visually rich and (hopefully) engaging nature. However, we needed also to ensure that the worlds were presented in such a way that interesting views on them would be relatively easy to find. In other words, there was a reciprocal relation between world design and view control issues. Technically, we addressed this problem on geometrical and optical grounds. For example, we “layered” world content within a sphere in such a way that chambers could be created and viewed inside and out by switching between an inner and an outer orbiting path. Practically, we provided performers with a variety of resources for creating worlds and, in Blink, choosing between cameras. Our intention was to give performers a rich range of resources for world creation and navigation but to structure them in such a way as to make them usable within the real time of an improvised performance setting.
169
Inhabited Information Spaces
As in the research on inhabited television, we realised early on that it was impractical and unnecessary for performers, inhabitants or the operators of virtual cameras to have unconstrained six degrees of freedom (6DOF) motion when trying to frame shots in the environment. It was all too easy to overshoot the target or end up at an inconvenient location. To remedy this, more application-specific movement methods were developed for the later inhabited television productions to enable camera operators to focus on particular persons or objects and move around them without losing them from sight. These camera vehicles could, for example, constrain their motion to the surface of a sphere centred on the relevant object, thus simplifying movement to a 2DOF (plus zoom) operation. In Out of this World, the virtual cameras interworked with a production management application so that, for example, cameras could be moved to set locations at the beginning of a new phase or be able to take as the object of their shot a particular entity which might be important to the events in the current phase. In all these respects, production design and management is a matter of conjointly designing virtual environments and navigation control so as to simplify the real time burden on participants in live performance.
10.4.1 Finding and Framing the Action We have already mentioned that the director of Out of this World experimented with a number of ways of organising the camera operators and selecting from the shots they gave. One of the reasons for this is that in television, as in cinema shot composition, it is not just a matter of selecting the right target and framing it. One has to compose shots and edit between them to convey a sense of the action in the scene. Sometimes it is enough to select a particular target (e.g. an actor or avatar performing a critical and clearly legible action). At other times it is vital to show characters in relationship to each other, or in mutual relationship to objects in their environment, or juxtapose one shot with another to capture the action. The problem of finding the action and framing it is even more acute when there is a large number of participants (e.g. in Heaven and Hell – Live). Clearly, this is a different order of view control problem from those classically discussed in 3D computer graphics or virtual reality. How can we find and frame the action? Is it possible to design view control and navigation techniques which directly support this requirement? To begin to investigate these issues, we developed SVEA (Sonification and Visualisation for Electronic Arenas), a tool which enabled us to view the patterns of people moving in the space and to place cameras in the environment. The intent was that these cameras would have semiautonomous behaviour, left to themselves they would roam the environment, finding the hotspots of activity and frame a shot so that it would
170
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
capture the group of actors at that point, attempting to (depending on the settings) maximise either the number of faces or profiles in view. It was also possible to add additional weight to specific performers, so that they would be “favoured” in view. A human operator could then take over to make any adjustments, or even move the camera somewhere else deemed more important at the time. We give further details of our techniques for camera deployment shortly. Attempting to further enrich the impression one could have of the behaviour of a populous crowd we experimented with sonifying a number of parameters, so that one would be able to hear, for example, the levels of aggregation. (Further details of our sonification strategies and a preliminary experimental test of the sound model we used can be found in Bowers et al., 1999). To find the potential areas of interest in an environment, we made a novel use of the Spatial Interaction Model (Benford et al., 1995b). The central ideas in this model are that in a virtual environment (potentially) every object has a focus and a nimbus, where the former is (approximately) an abstraction of an object’s “attention”, and the latter (approximately) an abstraction of an object’s “projection of presence”. Typically these are taken to be functions over space, but it is possible to extend the notion over time as well (S¸andor et al., 1997). The combination of focus and nimbus can then be used to model an object’s awareness of another object: the awareness A has of B is some function of A’s focus on B and B’s nimbus on A. An ordinary language approximation of these notions might be: my awareness of you depends upon my level of attention to you and the degree to which you are making your presence felt. Benford et al. (1995b) and S¸andor et al. (1997) show how (given certain assumptions) awareness levels can be quantified in continuous and graphed “spaces” respectively. These notions have been used in a number of systems for collaborative virtual environments to control the level and kind of detailing in rendering in shared environments among other applications. To support camera control and view selection in virtual environments on the basis of finding and framing activity, we make use of the spatial model on the assumption that objects (most specifically avatars) in the world will be differentially aware of the various parts of the scene and will themselves orient towards things of current interest. In principle, we can compute various “activity landscapes” using focus, nimbus or calculated awareness. For example, we can determine the sum for every point in space of the nimbus at that point contributed by every object in the environment, giving a “nimbus landscape”. An awareness landscape could be determined in the corresponding way using the awareness level for every object of others at each point. A nimbus landscape would give high values at points where many objects are nearby, the latter gives high values at points where, in addition, many objects are focusing upon each other. Such landscapes can then be visualised as a “heat map” of
171
Inhabited Information Spaces
the environment, where the levels in the activity landscape are displayed. In SVEA, we simplified this notion in a number of ways. We visualised only avatars. As motion in these particular settings was taking place in a horizontal plane we could justifiably ignore the avatars’ heights above the ground plane. We computed the heat maps for just the points where avatars were located. This allowed us to visualise the avatars as small triangles (showing position and orientation) in a 2D “overview” environment and colour them to show the landscape value at their location. Naturally, other visualisation techniques could be experimented with – see (Hirtes et al., 1999) for discussion. A visualised activity landscape (map) could be used for navigation and view control in a number of ways. First, directly: a user could move to areas of interest on the activity map by directly interacting with it. That is, one might click on a location and be teleported to it. Alternatively, the map could be used to inform the use of conventional navigation controls. That is, one might see an area of interest on the map and move towards it using whatever one’s current navigation vehicle might be. Finally, one could use the activity map algorithmically. For example, a region of the map might be selected (e.g. by drawing around it) and a camera location and orientation computed which maximises the number of avatar “faces” or “profiles” in the region that are in shot. In Hirtes et al. (1999), we describe a number of algorithms which could do this and similar view computations given a selected set of objects to display in shot. To give another example of how an activity map can be used algorithmically, consider its use as a map of dynamical “potential” which would govern an autonomously determined camera movement. As an example, we implemented what we termed a “puppy camera”, a camera which would follow the gradient of awareness until reaching a local maximum and, if this point was stable, stay there for a while until the camera gets “bored” and sets off in a random direction to find a new local maximum – a behaviour analogous to that observed in puppy dogs that the authors have encountered. This camera thus roams an environment seeking out interesting places and capturing views from these. We experimented with a population of four puppy cameras. In order to keep them from flocking to the same highly interesting event (something that was a concern even with human camera operators in the inhabited television shows we studied), the cameras could avoid each other by associating a force of repulsion to each of them. As a proof of principle, we used SVEA to present data captured during Heaven and Hell – Live (for more information, see Hirtes et al., 1999 and Figure 10.8).
10.4.2 The Round Table: A Physical Interface To concretise our work with SVEA, we needed to take the application beyond a proof of principle and prototype a use scenario. In the work
172
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
011
011
011
011
11
Figure 10.8 SVEA visualisation of a crowd based on the data recorded from Heaven and Hell – Live. The inhabitants are represented by coloured triangles, showing both their awareness levels (represented by saturation changes rather than colour scales in order to accommodate colour-deficient users) and their orientation in the plane. Open triangular objects represent cameras and their immediate field of view. As we did not have access to the geometry data of Heaven and Hell – Live, the background is empty, but the geometry is of course trivially added. At the bottom is a timeline showing the current time in the data set – in the prerecorded data set a user could move back and forth in time at will, otherwise the timestep would be updated at the rate of incoming data.
on inhabited television we have discussed, shot selection and vision mixing was done by the director using conventional television equipment, a video line entering a mixing desk from each workstation running virtual camera software. In our initial version of SVEA, shot selection and camera deployment were accomplished by mouse operations. While our camera algorithms worked very well, it was awkward to manually orient cameras in the environment with a mouse-driven interface. Furthermore, if the mouse was moved to select a new camera, time delays were introduced in comparison with the simple button press used to
173
Inhabited Information Spaces
select an input channel at a conventional video mixer. The direct yet speedy manipulation of a SVEA visualisation to compute camera locations and select views seemed to require a novel interface solution. In collaboration with Michael Hoch (then at the ZKM in Karlsruhe in Germany), we worked with the Round Table as a physical interface approach. The Round Table itself is composed of a projection surface and a camera (see Figure 10.9). Optically tracked physical objects (“phicons”) can be manipulated to accomplish interaction operations, their positions and orientations being reported on by image analysis software. Bowers et al. (2000b) describe a number of applications of the Round Table including a virtual space sound mixer and as a presentation device for interactive artworks. We projected SVEA visualisations on the table, placing differently shaped physical objects to represent cameras, to select groups of objects for framing by algorithmically deployed cameras or for zooming up for detailed scrutiny. These latter two facilities made another use of the idea of an awareness landscape. Selection of a group for the camera would be done by placing a phicon at a particular point. Those avatars that were aware of that point would then be selected for a camera shot. Likewise a zoom would be done so that the view grew to contain just those objects aware of the given point. By default there were four puppy cameras roving around, but the placement of a camera phicon would snap one of the cameras to that view overriding the puppy behaviour. If a marker was placed on the camera phicon, that camera was selected for transmission (TX). In the real world environment of the Round Table, we
Figure 10.9 Michael Hoch using the Round Table. Inside the cylindrical bottom is a projector displaying images on the matt glass tabletop. In the can above the table is an infrared lamp and a camera with an IR filter. The manipulable props are coated with IR-reflective tape making them visible to the camera. Reproduced with permission from Michael Hoch.
174
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
placed other screens to display views from cameras and TX. Thus, we are envisaging a scenario in which users can share the physical environment of the table and co-operatively work using it as a shared display to manipulate views of a large-scale virtual environment. Depending on the application these users could be behind-the-scenes production crew, selected inhabitants, or other participants. We found that interacting with physical props, while allowing all the advantages we had foreseen – rapid manual interaction by multiple users in close proximity – raised new issues. Of course, the phicons would not move of their own accord when the display was updated. This meant, for example, that when zooming in on a section of the view, any previously placed camera markers would remain where they were while the objects they were presumably trained at would move away. Obviously this could not be taken to mean that their corresponding cameras would jump to those new points, so instead we chose to de-assign those cameras and let them continue moving on their own until picked up again by a camera phicon. This meant we had to change the metaphor from placing representations of cameras to placing a tool that would in turn place a camera. This is somewhat indirect, but perhaps not more so than the indirection inherent in using a mouse to move objects on a separate screen. If such a solution seems opaque, then alternatives would be to disallow zooming or to compute zooming algorithms which distorted the display to maintain at least some of the camera-phicon associations, among other possibilities.
10.4.3 Conclusions The most important points of our work with production management are the following: ●
●
011
11
●
Real-time interaction. As noted in the introduction, all applications had real-time requirements, thus our camera-control algorithms also had to operate in real-time. We could not prepare camera paths beforehand but had to be able to find at least approximate shots automatically as events unfolded. Large-scale participation. We wanted to be able to accommodate large numbers of participants in virtual worlds, and therefore could not restrict ourselves to only following a few select performers, but rather we had to be able at any time to move the viewpoint to somewhere else in a possibly large space in order to capture an interesting event and also being able to notice that such an event was taking place. Understanding rules of practice. We need to start from how television actually is produced. Our suggested interfaces indeed are quite different from what is used by television producers today, but they do
175
Inhabited Information Spaces
●
●
●
not in themselves preclude the tasks that need to be done, even if they suggest alternative fashions in which to perform them. Division of labour. Our systems are created with the goal of allowing a working group to fluently and in real time divide the work between themselves as well as between computer system and human workers – a camera can be passed from autonomous algorithm to human camera operator, to a different operator and back to the computer process, all by simply moving a wooden block over the projection surface. Improvised action. A script can help camera operators and directors to plan their work but an inhabited television event is by necessity at least partially improvised on account of the invited inhabitants which make every show a live event, so it must be possible to fluently adapt one’s work as needed. Activity-orientation. Camera direction is not concerned with geometry but with activity. Therefore we have developed methods for finding activity and orienting cameras in relation to it.
10.5 Discussion: Navigation, Presence and Avatars We have been discussing some of the work from the eRENA project which was concerned with supporting large-scale real-time interaction in cultural and entertainment events using virtual and mixed reality technologies. In these settings we have described the interaction techniques and world design principles we have worked with in a number of artistic and entertainment applications. While we have presented some specific applications, these demonstrate some general design principles. By way of discussion, we situate our design principles in relation to general questions of interaction, navigation and avatar design for virtual and mixed reality environments.
10.5.1 Avatar-centred Navigation Classically, navigation in virtual environments is taken to mean something like “move my avatar in a certain direction with a certain orientation at a certain speed”. Various restrictions may be placed on what directions and speeds are allowed – e.g., one may use a “walking” or “driving” metaphor where the avatar is restricted to move along a ground plane, walls and other objects may be impenetrable, or one may use unrestricted six-degree of freedom motion, passing through any object at will. A typical experience is that it is preferable to have additional constraints on motion, in order to avoid ending up at an awkward angle halfway through the groundplane, unsure of how to manipulate one’s
176
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
interaction device so as to straighten up and stand on the ground again. In part this may be due to mismatches between the input devices and the available degrees of freedom, mouse-based interaction, for example, requiring modifiers and/or modes in order to support all six degrees of freedom. Still, even when using 6DOF devices, it is often difficult to reach one’s intended goal at the intended orientation. We also questioned whether it is necessary to actively steer one’s avatar every single step of the way, if one already knows where one is going. A common experience is that most of the time in virtual environments is spent on navigation, getting desired objects into view and then moving to them, both taking up valuable time and being a cause of frustration when one’s path goes astray or the intended target cannot be found. Accordingly, when we have encountered avatar-centred navigation in the work reviewed above (e.g. to support the inhabitants in inhabited television), we have found it most effective to constrain motion and map it in an activity-informed way onto simple (low DOF) navigation devices. Not only does this make for easier participation, it also makes the distribution of avatars more predictable, thus facilitating production and camera work. In short, rather than work with generic motion and selfrepresentation notions, we believe that the capabilities of an avatar should be specifically configured for the activities it is to engage in and so that its conduct can be appropriately picked up by others.
10.5.2 Object-centred Navigation Another common navigational paradigm is based on objects in the environment. Viewpoints are predefined points that the designer of the environment deemed likely to be interesting for a visitor. By choosing the name of a viewpoint in a list, one’s avatar is transported to that spot and encounters an intended set of objects there. The transportation may be a smooth animation through the environment, or it may be an instantaneous relocation, a teleport. A teleport may also be undertaken to some arbitrary co-ordinates; this requires that there is some kind of map of the environment, or that the co-ordinates of some likely spot can be easily ascertained and saved, for example, as a URL. Target-based movement allows the user to point at some visible object in the environment and be brought there, again either through a smooth animation or through a teleport. We have reported explorations which have further refined objectcentred navigation so that, as in the later inhabited television experiments, significant individual avatars or the “centre of gravity” of a group of them can be targeted. The camera control application used in Out of this World also supported viewpoints but, significantly, these were integrated with a production management application to support a working division of labour between a show producer and a camera operator.
177
Inhabited Information Spaces
Furthermore, once targets were selected, view control was informed by the need to produce coherent camera shots, rather than only allowing unconstrained motion. Our experience, then, is again that navigation and view control need to be informed by the practical activities that participants are performing (e.g. as a camera operator or as a show producer) and specifically configured with those ends in mind.
10.5.3 Activity-oriented Navigation We wanted to go further in the direction of informing interaction in terms of the practical activities that participants are engaged in by supporting navigation and view control where the intended target is not a geographical spot as such, but a place of interest and activity. We have described an approach which employs a model of awareness and activity in virtual environments to make inferences about where the “hot spots” might be so as to enable transportation there – in effect like a helpful taxi driver in a foreign city when asked: “Take me where the action is.” We described how navigation informed by representations of activity (activity maps/landscapes) might interwork with algorithms for view computation, for example, to maximise the number of faces of a group of avatars who are at a particular hot spot. While we presented these notions in terms of the deployment of cameras in the production of events, similar tools could be made available to inhabitants and other participants so that, for example, they can visualise an overview of activities and occurrences that they happen to be interested in and manipulate this to control their movement in the environment.
10.5.4 Navigation as Montage, Dispersed Avatars A consequence of the above is that we may loosen the idea that navigation is the moving of an avatar through space and make it more akin to real-time image cutting in television or film production, selecting from multiple viewpoints. A teleport still uses the metaphor that there is a single avatar, though one that can be instantaneously transferred from one place to another. But consider the following scenario: there are multiple users, each represented with an avatar, indicating their position in the environment and serving as a focus for the interaction of others with them; at the same time there is a pool of cameras that any user can access to check out what’s going on in the rest of the world, and on finding an interesting place, can ask to be teleported there. Now, in a sense the cameras in this scenario are parts of the user’s avatar, as the user is able to perceive the environment from the viewpoint of a given camera, yet it is not a unique access point, as several users may be peering through the same camera. If the cameras are
178
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
New Ideas on Navigation and View Control Inspired by Cultural Applications 10 ● ● ● ● ● ● 1 ●●●●●●●●●
extended with the ability to manipulate the environment, we have completed the dispersion of the avatar. Furthermore, from this perspective it can be noted that our Round Table implementation of SVEA disperses views not only over multiple cameras, but also their control over several physical devices which in turn can be shared between several users. In this way, we allow for flexible mappings between users, peripheral devices, displays, views, cameras and avatars.
011
011
011
011
11
10.5.5 Accomplishing Presence and Intelligibility We are prepared, therefore, to entertain some radical consequences of our work for interaction concepts in virtual and mixed realities. Components of the avatar “function” might be distributed and disembodied (e.g. there is no necessity for cameras to be “owned” or visualised). Of course, we may wish to constrain the picture for particular purposes in design. Our emphasis throughout has been on informing navigation and related issues on the basis of general event design and production considerations. Particular events may indeed require a traditional avatar concept. Other events may allow or require very different approaches. Furthermore, one may need to support different realisations of the relationships between navigation, view control and embodiment for different participants in the same event (e.g. a traditional avatar for an inhabitant, a dispersed capability for production work). By decoupling navigation from view control and by varying all these matters with respect to avatar-like embodiment, it might be argued that we are inviting complexity and unintelligibility. After all, much conventional thinking about traditional avatars and interaction in multi-user virtual environments is based around assumptions about how avatars can be used as a resource for mutual inferencing about participants’ conduct and perception. For example, if I can see your avatar at such a spot oriented in such a way, I should be able to make inferences about what you can see and, perhaps, what you are doing. On our more distributed and variably embodied model, how can this be true any more? Interestingly, Bowers et al. (1996) present ethnographic evidence that it is not the case that such inferencing occurs automatically and nonproblematically in all circumstances in conventional shared virtual environments. Participants commonly check whether their sight of another’s avatar does indicate the other’s presence before interacting with them, for example, by pre-calling the other’s name. Avatars might be creatively used to indicate the momentary unavailability of their user, for example, by laying them down. Avatars might change “ownership” as someone else in the real world takes over at the workstation. Other communication media might be used alongside the virtual environment itself to troubleshoot problems and sort out identities. In short, the presence of
179
Inhabited Information Spaces
persons in a virtual environment and the intelligibility of their actions are matters which are accomplished using designed technologies alongside whatever other resources participants have for making sense of what is going on. Participants “naturally” decouple navigation, view control, embodiment and activity, and reassemble them in their practical activity within virtual environments in whatever way is appropriate to the task at hand. From this point of view, our advocacy of an approach to interaction which regards navigation as montage and disperses the traditional capabilities of the avatar over multiple perceptual loci turns out to be a modest proposal.
180
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
11
Presenting Activity Information in an Inhabited Information Space Wolfgang Prinz, Uta Pankoke-Babatz, Wolfgang Gräther, Tom Gross, Sabine Kolvenbach and Leonie Schäfer
11.1 Introduction
011
011
011
11
A group of people that work together in the same spatial environment relies on various possibilities to observe the activities of partners, to monitor the progress of a project or to recognise the development of group structures. Such activities are often recognised peripherally; nevertheless, they stimulate spontaneous and informal communication. Although the computer supported co-operative work (CSCW) research area yielded a number of systems and solutions that enable and support distributed co-operation processes, distributed work is still significantly more difficult to manage than co-located work. A significant reason for this is the missing perception of the activities and actions within a distributed group. Therefore, distributed groups often suffer from a lack of awareness of the common activities. Co-operation partners are often not aware of activities of other co-operation partners that are relevant for their own work. The synchronisation problems resulting from this often lead to decision problems, misunderstandings or duplicated work. Thus, effectiveness, spontaneity, and social orientation possibilities in distributed teams are limited. The social forces which facilitate the behaviourmilieu synomorphy in an environment – that is, in a behaviour setting (Barker, 1968) – are very limited in electronic spaces. Awareness support can make the difference between an electronic behaviour setting and a pure electronic space (Pankoke-Babatz, 2000). Apart from the lack of awareness of actions that could be co-operative, there are limited opportunities for chance meetings. In the local working environment, coincidental meetings often initiate communication and the exchange of experience and knowledge. Prussak (1997) describes this phenomenon very appropriately: “If the water cooler was
181
Inhabited Information Spaces
a font of useful knowledge in the traditional firm, what constitutes a virtual one?”. Social contacts that are initiated by chance encounters at copiers, printers or coffee machines are important for the social orientation, the mutual exchange of information and knowledge, and the coordination of shared activities (Swan et al., 1999). The Tower system that is presented in this chapter addresses this problem. Tower (Theatre Of Work Enabling Relationships) provides different approaches for the support of awareness and the creation of chance encounters for local and distributed teams. First, we describe requirements and the methods we applied for requirements analysis. This is followed by the architecture of our awareness environment. Then we introduce different means for the presentation of activity information using a portal, Smartmaps, a 3D environment, or ambient interfaces. The chapter concludes with lessons learned about awareness in distributed electronic settings.
11.2 Related Work and Requirements The importance of awareness for CSCW was initially described and analysed for synchronous co-operation processes (Dourish and Bellotti, 1992). Later, different approaches were presented to stimulate awareness of shared activities through different awareness widgets such as multiuser scroll bars, or radar views (Roseman and Greenberg, 1996). For the generic support of synchronous applications, infrastructures have been developed that support the exchange of synchronisation and notification events between different applications by the provision of a notification server (Patterson et al., 1996; Segall and Arnold, 1997). Awareness is of equal importance for the support of asynchronous co-operation processes. For successful co-operation it is essential that users are informed about relevant activities of co-operation partners in a situated and intuitive way (Schlichter et al., 1998). Most approaches that provide asynchronous awareness visualise user actions by appropriate awareness icons, that indicate current or past activities on a shared object (Sohlenkamp et al., 2000). Infrastructures to support asynchronous awareness have been presented in Lövstrand (1991) and Prinz (1999). Video-based media space (Gaver, 1992; Lee et al., 1997) or video walls have been developed to bridge spatial distance between different locations and to support chance encounters and ad hoc communication between different places. In Benford et al. (1997c) such an approach is applied in a VR environment to provide chance encounters for people who browse the web. The Session Capture and Replay system is an example of a system that allows other users to replay past actions. This is particularly helpful for users who join the group process later. The system captures users’ interactions with an application and stores data into a session object. Group
182
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Presenting Activity Information in an IIS 11 ● ● ● ● ● 1 ●●●●●●●●●●
members can annotate, modify and exchange the session objects. Session objects consist of data streams representing captured interactions and audio annotations, which users can add while their interactions are captured. When a session is replayed, data streams are re-executed by the application (Manohar and Prakash, 1995). Begole et al. (2002) did statistical analyses on the availability of users of an instant messenger system and found out work rhythms. These work rhythms are used to project the current or future availability of other users. Common to all these approaches is the concentration on a particular co-operation type (synchronous/asynchronous) or a restricted application domain. However, co-operation processes are not limited to a particular application or a specific co-operation type. This is why Tower aims to develop a model that supports awareness across the boundary of different applications and that can visualise activity information using different presentations. The system provides an application-independent infrastructure that receives event information submitted by user activities or activity sensors. It stores and administers the event database and forwards events for the presentation by different indicators to users based on user-specific interest. Such an infrastructure is able to provide a comprehensive awareness support for different co-operation situations. In the following, we describe different scenarios and the resulting requirements for an awareness environment. A prerequisite for the smooth operational sequence of a co-operative process is the seamless interlinking of the actions of different team members. The availability or explicit provision of information or working objects that are required for the next activity often initiate user activities. To minimise meta-communication by which co-operation partners tell each other what they have done, it is necessary that notifications about relevant activities are produced automatically and presented to the user in the appropriate situation (Pankoke-Babatz and Syri, 1997). In addition to the immediate notification, users require a presentation that provides an overview of the current activities in the co-operative environment. Such an overview must be user configurable to allow each user to adapt it to their specific needs. We describe such a user configurable awareness portal later in this chapter. Users who work on a shared object or within a shared workspace require object-related information about the activities of others. Most systems provide activity symbols that indicate recent activities by others on an object. In addition, it is helpful to get an overview of all activities in a shared workspace or environment. We will show how such an overview can be provided using Smartmaps indicators that are based on tree-maps (Johnson and Shneiderman, 1991). In addition to the indication of activities, they also support an activity-based navigation in a shared workspace. This is exemplified by the integration of Smartmaps with BSCW shared workspaces (Appelt, 1999).
183
Inhabited Information Spaces
The relevance of peripheral awareness to support mutual orientation was identified in Heath and Luff (1991). In addition to the summarised presentation of awareness information via a portal, an environment such as Tower must provide possibilities for the presentation of user activities in the user’s periphery. For this purpose, Tower applies a 3D environment in which user activities are visualised automatically. The layout of the 3D environment is based on shared working contexts. Thus, the environment also represents meeting places where people who work in a similar context but at different geographical locations meet coincidentally. We will show in this chapter how such a world is created and how user activities are represented in the 3D environment. The settings discussed so far describe synchronous communication patterns. However, the support for long-term project work requires the support of asynchronous awareness. As also shown for video-based media spaces (Fussell et al., 2000), the continuous observation of remote partners does not suit team needs. Therefore, support for the presentation of past activities was developed in Tower. This supports users who need information about recent activities after a temporal absence from the co-operation process. A similar approach is described in Greenhalgh et al. (2000a). In the DocuDrama section of this chapter (Section 11.8) we describe how Tower addresses this issue. Although awareness is studied extensively in synchronous and collocated work situations (e.g. Heath and Luff, 1991), needs in asynchronous and distributed settings are still under-researched. Tower has provided the opportunity to study awareness needs in different settings and to use a tangible artefact (Brave et al., 1998) to elicit further needs and to discuss the potentials of technical support in more detail with potential users. The results of these user studies augment the description of the presentation tools described in the following. Finally we present the lessons learned about awareness.
11.3 User Involvement and Studies From the very beginning of Tower development, user groups participated in the project team. Their participation enhanced the project team’s understanding of user needs with respect to awareness. Planned Tower features or prototypes were discussed with the users and their early feedback improved the system design. We selected such an iterative design process based on qualitative evaluations since the co-operation processes supported by Tower were also long term and asynchronous. Studies of awareness needs and requirements for technical support were performed in two different ways. On the one hand, discussion of needs and the usability of awareness features were discussed with potential users of application partners. Here, needs for awareness support were studied by means of interviews and workshops. On the other hand, the
184
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Presenting Activity Information in an IIS 11 ● ● ● ● ● 1 ●●●●●●●●●●
team process of the distributed Tower team itself was observed throughout the whole course of the project.
11.3.1 Partner Settings and Evaluation Methods One application partner was a small German company with about 20 staff members. Its major aim was the provision of web-based support mainly to engineers. The other partner was one of the world’s leading providers of civil engineering consultancy and support services currently expanding to about 10,000 employees. Representatives of the two Tower application partners also became members of the Tower team. The collaboration began with expert interviews and expert workshops. We discussed the Tower features planned and under development as well as particular needs at their application partners’ sites. This was followed by five user workshops performed at the application partners’ sites. Between three and ten staff members from the respective application partners participated per workshop. Workshops comprised three kinds of topics: the need for awareness support in the respective settings, the demonstration of Tower features and finally the discussion of the potential usability of those features. Thus, Tower features were used as tangible interfaces to elicit further user needs. The Tower development team was distributed across four companies in two countries. In addition one expert from each application partner joined this team. In total, about 40 people had access to the Tower workspace that was used to co-ordinate the project work. Among them about 15 used it regularly. In the course of the project we could study how awareness needs changed depending on the team process and on particular work situations. Seven persons joined the project at a later stage. From them we could study the particularities of the awareness needs of a newbie. The course of the Tower team process was documented continuously from the members’ perspective. In addition, technically measured data – that is, recorded event data – were analysed to understand the team process, the relevance of media usage, and the requested awareness support in the respective phases of team work. This enabled a detailed view into the awareness needs but also into the way in which awareness was achieved in a distributed work setting. This also informed our study on how work activities are reflected in the recorded event data and how these can be processed to provide suitable awareness notifications. The diversity of application partners in Tower has provided the opportunity to study various different work settings and to find out about their particularities with respect to needs for awareness support. With the introduction of early prototypes we were able to test potential effects on users. Where there was positive acceptance of a feature, we could learn from users how to improve it. Where users objected to or rejected a
185
Inhabited Information Spaces
feature, more detailed analysis of the underlying rationale was needed. We had to analyse whether the rejections were due to shortcomings of the feature, which could be overcome, or whether they were due to misfits with their needs and misassumptions about awareness. Interestingly, any feedback we gained from the users also disclosed user-specific needs for awareness support on the other hand. Thus we could learn a lot about awareness needs in general. We found that the needs for awareness depend on the different settings, the particularities of the co-operation cultures, the modes of working and the current work process and the individual interests of an actor in a particular situation. When Tower started the major aim was to design a theatre of work, i.e. a 3D stage which augments an existing shared workspace system with the requested awareness about ongoing activities. In the course of the work, however, awareness turned out to be much more complex than we had thought and we realised that it cannot be supported with a single tool. In real world settings, awareness is a multi-channel phenomenon, thus it requires multiple means for the indication of activity information in an electronic setting. Therefore, in the course of the project several different features were developed, from among which a user may choose the most appropriate one.
11.3.2 Do Users Meet at all in a Shared Workspace ? Being present in the same place usually provides an opportunity for chance encounters. One motivation for Tower is the provision of an awareness space that notifies users about the presence of others in their working context. We have analysed the log files of user activities in a shared project workspace to check whether situations occur where more than one user has been present at the same point in time. In such a case, knowing about the local presence would have opened up a chance to meet in the context of the actual work. Such functionality goes beyond the services offered by simple presence awareness systems such as ICQ (“I Seek You”, a service to get in touch with people across the network: http://www.icq.com/). The log file contains data for each user action in the shared project workspace for the period from January until October 2001 (approximately 200 working days). In this time period, about 15 users had access to the workspace, but there were only 8 active users who used the workspace frequently. The project workspace contained more than 1000 objects, e.g. documents, presentations, figures, etc. The result of our analysis is shown in Figures 11.1 and 11.2. Each figure shows the number of meetings between two or more users within a certain time interval. A meeting is defined by two consecutive actions of different users on the same object. The x axis indicates the time interval between two actions in minutes. The y axis denotes the total number of
186
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
Presenting Activity Information in an IIS 11 ● ● ● ● ● 1 ●●●●●●●●●●
meetings that occurred within a time interval. Figure 11.1 indicates that only 20 meetings occurred within a short time space of 1–2 minutes, while 60 meetings took place over a 10-minute period. This means that every third day two users met on exactly the same object. If we consider folders within the project workspace as meeting places, then the number of meetings increases drastically. The grey line in Figure 11.1 indicates that within 5 minutes 60 meetings between two or more users took place in the same folder. Using 15 minutes as the time interval we count 90 meetings, i.e. every second day people have met coincidentally in the same folder. These numbers increase further if we consider the whole project workspace as a meeting place. In this case more than 400 meetings took place within a 5-minute period, i.e. two or more meetings on every working day. This frequency of coincidental meetings is similar to the number of meetings between two people in the same spatial environment. The statistics provide evidence that users actually meet in a shared virtual project or workspace and they further indicate that it is important to choose the right level of detail for the construction of the meeting
011 Same object 140
100 80 60 40 20
011
011
11
Same folder
120
– –
10
20
30
40
50
60
Figure 11.1 Number of meetings on the same object and in a folder.
Same project space 1200 1000 800 600 400 200 0 –
10
20
30
40
50
60
Figure 11.2 Number of meetings in the project workspace.
187
Inhabited Information Spaces
places. If the level of detail is too high, then the number of coincidental meetings is too low. If it is too low, more meetings will occur but then the context of the meetings is too general. The empirical observations are confirmed by our practical experiences with the construction of the Tower world that are reported later in this chapter.
11.4 The Tower Architecture In order to support awareness in a distributed electronic environment, the events that take place at one location or application, must be conveyed to the locations of the other users sharing the environment. First, this requires a means to detect events caused by user actions. Secondly, these events need to be recorded and processed. Finally, the events must be presented and situated in the local action field of the other users. Consequently, the Tower system provides three basic components: a set of various sensors that recognise user actions; an event and notification server that stores, administers, filters, and distributes events; and a set of different visualisation tools. In the following we describe the architecture of Tower. Figure 11.3 illustrates the components and architecture of the Tower system. Different sensors recognise user actions. These are either integrated in applications or realised as agents that observe user actions, e.g. modifications of shared files systems or web pages. Sensors forward events to the central event and notification server (ENI) by calling appropriate common gateway interface (CGI) methods of a web server. This web-based approach was chosen to provide a simple, yet powerful interface that can be used by almost all applications (Prinz, 1999). The ENI-server stores and administers events. It forwards events to appropriate indicators that have a registered interest in events using predicates over event attributes. In addition, the server implements functions for the authentication of users, the authorisation of access rights, and the aggregation of events in a history. The transformation module allows a semantic transformation of events to satisfy requirements of different applications. Further, it supports the interworking of different ENI servers by enabling servers to exchange events. The reciprocity module informs users about the interest of other users. This provides transparency to avoid the misuse of the awareness environment for control purposes. Additional modules can extend the ENI-server. For example the context module has been developed to contextualise incoming events. Examples for contexts are a project, a task or collection of document folders that belong to a work package. The classification is done by matching event attributes with a predefined context description (Gross and Prinz, 2000). In the following sections different indicators are described for the presentation of activity information in inhabited information spaces.
188
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Presenting Activity Information in an IIS 11 ● ● ● ● ● 1 ●●●●●●●●●●
Work environment Activity sensors
Agent sensors
Event processing modules
Theatre of work Ambient interfaces
HTTP-server
011
011
011
011
11
3D multi-user environment
DocuDrama
Symbolic acting module Space module
CGI-scripts Event processing modules Authorisation Key action module
Context module
Event and notification server Interest profiles
Access control
Reciprocity
Aggregation
History
Transformation
Figure 11.3 Tower architecture.
11.5 Personalised Overview of Activities: The Tower Portal Our user studies revealed that an individual user may like to be made aware of several electronic locations. For example, a user may be involved in a project organising an electronic seminar and finally being a shareholder. Such a user may want to perceive awareness information about all relevant contexts. In case that these contexts are relevant over a certain period of time, this user may assemble awareness information about all these contexts in a personalised Tower portal. Figure 11.4 shows such a portal of a Tower team member. It provides awareness about people currently present in the Tower project space and the activities in the Tower workspace as well as information about the current German stock market. The Tower portal provides users with a personalised visualisation of awareness relevant for their personal work context. For this purpose users configure their own Tower web page, called MyTower. For the individual configuration of MyTower users specify interest profiles to express which events and awareness information they are interested in (Fuchs, 1999; Fuchs et al., 1995). This may be the notification of changes to documents in shared workspaces, activities of group members, or
189
Inhabited Information Spaces
Figure 11.4 An example of MyTower.
information delivered by agents observing web contents. In addition to the selection of contents in the Tower web page, users decide on the visualisation of the event information. For this purpose the Tower portal offers a multitude of configurable indicators. The configuration of MyTower in Figure 11.4 contains five different indicators. In the first line the RandomImage-Indicator shows the photos of all users currently online in Tower. The Smartmaps gives an overview of the user actions in the Basic Support for Co-operative Work (BSCW) workspace of the Tower project. In the next row a ticker tape informs in more detail about the project activities of the Tower team members. The last indicator is a URL-Indicator. It displays the picture of the current stock development of the German stock exchange. Figure 11.5 illustrates the concept and the architecture of the Tower Portal. MyTower is an HTML document containing Java applets that inform users about new events and represent this awareness information in the web page. One of these Java applets is responsible for the communication with the ENI server. It administrates the user-defined interest profiles, asks the ENI server for new events, and distributes these events to the appropriate indicators, likewise Java applets. With the modular architecture and the implementation in Java, the Tower Portal is application independent. All indicators can also be integrated in external applications, so that existing systems can be extended with the Tower awareness widgets.
190
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
Presenting Activity Information in an IIS 11 ● ● ● ● ● 1 ●●●●●●●●●●
TOWER portal
Indicator
User login
MyTOWER User registration
Indicator
Communication applet
Indicator Indicator
Credentials
Event query MyTOWER
User authorisation
Events CGI-scripts
ENI server Personnel myTOWER user web page
Figure 11.5 Concept and architecture of the Tower portal.
011
011
011
11
11.6 Awareness in a Working Context: Smartmaps The users asked for simple means to provide awareness. To them awareness information and the possibility to access the respective objects were integral. When connecting to a shared workspace the traces of activities which took place since the last time an actor was connected were requested. In addition, while connected, awareness about current activities was required. Like in real world settings, situated action (Suchman, 1987) requires awareness information about the working space in which action takes place but, in contrast, in electronic workspaces situations may span over time. To meet these requirements, Smartmaps were developed. Smartmaps provide both task-oriented and social awareness (Prinz, 1999), i.e. they yield information about the state of artefacts as well as the presence and activities of people. Furthermore they provide a twodimensional layout of the electronic workspace which may ease the local orientation in the workspace itself. Traces of past or current actions are indicated as “footprints” in the Smartmaps. Smartmaps thus span synchronous and asynchronous work situations. Past actions as well as current actions are indicated in the space. Smartmaps provide an overview of all activities in shared information spaces like shared file systems, web sites, or shared workspaces in 2D graphics. Figure 11.6 shows the Smartmap of a large file system. All files in the tree of folders are represented as small rectangles. The hierarchy of sub-folder becomes visible through the thick lines. Files in the same
191
Inhabited Information Spaces
Figure 11.6 A Smartmap representing the files and user activities in a large file system.
sub-folder are represented close to each other. Some rectangles, i.e. artefacts, are highlighted in a different colour to indicate activity. Smartmaps is a Java applet implementation based on the tree-map visualisation technique (Johnson and Shneiderman, 1991). It shows artefacts of information spaces, preserving a lexicographic ordering. Different actions like read, write, rename, create, delete, etc. in information spaces are reported by the ENI-server as events to the Smartmaps. The events carry, for example, data about producer, artefact name, artefact type and operation. This information is interpreted and indicated in the Smartmaps by colouring the corresponding rectangles. The default presentation mode conveys the overall activity and their distribution in the information space. Tool tips, which are activated when users move the mouse over the corresponding region, indicate the artefact’s name. The tool tips also present the names of the persons currently acting on the artefacts. This enables people to know that there are currently others present and working in the same information space, the same part or even on the same artefact. To explain the application of Smartmaps for the provision of awareness in shared workspaces, we describe the integration of Smartmaps into BSCW shared workspaces (Figure 11.7). Integrating it into another groupware platform, e.g. Lotus Notes, is also possible. When combined with shared workspaces, Smartmaps augment the often-applied list mode presentation of the shared objects with the 2D spatial representation of
192
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Presenting Activity Information in an IIS 11 ● ● ● ● ● 1 ●●●●●●●●●●
011
011
011
011
11
Figure 11.7 A Smartmap showing activity in a BSCW shared workspace.
the complete workspace. When navigating in the shared workspace, the Smartmap highlights the actual position of the user in the overall workspace by an orange rectangle in the Smartmap. Thus, beyond the provision of the supply of awareness information it also eases navigation in the BSCW workspace. Users have the following options to interact with the Smartmaps. When moving the mouse over a Smartmap the complete name of the represented artefact or information about recent user activities is displayed. A mouse click presents the pathname of the artefact in the status bar of the browser window, a shift-mouse click opens the artefact itself, a control-mouse click opens the enclosing “folder”. A control-right-mouse click opens a pop-up menu to open the artefact, the enclosing workspace, and further enclosing workspaces up to the top-level workspace. These interactions enable an activity-based navigation in the information space and reduce the amount of time that is normally needed to navigate hierarchical structures. Several parameters configure the visualisation. The colour for highlighting can be chosen, the highlighting intensity can be specified, only artefacts, or artefacts and enclosing “folders” up to a configurable level can be highlighted with decreasing intensities. Furthermore the granularity of the visualisation is adjustable: complete (default); only enclosing “folders”; only the first x-levels of the hierarchy; and up to the last y-levels of the hierarchy. In addition the duration of highlighting can be set. Usage experiences have shown that a length of three minutes for Smartmaps representing busy web sites, and 4 hours for shared workspaces and shared file systems is a good duration. An overview of the activities that happened during the last day can be achieved by selecting a duration of 24 hours.
193
Inhabited Information Spaces
We have integrated the Smartmaps applet with several large project workspaces and we could observe that users quickly apply the Smartmaps to navigate the information space as well as for activity-based navigation. For example, they first check the places for which activity of other users is indicated. They move the mouse to the corresponding highlighted rectangle to see who did which operation on the object. Then, often that link is followed and the corresponding folder or object is opened. The Smartmaps applet in the header of a workspace provides users with an overview of the complete workspace structure, easy navigation, direct access to workspaces and artefacts, and informs about group members carrying out actions, currently or in the past. In fact such awareness-enhanced workspaces become an inhabited place for social encounters and activity-based communication.
11.7 Symbolic Actions in a Context-based 3D Environment All these presentation modes are text-based or use 2D graphics. To come closer to an intuitive way of presenting awareness information the Tower world was developed. It presents awareness information in a 3D world. Actions in the electronic space are visualised by symbolic actions of avatars representing the respective actors. The avatars are located at the representation of the corresponding object. In the project the Tower world was understood as the “Theatre of Work”, i.e. the stage on which the activities are played out that take place in an electronic environment.
11.7.1 The Tower World The Tower world realises context-based presentation of user activities in a multi-user 3D environment. The environment consists of a landscape containing representations of shared working objects such as documents, folders, etc. In the following figures, boxes are used for the representation of these objects. This representation can be very detailed, e.g. each shared document is represented by one object in the 3D world. The construction of such a detailed world can be done automatically using the space module (Gavin et al., 2000). This module allows users to select the type of objects that shall be represented as well as the grouping and arrangement of objects based on their semantics. That is, objects can be grouped based on their location in a shared folder, or on other attributes such as keywords, owner, modification dates, etc. More abstract worlds represent only contexts such as folders or an aggregation of folders in a work package or task context. Figures 11.8 and 11.9 show examples for detailed and abstract worlds. Our experiences have shown that most users prefer a more abstract representation, since this allows a better overview.
194
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Presenting Activity Information in an IIS 11 ● ● ● ● ● 1 ●●●●●●●●●●
011
011 Figure 11.8 An abstract overview showing different project contexts.
011
011
11
Figure 11.9 A detailed Tower world representing individual documents.
195
Inhabited Information Spaces
The Tower world is populated with avatars that represent users and their actions in a symbolic way (McGrath, 1998). This is realised by automatically moving avatars to places in the 3D world that correspond to the working context or the document the user is currently working on. Symbolic gestures of the avatar represent the current action of the user. For example, a read operation is visualised by an avatar reading a newspaper, while an avatar indicates a write operation with a large typewriter. The automatic placement of the avatar and the symbolic actions are controlled by the space and symbolic acting module of the Tower system (see the architecture in Figure 11.3). Team members are represented in Tower only when they are working on shared documents or when they perform activities in a public space. This distinguishes the Tower approach from video-based media spaces, where users are visible independently of their current working context. The exaggerated presentation of user activities by symbolic actions provides a good overview on the activities in a world and the overall situation, also from a distance. Users can easily recognise which documents, tasks, or colleagues currently perform related processes. Users whose avatars are close to each other in the Tower world are also working in a similar working context. Communication channels such as audio or chat are provided to enable a spontaneous conversation. Thus, the Tower world could serve as a chance encounter and context-based meeting space that facilitates coincidental meetings. For a peripheral awareness on user activities, the Tower world can be projected in the user’s office environment. Alternatively, it can also be included as a plug-in in the user’s web browser (Figure 11.10). Camera agents offer a guided tour inside Tower world. The agents are configurable and provide a personalised entry to Tower world and the represented events. A history camera agent guides the visitors to places at which in the past interesting events have taken place.
11.7.2 User Feedback User feedback gave evidence that the easy orientation in the world representing a workspace is of high relevance. Therefore, simplified views had to be provided. Now several worlds representing different levels of detail are available. One overview world displaying the major folders in a workspace and in parallel a detailed world with all the documents is provided. An experience with using the Tower world as well as feedback from users indicated that specific navigation support is needed. Users want to navigate to points of interests or to points of activities to take a more detailed look. This is an interesting finding, since at the beginning of the project, it was considered to be beneficial, that in contrast to other 3D worlds, we “let the system do the walking”. That implies that the world was intended to be a display medium, but not
196
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Presenting Activity Information in an IIS 11 ● ● ● ● ● 1 ●●●●●●●●●●
011
011
011
011
11
Figure 11.10 Tower world as an integral part of the web browser.
for navigation. This is still true for the control of the avatars that represent the actions of its corresponding user. But in the position of a visitor, users considered it necessary to have easy means available to select suitable viewpoints. To this end, users requested that the positions of avatars were clearly indicated. The position of an avatar indicates a location of interest, that is, a location of activity. Therefore, users requested large avatars relative to the size of the work objects. Furthermore, the feedback from various user groups pointed out that the appearance of avatars may depend on the formality of the social context. Workshop participants suggested starting with simple representations. Nevertheless, the avatars should be complex enough to be recognised and personalised to some degree. Participants demanded that it be easy to identify the person represented by the avatar. Furthermore, the avatars should look friendly. Among the participants, there was a clear preference for stylised avatars as opposed to naturalistic ones. Participants proposed to use team colours for the avatars’ clothing. To satisfy these needs, Tower provides a tool where
197
Inhabited Information Spaces
users can select among six different avatars and change colours of hair, eyes and dress and thus shape their personal avatar. Another interesting finding was that the users wanted to immediately access the workspace objects represented in the Tower world, although we intended the world in the first instance only as a visualisation and not as an interaction medium. All this gives a clear indication of the need to integrate the supply of awareness tightly with the action options of an actor. Actually, the users are actors and not spectators. They want to interact through navigation but also through interaction with the presented objects, just like they would interact in a 3D game environment. We can conclude that like in real worlds, awareness is an integral part of action planning (Lantermann, 1980; Leont’ew, 1977). The various user groups proposed to use the Tower world mainly for places with high levels of activity. For example, one user group suggested using it in a public space to show the activities on the web portal they provide. Another group suggested displaying the Tower world in their coffee room to show the overall team activity in their office. However, in the usual team setting the level of activity was too low to benefit from the synchronous presentation of awareness in the Tower world. The same finding also occurred for the usage of video spaces to support ongoing team work (Fussell et al., 2000). Instead, the support of medium and longterm teamwork requires that awareness spans activities over time. To meet these needs, we tried to replay the recorded event data over a given time period. However, the pure temporal order of events did not convey the right information about the activities in the electronic workspace either. Instead, more complex processing of event data is required. A possible solution for this requirement is presented in the next section.
11.8 DocuDrama DocuDrama aims at recording the history of desktop events and activities generated by a project team in the Tower collaborative work environment. Avatars that enact the events as they occurred in the shared workspace visualise the replay of the team’s interactions with documents. DocuDrama Conversation, a DocuDrama approach that focuses on interaction between people on documents, refines the idea of the history replay. DocuDrama Conversation rearranges the order of events for the history replay instead of a play-out strictly organised by time. This approach enables the user to focus on activities that have taken place on a certain document. For example, a team member uploads a document in the team’s workspace and emails the project team about its existence and location. As a follow-up activity, other team members will open the document and read, change or annotate it. DocuDrama Conversation groups all these activities by different team members into a single scene. The replay of the events in that scene is performed in the order of their
198
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
Presenting Activity Information in an IIS 11 ● ● ● ● ● 1 ●●●●●●●●●●
occurrence although the scene is not thematically interrupted by concurrent activities on a different topic. In DocuDrama Conversation, avatars and their symbolic actions are at the centre of interest. The avatars look and turn towards each other to enhance the impression of an ongoing conversation. To enrich the story and to keep the user’s attention, special focus has been given to camera navigation and positioning. At the beginning of a scene, the camera approaches the centre of activity, the box of the current document, and remains in an overview position. The avatars appear one after the other and perform their symbolic actions. The camera chooses randomly between a variety of close-up views on the avatars. Figure 11.11 shows an example of possible camera positions and views in DocuDrama Conversation. The position of the avatars is dynamically determined. In scenes with only two or three avatars the avatars are grouped facing each other. If there is a larger number of actors involved, the avatars are grouped in circles on top of the document boxes. Figure 11.12 shows an arrangement of avatars in circles to give the impression of a conversation. To start DocuDrama users define a timeframe, which, by default, includes all past events. The user chooses a timeframe of the past, a day or a week, and defines subjects, authors or activities, which are relevant to him. Figure 11.13 shows the user interface of DocuDrama Conversation. The replay of events takes place in a single-user version of Tower world. Text fields in the middle section give information about user actions currently played out by avatars, the document as the centre of activity and about the period in time, in which activities on this document have taken place. The section below includes the elements for the user’s personal configuration of DocuDrama.
011
011
11
Figure 11.11 Camera positions.
199
Inhabited Information Spaces
Figure 11.12 Avatars positioned in circles.
Users of DocuDrama Conversations reported that the history replay enabled a better understanding of the process of events and their coherence, compared to a sequence of event reports of the same timeframe. The choice of selection criteria has proven to be useful and will be refined and extended in a future version of DocuDrama.
Figure 11.13 DocuDrama user interface.
200
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Presenting Activity Information in an IIS 11 ● ● ● ● ● 1 ●●●●●●●●●●
11.9 Ambient Interfaces Ambient interfaces support presenting and capturing information beyond the PC. They go beyond the classical PC with the traditional hardware and the desktop metaphor. Ambient interfaces include the whole environment of the user as a medium for the interaction between the user and the system. The subtle presentation of information exploits people’s ability to be peripherally aware of others, thus enabling users to be aware of activities without disturbance. Furthermore, ambient interfaces allow the user to capture information permanently – even if they are not working with their PC (Gross, 2002). Ishii et al. at the MIT Media Lab developed some early systems (e.g., ambientROOM). However, these systems were called ambient displays and primarily focused on presenting information and not on capturing information (Wisneski et al., 1998). We explicitly use the term ambient interfaces to denote systems that use the physical environment of the user to present information and to capture events. For the Tower environment we developed two types of ambient interfaces: binary ambient interfaces, and AwareBots. The technological bases for binary ambient interfaces are relayboards, which can be connected to the parallel or serial port of the PC. A special client controls the individual relays. Examples of binary ambient interfaces that we developed are a fish tank with plastic fish (Figure 11.14a), a fan, a coffee machine, and so forth. The system can release bubbles into the fish tank, which can be seen and heard by the users. The different binary ambient interfaces can be configured in various ways. For instance, releasing bubbles can indicate that the user’s web pages are accessed; the fan can blow air into the user’s face when a document is uploaded into the shared workspace; and the coffee machine can be switched on when the user logs into the Tower system from within the office. AwareBots are ambient interfaces that have the shape of robots. The Tower AwareBots were built based on the LEGO Mindstorms Robotics Invention System. The first generation of AwareBots was RoboDeNiro (Figure 11.14b). RoboDeNiro uses two motors for the presentation of information and a touch sensor for capturing information. It can lift its hat, rotate its torso, and senses when its arm is pressed. RoboDeNiro was used both in individual users’ offices and in a public space. The following configuration was chosen: RoboDeNiro can wait for a specific user, when this user logs in RoboDeNiro lifts its hat; when changes in a specific shared workspace occur, RoboDeNiro rotates its body; when its arm is pressed, it sends a specific message to the ticker tape. On the whole, the user reactions were positive and the users could understand the semantics of the individual movements of RoboDeNiro. However, more and more users stated that they would like to personalise their robot or build their own robot.
201
Inhabited Information Spaces
a
b
c d Figure 11.14 (a) Fish tank; (b) RoboDeNiro; (c) EyeBot; (d) ClassificatorBot.
Therefore, we distributed LEGO Mindstorm packages to the users and asked them to build their own AwareBots. This resulted in the second generation of AwareBots. One user built the EyeBot, which can roll its eyes and has a switch with a flag that can be moved to the front and to the back (Figure 11.14c). Whenever changes occur in the shared workspace, the EyeBot rolls its eyes. The switch can indicate the user’s availability. Another user built the ClassificatorBot – a mobile robot that can rotate to the left and to the right (Figure 11.14d). The ClassificatorBot can be used to indicate any kind of relationship (e.g., it was used to indicate access to web pages; on an access to the personal pages of the user the ClassificatorBot turns to the left, on access to other web pages of the research group it turns to the right).
202
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Presenting Activity Information in an IIS 11 ● ● ● ● ● 1 ●●●●●●●●●●
On the whole, both the binary ambient interfaces and the AwareBots allow the subtle presentation of up-to-the-moment information and events as well as capturing information over time. Furthermore, several AwareBots remain in a certain position and can indicate status information that the users can capture later (e.g., when they return to their office). The users of the ambient interfaces can not only design and construct their ambient interface, but they can also specify their individual mapping of the TOWER events to the symbols of the ambient interfaces – they do not need to use the default mapping. For instance, a user can specify which action the pressing of the arm of the RoboDeNiro triggers. Another user might want to have a login or logout event triggered, because she imagines that pressing the arm symbolises a greeting of saying hello or goodbye; another user might want to trigger a message to the ticker tape, asking if some colleagues are in the same building and would like to have a short coffee break together. Another example is rotating the body: the default mapping is that rotating the body shows changes in a specific shared workspace. Some users preferred the position of the torso to represent the number of new emails in the inbox: for each new email, RoboDeNiro rotates the torso a little more (some users even draw a scale, so they could see exactly at which position the torso was standing). For the public places, such as a coffee room, the default settings were kept in order to avoid confusion. Users of personal ambient interfaces also reported that they used a personal mapping for privacy reasons: when other persons were in their office, the guests could see the changes in the ambient interfaces, but they could not tell what the mapping was. Therefore, only the respective owner could interpret the mapping.
11.10 Lessons Learned About Awareness The demonstrations of the Tower features and the discussions with the different groups of users have shown that the needs for awareness depend on many different factors. Specifically, we found differences between the needs of tele-workers compared with those of collocated team members or those of members of distributed teams. Another interesting finding was that not only the events that occurred were of interest but also those that were expected but did not occur in time. For example, if a deadline in a project approaches and a requested contribution does not arrive in time, users wanted to be notified about those outstanding events.
11.10.1 Awareness Is Something One Is Not Aware of All discussions with users in the various settings, however, have confirmed that awareness is something they are not aware of. Instead of
203
Inhabited Information Spaces
permanently watching awareness information, they are performing task-related activities. “Seeking awareness information is an out-of-task activity”, a user said. Any extra effort is considered to be troublesome. Furthermore, the team observation disclosed that – like in real world settings – all available information was read as conveying awareness information almost unconsciously. For example, the stream of emails in the project was interpreted as indicating the level of activity. It conveyed presence information about the senders of the mails. Information about the topics of relevance was read from the subjects used in the emails. This was combined with the information gained in the BSCW daily activity report to achieve awareness about the team and the work process. In interviews the team members confirmed this intuitive interpretation of email as awareness means. From real world environments one could learn that awareness is one of the most unconscious and intuitive ways of human orientation in a surrounding environment. One is always aware without knowing. It is impossible not to be aware since this would imply not perceiving the environment. Tightly interrelated with perception is interpretation, that is, associating meaning to what one perceives (J. Gibson, 1986). Perception and interpretation are preconditions for being aware also in electronic settings. Accepting the unconsciousness of awareness it is not surprising that requirements analysis for electronic support of awareness using verbal discussions, interviews and so on, did not give a complete picture. The availability of tangible interfaces in Tower often disclosed new aspects. They helped to improve the understanding of awareness needs in electronic environments considerably.
11.10.2 Synchronicity of Awareness Purely synchronous supply of awareness information turned out to be much less useful than expected (see also Fussell et al., 2000). In particular it was not suitable for the work of distributed teams as the team observation disclosed. Also the fixed time interval of the daily BSCW activity report that is compiled and distributed every night was not suitable in all situations. Our team observations revealed changing interaction rhythms depending on the actual work processes. For example, in case of an approaching deadline and in case of joint document production, a high level of awareness was requested to support co-orientation in the team. While in the phase of programming, continuous mutual awareness was irrelevant but immediate reactions of other partners were requested once a problem occurred and immediate help from others was needed. The provision of awareness information should acknowledge these rhythms and also changes of the rhythm. Thus in phases with more activities, awareness information may be provided more often than in phases
204
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Presenting Activity Information in an IIS 11 ● ● ● ● ● 1 ●●●●●●●●●●
with low level of activities. Actually, real world places also have their own rhythm and tempo, depending on the surrounding culture (Levine, 1997). Simultaneous awareness is required in situations where one wants to establish a synchronous contact. To this end awareness about people and their actual availability is required. Orientation in the work process requires asynchronous awareness. This implies compiled stories about time periods.
11.10.3 Walking and Talking Are Means to Achieve Awareness In order to understand particularities of asynchronous awareness better, we asked team members how they achieve awareness in their local real world settings after a time of absence. “I gain awareness from going for a walk”, an interviewee said. Walking around and talking to colleagues were named by all interviewees as an important means to gain awareness. After some time of absence they ask colleagues about what happened in the meantime: “I like to listen by first getting the current state of things, what is going well, what is going badly, are there any hitches, and effects of hitches on the overall program.” Length of time for discussion is for most of them about 5 minutes. This gives clear evidence that summaries are requested, rather than detailed lists of all events. The feedback from interviewees explains why the pure presentation of events in the temporal order failed. Instead the challenge is to process the stream of events as recorded in Tower into meaningful stories about what happened over time. It also explains why the basic motto of the project, “Let the system do the walking”, did not comprise the whole story. Instead, the environment should “talk” to the users. In effect, this also means getting summaries and an overview kind of awareness stories. In case a user wants to get more detail, they want to be able to “walk around” in the electronic environment and ask the environment for more awareness information, i.e. to retrieve awareness information on request. The Docudrama development is a first step towards this direction, but further research is needed.
11.10.4 Peripheral Awareness in Electronic Settings With Tower we wanted to provide means for peripheral awareness similar to real environments. Instead of permanently watching events in the workspace, awareness should be immediately at hand, when needed and it must be adjusted to temporal fragmentation. For presence awareness this implies that a table of people present must be accessible with the level of temporal accuracy that is requested for a personal contact. For workspace awareness this implies that the environment provides on request the story of intermediate events for the location a user is
205
Inhabited Information Spaces
interested in. Peripheral awareness in electronic environments means immediate situative provision. Also in these cases the awareness information spans over time, that is, comprises awareness information over a time interval.
11.10.5 Awareness Is Double-situated: The Workspace’s and the Observer’s Situation Instead of constant need for the same kind of awareness information, the relevance of such information depends on two situations, the situation in the actual shared electronic workspace and the current situation of an observer, respectively a potential actor. For example, to keep all members of an electronic setting informed about the progress of work, a daily activity report that is sent out every day to all members has proven to be helpful. All team members in Tower found the BSCW daily activity report useful. “I look at BSCW daily activity reports everyday, it gives you information about who is doing what”. However, after an absence of several days, reading all intermediate reports does not suit the user needs. Instead, this user situation requires summaries of all intermediate events. The environment should compile a story from those events that took place since the actor’s last presence in the environment. This implies that instead of receiving a report giving the awareness information in fixed intervals of time, users want them to be compiled dynamically according to their individual observation rhythm. On the other hand, when something unusual happens and the progress of work in the workspace needs the actor’s attention, awareness information must be pushed – e.g. by email or by activating an ambient interface in the local work setting – to the respective actor. Thus, the awareness support in a workspace must keep track of what happens in the environment to provide the respective information to its actors. The stories compiled should be adjusted to the interaction rhythm of the environment as well as to the observation rhythms of its actors. We can, for example, differentiate the following potential situations of an actor who joins a workspace: taking a break and looking around; finishing a document under time pressure; supervising a project’s progress; searching for a particular document or for information on a particular topic; or a newbie may want to find out about places of relevance and about the history of the project. All these situations differ with respect to the awareness needed. The awareness information supplied must be dynamically adapted to these user situations. Instead of configuring tools, a user may be enabled to choose among the most suitable means of presenting awareness information, just like the choice of features offered by Tower. Furthermore, the environment should be able to detect as far as possible the user’s
206
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Presenting Activity Information in an IIS 11 ● ● ● ● ● 1 ●●●●●●●●●●
situation (by means of the events it has recorded from the user and by means of the presentation tool selected) and to adapt the provision of awareness information respectively. This requires more research to investigate concepts of adaptability to the situation of the shared environment, the actual situation of the user and the group. Mechanisms to compile events into meaningful episodes and stories need further investigations. Similarly, the requirements of synchronicity for the provision of awareness have to be studied in more detail.
011
011
011
011
11
11.11 Summary and Conclusion This chapter described an awareness environment for co-operative activities that is based on inhabited information spaces. It provides a number of user-configurable indicators. The 3D environment creates a shared space presenting users and their working context as an inhabited information space. The portal summarises all notifications in an overview presentation. We have described Smartmaps as a compact visualisation for an inhabited environment. They can be easily integrated into the information space itself to build social spaces. Furthermore, we presented ambient interfaces. The complexity of the Tower system became evident upon each demonstration and workshop and so does the complexity of its usability. Many items arose which could have not been foreseen until such a tool really existed. For example, the relationship of size between avatars and documents, the suitable level of detail with respect to documents and folders, a suitable mapping which facilitates navigation in the file system and the 3D world in parallel, the navigation in the 3D world to find points of interest had to be revised. In particular the relevance of DocuDrama became evident and many new requirements could be raised. Discussions with the different groups of users have shown that the needs for awareness depend on the different settings, modes of working and the particularities of the co-operation cultures. Awareness may contribute to project management, to knowledge management – e.g. for finding documents by patterns of their usage – and provide chance encounters in non-collocated teams. However, all discussions with users in the various settings have confirmed that awareness is a phenomenon of which one is not aware. Therefore any extra effort required to the provision of awareness is considered to be troublesome. Instead, awareness notifications must be compiled into meaningful stories that span over time. Details must be available on request. The presentation of activity notifications must fit with the changing local work situation of the particular observer. The availability of medium or long-term history of awareness may be useful in many situations. All this requires sophisticated mechanisms to aggregate and select the relevant data.
207
Inhabited Information Spaces
The effects of awareness support became evident when an interviewee said that he now worries “how I may appear to others, when they see what I am doing”. This is an aspect that requires further attention. Awareness is not only a means to see what others are doing. Instead, it will become a means for presenting oneself to a distributed team.
Acknowledgments The research presented here was carried out by the IST-10846 project TOWER, partly funded by the EC. We would like to thank all our colleagues from the TOWER team at Aixonix, Atkins, blaxxun, BTexact, Fraunhofer FIT, and UCL for their co-operation.
208
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
Part 5 Construction
011
011
011
11
209
This page intentionally left blank
011
011
011
011
11
DIVE: A Programming Architecture for the Prototyping of IIS
12
Emmanuel Frécon
12.1 Introduction This chapter presents the Distributed Interactive Virtual Environment (DIVE) system. DIVE realises an architecture and a programming toolkit for the implementation and deployment of wide-area, Internet-based multi-user virtual environments. The platform is designed to scale to both a large number of simultaneous participants and a large number of objects populating the environments, while ensuring maximum interaction at each site. Started as a lab tool in 1991, DIVE has now reached a stability and state that allows its use outside the research niche that it has occupied since its birth. DIVE can be compiled for a number of UNIX and Windows platforms. It benefits directly from the recent advances in three-dimensional (3D) graphics hardware that have moved 3D capabilities from professional workstations into everyday home personal computers (PCs) and probably soon to mobile terminals (see ATI, 2003; and Intel, 2003). In this chapter, we present the DIVE system from the perspective of Inhabited Information Spaces (IIS). Our intention is to describe an example system in order to give a better understanding of the technological issues that are involved by the realisation of large-scale shared interactive 3D spaces: networking solutions to keep interaction high and ensure the illusion of a shared space, various interaction methods for input and output (including live audio and video communication), support for large spaces both in extent and detail, openness of the platform towards the outside world, etc. Our focus on programming interfaces to the system is justified by the very eclectic nature of IIS and the recurrent necessity to couple the space with a database. At many different levels, DIVE tries to ensure an adequate trade-off between complexity for the programmer and capability. For example, the
211
Inhabited Information Spaces
different programming interfaces that it offers will allow the programmer to pick the interface and language that are most appropriate for the tasks at hand. However, offering such a palette of interfaces lends to complexity in mastering all or some of the interfaces and integrating all the components forming an application together. Similarly, DIVE provides a number of building blocks to interface its communication architecture. Assembling these blocks in various ways will let the application developer experiment with different interest management techniques at the cost of more programming complexity compared to other systems. However, the “standard” DIVE-based application instance will generally be developed without the need to focus on such complex problems. The remainder of this chapter is structured as follows. In the next section, we describe the conceptual abstraction through which DIVE applications transparently communicate with each other, namely via virtual worlds. We then focus on the wide-range of programming interfaces and techniques and describe their use for IIS. Following this, we present the run-time architecture and a summary of the main modules composing the system. Finally, we examine a specific application in more detail to hook its architecture and design choices into the various key techniques presented in the rest of this chapter.
12.2 The Virtual World as a Common Interaction Medium DIVE provides both an architecture and a programming model for the implementation of multi-user interactive virtual environments over the Internet. The architecture focuses on software and networking solutions that enable highly responsive interaction at each participating peer, that is, interaction results are whenever possible immediately shown locally at the interacting peer but slightly postponed at remote peers. The programming model hides networking details, allowing the programmer to focus on the space, its content and its application logic. By peer, we mean an application process running at a specific host (computer) connected to the Internet. An application or process is any active program interfacing to the virtual environment by presenting and modifying entities (see below), monitoring and reacting on different types of events, and so on. A typical application is the 3D browser that handles interaction and visualisation of the environment for a specific user. Other applications may, for instance, perform animations and complex simulations, and present 3D user interfaces for different purposes. A number of IIS examples are presented at the end of this chapter and elsewhere in this book, e.g. the Web Planetarium (see Chapter 2) and the Pond (see Chapter 4). A central feature in the programming architecture of DIVE is the shared distributed world database. All user and application interactions take place through this common medium. The world database acts as a
212
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
DIVE: A Programming Architecture for the Prototyping of IIS 12 ● ● ● ● 1 ●●●●●●●●●●●
world abstraction, since DIVE applications operate solely on the database and do not communicate directly with each other. This technique allows a clean separation between application and network interfaces. Thus, programming will not differ when writing single-user applications or multi-user applications. This model has proven to be successful since DIVE has changed its inter-process communication package a number of times without applications requiring any redesign, only minor code tweaking. A DIVE world is a hierarchical database of what are called entities. DIVE entities can be compared to objects in object-oriented programming, although DIVE is written in plain ANSI C. The database is hierarchical to ease ordering of information and rendering. In addition to graphical information, DIVE entities can contain user-defined data and autonomous behaviour descriptions. We believe that the 3D rendering module should not dictate a structure to the content of the database. By making this distinction the system becomes more flexible, and can more easily be adapted to a large variety of rendering techniques. Entity persistence is ensured as long as one application is connected to a world. When the last application “dies”, the world dies and entities will stop living. The next time an application connects to this world, it will reload its state from some initial description files or a Uniform Resource Location (URL). Currently, active persistence is achieved by running monitoring processes that periodically save the state of the world to files, from which the world database can be restored if the system needs to be restarted.
12.3 Partial, Active Database Replication The DIVE architecture is based on active replication of (parts of) the database, so that a copy resides at each application process (see Figure 12.1). This model allows applications to access the world database directly from memory, which provides low-latency user interaction. Typically, entity additions, removals and modifications are done on the local copy first, then distributed to all connected peers through network messages and applied to the local copy at each receiving peer. By this we mean that the replication of the database is active. Conceptually, programmers can think of a “global” central database residing somewhere on the network, but the database is indeed replicated at each process. To achieve scaleable real-time interaction, DIVE: ● ●
uses a peer-to-peer multicast communication model where partitions of the world are associated to multicast groups; tolerates world copies that differ slightly and implements services that ensure their equality over time. Dead-reckoning techniques and runtime object update mechanisms are used to achieve this;
213
Inhabited Information Spaces
P2
P1 P6 W2
W1 W2
W1
P3 H1
H3
W1
W2
W2
P5
W1
W2
Network
W1
H2
P4
Figure 12.1 The replication architecture of DIVE. A number of processes Px, hosted on several machines Hx are sharing two different virtual worlds W1 and W2, conceptually located “on the network”. Only P3 and P6 share both worlds, all other processes share one or the other world.
●
divides the world into sub-hierarchies that are only replicated and used in-between the small number of applications that have expressed an interest in a particular hierarchy.
Modifications to the database are applied locally first, then transmitted and applied to all receiving copies. Thus, on the receiving side, world events are captured and transcribed into the database some time after they occur. The length of this time is highly dependent on the networks that packets will cross. At all levels, DIVE is tuned to accommodate varying round-trip times and packet losses. Despite network latency, DIVE does not introduce differences between peers that are excessively large. The distribution principle fits with our experience of virtual environments: typically, entities will be modified in bursts and then stabilise into a common “inertia”, that is, a common state that lasts. Differences introduced by latency and lost network packets are “healed” over time by periodic synchronisation using sequence numbers to keep track of entity versions, followed by possible update requests. As a result, with time, all connected peers will achieve the same “inertia” for these entities. DIVE pursues an idea originally in Scaleable Reliable Multicast (SRM) (Floyd et al., 1995) to reduce the amount of message passing and thereby minimise network load and increase scalability. The method uses multicasting heavily, makes communication entity-based, and bases reliability on a negative acknowledgement request/response scheme.
214
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
1 11
DIVE: A Programming Architecture for the Prototyping of IIS 12 ● ● ● ● 1 ●●●●●●●●●●●
To remedy problems with network congestion and traffic overhead, and to offer the possibility for hundreds of participants to share a common place, DIVE provides a mechanism for dividing the world into sub-hierarchies that are only replicated and used in-between the small number of applications that are actually interested in them. This mechanism forms logical and semantic partitions. Each sub-hierarchy is associated with a multicast communication channel, called a lightweight group. As a result, processes that are not interested in these sub-hierarchies can simply ignore this branch of the entity tree and reduce network traffic. The top-most entity, i.e. the world, is itself always associated with a multicast channel, and every world member process must listen to it. More information on lightweight groups can be found in Frécon and Stenius (1998). On top of lightweight groups, DIVE offers an abstraction called “holders”. This abstraction deals with the initialisation of the database branch encapsulated by the group, for example through a URL. Holders can be associated to empty multicast groups, which implies total locality of part of the virtual world and can be used for scenery or other parts of an environment that are guaranteed to never change. Holders, described in more detail in Frécon and Smith (1999) and Frécon et al. (2001), can also be used to drastically reduce network traffic through semantically richer application-driven protocols and the realisation of predictive behaviours.
12.4 Programming the System In this section, we present a summary of the palette of techniques offered to DIVE application programmers. We will see that DIVE offers a wide range of programming interfaces, from a high-level scripting language to low-level C programming. The variety of languages, the openness of the platform and the variety of architectural options (from stand-alone applications to distributed scripts) make DIVE a toolkit of choice for IIS systems. It fits requirements such as advanced rendering techniques, e.g. through specific plugins, and connectivity to external databases, e.g. through its ability to embed application data in the environment and its component-based approach. Above all, DIVE offers a framework for the implementation of applications. Years of research development have led to a number of features that are present in few systems, such as the ability to support a wide range of avatars, spatialised audio communication between connected users, visual environment subjectivity (Jää-Aro and Snowdon, 2001). This makes it a software platform ideally adapted to the prototyping of IIS systems.
215
Inhabited Information Spaces
12.4.1 The DIVE Programming Model DIVE applications perform three distinct steps to exist and interact within the environment. Each of these steps is optional. An application will: ● ● ●
introduce one or more (shared) entities to the environment; register for events on selected parts of the environment, usually the entities that it has added to the environment itself; react to events occurring in the environment through modifications to the database, i.e. modify the environment.
Consider the example of a multi-user whiteboard application such as the one described in Ståhl (1992). Such an application follows the steps above. It introduces a 3D model of a whiteboard together with icons for the different drawing tools available. It listens to user interaction events on the surface and icons and reacts by drawing graphical elements on the surface or changing the drawing tool, e.g. from ellipse to free-hand tool.
12.4.2 Programming Interfaces Given this very general principle, DIVE, written itself in ANSI C, offers a wide range of programming interfaces (see Figure 12.2). Monolithic applications can be implemented by compiling some or all of the DIVE component libraries into a DIVE stand-alone application (or part of another application if necessary). Examples of such applications are earlier versions of VR-VIBE (Benford et al., 1995a) the Web Planetarium (Chapter 2), and The Pond (Chapter 4). DIVE offers interfaces for two major groups of languages: C/C++ and Java. ● ●
The C/C++ interface gives full access to the component libraries of the system. This leads to both power and complexity in some cases. The Java interface translates the DIVE object-based approach into true object orientation. For example, all DIVE entities are mirrored by Java objects that instantiate a class in a class hierarchy that mirrors the DIVE class hierarchy (see Chapter 4 in Pettifer, 1999).
Applications can also be loaded dynamically at run-time. These typically need one or several monolithic applications to exist. There are two major interfaces for dynamic loading of applications: ●
216
DIVE has support for plug-ins written in C or C++. By construction, plug-ins have full access to the components libraries of the system without any restrictions. Plug-ins can be either loaded when a standalone application starts or later on, programmatically or as a result of user actions. The standard 3D browser that comes with the system offers the user the option to select a plug-in from the local file system and load it into the 3D browser.
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
DIVE: A Programming Architecture for the Prototyping of IIS 12 ● ● ● ● 1 ●●●●●●●●●●● ●
More importantly, DIVE offers a high-level programming interface in the form of a scripting language. This interface is based on TCL (Tool Command Language). Any entity in the database can be associated with a script that will describe its behaviour over time. Scripts follow the DIVE programming model: they will register for events and react to events. To this end, the set of standard TCL commands is enhanced with a number of DIVE specific commands for event registration, modification of the environment, etc. This forms DIVE/TCL (see Frécon and Smith, 1999 for more information).
Finally, external applications are allowed to connect to a running standalone application through the DIVE Client Interface (DCI). In this case, the DIVE application acts as a server and the external application as a client (note that the communication that takes place is totally different from the standard DIVE communication mechanisms: it is a pure client–server solution tuned for usage within a machine or on a local network). External applications will be represented in the environment by an entity of a specific type. This entity will operate in the environment on their behalf. The language used here is, again, DIVE/TCL. This interface is tuned for situations where the external application cannot be easily modified but needs some DIVE facilities. Finally, standard implementations of the client interface exist in many languages: C, TCL, Java, Oz, Prolog.
Java inverface cation interface Appli o C re libraries
011
DCI
DIVE/TCL
C/C++ Interface
11
Plugin
011
External application
Figure 12.2 DIVE offers a wide range of programming interfaces. Most of them operate on the hierarchical database as a central abstraction. The Java interface mirrors part of this database in a pure object-oriented manner. The script interface (DIVE/TCL) is part of the database in the sense that scripts are associated to the replicated objects and describe their behaviour in a distributed manner.
217
Inhabited Information Spaces
12.4.3 Building your Application Given the wide palette of programming interfaces and architectures, we discuss now the various ways of writing DIVE applications and the typical applications that these target.
“Monolithic” Programming It is possible to write DIVE applications that apply the programming model in one sweep, i.e. within a single piece of code. There are different ways to implement these types of applications: ● ●
●
C, C++ or Java applications can be programmed and linked together with the DIVE core libraries. It is also possible to build whole applications using the plug-in interface. These will be hosted within a stand-alone application such as above (typically the 3D browser). External applications programmed in any language of choice that has support for TCP/IP and connected to a running DIVE application through the DCI.
Such applications will be integrated to a DIVE world through event registration. This offers transparent access to a networked environment. Monolithic DIVE Programming is suitable for applications such as simulation engines, complex applications or simple renderers. An example of IIS application that uses this model is VR-VIBE (Benford et al., 1995a). VR-VIBE creates visualisations of sets of documents. Users specify keywords that they wish to use to generate the visualisation and place these keywords in 3D-space. Representations of the documents are then displayed in the space according to how relevant each document is to each of the keywords (this relevance is computed by searching the documents for the keywords specified by the user and recording the number of matches). Earlier versions of VR-VIBE were written as a stand-alone application written entirely in C. This is mainly for scalability reasons. Above all, VRVIBE is typically a compute intensive application, e.g. computation of document icons positions upon interaction and initialisation. It postulates the acceptance of networking delays, for example, the time for new document position to arrive at all visualising sites. Along the years, VRVIBE has benefited from the incorporation of DIVE/TCL. Consequently, it has now become a mixed mode application (see below) where the core component is still written in C for scalability and the user interface uses the behaviour language. Another example of IIS application that uses this centralised programming model is the Pond (Chapter 4). The Pond is a multi-user horizontally projected system for access and manipulation of information
218
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
DIVE: A Programming Architecture for the Prototyping of IIS 12 ● ● ● ● 1 ●●●●●●●●●●●
elements from the Internet as well as for communication and collaboration between users being present both remotely and physically. The current demonstrator provides a visualisation of a record database and focuses on touch and sound interfaces, “prohibiting” textual entry through a keyboard. The Pond is mostly implemented as a stand-alone application (in Java), for performance reasons. The logic of the Pond application, e.g. flocking behaviour, Internet search initiated by users, is on purpose separated from the visualisation (a specific DIVE 3D browser, with a number of rendering plug-ins). Yet another application largely following this technique is the Web Planetarium (Chapter 2). The application visualises the structure behind World Wide Web documents and hyperlinks as a 3D virtual world of planet-like abstract objects and connection beams. An object is a 3D representation of a web page and, once the user is inside, displays the hyperlinks on that page as additional small objects on which the user is able to click in order to fetch new pages, and thus extend the 3D graph with new site representations. The Web Planetarium is implemented as a stand-alone application (in C), for performance reasons. As for the Pond above, the logic of the application (parsing of Web pages, placement of documents, interaction with the users) is contained in an application separated from the visualisation. This allows for a number of different visualisation set-ups: standard desktop, multi-screen displays, etc.
Script Programming A radically different way of programming consists of creating a DIVE world (or set of objects) with scripts attached to objects in the database. The TCL layer isolates clients from executable changes to the DIVE, which is of importance as DIVE/TCL has stabilised over the years and is generally kept backward compatible. Of interest to this style of programming is the fact that the standard 3D browser provides a generic actor environment (with audio and visual input and output). On top of this generic environment, there can be any number of TCL/TK-based “skins”, presenting different user interfaces. This environment allows applications that live within the presented DIVE worlds and programmed in the form of scripts to modify the layout of the 2D user interface at run-time, for example to add application specific menus (see Steed et al., 1999 for an example). Script programming is tuned for rapid prototyping, simple applications, interface experiments and animations. Script programming is ideal when the behaviour of the application can be described through a number of somewhat independent visual objects that consume little CPU power. It is particularly suitable to adapt to changing environments of
219
Inhabited Information Spaces
all sorts: adding new objects with new scripts to an environment will make it behave differently. This can even happen at run-time and is therefore very dynamic by nature. An example of an IIS that uses this programming model is the DIVE Room System (Frécon and Avatare-Nöu, 1998). This application actively supports collaborative as well as individual work through the concept of rooms. The application uses a real-life metaphor and introduces virtual counterparts of objects that are usually found in meeting and shared group rooms: overhead projectors, notebooks, screens, documents, etc. Every object in the environment has only a slight dependence on other objects. This design fact has driven the choice towards a scripting programming model. Some objects have to know about one another, for example, placing an overhead on the projector will have the effect of showing it in a bigger format on the screen. To achieve this, the application uses script-to-script communication, which is part of the scripting interface.
Mixed Mode As already alluded to several times already, a key design of the DIVE system is the possible interconnection of all programming components. As such, the system encourages application programmers to mix
Figure 12.3 Several views of the DIVE room system. Main picture: At the forefront is the remote control to allow remote operation of the screen that is currently enlarging the document that has been placed on the overhead projector. Close to the projector is a set of virtual folders containing various documents. Inset: An avatar sitting at the conference table and examining a document.
220
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
DIVE: A Programming Architecture for the Prototyping of IIS 12 ● ● ● ● 1 ●●●●●●●●●●●
programming interfaces in order to best suit the needs of the various components on which they are working. A typical example is the combination of plug-ins for services and DIVE/TCL for application logic. This combination provides a very powerful prototyping process. Plug-ins will register functions with the DIVE/TCL layer and the TCL layer is able to inspect loaded plug-ins or request a plug-in to be loaded/unloaded. Another example is external applications connecting through DCI. Such applications will often extend the entity that represents them in DIVE with some application-specific DIVE/TCL code in order to relieve the burden placed on the socket connection between DIVE and the application. WebPath (Frécon and Smith, 1998) is an example of such an application. Mixed mode offers many possibilities; examples of use are face and body animation, new navigation styles, simulation of crowds, etc. This mixed mode is used in many of the latest IIS based on DIVE, since it combines rapid application development (using the scripting language), a component-based approach (allowing components of the application to be associated to specific programmers) and component reuse (both at the plug-in and scripting level). An example of an IIS that uses this model is the London Traveller application described later on in this chapter (Section 12.6). The application supports travellers by providing an environment where they can explore London, utilise group collaboration facilities, rehearse particular journeys and access tourist information data. To adapt to the amount of data to visualise (3D model over 16 × 10 km of London) the core DIVE rendering engine was modified. Additionally, the model is brought to life using avatar crowd and face animations. For performance reasons, this is implemented using several plug-ins. Finally, the different applications that are embedded within the London model were implemented using a set of objects and scripts, designed and implemented by different programming teams. Another example of IIS that uses this model is the Library Demonstrator application (Mariani and Rodden, 1999). The overall goal of this application is to provide visitors to the library with an alternative search method, that will help in “fuzzy” searches, that is, searches that typically would require the visitor to speak to a librarian to get a grip of what to look for in the library. Rather than replacing the existing computer-based search methods, which work well for the initiated and focused users, the demonstrator provides a complementary means of searching, adapted to the users outside of this category. The core of the library demonstrator is built on top of the Java interface in order to benefit from Java’s ability to interact easily with external databases. A large part of the user interface is written using the DIVE/TCL interface, which makes it easy to experiment with a number of different approaches in order to find a suitable design.
221
Inhabited Information Spaces
Figure 12.4 The main window shows the visualisation part of the library demonstrator. At the centre of the window is a focusing circle. The information for the document contained within that circle is shown in a semi-transparent legend in the top left corner of the visualisation window. To the right are a number of windows to perform queries within the library database.
Remote Dynamic Programming The DIVE/TCL layer supports a remote procedure declaration and activation protocol that allows distributed script execution. Indeed, the dynamic nature of scripting languages and the distributed nature of the DIVE system make possible the real-time extension and modification of existing scripts within existing entities. Building applications in such a context becomes interesting. For example, this allows for per-user customised behaviour, e.g. 2D versus 3D presentation or the ability to extend the user interface of other participants by sharing scripts. Finally, it allows distributed application development where the code of the application can be edited between multiple machines interactively from within the system. There are a number of situations where this model can be useful. For example, the various parts of the London Traveller application dynamically add a menu to the standard 3D browser to improve control over
222
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
DIVE: A Programming Architecture for the Prototyping of IIS 12 ● ● ● ● 1 ●●●●●●●●●●●
the information shown. The code for menu extension is contained in the code of the London Traveller application and brought to all users successively connected. Similarly, later versions of VR-VIBE added an extra window to any person (process) entering the world.
12.5 DIVE as a Component-based Architecture To complete our description of the system, the following sections discuss the main modules that compose the DIVE system and that are offered as part of the standard libraries.
12.5.1 System Components In DIVE, an event system realises the operations and modifications that occur within the database. Consequently, all operations on entities such as 3D transformations will generate events to which applications can react. Additionally, there are spontaneous and user-driven events such as collision between objects or user interaction with input devices. An interesting feature of the event system is its support of high-level application-specific events, enabling applications to define their content and utilisation. This enables several processes composing the same application (or a set of applications) to exchange any kind of information using their own protocol. Most events occurring within the system will generate network updates that completely describe them.
User
011
011
11
Input devices
Output devices Rendering
User interface
Audio and video i/o
MIME
Tcl scripting
Event system
3D i/o Database Distribution SRM mechanisms Network
Figure 12.5 The different modules composing the system and their approximate layering and dependencies between the user and the network.
223
Inhabited Information Spaces
In any application, the content of the database must be initialised. DIVE uses a module that manages several three-dimensional file formats and translates them into the internal data structures that best represent their content. Usually only one peer will load, and parse, a particular file and the resulting entity hierarchy will be distributed to other connected peers through a series of (multicast) updates that describe the resulting entities. This specific mechanism differs from many other systems that rely on being able to access the description files or URLs from all connected peers. DIVE has an embedded scripting language that provides an interface to most of the services of the platform. Scripts register an interest in, and are triggered by, events that occur within the system. They will usually react by modifying the state of the shared database. Moreover, these modifications can lead to other events, which will possibly trigger additional scripts. A series of commands allow the logic of the scripts to gather information from the database and decide on the correct sequence of actions. For example, the simplistic script below would move the associated object 1.0 m upwards at every interaction with the mouse (2D or 3D) or any other interaction device connected. The procedure on_interaction is bound to any interaction event on the object (which identifier is returned by [dive_self]). proc on_interaction {event_type object_id type\origin_id src_id x y z} { dive_move [dive_self] 0.0 1.0 0.0 LOCAL_C } dive_register INTERACTION_SIGNAL DIVE_IA_SELECT\[dive_self] “” on_interaction
12.5.2 User-oriented Components The services described previously are independent of any DIVE application. This section focuses on the different modules present within the 3D browser. The primary display module is the graphical renderer. Traditionally, the rendering module traverses the database hierarchy and draws the scene from the viewpoint of the user. This module has several implementations, on top of various graphical libraries such as Performer, OpenGL, Direct3D or PocketGL. Some versions support a constant frame-rate rendering mode (Steed and Frécon, 1999). DIVE has integrated audio and video facilities. Audio and video streams between participants are distributed using unreliable multicast communication. Audio streams are spatialised so as to build a soundscape, where the perceived output of an audio source is a function of the
224
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
DIVE: A Programming Architecture for the Prototyping of IIS 12 ● ● ● ● 1 ●●●●●●●●●●●
distance to the source, the inter-aural distance and the direction of the source. The audio module supports mono-, stereo- or quadra-phonic audio rendering through speakers or headphones connected to the workstation. Input can be taken from microphones or from audio sample files referenced by a URL. Similarly, the video module takes its input from cameras connected to the workstations or video files referenced by URLs. Video streams can either be presented to remote users in separate windows or onto textures within the rendered environment. Users may also be presented with a two-dimensional interface that offers access to rendering, collaboration and editing facilities. The interface itself is written using the same scripting language as offered by the world database. Consequently, applications can dynamically query and modify the appearance of the 2D interface. For example, the London Traveller application exploits this feature by adding an applicationspecific menu to the standard interface of the DIVE browser (see Section 12.6). Finally, a MIME (Multimedia Internet Mail Extensions) module is provided to better integrate with external resources. It automatically interprets external URLs. For example, an audio stream will be forwarded onto the audio module where it will be mixed into the final soundscape.
12.5.3 The DIVE Run-time Architecture Not all components are present within the different programs of the DIVE run-time architecture: ●
011
●
011
11
●
Sessi