Lecture Notes in Computer Science
Edited by G. Goos, J. Hartmanis and J. van Leeuwen
1614
3 Berlin Heidelberg New York Barcelona Hong Kong London Milan Paris Singapore Tokyo
Dionysius P. Huijsmans Arnold W.M. Smeulders (Eds.)
Visual Information and Information Systems Third International Conference, VISUAL’99 Amsterdam, The Netherlands, June 2-4, 1999 Proceedings
13
Series Editors Gerhard Goos, Karlsruhe University, Germany Juris Hartmanis, Cornell University, NY, USA Jan van Leeuwen, Utrecht University, The Netherlands
Volume Editors Dionysius P. Huijsmans Leiden University, Computer Science Department Niels Bohrweg 1, 2333 CA Leiden, The Netherlands E-mail:
[email protected] Arnold W.M. Smeulders University of Amsterdam, Research Institute Computer Science Kruislaan 403, 1098 SJ Amsterdam, The Netherlands E-mail:
[email protected]
Cataloging-in-Publication data applied for Die Deutsche Bibliothek - CIP-Einheitsaufnahme Visual information and information systems : third international conference ; proceedings / VISUAL ’99, Amsterdam, The Netherlands, June 2 - 4, 1999. D. P. Huijsmans ; Arnold W.M. Smeulders (ed.). - Berlin ; Heidelberg ; New York ; Barcelona ; Hong Kong ; London ; Milan ; Paris ; Singapore ; Tokyo : Springer, 1999 (Lecture notes in computer science ; Vol. 1614) ISBN 3-540-66079-8
CR Subject Classification (1998): H.3, H.5, H.2, I.4, I.5, I.7, I.3 ISSN 0302-9743 ISBN 3-540-66079-8 Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. c Springer-Verlag Berlin Heidelberg 1999 Printed in Germany
Typesetting: Camera-ready by author SPIN 10705199 06/3142 – 5 4 3 2 1 0
Printed on acid-free paper
Preface Visual Information at the Turn of the Millenium Visual information dominates the senses we have been given to observe the world around us. We tend to believe information most when it is in visual form. Television and Internet have accelerated the perfusion of visual information to unprecedented heights. Now that all sensors are turning digital, and personal computers and the Net are powerful enough to process visual information, a new era is being born: the age of multimedia information. The dominant component of multimedia information is visual. Hence the conclusion, we are on the threshold of the age of visual information. The approach of the new millenium provokes these sweeping thoughts. Five hundred years after the invention of printed books, visual information has returned to the forefront of information dissemination, on equal par with textual and numerical information. The practice of designing visual information systems is far removed from such grandiose thoughts. Visual information systems are radically different from conventional information systems. Many novel issues need to be addressed. A visual information system should be capable of providing access to the content of pictures and video. Where symbolic and numerical information are identical in content and form, pictures require a delicate treatment to approach their content. To search and retrieve items on the basis of their pictorial content requires a new, visual or textual way of specifying the query, new indices to order the data, and new ways to establish similarity between the query and the target. A novel element, still lacking research, is the display of the information space of all visual items in the system. Derived from the Third International Conference on Visual Information Systems, held in Amsterdam, this issue of Springer’s Lecture Notes in Computer Science provides a state-of-the-art view on visual information systems. Among the building blocks of visual information systems, the computation of features is currently attracting the most attention. Good features are instrumental in reducing the abundance of information in the picture or in the video to the essence. Ideally speaking the feature is insensitive to irrelevant variations in the data, and sensitive to variations in semantic differences in the data. In the proceedings you will find features of various kinds, where invariance is of specific importance to features for image databases. For browsing and searching for unspecified items in the information space of all items in the system, visual interaction on the ensemble of all items can provide an overview to the surfing user. In the proceedings you will find contributions on query by iterative optimization of the target, displaying the information space, and other ways to trace semantically similar items or documents. It is expected that the topic will attract more attention, more completely fulfilling the name: visual information systems. An important issue of visual search is the similarity measure. It is not easy to decide what makes two objects, example and target, experienced as equal. Simi-
VI
Preface
larity is currently approached as either an exact correspondence (as in standard databases), as a statistical problem (as in object classification), or as a metrical problem (in feature space). It is quite likely that similarity search as a cognitive problem will gain in weight where human-perceived similarity will be core. Similarity search for all practical purposes is proximity search: the subject and the target match by proximity. In the proceedings you will find many different implementations of the notion of proximity. Underlying any information system, there should be a database proper with data structures, query specification, and indexing schemes for efficient search. Where the main emphasis of the techniques embodied here is on processing visual information, the connection to databases, and the database parlance is still underrated. In the proceedings you will find contributions on extensions of the database tradition towards unstructured multimedia items, on data structures especially suited for spatial data, and on new ways to access spatial data. An essential part of visual information processing is the success of capturing the information in the image. Where the biggest problem in computer vision is a successful segmentation step, in image databases several authors find their way around this step. In the proceedings you will find contributions based on characterizing internally similar partitions in the image, salient details, or total image profiles. Contributions on all these and many more aspects of many more topics can be absorbed from the proceedings. Their combination in one LNCS Volume gives an up-to-date overview of all components of visual information systems. All the contributions in this book have been reviewed thoroughly. The editors of this book wish to thank the members of the program committee and the additional reviewers for their effort. Their work has enhanced the final submission to this book. You will find their names on a separate sheet. We thank them cordially. With this book we hope that the conference series on visual information systems will continue on to a long-lived future. The conference chair would like to seize the opportunity to thank the members of the local committee and the conference bureau for making the conference happen. Finally, the support of the members of the visual information systems steering committee has been much appreciated.
March 1999 Arnold W.M. Smeulders Nies Huijsmans
Visual99 Conference Organization
Conference Chair Arnold W.M. Smeulders
University of Amsterdam, NL
The Visual Information Systems Steering Committee S.K. Chang Ramesh Jain Tosiyasu Kunii Clement Leung Arnold W.M. Smeulders
University of Pittsburgh, USA University of California, USA The University of Aizu, J Victoria University of Technology, AU University of Amsterdam, NL
Program Chairs Ruud M. Bolle Alberto Del Bimbo Clement Leung
IBM Watson, USA University of Florence, I Victoria University of Technology, AU
Program Committee Jan Biemond Josef Bigun S.K. Chang David Forsyth Theo Gevers Luc van Gool William Grosky Glenn Healey Nies Huijsmans Yannis Ioanidis Horace Ip Ramesj Jain Rangachar Kasturi Martin Kersten Inald Lagendijk Robert Laurini Carlo Meghini
Technical University Delft, NL Halmstad University, S Pittsburgh, USA Berkeley, USA University of Amsterdam, NL Catholic University, Leuven, B Wayne State University, USA University of California, Irvine, USA Leiden University, NL University of Athens, G City University of Hong Kong, HK University of California, San Diego, USA Penn State University, USA CWI, Amsterdam, NL Technical University Delft, NL Universite C. Bernard Lyon, F IEI CNR, Pisa, I
VIII
Conference Organization
Erich Neuhold Eric Pauwels Fernando Pereira Dragutin Petkovic Hanan Samet Simone Santini Stan Sclaroff Raimondo Schettini Stephen Smoliar Aya Soffer Michael Swain Hemant Tagare George Thoma Remco Veltkamp Jian Kang Wu
University of Darmstadt, D Catholic University, Leuven, B Instituto Superior Tcnico, Lisbon, P IBM, Almaden, USA University of Maryland, USA University of California, San Diego, USA Boston University, USA ITIM CNR, Milan, I Xerox, Palo Alto, USA Technion, Haifa, IL DEC, USA Yale University, USA National Library of Medicine, USA Utrecht University, NL National University of Singapore, SP
Additional Reviewers Giuseppe Amato Sameer Antani Frantisek Brabec Andr Everts Ullas Gargi Sennay Ghebreab Henk Heijmans Gisli R. Hjaltason Bertin Klein Thomas Klement Martin Leissler Michael Lew Ingo Macherius Giuseppe De Marco Vladimir Y. Mariano TatHieu Nguyen S.D.Olabarriaga Patrizia Palamidese Fabio Patern P. Savino Geert Streekstra V.S.Subrahmanian Ulrich Thiel Jeroen Vendrig Marcel Worring
IEI CNR, Pisa, I Penn State University, USA University of Maryland, USA University of Darmstadt, D Penn State University, USA University of Amsterdam, NL CWI, Amsterdam, NL University of Maryland, USA University of Darmstadt, D University of Darmstadt, D University of Darmstadt, D Leiden University, NL University of Darmstadt, D IEI CNR, Pisa, I Penn State University, USA University of Amsterdam, NL University of Amsterdam, NL IEI CNR, Pisa, I IEI CNR, Pisa, I IEI CNR, Pisa, I University of Amsterdam, NL University of Maryland, USA University of Darmstadt, D University of Amsterdam, NL University of Amsterdam, NL
Conference Organization
IX
Local Organizing Committee Theo Gevers Nies Huijsmans Dennis Koelma Carel van den Berg Remco Veltkamp Marcel Worring
University of Amsterdam, Leiden University, University of Amsterdam, PGS, Amsterdam, Utrecht University, University of Amsterdam,
Sponsors Shell Nederland B.V Netherlands Computers Science Research Foundation Advanced School for Computing and Imaging University of Amsterdam Royal Academy of Arts and Sciences
NL NL NL NL NL NL
Table of Contents
Visual Information Systems Supporting Image-Retrieval by Database Driven Interactive 3D InformationVisualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M. Leissler, M. Hemmje, E.J. Neuhold
1
Video Libraries: From Ingest to Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 R.M. Bolle, A. Hampapur Querying Multimedia Data Sources and Databases . . . . . . . . . . . . . . . . . . . . . 19 S.-K. Chang, G. Costagliola, E. Jungert General Image Database Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 P.L. Stanchev System for Medical Image Retrieval: The MIMS Model . . . . . . . . . . . . . . . . . 37 R. Chbeir, Y. Amghar, A. Flory An Agent-Based Visualisation Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 J. Meddes, E. McKenzie Error-Tolerant Database for Structured Images . . . . . . . . . . . . . . . . . . . . . . . . 51 A. Ferro, G. Gallo, R. Giugno
Interactive Visual Query Query Processing and Optimization for Pictorial Query Trees . . . . . . . . . . . . 60 A. Soffer, H. Samet Similarity Search Using Multiple Examples in MARS . . . . . . . . . . . . . . . . . . . 68 K. Porkaew, S. Mehrotra, M. Ortega, K. Chakrabarti Excluding Specified Colors from Image Queries Using a Multidimensional Query Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 D. Androutsos, K.N. Plataniotis, A.N. Venetsanopoulos Generic Viewer Interaction Semantics for Dynamic Virtual Video Synthesis 83 C.A. Lindley, A.-M. Vercoustre Category Oriented Analysis for Visual Data Mining . . . . . . . . . . . . . . . . . . . . 91 H. Shiohara, Y. Iizuka, T. Maruyama, S. Isobe User Interaction in Region-Based Color Image Segmentation . . . . . . . . . . . . . 99 N. Ikonomakis, K.N. Plataniotis, A.N. Venetsanopoulos
XII
Table of Contents
Using a Relevance Feedback Mechanism to Improve Content-Based Image Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 G. Ciocca, R. Schettini Region Queries without Segmentation for Image Retrieval by Content . . . . 115 J. Malki, N. Boujemaa, C. Nastar, A. Winter Content-Based Image Retrieval over the Web Using Query by Sketch and Relevance Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 E. Di Sciascio, G. Mingolla, M. Mongiello Visual Learning of Simple Semantics in ImageScape . . . . . . . . . . . . . . . . . . . . 131 J.M.Buijs, M.S. Lew
Browsing Information Space Task Analysis for Information Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 S.L. Hibino Filter Image Browsing: Exploiting Interaction in Image Retrieval . . . . . . . . . 147 J. Vendrig, M. Worring, A.W.M. Smeulders Visualization of Information Spaces to Retrieve and Browse Image Data . . 155 A. Hiroike, Y. Musha, A. Sugimoto, Y. Mori Mandala: An Architecture for Using Images to Access and Organize Web Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 J.I. Helfman A Compact and Retrieval-Oriented Video Representation Using Mosaics . . 171 G. Baldi, C. Colombo, A. Del Bimbo
Internet Search Engines Crawling, Indexing and Retrieval of Three-Dimensional Data on the Web in the Framework of MPEG-7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 E. Paquet, M. Rioux A Visual Search Engine for Distributed Image and Video Database Retrieval Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 J.-R. Ohm, F. Bunjamin, W. Liebsch, B. Makai, K. Mueller, B. Saberdest, D. Zier Indexing Multimedia for the Internet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 B. Eberman, B. Fidler, R. Iannucci, C. Joerg, L. Kontothanassis, D.E. Kovalcin, P. Moreno, M.J. Swain, J.-M. Van Thong Crawling for Images on the WWW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 J. Cho, S. Mukherjea
Table of Contents
XIII
A Dynamic JAVA-Based Intelligent Interface for Online Image Database Searches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 V. Konstantinou, A. Psarrou
Video Parsing Motion-Based Feature Extraction and Ascendant Hierarchical Classification for Video Indexing and Retrieval . . . . . . . . . . . . . . . . . . . . . . . . 221 R. Fablet, P. Bouthemy Automatically Segmenting Movies into Logical Story Units . . . . . . . . . . . . . . 229 A. Hanjalic, R.L. Lagendijk, J. Biemond Local Color Analysis for Scene Break Detection Applied to TV Commercials Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 J.M. S´ anchez, X. Binefa, J. Vitri` a, P. Radeva Scene Segmentation and Image Feature Extraction for Video Indexing and Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 P. Bouthemy, C. Garcia, R. Ronfard, G. Tziritas, E. Venau, D. Zugaj Automatic Recognition of Camera Zooms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 S. Fischer, I. Rimac, R. Steinmetz A Region Tracking Method with Failure Detection for an Interactive Video Indexing Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 M. Gelgon, P. Bouthemy, T. Dubois Integrated Parsing of Compressed Video . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 S.M. Bhandarkar, Y.S. Warke, A.A. Khombhadia Improvement of Shot Detection Using Illumination Invariant Metric and Dynamic Threshold Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 W. Kong, X. Ding, H. Lu, S. Ma Temporal Segmentation of MPEG Video Sequences . . . . . . . . . . . . . . . . . . . . 283 E. Ardizzone, C. Lodato, S. Lopes Detecting Abrupt Scene Change Using Neural Network . . . . . . . . . . . . . . . . . 291 H.B. Lu, Y.J. Zhang Multi-Modal Feature-Map: An Approach to Represent Digital Video Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 U. Srinivasan, C. Lindley Robust Tracking of Video Objects through Topological Constraint on Homogeneous Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 M. Liao, Y. Li, S. Ma, H. Lu
XIV
Table of Contents
Spatial Data The Spatial Spreadsheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 G.S. Iwerks, H. Samet A High Level Visual Language for Spatial Data Management . . . . . . . . . . . . 325 M.-A. Aufure-Portier, C. Bonhomme A Global Graph Model of Image Registration . . . . . . . . . . . . . . . . . . . . . . . . . . 333 S.G. Nikolov, D.R. Bull, C.N. Canagarajah A Graph-Theoretic Approach to Image Database Retrieval . . . . . . . . . . . . . . 341 S. Aksoy, R.M. Haralick Motion Capture of Arm from a Monocular Image Sequence . . . . . . . . . . . . . . 349 C. Pan, S. Ma
Visual Languages Comparing Dictionaries for the Automatic Generation of Hypertextual Links: A Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358 I. Gagliardi, B. Zonta Categorizing Visual Contents by Matching Visual ”Keywords” . . . . . . . . . . . 367 J.-H. Lim Design of the Presentation Language for Distributed Hypermedia System . 375 M. Katsumoto, S.-i. Iisaku A Generic Annotation Model for Video Databases . . . . . . . . . . . . . . . . . . . . . . 383 H. Rehatschek, H. Mueller Design and Implementation of COIRS(a COncept-Based Image Retrieval System) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 H. Yang, H. Kim, J. Yang Automatic Index Expansion for Concept-Based Image Query . . . . . . . . . . . . 399 D. Sutanto, C.H.C. Leung
Features and Indexes for Image Retrieval Structured High-Level Indexing of Visual Data Content . . . . . . . . . . . . . . . . . 409 A.M. Tam, C.H.C. Leung Feature Extraction: Issues, New Features, and Symbolic Representation . . . 418 M. Palhang, A. Sowmya Detection of Interest Points for Image Indexation . . . . . . . . . . . . . . . . . . . . . . 427 S. Bres, J.-M. Jolion
Table of Contents
XV
Highly Discriminative Invariant Features for Image Matching . . . . . . . . . . . . 435 R. Alferez, Y.-F. Wang Image Retrieval Using Schwarz Representation of One-Dimensional Feature 443 X. Ding, W. Kong, C. Hu, S. Ma Invariant Image Retrieval Using Wavelet Maxima Moment . . . . . . . . . . . . . . 451 M. Do, S. Ayer, M. Vetterli Detecting Regular Structures for Invariant Retrieval . . . . . . . . . . . . . . . . . . . . 459 D. Chetverikov Color Image Texture Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467 N. Nes, M.C. d’Ornellas Improving Image Classification Using Extended Run Length Features . . . . . 475 S.M. Rahman, G.C. Karmaker, R.J. Bignall Feature Extraction Using Fractal Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483 B.A.M. Schouten, P.M. de Zeeuw
Object Retrieval Content-Based Image Retrieval Based on Local Affinely Invariant Regions . 493 T. Tuytelaars, L. Van Gool A Framework for Object-Based Image Retrieval at the Semantic Level . . . . 501 L. Jia, L. Kitchen Blobworld: A System for Region-Based Image Indexing and Retrieval . . . . . 509 C. Carson, M. Thomas, S. Belongie, J.M. Hellerstein, J. Malik A Physics-Based Approach to Interactive Segmentation . . . . . . . . . . . . . . . . . 517 B.A. Maxwell
Ranking and Performance Assessment of Effectiveness of Content Based Image Retrieval Systems . . . 525 A. Dimai Adapting k-d Trees to Visual Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533 R. Egas, D.P. Huijsmans, M. Lew, N. Sebe Content-Based Image Retrieval Using Self-Organizing Maps . . . . . . . . . . . . . 541 J. Laaksonen, M. Koskela, E. Oja Relevance Feedback and Term Weighting Schemes for Content-Based Image Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549 D. Squire, W. Mueller, H. Mueller
XVI
Table of Contents
Genetic Algorithm for Weights Assignment in Dissimilarity Function for Trademark Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557 D. Y.-M. Chan, I. King
Shape Retrieval Retrieval of Similar Shapes under Affine Transform . . . . . . . . . . . . . . . . . . . . . 566 F. Mokhtarian, S. Abbasi Efficient Image Retrieval through Vantage Objects . . . . . . . . . . . . . . . . . . . . . 575 J. Vleugels, R. Veltkamp Using Pen-Based Outlines for Object-Based Annotation and Image-Based Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585 L. Schomaker, E. de Leau, L. Vuurpijl Interactive Query Formulation for Object Search . . . . . . . . . . . . . . . . . . . . . . . 593 T. Gevers, A.W.M. Smeulders Automatic Deformable Shape Segmentation for Image Database Search Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601 L. Liu, S. Sclaroff A Multiscale Turning Angle Representation of Object Shapes for Image Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609 G. Iannizzotto, L. Vita Contour-Based Shape Similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617 L.J. Latecki, R. Lakaemper Computing Dissimilarity Between Hand Drawn-Sketches and Digitized Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625 F. Banfi, R. Ingold
Retrieval Systems Document Generation and Picture Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . 632 K. van Deemter FLORES: A JAVA Based Image Database for Ornamentals . . . . . . . . . . . . . 641 G. van der Heijden, G. Polder, J.W. van Eck Pictorial Portrait Indexing Using View-Based Eigen-Eyes . . . . . . . . . . . . . . . 649 C. Saraceno, M. Reiter, P. Kammerer, E. Zolda, W. Kropatsch Image Retrieval Using Fuzzy Triples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657 S.H. Jeong, J.D. Yang, H.J. Yang, J.H. Choi
Table of Contents
XVII
Image Compression Variable-Bit-Length Coding: An Effective Coding Method . . . . . . . . . . . . . . . 665 S. Sahni, B.C. Vemuri, F. Chen, C. Kapoor Block-Constrained Fractal Coding Scheme for Image Retrieval . . . . . . . . . . . 673 Z. Wang, Z. Chi, D. Deng, Y. Yu Efficient Algorithms for Lossless Compression of 2D/3D Images . . . . . . . . . . 681 F. Chen, S. Sahni, B.C. Vemuri
Virtual Environments Lucent VisionT M : A System for Enhanced Sports Viewing . . . . . . . . . . . . . . 689 G.S. Pingali, Y. Jean, I. Carlbom Building 3D Models of Vehicles for Computer Vision . . . . . . . . . . . . . . . . . . . 697 R. Fraile, S.J. Maybank Integrating Applications into Interactive Virtual Environments . . . . . . . . . . 703 A. Biancardi, V. Moccia
Recognition Systems Structural Sensitivity for Large-Scale Line-Pattern Recognition . . . . . . . . . . 711 B. Huet, E.R. Hancock Complex Visual Activity Recognition Using a Temporally Ordered Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 719 S. Bhonsle, A. Gupta, S. Santini, M. Worring, R. Jain Image Database Assisted Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 727 S. Santini, M. Worring, E. Hunter, V. Kouznetsova, M. Goldbaum, A. Hoover Visual Processing System for Facial Prediction . . . . . . . . . . . . . . . . . . . . . . . . . 735 C. Xu, J. Wu, S. Ma Semi-interactive Structure and Fault Analysis of (111)7x7 Silicon Micrographs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 745 P. Androutsos, H.E. Ruda, A.N. Venetsanopoulos Using Wavelet Transforms to Match Photographs of Individual Sperm Whales Identified by Contour of the Trailing Edge of the Fluke . . . . . . . . . . 753 R. Huele, J.N. Ciano From Gaze to Focus of Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 761 R. Stiefelhagen, M. Finke, J. Yang, A. Waibel
XVIII Table of Contents
Automatic Interpretation Based on Robust Segmentation and Shape-Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 769 G. Frederix, E.J. Pauwels A Pre-filter Enabling Fast Frontal Face Detection . . . . . . . . . . . . . . . . . . . . . . 777 S.C.Y. Chan, P.H. Lewis
Visualization Systems A Technique for Generating Graphical Abstractions of Program Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 785 C. Demetrescu, I. Finocchi Visual Presentations in Multimedia Learning: Conditions that Overload Visual Working Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 793 R. Moreno, R.E. Mayer Visualization of Spatial Neuroanatomical Data . . . . . . . . . . . . . . . . . . . . . . . . . 801 C. Shahabi, A.E. Dashti, G. Burns, S. Ghandeharizadeh, N. Jiang, L.W. Swanson Visualization of the Cortical Potential Field by Medical Imaging Data Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 809 M.C. Erie, C.H. Chu, R.D. Sidman Applying Visualization Research Towards Design . . . . . . . . . . . . . . . . . . . . . . 817 P. Janecek
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825
Supporting Image-Retrieval by Database Driven Interactive 3D Information-Visualization Martin Leissler, Matthias Hemmje, Erich J. Neuhold GMD – German National Research Center for Information Technology IPSI – Integrated Publication and Information Systems Institute Dolivostr. 15, 64293 Darmstadt, Germany [leissler, hemmje, neuhold]@darmstadt.gmd.de Abstract. Supporting image-retrieval dialogues between naive users and information systems is a non-trivial task. Although a wide variety of experimental and prototypical image retrieval engines is available, most of them lack appropriate support for end-user oriented front ends. We have decided to illustrate the possible advantages of a tight coupling between interactive 3D information visualization systems and image retrieval systems based on database management systems by deriving requirements from a characteristic application scenario. By means of an “interactive 3D gallery” scenario, the paper provides an overview of the requirements, components, and architecture of a general database-driven 3D information visualization system on the basis of an RDBMS and VRML. The given approach supports loading time as well as runtime database access in various forms. It reflects the overall conceptual framework of our activities in this highly dynamic area of research and forms a basis for many other applications where information objects have to be visualized for interacting users or user groups.
1. Introduction Supporting image retrieval dialogues between naive users and information systems is a non-trivial task. Although supporting the basic “pattern matching” process within the image retrieval mechanism has been tackled by various research activities (e.g. [Pentland et al. 95] [Picard et al. 93] [Wang et al. 97] [Müller & Everts 97]) during the last years, supporting the user interface front end in an image retrieval dialogue in an appropriate way has been neglected to some extent. Most of the work conducted in this area (e.g. [VVB], [Chang et al 97a], [Chang et al 97b], [Chang et al 96a], [Christel et al 96]) applies user-interface paradigms implemented on the basis of 2D interface-toolkits. In contrast to these works we want to outline in this paper how image retrieval user interfaces in distributed front-end scenarios can be supported by means of interactive 3D information visualization technologies. The work presented in this paper is based on concepts, experiments, experiences and insights gained from our work aiming at supporting, e.g., full-text-retrieval and multimedia retrieval dialogues in a similar way. The paper introduces the available base technologies. Furthermore, it outlines an overall architectural system model based on a requirement
Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 1-14, 1999. Springer-Verlag Berlin Heidelberg 1999
2
Martin Leissler et al.
analysis derived from an example scenario. An overview of implementation aspects of the proposed architectural framework and an outlook on potential future works conclude the paper.
2. Base Technologies Looking at the development of non-immersive interactive information visualization applications over the past few years, the Virtual Reality Modeling Language (VRML) has clearly become the de facto standard for representing interactive 3D worlds on the web or in offline applications. Furthermore, in 1998, VRML97 (the actual specification, [VRML97]) has made its way to become an ISO standard. If we examine the historical development of VRML in detail, it has to be recognized that the first version of the language standard (VRML 1.0) was directly derived from the file format which the OpenInventor toolkit [Wernecke 94] from Silicon Graphics Inc. had defined to exchange 3D scenes between different applications. This version of VRML was completely static, i.e., no user interactivity and even no animation was supported in a VRML1.0 scene. Soon after this first version came the second version (VRML2.0) which is today mostly identical with the actual ISO standard. It incorporates complex interactive behavior, advanced animation features and custom extensibility. However, VRML 97 is still a closed file format which uses its own internal and proprietary event model to describe behavior and user interaction. All information about a scene is contained exclusively within the VRML code. In consequence, there is no “natural” way for a VRML scene to communicate and integrate with other external applications or software components. The task of identifying concepts to solve this problem has mainly been tackled through single working groups within the VRML consortium [Web3D]. One working group, for example, has defined the so called External Authoring Interface (EAI) [EAI] to handle biderectional communication between a Java applet and a VRML97 scene coexisting on a web page. If VRML is, based on its promising starting point, ever to become a seriously used interactive information visualization medium and development platform for all kinds of information system applications, a flexible and efficient integration of the language with existing technological standards such as interfaces to database management systems or application servers has to be achieved. This means built-in standard mechanisms for communication between VRML and external systems have to be derived from the existing standard. Clearly, this problem has to be tackeled from both sides, the VRML language side and the side of external standard application programming interface (API). Situated in this interactive information visualization working context, our research concentrates on VRML and database integration which, in our opinion, is the most urgent problem at hand. By working on the problem of how VRML could communicate with existing database management systems (DBMS), we can also learn a lot about supporting more complex information visualization scenarios, e.g., persistent multi-user scenarios. Until today, all applications that used VRML in
Supporting Image-Retrieval
3
connection with a DBMS had to rely on using custom-coded database access with proprietary APIs such as the Java Database Connectivity (JDBC) [JDBC] or the Microsoft Open Database Connectivity (ODBC) [ODBC]. This is highly unsatisfactory, because people work on the implementation of the same interfacing problems over and over again. A standardization of the VRML side of the DBMS integration has been partially proposed by the database working group of the Web3D consortium [Web3D]. However, we believe that, although the overall observations of the working group are correct and sound, some more work will have to be done in the detailed specification of the features of the database integration – on API side as well as on VRML side. Some examples of necessary features will be provided in later sections of this paper. To derive the neccesary extended requirements, this paper first takes a look at a complex example scenario and later describes how this scenario could possibly be conceptually solved and supported by corresponding architectural system models and implementations.
3. Example Scenario Given that, for example in an electronic commerce application, users want to browse through a visually rich 3D virtual gallery environment (as described in [Müller et al. 99]) filled with, from their subjective point of view, more or less interesting paintings of various artists. They want to have the option to buy a painting or just enjoy reviewing works of their favorite artists, for a while. A soon as they have entered the system, a so called “query interface” is presented in which they can enter a description of their interest. After choosing a few of their favorite artists and selecting some paint styles they like, it is time to submit the query containing the so called “search criteria” to the system. Now, a 3D visually interactive presentation of an art gallery is generated in an area of the screen. The selection of the works-of-art is based on the users’ search criteria. Somewhere else on the screen, an interactive 2D overview visualization which explains the overall architectural structure of the gallery building is visualized. The displayed architectural topology of the gallery building is structured hierarchically and is therefore easy to navigate by selecting areas related to the different artists, painting techniques, and styles. As the users move around in this virtual gallery environment, they orient themselves with the help of so called “landmarks” and “signposts” inside the 3D environment as well as in the 2D overview visualization of the gallery. After a short spatial navigation, a room entrance is reached. A label describes that the works of art displayed in this room match a couple of search criteria defined earlier in the form based query construction dialogue by the user (e.g., a certain artist and his self-portrait style paintings). After navigating through the entrance, a room shows up which contains only the paintings expected in this section of the gallery. As the users navigate into the room, they take their time to study the paintings hanging on the walls of the room. By clicking on one of the paintings in the 3D environment, all information about the painting stored in the database is displayed on a separate area of the screen together
4
Martin Leissler et al.
with a more detailed high-quality image of the painting and further meta-information like, e.g., pricing and sales status. While striving around in some room, it can be recognized that one of the paintings is suddenly marked with a small “not-available” banner. By some coincidence, it must have been removed from the gallery. Either it has been sold to someone else and is therefore no longer available in the 3D gallery or someone has, e.g., not paid his bill to the gallery service provider for renting the space in the gallery. The application aspects and the users’ experience described above demand a set of different interactive information visualization functions to be supported by an overall technical architecture. The concepts which can define a formal basis for the implementation of such an architectural framework and its components are described below. The complete conceptual interactive information visualization model will be described in more detail in a different paper.
4. Other Work Before we describe the detailed conceptual system model and implementational architecture of our approach, we take a short look at existing systems which support scenarios that are similar but not identical to the one described above. 4.1. Virgilio The Virgilio system as described in [Massari et al. 98] and [Constabile et al. 98] is a software system architecture which allows the user to submit a generic query on a database which typically contains multimedia content. As a result, the system constructs a 3D metaphorical visual representation of the hierarchical query result structure through which the user can browse the query result set interactively. One of the main features of Virgilio is that the querys, the visual representation, and the mapping between the query and the visualization are all stored in persistent repositorys and can therefore be easily exchanged. The mapping between visual objects and query structures is called a “metaphor” in Virgilio. This metaphor can be completly user-defined, which means that the appearance of the visual environment can be dynamically adjusted. On the basis of a set of properties of the visual objects (e.g. a room object can contain other objects), the metaphor is intelligently applied to the given query result. A prototypical implementation of the Virgilio architecture exists on the basis of a custom application using a proprietary API (OpenInventor). After choosing a different query, the system has to completely reconstruct the 3D environment. If the underlying data changes during the runtime navigation, this will have no immediate effect on the scene. Furthermore, the querys in the query repository are fixed. The user cannot query the system freely. 4.2. TVIG The TVIG system (The Virtual Internet Gallery) [Müller et al. 99] implements a scenario very similar to the 3D gallery scenario described above. Users can use a
Supporting Image-Retrieval
5
standard web browser to query a relational database for information about artworks (e.g., their favorite artist by name) and retrieve a dynamically constructed 3D gallery visualized in the same web page. While browsing through the gallery, users can click on door handles to enter the rooms containing the paintings. Rooms and paintings are queried and constructed on demand, at runtime, to keep the system load as low as possible. The mapping between the gallery building structure and the search results are user definable from within the system. The implementation of TVIG uses some standards such as HTML, VRML, Java and JDBC, but is mainly based on custom-written code. As in Virgilio, the visual environment does not react immediatly to changes in the underlying database. The visualization is reconstructed if a new search query is submitted by the user. In both systems there are no generic standard mechanisms to communicate between the database and the 3D visualization.
5. Requirements Any scenario similar to the one described above should demand for the following general requirements: First of all, we definitetly need an appropriate information distribution mechanism for the given scenario. In today’s world of globally networked computer systems, it is obvious that an application like a virtual gallery should be a completely web based application in order to be able to reach as many users as possible. The numerous versions of web browsers with all kinds of mutimedia plugins available on client machines makes the www an ideal application platform for our scenario. In addition to browser technology defining the web-based application front end, a generally VR-capable application server is required in the back end of the application solution. Next, we need a standard way of representing and rendering a real time interactive 3D environment for the virtual gallery. This technology should be able to run on as many client platforms as possible and has to support user interaction, scene object behavior and scene animation. Furthermore, the visualization environment has to support interactive navigation in a straightforward way and should seamlessly scale with the available client resources. Since the data for the information objects of interest (in this case paintings) should be available to all users of the application at any time, a peristent storage database is required. This mechanism should also be able to serve a broad bandwith of different client platforms and a large number of users, in parallel. Because the visual environment is based on the highly dynamic content of the persistent storage mechanism as well as on the highly dynamic interests of the users there has to be an integrated mechanism to parametrically construct, reconstruct and adapt the whole visual environment in a very flexible fashion.
6
Martin Leissler et al.
Any user interaction has to be mapped to operations on the persistent storage which are in turn reflected in the 3D environment. This muhas to be enabled while the application is running. Therefore, we need a bidirectional communication mechanism between the running 3D environment and the underlying persistent storage mechanism [Hemmje 99]. Changes in the storage should be reflected in the 3D environment immediatly. Therefore we need a mechanism to automatically notify the 3D scene about changes occuring in the underlying database managed by the persistent storage mechanism.
6. Architectural Model for a Database Driven 3D Visualization Environment Figure 1 displays the general architectural model of a an application scenario like the virtual gallery. It supports the requirements derived in Section 3. The architectural model consists of a browsing client, an application server, and a persistent storage system supporting the information object database displayed in figure 1. Since the communication between the components (depicted by arrows) can potentially take place over a networked connection, the single components can be arbitrarily assigned to hardware platforms, e.g., from all three components on one platform to all on different platforms. The left side of the diagram displays the users’ VR-capable browsing client. After entering the application, the VR client displays the user interface with all its components like, for example, a form-based user interface component which can be used to enter search criteria describing the users’ interest in the customized gallery experience. runtime access
VR Client
results & notification query
request 3D scene
Application Server
DB
result
VR Extension Fig. 1. The general architectural model of a data driven 3D visualization application
If a query (i.e., an information request) containing the search criteria is submitted to the application server, the server recognizes an appropriate 3D scene has to be delivered back to the client. Since the server has to produce the actual data that matches the users’ request, it has to translate user requests into a query which can be sent to the database. The database system processes the query and sends the result back to the application server. Now, the application server can use the retrieved data
Supporting Image-Retrieval
7
to construct the 3D scene with the help of some special server extension. The 3D scene is then sent to the users VR browsing client and displayed properly. As the dynamically constructed 3D scene is interactively browsed, users can interact with certain objects of the surrounding environment, which, in turn, may lead to the necessity to get additional data from the database storage mechanism. If, for example, the user enters an area in which the images for the paintings on the walls have not been retrieved from the database, they have to be retrieved, downloaded, and integrated into the scene during application runtime. Furthermore, if the user clicks on the image of a painting to retrieve information about the artist, price etc. the same holds true. In these cases, the client runtime access mechanism has to be used to query the database and retrieve the results. These have to be directly integrated into the scene, at runtime, too. Furthermore, as soon as any data manipulation action is performed on the database storage mechanism which effects the data visualized in the running 3D scene - be it from external manipulation or from the scene itself – an immediate notification of the client has to be performed to which the clients 3D environment can, in turn, react. By now, we have described the application interaction cycle completely. Note that even though the arrows in the above figure are drawn straight from the client to the database storage mechanism, the runtime communication and notification mechanisms do not necessarily have to go directly from the client system to the storage system. It is possible (and in many cases appropriate) to handle the communication through an additional middleware layer. Taking the described scenario, the derived reqirements, and the proposed general architectural system model as a basis for our implementation approach, we can now look at the technical details which are needed to implement such a system.
7. Implementation Aspects in the Architectural Model Assuming that we use a commonly available web browser as the platform for our application front end, we have furthermore decided to store the information about the artists, paintings, styles, techniques, prices, etc. in the tables of a conventional relational database management system (DBMS) as the platform for the persistent storage mechanism, i.e., the back end of our application. Next we assume that VRML97 is used to present the 3D visualization of the gallery. As stated before, VRML97 is the most advanced and widely-accepted standard available on the market. Therefore, it is most likely to be installed on a large number of client web browsers. However, as stated in the first section, VRML97 has a few limitations. Since it is a closed file format with its own event model, all interactions which could influence external systems have to be translated from and to the VRML event model. To date, there is no predefined way to communicate between a VRML scene and a database system. This limitation applies to both, VRML scenes lying on a web server (loading time database access) and VRML scenes currently running in a client browser (runtime database access).
8
Martin Leissler et al.
If a scenario as the described virtual gallery has to be implemented with reusable components, we have to define and implement standardized and general ways of database communication. 7.1. Loading Time Database Access in VRML Since the 3D gallery is dynamically constructed on the basis of results corresponding to the users search criteria, we cannot use a static VRML scene to represent the visual environment. Therefore, the scene has to be somehow constructed dynamically and more or less on the fly after the user has submitted the search criteria to the system. Because all data for the paintings is stored in a relational database, the application has to translate the users’ search criteria into some query language statements (proposedly SQL-statements) which are then sent to the DBMS to be executed. Furthermore, the results of the query have to be integrated into the process of dynamically constructing the 3D gallery scene. In a typical implementation this would work by letting the custom coded application (typically a Java applet or an ActiveX control) send an SQL statement to the DBMS via some API. Then, while iterating over the query statements’ result set, the VRML scene code is generated by some code segment, deep inside the application, and sent to the VRML browser component. Because of the broad variety of possible visualizations of the 3D content, the probability for being able to reuse the VRML scene constructor code in a different but to some extent similar application scenario is typically very low. Furthermore, even if this implementation technique may work well, what we really want is a flexible and efficient standard way to integrate database results into a dynamically-built VRML scene. This can be achieved by integrating a loading time database access capability into VRML with the help of a server side include mechanism as presented in [Risse et al. 98] and [Müller et al. 98]. This mechanism allows the scene designer to define unfinished VRML templates which contain SQL statements. At loading time of the scene the responsible server on which the VRML scene templates are stored (i.e., either a web server or a DBMS with a custom extension) fills the templates with actual result data from the SQL query results. The client browser retrieves a perfectly VRML compliant scene description which visually represents the actual data from the DBMS. Figure 2 displays a possible implementation approach of the server extension module during the dynamic information visualization process. On the upper left of Figure 2 the VRML scenegraph containing a server side include node and a template subgraph is displayed. Before the server returns the VRML scene, the SQL statement is executed. Then the template subgraph is instanced for each returned row and filled with the query results. The so constructed VRML-compliant scene is returned to the client browser. 7.2. Runtime Database Access in VRML For HTML pages which download once from a web server and are only statically viewed by the user, a server-side include mechanism may be enough database
Supporting Image-Retrieval
9
interaction. But the main difference between a static HTML page and a VRML scene is that, after download, the VRML scene starts to run, i.e., it has a runtime. The user interacts with scene elements, and predefined animation sequences may be running, triggered by scene events. Mapping this to the given scenario means that the user, for example, interactively opens a door of the gallery by clicking on the handle. If such an interaction should trigger a database access (read or write) to dynamically construct the room behind the door, we clearly need some runtime interaction component in VRML capable of sending SQL statements to a database and retrieving the results. Furthermore, it should be possible to distribute the result data into the running VRML scene. Generally, this means a mapping from VRML runtime user interaction to database interaction. In this case, the events occur in the VRML scene and are mapped onto DBMS manipulation sequences. SSI-Node
VRML extension call SELECT s1,s2,... FROM tabelle WHERE C
sqlStatement "SELECT s1,s2,... FROM tabelle INTO f1, f2,... WHERE C“
Template
f1
f2
f3
s1
s2
s3
s4
...
V11
V12
V13
V14
...
V21
V22
V23
V24
...
VN4
...
f4
... VN1
VN2
VN3
Instancing process
... V11
V12
V13
V14
V21
V22
V23
V24
VN1
VN2
VN3
VN4
Fig. 2. Loading time database access with a VRML server side include mechanism.
Again this functionality can be achieved by using proprietary Java code inside the VRML scripting nodes. In a typical VRML scene which uses such a database access mechanism, the custom Java code inside a scripting node reacts to some scene event (representing a user interaction) by accessing a DBMS via a defined API (e.g., JDBC). The results are then collected in a VRML multiple value type field and further distributed in the scene by the VRML event mechanisms. Once again, we have custom code which has to be rewritten from scratch or modified each time the application requirements change. This of course, is costly and a never ending source of bugs which are potentially hard to trace. Moreover, this solution is highly dependent on the VRML browser component which typically leads to unpredictable behavior in cross browser application scenarios.
10
Martin Leissler et al.
What would make the VRML application developer’s life a lot easyer is a standard mechanism for runtime SQL database access out of a VRML scene. The approach has to be general enough to cover all possible cases within a runtime database access scenario. This mechanism is provided by a VRML extension node (prototype) which allows to define arbitrary SQL statements and the distribution of the possible results in the scene graph, as shown in Figure 3. Web Browser VRML Scene
SQL node
(3)
SQL Statement results
(1) (2) RDBMS
Fig. 3. The VRML runtime SQL-node. First, an SQL statement is sent to the database (1). Next, the results are returned into the VRML scene graph (2). Finally, the results are distributed to the defined positions in the graph (3).
An extension node with a similar functionality has been proposed by the database working group (now enterprise working group) of the VRML consortium in their “Recommended practices for SQL database access” [Lipkin 98]. However, while the working groups proposal covers the steps (1) and (2) of Figure 3, our solution also lets the user of such an extension node directly define exactly how the result data is to be distributed across the nodes of the VRML scene graph. Thereby, the developer is spared the burden of writing a couple of custom script nodes for every different scene graph, just in order to distribute data. This leads to less, and more efficient, VRML code. Communication between the SQL node and the DBMS can be implemented through a direct database connection. Because in many application scenarios DBMS connections are expensive, it is also possible to connect all clients to some middleware component via lightweight communication protocols and let this component handle all requests through one keep-alive DBMS connection. The implementation specific details of this approach will be presented in a separate paper. 7.3. Automatic Database Event Triggering with VRML The last requirement mentioned in our analysis is derived from the fact that, although the 3D scene in some way visualizes the actual database content, it does not automatically react to changes in the underlying data. If, for example, one of the paintings in our example scenario is marked as “not-available” in the database, for
Supporting Image-Retrieval
11
some reason (e.g., because the painting has been sold), the VRML scene should react instantly by visualizing this event, e.g., by visualizing a banner on top of the painting. Therefore, we need a mechanism which enables the database to automatically notify the running VRML scene about occuring events. More general, this means a mapping from database events to VRML scene events is required. Typically, in existing applications, this has to be done by reloading and reconstructing the whole scene based on the changed database content, which is unflexible and time consuming. Another possibility is to query the database in regular intervals from within the VRML scene to detect possible changes in the underlying data. This approach could be implemented with the help of the above mentioned runtime query component. However, this unneccesarily consumes network bandwidth and runtime resources even if the database content remains unchanged. An elegant solution to this problem is an active database trigger mechanism which enables the database to contact the VRML scene if some given part of the database has changed in a predefined way. Such a technology needs to define standards how the database should invoke a notification mechanism, how the database events are translated to VRML events and sent to the running scene and, finally, how the VRML scene can handle such events in order to delegate them to the relevant parts in the scenegraph. VRML Client
DB Client
VRML Scene notification
Trigger
Trigger Node
DBMS
access
DB Client DB Client
SQL Node
Fig. 4. Architecture for automatic notification through database triggers
Many different clients of the database system (including the VRML scene itself !) can access the database and change its internal data. Database triggers can be assigned to arbitrary parts of the data (i.e., tables in an RDBMS) and fire a predefined action sequence. In this case the the trigger action launches a notification mechanism which contacts a defined node in the VRML client scene. After distributing the event in the scene a new query can be sent to retrieve the updated data from the database. Note that this last aspect of the presented scenario already supports a shared multi client architecture. As soon as we have multiple clients connected to the system which display different parts of the scene (or even different scenes) based on the data in the storage, the trigger notification mechanism could also be used in shared virtual environment applications which have to synchronize a global application state across multiple connected clients. Again, this is best done by using a middleware component to handle the communication between the database trigger mechanism and the client
12
Martin Leissler et al.
machine. The middleware component can distribute the incoming notification events from the database to the appropriate clients and, at the same, time merge the expensive database connections. The details of this approach are presented in a separate paper. 7.4. Overall Architectural Framework After describing all crucial system components, we can now define a generic overall architectural framework which matches our given requirements and is able to run database driven interactive 3D information visualization applications similar to the described gallery scenario. (Figure 5) Runtime trigger Runtime Loading time
VRML Scene
Application Server
Trigger Node notification SQL Node results
. . .
Trigger notification server
SQL statement
JDBC server
Trigger SQL statement results
VRML Scene Search request
Web server Web server
SQL statement Result rows
Trigger Node VRML scene
VR Extension
DB Client
notification
RDBMS
DB Client . . . DB Client
SQL Node
Fig. 5. Technical architecture of a database driven interactive 3D information visualizatiom system
The above figure displays the interaction of all components. Before VRML clients go into runtime interaction mode, they log into the system and request the 3D environment from the web server based on custom search criteria. The web server, as all other intermediary sever components, is combined under the concept of an application server. Note again that all components in the diagram, including the application server components, can be arbitrarily assigned to physical machines. During the loading-time process (dotted arrows), the web server querys the DBMS, fetches the result data matching the users search criteria, and, finally, returns the customized VRML templates to the client via the VR extension module. As a result of certain interactions (e.g., opening a door in the gallery), an SQL node in the running VRML scene querys the database via a middleware database driver (typically JDBC type 3) and distributes the query results to the proper positions in the scene graph (e.g., puts the pictures on the gallery walls).
Supporting Image-Retrieval
13
If the underliying data for a VRML scene is affected now, either by a VRML client (through an SQL node) or an external DBMS client, the trigger notification mechanism may be launched (normal arrows). The notification event is distributed to the clients via a middleware trigger server. This component notifies exclusively those clients which are affected by the current change in the underlying data (e.g., clients displaying the same scene), and thereby optimizes the network load. Notificatications are distributed in the running VRML scene as events which, in turn, may launch a query from an SQL node to retrieve the most recent data. Note how the trigger mechanism can be used to propagate user interactions (affecting the global database state) on one client across multiple other clients connected to the system.
8. Conclusions and Outlook In this paper, we have presented an architectural framework consisting mainly of three components (VRML server side includes, SQL runtime node and active database triggers) which is capable of running highly dynamic database driven interactive 3D information visualization applications. We have outlined how all architectural components can work together in a complex visualization scenario such as the virtual gallery. However, the presented architectural components can also be used as completely independent stand-alone components in applications with different requirements. Indeed, in most application scenarios not all of the three components need to be used. Our experience has taught us that a combination of only some of them is in most cases already sufficient.
References [Chang et al 97a] Chang, S.-F., Chen, W., Meng, H., Sundaram, H., Zhong, D. (1998). Videoq: An automated content-based video search system using visual cues. In Proceedings of ACM Multimedia 1997. [Chang et al 97b] Chang, S.-F., Smith, J., Meng, H., Wang, H., Zhong, D. (1998). Finding images/video in large archives.‘ In: D-Lib Magazine, February 1997. [Chang et al 96] Chang, Y.-L., Zeng, W., Kamel, I., Alonso, R. (1996). Integrated image and speech analysis for content-based video indexing. In: Proceedings of ACM MM 1996 [Christel et al 97] Christel, M., Winkler, D., Taylor, C. (1997). Multimedia abstraction for a digital video library. In: Proceedings of ACM Digital Libraries '97, pages 21--29, Philadelphia, PA [Costabile et al. 98] Costabile, M. F., Malerba, D., Hemmje, M., Paradiso, A. (1998) Building Metaphors for Supporting User Interaction with Multimedia Databases In: Proceedings of 4th IFIP 2.6 Working Conference on Visual DataBase Systems - VDB 4, L'Aqulia, Italy, May 27-29, p. 47-66, Chapman & Hall 1998 [DBWork] Enterprise Technology Working Group of the Web3D consortium http://www.vrml.org/WorkingGroups/dbwork/ [EAI] Information technology -- Computer graphics and image processing -- The Virtual Reality Modeling Language (VRML) -- Part 2: External authoring interface Committee Draft ISO/IEC 14772-2:xxxx http://www.web3d.org/WorkingGroups/vrml-eai/Specification/
14
Martin Leissler et al.
[Hemmje 99] Hemmje, M. (1999). Supporting Information System Dialogues with Interactive Information Visualization.. To appear in: Dissertation Thesis, Technical University of Darmstadt, 1999 [JDBC] Sun Microsystems, Inc. The JDBCTM Data Access API http://java.sun.com/products/jdbc/index.html [Lipkin 98] Lipkin, D. (1998). Recommended Practices for SQL Database Access http://www.web3d.org/Recommended/vrml-sql/ [Massari et al. 98] Massari, A., Saladini, L., Sisinni, F., Napolitano, W., Hemmje, M., Paradiso, A., Leissler, M. (1998). Virtual Reality Systems For Browsing Multimedia. In: Furth, B. (ed.): Handbook of Multimedia Computing [Müller et al. 98] Müller, U., Leissler, M., Hemmje, M. (1998). Entwurf und Implementierung eines generischen Mechanismus zur dynamischen Einbettung multimedialer Daten in VRMLSzenen auf der Basis eines objektrelationalen DBMS. GMD Research Series, No, 23/1998, GMD – Forschungszentrum Informationstechnik, St. Augustin 1998 [Müller et al. 99] Computing and Systems (ICMCS'99) Mueller A., Leissler M., Hemmje, M., Neuhold E.(1999). Towards the Virtual Internet Gallery. To appear in: Proceedings of IEEE International Conference on Multimedia [Müller & Everts 97] Müller, A.,Everts, A. (1997). Interactive image retrieval by means of abductive inference. In RIAO 97 Conference Proceedings -- Computer-Assisted Information Searching on Internet, pages 450--466, June 1997. [ODBC] Microsoft Press (1997). Microsoft ODBC 3.0 software development kit and programmer’s reference. Microsoft Press, Redmond Washington [Picard et al. 93] Picard, R. W., Kabir, T. (1993). Finding Similar Patterns in Large Image Databases. In: IEEE ICASSP, Minneapolis, Vol. V, pp. 161-164, 1993 [Pentland et al. 95] Pentland, A., Picard, R. W., Sclaroff, S. (1995). Photobook: Contentbased Manipulation of Image Databases. In: SPIE Storage and Retrieval Image and Video Databases II, San Jose, CA, 1995 [Risse et al. 98] Risse, T., Leissler, M. , Hemmje M., Aberer, K. (1998). Supporting Dynamic Information Visualization With VRML and Databases. In: CIKM '98, Workshop on New Paradigms in Information Visualization and Manipulation, Bethesda, November 1998 [VRML97] International Standard ISO/IEC 14772-1:1997 http://www.web3d.org/Specifications/VRML97/ Information technology -- Computer graphics and image processing -- The Virtual Reality Modeling Language (VRML) -- Part 1: Functional specification and UTF-encoding. [VVB] The virtual video browser http://hulk.bu.edu/projects/vvb_demo.html [Wang et al. 97] Wang, J. Z., Wiederholg, G., Firschein, O., Wie, S.X. (1997). Content-based image indexing and searching using Daubechies’ wavelets.In: International Journal on Digital Libraries, Vol.1, Number 4, December 1997, Springer Verlag, pp.311-328 [Web3D] Web3D Consortium (formerly: VRML consortium) http://www.web3d.org Home of the Web3D consortium (formerly http://www.vrml.org) [Wernecke 94] Wernecke, Josie (1994) The Inventor Mentor, Programming Object-Oriented 3D-Graphics with Open Inventor Release 2. Open Inventor Architecture Group; AddisonWesley Publishing Company, Inc. 1994
Video Libraries: From Ingest to Distribution
Ruud M. Bolle and Arun Hampapur IBM T. J. Watson Research Center Yorktown Heights, NY 10598 {bolle,arunh}@us.ibm.com
Abstract. Production, transmission and storage of video will eventually all be in digital form. Additionally, there is a need to organize video efficiently in databases so that videos are easily ingested, retrieved, viewed and distributed. We address and discuss many of the issues associated with video database management.
1
Introduction
The digital form of video will allow us to do many things – some of these things can be envisioned today, others will be discovered during the years to come. The digital form permits computationally extracting video content descriptors. Ideally, video is completely annotated through machine interpretation of the semantic content of the video. In practice, given the state-of-the-art in computer vision, such sophisticated data annotations may not be feasible. Much of the focus in multimedia library efforts has been on text and image databases [1], not on video libraries. We touch upon techniques for processing video as an image of space-time and we argue that the type of video analysis that has to be performed should be geared toward the specific video genre or category – e.g., sports versus soap operas. Processing video to derive annotations is one thing. Another thing, as we discuss first, is the infrastructural demands for putting such video management systems together.
2
Base Video Management System
Figure 1 shows the functional components of a base video management system and their relationship to each other. Ingest and annotation: Video data can be analog or digital. Ingest deals with video digitization in the case of analog video, and parsing a wide variety of digital video formats. Associating annotations with video segments is another part of the ingest process. Annotation consists of 1. Real time logging: Extracting a first level of time dependent indices from a video stream. This may include keyframe extraction and basic audio index information.
The work reported in the paper has been funded in part by NIST/ATP under Contract Number 70NANB5H1174.
Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 15–18, 1999. c Springer-Verlag Berlin Heidelberg 1999
16
Ruud M. Bolle and Arun Hampapur
Meta Database
Retrieval and Browsing
Video Input Ingest and Annotation
Media Database
Media Distribution
Fig. 1. Base video management system 2. Manual annotation and cataloging: In many applications, manually associating information with different time segments of video and linking a unit of video to other media is essential. 3. Automatic offline annotation: Providing content based access to video requires content analysis and annotation. Most of these processes run offline. The data generated by the ingest and annotation process is stored in two locations, namely: Meta database: This can be a traditional relational database system like DB2. A data model for video data management [2] includes a time independent part like, title, producer, directors, length, etc. And, more interestingly, a time dependent part which uses a relational table structure. Media database: These databases handle both the storage and distribution aspects of managing the actual digital video data. They are file servers which are designed to handle streaming media like video. Finally, there is the issue of component inter-operability. Each of the functional blocks shown in Figure 1 is a complex subsystem. This gives rise to issues of inter-operation between the components using a standardized command and control protocol. A research effort that addresses this issue can be found in [3].
3
Video Annotation
Annotation is ideally achieved in a completely automatic fashion [4]. Video is a concatenation of shots. As described in [5], the analysis of video should not depend too much on the reliability of the shot detection algorithm that is used. Moreover, the analysis of video should go beyond computations on pixels just within shots, i.e., between-shot processing is important. Between-shot processing has as goal to derive high-level structure for automatic annotation of, possibly long, video segments. The scene structure of, for example, sitcoms can be rediscovered using clustering of shots [5]. In [6] the concept of motion picture grammars is introduced. The thesis is that video data can be represented by grammars (e.g., [7]). The grammars need to be stochastic [8];
Video Libraries: From Ingest to Distribution
17
stochastic, context-free grammars and hidden Markov models [9] are closely related. Hidden Markov models are used in [10] to detect commercials.
4
Video Retrieval and Distribution
Retrieving video through the formulation of a query is inherently more complicated than retrieving text documents. In addition to text, there is visual and audio information; moreover, there is temporal visual dynamics. Very much like text query formulation, a video query is a sequence of steps. Each step is an active filtering to reduce the number of relevant candidates. Each step allows interactive query formulation, and each gives a more refined query to the next step. Video query (see [5]) can be broken down as: query on the category of video (navigating), query on the text, and/or audio and visual feature descriptions (searching), query on the semantic summary of visual content (browsing) and query on the full-motion audio-visual content (viewing).
5
Specialized Video Management System
Base video systems are currently available as products. Such systems provide most of the infrastructural requirements for managing video. However, effective video management requires the ability to retrieve video based on much higherlevel semantic concepts. This demands the development of specialized video data management systems which are tailored to different domains. Each new application domain will require several additional functionalities which include specialized indexing algorithms, user interfaces, and data models: Indexing algorithms: Depending on the application domain, new indexing strategies are needed. For example, for sports, new event indexing algorithms need to be developed. Say, for basketball, algorithms for detecting events like scoring become critical. User interfaces: The browsing and viewing patterns for video will differ significantly across domains. For example, in a news video management system, the searching will be based on the content of the speech. For sports it will be based more on visual content such as different play-patterns. Thus both the query interface and the video control interface need to suit the domain. Data models: Certain domains may require that the management system be capable of managing several types of media and associations between them. This implies that data models for such systems have to be augmented beyond the simple models used in base video management systems.
6
Discussion
We have described many of the aspects of video database management systems. Video indexing is but one of the components of such systems; video ingest is another important and often neglected component. Finally, complicated!l infrastructures are needed for complete end-to-end systems.
18
Ruud M. Bolle and Arun Hampapur
References 1. A. Gupta and R. Jain, “Visual information retrieval,” Comm. ACM, vol. 40, pp. 70– 79, May 1997. 15 2. A. Coden, N. Haas, and R. Mack, “A system for representing and searching video segments defined by video content annotation methods,” tech. rep., IBM T.J. Watson Research Center, 1998. 16 3. N. Haas, PROPOSED SMPTE STANDARD for television Digital Studion Command and Control (DS-CC) Media and Metadata Location. NIST/ATP HD Studio Joint Venture, 1998. 16 4. A. Nagasaka and Y. Tanaka, “Automatic video indexing and full-motion search for object appearances,” in Proc. IFIP TC2/WG2.6 2nd Working Conf. on Visual Database Systems, pp. 113–127, Sep.-Oct. 1991. 16 5. R. M. Bolle, B.-L. Yeo, and M. M. Yeung, “Video query: Research directions,” IBM J. of R & D, vol. 42, pp. 233–252, March 1998. 16, 17 6. R. Bolle, Y. Aloimonos, and C. Fermuller, “Toward motion picture grammars,” in Proc. IEEE 3rd ACCV, pp. 283–290, Jan. 1998. 16 7. K. S. Fu, Syntactic Pattern Recognition and Applications. Englewood Cliffs, NJ: Prentice Hall, 1982. 16 8. E. Charniak, Statistical Language Learning. MIT Press, 1993. 16 9. X. D. Huang, Y. Ariki, and M. A. Jack, Hidden Markov Models for Speech Recognition. Edinburgh University Press, 1990. 17 10. Y.-P. Tan and R. Bolle, “Binary video classification,” Tech. Rep. RC-21165, IBM T.J. Watson Research Center, 1998. 17
Querying Multimedia Data Sources and Databases1 Shi-Kuo Chang1, Gennaro Costagliola2, and Erland Jungert3 1
Department of Computer Science University of Pittsburgh
[email protected] 2 Dipartimento di Matematica ed Informatica Università di Salerno
[email protected] 3 Swedish Defense Research Institute (FOA)
[email protected]
Abstract. To support the retrieval and fusion of multimedia information from multiple sources and databases, a spatial/temporal query language called ΣQL is proposed. ΣQL is based upon the σ−operator sequence and in practice expressible in SQL-like syntax. ΣQL allows a user to specify powerful spatial/temporal queries for both multimedia data sources and multimedia databases, eliminating the need to write different queries for each. A ΣQL query can be processed in the most effective manner by first selecting the suitable transformations of multimedia data to derive the multimedia static schema, and then processing the query with respect to this multimedia static schema.
1 Introduction The retrieval and fusion of spatial/temporal multimedia information from diversified sources calls for the design of spatial/temporal query languages capable of dealing with both multiple data sources and databases in a heterogeneous information system environment. With the rapid expansion of the wired and wireless networks, a large number of soft real-time, hard real-time and non-real-time sources of information need to be processed, checked for consistency, structured and distributed to the various agencies and people involved in an application [12]. In addition to multimedia databases, it is also anticipated that numerous web sites on the World Wide Web will become rich sources of spatial/temporal multimedia information. Powerful query languages for multiple data sources and databases are needed in applications such as emergency management (fire, flood, earthquake, etc.), telemedicine, digital library, community network (crime prevention, child care, senior citizens care, etc.), military reconnaissance and scientific exploration (field computing). These applications share the common characteristics that information from multiple 1
This research was co-funded by the National Science Foundation, USA, the Swedish National Defence Institute and the Italian National Council of Research (CNR).
Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp.19 -28, 1999. Springer-Verlag Berlin Heidelberg 1999
20
Shi-Kuo Chang et al.
sources and databases must be integrated. A typical scenario for information fusion in emergency management may involve live report from a human observer, data collected by a heat sensor, video signal from a camera mounted on a helicopter, etc. Current systems often have preprogrammed, fixed scenarios. In order to enable the end user to effectively retrieve spatial/temporal multimedia information and to discover relevant associations among media objects, a flexible spatial/temporal multimedia query language for multiple data sources and databases should be provided. To support the retrieval and fusion of multimedia information from multiple sources and databases, a spatial/temporal query language called ΣQL is proposed. ΣQL is based upon the σ−operator sequence and in practice expressible in an SQL-like syntax. The natural extension of SQL to ΣQL allows a user to specify powerful spatial/temporal queries for both multimedia data sources and multimedia databases, eliminating the need to write different queries for each. Query language for heterogeneous multimedia databases is a new and growing research area [9, 13]. There has been substantial research on query languages for images and spatial objects [2], and a survey can be found in [5, 6]. Of these query languages, many are based upon extension of SQL [14], such as PSQL [15] and Spatial SQL [8]. Next come video query languages where the focus is shifted to temporal constraints [1] and content based retrieval [3]. While the above described approaches each address some important issues, there is a lack of unified treatment of queries that can deal with both spatial and temporal constraints from both live data sources and stored databases. The proposed approach differs from the above in the introduction of a general powerful operator called the σ−operator, so that the corresponding query language can be based upon σ−operator sequences. The paper is organized as follows. The basic concepts of the σ−query is explained in Section 2. Section 3 introduces elements of Symbolic Projection Theory and the general σ−operator, and Secction 4 describes the SQL query language. An illustration of data fusion using the σ−query is presented in Section 5. Section 6 formalizes the representation for multimedia sources and then gives a query processing example. In Section 7 we discuss further research topics.
2 Basic Concepts of the σ−Query As mentioned in Section 1, the σ−query language is a spatial/temporal query language for information retrieval from multiple sources and databases. Its strength is its simplicity: the query language is based upon a single operator - the σ−operator. Yet the concept is natural and can easily be mapped into an SQL-like query language. The σ−query language is useful in theoretical investigation, while the SQL-like query language is easy to implement and is a step towards a user-friendly visual query language. An example is illustrated in Figure 1. The source R, also called a universe, consists of time slices of 2D frames. To extract three pre-determined time slices from the source R, the query in mathematical notation is: σt (t1 , t2 , t3 ) R. The meaning of the σ−operator in the above query is SELECT, i.e. we want to select the time axis and three slices along this axis. The subscript t in σt indicates the selection of the time axis. In the SQL-like language a ΣQL query is expressed as:
Querying Multimedia Data Sources and Databases
21
SELECT t CLUSTER t1, t2, t3 FROM R
Fig. 1. Example of extracting three time slices (frames) from a video source
A new keyword "CLUSTER" is introduced, so that the parameters for the σ−operator can be listed, such as t1, t2, t3. The word "CLUSTER" indicates that objects belonging to the same cluster must share some common characteristics (such as having the same x coordinate value). A cluster may have a sub-structure specified in another (recursive) query. Clustering is a natural concept when dealing with spatial/temporal objects. The mechanism for clustering will be discussed further in Section 3. The result of a ΣQL query is a string that describes the relationships among the clusters. This string is called a cluster-string, which will also be discussed further in Section 3. A cluster is a collection of objects sharing some common characteristics. The SELECT- CLUSTER pair of keywords in ΣQL is a natural extension of the SELECT keyword in SQL. In fact, in SQL implicitly each attribute is considered as a different axis. The selection of the attributes’ axes defines the default clusters as those sharing common attribute values. As an example, the following ΣQL query is equivalent to an SQL query to select attributes’ axes "sname" and "status" from the suppliers in Paris. SELECT sname, status CLUSTER * FROM supplier WHERE city = "Paris" In the above ΣQL query, the * indicates any possible values for the dimensions sname and status. Since no clustering mechanism is indicated after the CLUSTER keyword the default clustering is assumed. Thus by adding the "CLUSTER *" clause, every SQL query can be expressed as a ΣQL query. Each cluster can be open (with objects inside visible) or closed (with objects inside not visible). The notation is t2o for an open cluster and t2c or simply no superscript for a closed cluster. In the ΣQL language the keyword "OPEN" is used: SELECT t CLUSTER t1 , OPEN t2 , t3 FROM R
22
Shi-Kuo Chang et al.
With the notation described above, it is quite easy to express a complex, recursive query. For example, to find the spatial/temporal relationship between objects having the same x coordinate values x1 or x2 from the three time slices of a source R, as illustrated in Figure 1, the ΣQL query in mathematical notation is: σx (x1 , x2)( σt (t1o, t2o, t3o ) R)
(1)
The query result is a cluster-string describing the spatial/temporal relationship between the objects 'a' and 'b'. How to express this spatial/temporal relationship depends upon the (spatial) data structure used. In the next section we explain Symbolic Projection as a means to express spatial/temporal relationships.
3 A General σ−Operator for σ−Queries As mentioned above, the ΣQL query language is based upon a single operator - the σ−operator - which utilizes Symbolic Projection to express the spatial/temporal relationships in query processing. In the following, Symbolic Projection, the cutting mechanism and the general σ−operator are explained, which together constitute the theoretical underpinnings of ΣQL. Symbolic Projection [7, 11] is a formalism where space is represented as a set of strings. Each string is a formal description of space or time, including all existing objects and their relative positions viewed along the corresponding coordinate axis of the string. This representation is qualitative because it mainly describes sequences of projected objects and their relative positions. We can use Symbolic Projection as a means for expressing the spatial/temporal relationships extracted by a spatial/temporal query. Continuing the example illustrated by Figure 1, for time slice Ct1 its x-projection using the Fundamental Symbolic Projection is: σx (x1 , x2 ) Ct1 = (u: Cx1,t1 < Cx2,t1)
(2)
and its y-projection is: σy(y1 , y2 ) Ct1 = (v: Cy1,t1 < Cy2,t1)
(3)
In the above example, a time slice is represented by a cluster Ct1 containing objects with the same time attribute value t1. A cluster-string is a string composed from cluster identifiers and relational operators. The single cluster Ct1 is considered a degenerated cluster-string. After the σy operator is applied, the resulting cluster Cy1,t1 contains objects with the same time and space attribute values. In the above example, the cluster-string (v: Cy1,t1 < Cy2,t1) has the optional parentheses and projection variable “v” to emphasize the direction of projection. The query σt(t1 , t2 , t3 ) R yields the following cluster-string α: α = (t: Ct1 < Ct2 < Ct3 )
(4)
When another operator is applied, it is applied to the clusters in a cluster-string. Thus the query σx (x1, x2) σt(t1o, t2o, t3o)R yields the following cluster-string β:
Querying Multimedia Data Sources and Databases
β = (t: (u: Cx1,t1 < Cx2,t1) < (u: Cx1,t2 < Cx2,t2) < (u: Cx1,t3 < Cx2,t3))
23
(5)
The above cluster-string β needs to be transformed so that the relationships among the objects become directly visible. This calls for the use of a materialization function MAT to map clusters to objects. Since Cx1,t1 = Cx1,t2 = Cx1,t3 = {a} and Cx2,t1 = Cx2,t2 = Cx2,t3 = {b}, the materialization MAT(β) of the above cluster-string yields: MAT(β) = (t: (u: a < b) < (u: a < b) < (u: a < b))
(6)
The query result in general depends upon the clustering that in turn depends upon the cutting mechanism. The cutting is an important part of Symbolic Projection because a cutting determines both how to project and also the relationships among the objects or partial objects in either side of the cutting line. Usually the cuttings are ordered lists that are made in accordance with the Fundamental Symbolic Projection. The cutting type, κ-type, determines which particular cutting mechanism should be applied in processing a particular σ−query. The general σ−operator is defined by the following expression where, in order to make different cutting mechanisms available, the cutting mechanism κ−type is explicitly included: σaxes, k-type σ-type (clusters)ϕ
= stype :
(7)
The general σ−operator is of the type σ−type and selects an axis or multiple axes, followed by a cutting mechanism of the type κ−type on (clusters)ϕ where ϕ is a predicate that objects in the clusters must satisfy. The σ−operator operates on a clusterstring that either describes a data source (e.g. data from a specified sensor) or is the result of another σ−operator. The result of the σ−operator is another cluster-string of type stype. Since the result of the σ−operator is always a cluster-string, a materialization operator MAT is needed to transform the cluster-string into real-world objects and their relationships for presentation to the user.
4 The ΣQL Query Language ΣQL is an extension of SQL to the case of multimedia sources. In fact, it is able to query seamlessly traditional relational databases and multimedia sources and their combination. The ΣQL query language operates on the extended multimedia static structure MSS which will be described in Section 6. A template of an ΣQL query is given below: SELECT dimension_list CLUSTER [cluster_type] [OPEN] cluster_val1, .., [OPEN] cluster_valn FROM source WHERE conditions PRESENT presentation_description which can be translated as follows: "Given a source (FROM source) and a list of dimensions (SELECT dimensions), select clusters (CLUSTER) corresponding to a list of
24
Shi-Kuo Chang et al.
projection values or variables ([OPEN] cluster_val1, ..) on the dimension axes using the default or a particular clustering mechanism ([cluster_type]). The clusters must satisfy a set of conditions (WHERE conditions) on the existing projection variables and/or on cluster contents if these are open ([OPEN]). The final result is presented according to a set of presentation specifications (PRESENT presentation_description)." Each σ−query can be expressed as an ΣQL query. For example, the σ−query σs,κ(s1, s2o, s3, .., sn)φ R can be translated as follows: SELECT s CLUSTER κ s1, OPEN s2, s3, .., sn FROM R WHERE φ
5 An Exmaple of Multi-Sensor Data Fusion In this section, ΣQL will be illustrated with a query that uses heterogeneous data from two different sensors -- a laser radar and a video. An example of a laser radar image is given in Figure 2. This image shows a parking lot with a large number of cars, which look like rectangles when viewed from the top. The only moving car in the image has a north-south orientation while all others have an east-west orientation. Laser radar images are characterized by being three-dimensional and having geometric properties, that is, each image point is represented by x-, y- and z-coordinate values. The particular laser radar used here is a product by SAAB Dynamics of Sweden, which is helicopter born and generates image elements from a laser beam that is split into short pulses by a rotating mirror. The laser pulses are transmitted to the ground, in a scanning movement, and when reflected back to the platform a receiver collects the returning pulses that are stored and analyzed. The result of the analysis is a sequence of points with a resolution of about 0.3 m. The video camera is carried by the helicopter as well and the two sensors are observing the same area. This means that most cars in the parking lot can be seen from both sensors. The moving car shown in two video frames in Figure 3 is encircled. Figure 4 shows two symbolic images corresponding to the two video frames in Figure 3. Almost identical projection strings can be generated from the laser radar image. Basically the query can be formulated as follows. Suppose we are interested in finding moving objects along a flight path. This can be done by analyzing only the video frames, but that may require too much computation time and the problem cannot be solved in real time. Laser radar images can, however, be used to recognize vehicles in real time, which has been shown by Jungert et al. in [9, 10]. However, it cannot be determined from the laser radar images whether the vehicles are moving. The solution is to analyze the laser radar image to first find existing vehicles, determine their positions in a second step, and then verify whether they are moving from a small number of video frames. Finally, in the fusion process, it can be determined which of the vehicles are moving. Subquery1: Are there any moving objects in the video sequence in [t1, t2]? Q1 = σmotion(moving)σtype(vehicle) σxy,interval_cutting(*) σt(To)T mod 10 = 0 and T>t1 and T
Querying Multimedia Data Sources and Databases
25
Subquery2: Are there any vehicles in the laser radar image in [t1, t2]? Q2 = σtype (vehicle) σxyz,interval_cutting(*)σt(To) T>t1 and T
Fig. 2. A laser radar image of a parking lot with a moving car (encircled)
Fig. 3. Two video frames showing a moving white vehicle (encircled) in the parking lot
Fig. 4. Symbolic images of two video frames of moving car and its close neighbors The subquery Q1 selects the video source and opens the video frames. The selection of video frames includes some conditions to specify which frames to accept and in what time interval. In this case we have chosen every tenth video frame within the interval [t1,t2]. In the next selection the σxy-operator is applied to the video frames using the interval cutting mechanism [5, 6]. This operator generates the (u,v)-strings from which the object types are determined by the σtype-operator in the selected frames. Finally the vehicles in motion are determined by the application of the motion operator. The motion string (m) is generated from the time projection string: t: (u: a0s < a 1s a 2s < a 0e < a 1e a 2e < a 3s a 4s a 5s < a 3e a 4e a 5e , v: a0s < a 0e < a 1s a 3s < a 1e a 3e < a 4s < a 4e < a 2s a 5s < a 2e a 5e) < (u: ... , v: ...) < … (u: a1s a 2s < a 1e a 2e < a 0s < a 0e < a 3s a 4s a 5s < a 3e a 4e a 5e , v: a0s < a 1s a 3s < a 1e a 3e < a 0e < a 4s < a 4e < a 2s a 5s < a 2e a 5e) < (u: ... , v: ...) < ...
26
Shi-Kuo Chang et al.
From this string the motion string is generated by the application of a σ-operator which generates a string similar to an implicit merge_or-operation, i.e.: m: t: (u: a0s < a 0e < a ´0s < a ´0e , v: a 0s < a 0e < a ´0s < a ´0e) The subquery Q2 returns first the (u,v)-strings for the time interval [t1,t2]. An intermediate result of the subquery thus becomes: u: a1s a 2s < a 1e a 2e < a 0s < a 0e < a 3s a 4s a 5s < a 3e a 4e a 5e , v: a1s a 3s < a 0s < a 1e a 3e < a 4s < a 4e < a 2s a 5s < a 0e < a 2e a 5e After applying the σxyz-operator, existing vehicles are detected by applying the σtypeoperator. The subqueries can now be fused. A fusion operator φ, called merge_and, is designed, which performs fusion of vehicle information with respect to equality of vehicle type and position (x,y) in time t. Its input comes from multiple data sources of equal type. All object types are consequently equal in both subqueries. Fusion is thus applied such that only those objects selected from the two subqueries that can be associated to each other, will remain in the motion object string that gives only a0 in response. The complete query now looks like: φxytmerge-and(*)(σmotion(moving)σtype(vehicle) σxy,interval_cutting(*)σt(To)T mod 10 = 0 and T>t1 and T
σmedia_sources (videoo)media_sources, σtype (vehicle) σxyz,interval_cutting(*)σt(To) T>t1 and T t1 AND T < t2, SELECT motion CLUSTER moving FROM SELECT type CLUSTER vehicle FROM SELECT x,y CLUSTER interval * FROM SELECT t CLUSTER OPEN (* ALIAS T) FROM SELECT media_sources CLUSTER OPEN video FROM media_sources WHERE T mod 10 = 0 AND T>t1 AND T
6 Multimedia Source Representation and ΣQL Query Execution In the previous sections we have described a data source as a simple projection string. However, in general, in order to describe data sources we need a more complex data structure. In this section we describe an extension of the MSS model proposed in [4] for the description of multimedia data. A multimedia source description is composed of a hierarchy of entities. Each entity has the format where the name is the entity identifier, the type is the entity type and each description is a triple ((d1..dm):{e1, e2, .., en}: rel_expr) with m ≥ 1 and n ≥ 0 containing (a) a list of dimensions di according to which the entity is being clustered; (b) a set of component entity identifiers resulting
Querying Multimedia Data Sources and Databases
27
from the clustering; and (c) a relational expression where relations (depending on the dimensions) are used to relate the component entities A description is legal if n = 0 or a clustering mechanism able to derive the description from the source is available. In the case of n=0, the entity is an atom with respect to the description dimension and the relational expression reduces to a simple value. Depending on the chosen description type (and consequently on the associated clustering mechanism) a source can be seen as a temporal sequence of entities, or a spatial disposition of entities, or as a set of attribute-value pairs, etc. As an example, let us consider a video clip segment showing two trees and two walking persons. The video clip is clustered according to the time dimension in three consecutive frames as shown in Figure 7. The video clip represents Cathy (c) and Bill (b) moving east, and two trees (a). The following entities constitute the MSS representation of the video clip: [R, video, (τ: {R1, R2, R3}: (t: R1 < R2 < R3))] [ R1, frame, (τ: {}: t1),((x, y): {a, b, c}: ((u: a=b< c < a), (v: a < b < a= c))) ] [ R2, frame, (τ: {}: t2),((x, y): {a, b, c}:((u: a < b < a=c),(v: a < b < a= c)] [ R3, frame, (τ: {}: t3), ((x, y): {a, b}: ((u: a < < a=b), (v: a < b < a ))) ] [ a, plant, (object: {}: {(name: tree)})] [ b, person, (object: {}: {(name: Bill)})] [ c, person, (object: {}: {(name: Cathy)})] t2
t1 a
R1 =
c
R2 =
b a
t3
aa
c
b
b
a
c
aa
c
t
a
R3 =
b a
Fig. 5. A video clip segment of three frames R1, R2, R3
The phases needed to process a ΣQL query are as follows: 1. Lexical analysis and syntactic analysis. 2. Transformational analysis to validate semantic correctness, where the following actions are taken: a) comfirm the compatibility between the dimensions and the sources in each sub-query; b) check if the overall query is consistent -- i.e., there exist representations for the intermediate results that make the query execution feasible -- and if so, keep track of the representations (intermediate result type inference); c) for each intermediate result, select a feasible representation (probably with the help from the user); d) if the main source is not structured, then build the MSS schema, else check if there exists a query-engine that supports the querying of the structured source. 3. Query optimization and query execution: If the main source is not structured then populate the MSS structure according to the schema defined during the semantic analysis by extracting the appropriate information from the source. (The semantic validation implies that the algorithms to extract the information from the source are available); execute the query against the MSS; present the results; else send the query to the database query-engine and present the results.
28
Shi-Kuo Chang et al.
7 Discussion As explained in previous sections, ΣQL can express both spatial and temporal constraints individually using the SELECT/CLUSTER construct and nested subqueries. Its limitation seems to be that constraints simultaneously involving space and time cannot be easily expressed, unless embedded in the WHERE clause. Although such constraints may be rare in practical applications, further investigation is needed in order to deal with such complex constraints. ΣQL can be applied directly to a video sequence such as the one shown in Figure 1, by visually selecting and clustering (decomposing) the 2D (or 3D, 4D, etc.) space. Applications to distance learning, remote sensing, etc. can be explored. Another important application of ΣQL, as suggested in Section 6, is to facilitate user-system interaction in selecting feasible representations in the transformational analysis of a σ−query for visual reasoning. References 1. Ahanger, G., Benson, D. and Little, T.D., "Video Query Formulation", Proc.s of Storage and Retrieval for Images and Video Databases II, San Jose, February 1995, SPIE, pp. 280-291. 2. Chan, E. P. F. and Zhu, R., "QL/G - A query language for geometric data bases", Proc. of the 1st Int. Conf. on GIS in Urban Regional and Env. Planning, Samos, Greece, April. 3. Chang, S. F., Chen, W., Meng, H. J., Sundaram, H., and Zhong, D., "VideoQ: An automated Content Based Video Search System using Visual Cues", Proceedings of the Fifth ACM International Multimedia Conference, November 1997. 4. Chang, S. K., "Content-Based Access to Multimedia Information", Proceedings of Aizu International Student Forum-Contest on Multimedia, (N. Mirenkov and A. Vazhenin. eds.), The University of Aizu, Aizu, Japan, Jul 20-24, 1998, pp. 2-41. (Available at www.cs.pitt.edu/Óchang/365/cbam7.html) 5. Chang, S.-K. and Jungert, E., Symbolic Projection for Image Information Retrieval and Spatial Reasoning, Academic Press, 1996. 6. Chang, S.-K. and Jungert E. Pictorial data management based upon the theory of Symbolic Projection, Journal of Visual Languages and Computing, vol. 2, no 3, 1990, pp 195-215. 7. Chang, S.-K., Shi, O. Y. and Yan, C. W., Iconic indexing by 2D strings, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 9, No. 3, pp. 413-428, 1987. 8. Egenhofer, M., "Spatial SQL: A Query and Presentation Language", IEEE Transactions on Knowledge and Data Engineering, Vol. 5, No. 2, 1991, pp. 161-174. 9. Holden, R., "Digital's DB Integrator: a commercial multi-database management system", Proc. of 3rd Int.l Conf.e on Parallel and Distributed Information Systems, Austin, TX, USA, Sept. 28-30 1994, IEEE Comput. Soc. Press, Los Alamitos, CA, USA, pp. 267-268. 10. Jungert, E., Carlsson, C., and Leuhusen, C.,"A Qualitative Matching Technique for Handling Uncertainties in Laser Radar Images", Proceeding of the SPIE conference on Automatic Target Recognition VIII, Orlando, Florida, April 13-17, 1998, pp 62-71 11. Lee, S.-Y. and Hsu, F.-S., Spatial reasoning and similarity retrieval of images using 2D Cstring knowledge representation, Pattern Recognition, vol. 25, 1992, pp 305-318. 12. Lin, C. C., Chang, S. K. and Xiang, J. X., Transformation and Exchange of Multimedia Objects in Distributed Multimedia Systems, ACM Journal of Multimedia Systems, Springer-Verlag, Vol. 4, Issue 1, 1996, 12-29. 13. Li, J. Z., Ozsu, M. T., Szafron, D. and Oria, V., "MOQL: A multimedia Object Query Language", Proc. 3rd Int. Worksh on Multim Info Systems, Como, Italy, Sept -97, pp 19-28. 14. Oomoto E. and Tanaka, K., Video Database Systems - Recent Trends in Research and Development Activities, in Handbook of Multimedia Information Management, (Grosky, W. I., Jain, R. and Mehrotra, R., eds.), Prentice Hall, 1997, pp 405-448. 15. Roussopoulos, N., Faloutsos, C. and Sellis, T., "An Efficient Pictorial Database System for PSQL", IEEE Trans. on Software Engineering, Vol. 14, No. 5, May 1988, pp. 639-650. 16. E. Waltz and J. Llinas, "Multisensor data fusion", Artect House, Boston, 1990.
General Image Database Model Peter L. Stanchev Institute of Mathematics and Computer Science, Bulgarian Academy of Sciences Acad. G. Bonchev St. 8, 1113 Sofia, Bulgaria, [email protected]
Abstract. In this paper we propose a new General Image DataBase
(GIDB) model. The model establishes taxonomy based on the systematisation of existing approaches. The GIDB model is based on the General Image Data model [1] and General Image Retrieval model [2]. The GIDB model uses the powerful features offered by object-oriented modelling, the elegance of the relational databases, the state of art of computer vision and the current methods for knowledge representation and management to achieve effective image retrieval. The developed language for the model is a hybrid between interactive and descriptive query languages. The ideas of the model can be used in the design of image retrieval libraries for an object-oriented database. As an illustration the results of applying the GIDB model to a plant database in the Sofia Image Database Management System are presented.
1 Introduction The image databases are becoming an important element of the emerging information technologies. They have been used in an a wide variety of applications such as: geographical information systems, computer-aided design and manufacturing systems, multimedia libraries, medical image management systems, automated catalogues in museums, biology, geology, mineralogy, astronomy, botany, house furnishing design, anatomy, criminal identification, etc. As well they are becoming an essential part of most multimedia databases. The first survey for image databases appeared in the early 1980 by Tamura and Yokoya [3]. They classify the image database systems into three categories: convention databases, conventional databases with extended function for image processing, and specialised systems designed for a particular application domain. On the other hand Grosky and Mehrota [4] classify the image databases into three categories: systems using relational databases, system based on object-oriented model and systems for image interpretation. Grosky elaborated the ideas of image databases to multimedia databases [5]. There are mainly five approaches towards image database system architecture:
Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 29-36, 1999. Springer-Verlag Berlin Heidelberg 1999
30
Peter L. Stanchev
(1) Conventional database system as an image database system. The use of a conventional database system as an image database system is based mainly on relational data models and rarely on hierarchical. The images are indexed as a set of attributes. At the time of the query, instead of retrieving by asking for information straight from the images, the information is extracted from previously calculated image attributes. Languages such as Structured Query Language (SQL) and Query By Example (QBE) with modifications such as Query by Pictorial Example (QPE) are common for such systems. This type of retrieval is referred as attribute based image retrieval. A representative prototype system from this class of systems is the system GRIM_DBMS [6]. (2) Image processing/graphical systems with database functionality. In these systems topological, vector and graphical representations of the images are stored in the database. The query is usually based on a command-based language. A representative of this model is the research system SAND [7]. (3) Extended/extensible conventional database system to an image database system. The systems in this class are extensions over the relational data model to overcome the imposed limitations, by the flat tabular structure of the relational databases. The retrieval strategy is the same as in the conventional database system. One of the research systems in this direction is the system GIS [8]. (4) Adaptive image database system. The framework of such a system is a flexible query specification interface to account for the different interpretations of images. An attempt for defining such kind of systems is made in [9]. (5) Miscellaneous systems/approaches. Various other approaches are used for building image databases such as: grammar based, 2-D string based, entity-attributerelationship semantic network approach, matching algorithms, etc. In this paper a new General Image DataBase (GIDB) model is presented. It includes descriptions of: (1) an image database system; (2) generic image database architecture; (3) image definition, storage and manipulation languages.
2 The GIDB Model Description This section we start with some definitions. Definition 1. A Data Model is a type of data abstraction that is used to provide the data conceptual representation. It is a set of concepts that can be used to describe the structure of a database. By the structure of a database we mean the data types, relationships, and constraints that should hold on the data. It can also include a set of operations for database retrieval and update [10]. Definition 2. An Image Database (IDB) is a logically coherent collection of images with some inherent meaning. The images usually belong to a specific application domain. An IDB is designed, built, and populated with images for specifics purpose and represents some aspects of the real world. Definition 3. An Image Database Management System (IDBMS) is a collection of programs that enable the user to define, construct and manipulate an IDB for various applications. An image definition involves specifying the characteristics of
General Image Database Model
31
the application domain, the image indexing mechanism, the image-object recognition mechanism, and the information about the images that will be extracted and stored together with the images. An image database constructing is the process of storing the images themselves on some storage media, together with the logical image description. An image database manipulation includes functions such as querying the database for a retrieval of a specific image and updating the image database to reflect changes of the images in the real word. As well the user could create his own set of programs and bind them into an image database system. Definition 4. An Image Database System (IDBS) is constituted from IDB and IDBMS. The main differences from a conventional database system environment are: (1) the existence of tools for image databases definition, including tools for image indexing and image-object recognition and (2) existence of image processing procedures. Definition 5. The way that the users think about data is called external view level, the way that the data are recognised by the database system is called internal or physical level, and the middle layer is called conceptual level. 2.1 The Generic Architecture of an Image Database The architecture of a generic image database system is given in Fig. 1. Three phases for interactions with the system are provided: domain definition, image entering and image retrieval. In order to introduce new application areas for the system the administrator uses the domain definition phase. At the second phase the images are entered into the system. The third phase is image retrieval. In it the end-users use the system for posing queries and viewing the image features. PHASE
INPUT
PROCESS
RESULT
1. Domain definition a. logical description
Logical Image Definition Language (LIDL)
LIDL processor
Procedure for image indexing
b. physical description
Physical Image Definition Language (PIDL)
PIDL processor
Procedure for physical image storage
2. Image entering a. input the image and image information
images
Image Storage Language (ISL) processor
Logical and physical IDB
b. image updating
ISL updating tools
ISL processor
Logical and physical IDB
c. image deletion
ISL deletion tools
ISL processor
Logical and physical IDB
3. Image retrieval a. image display
Image Manipulation Language-IML
b. logical image display
IML
Query processor
Query processor & Statistical processor
Fig. 1. Generic architecture of an IDBS
Images Semantic data Statistical data
32
Peter L. Stanchev
2.2 Image Data Model Description The proposed Image Data model establishes taxonomy based on the systematisation of the existing approaches. The proposed approach for the image modelling includes: • using language approach, where language structures are used for physical and logical image content description; • using object oriented approach, where the image and the image objects are treated as objects containing appropriate functions calculating its functions. The data model is object oriented. The image itself together with its semantic descriptions is treated as an object in terms of the object oriented approach. The image is presented in two layouts (classes) - logical and physical. A semantic schema of the proposed model is shown in Fig. 2. Image R Logical view Global view Meta attributes
Content-based view
Semantic attributes Objects
Physical view R R
Modelbased view = = Relations
Image header
Image matrix
General purpose view Colour
Texture
Topological Vector Metric
Spatial
Colour Texture Shape Logical Semantic attributes attributes Legend: is-an-abstraction-of (multi-valued) is-an-abstraction-of (one-to-one) = domain dependant R - required
Fig. 2. Semantic schema of the GID model
2.3 Image Retrieval The retrieval model is unique in the sense of its comprehensive coverage of the image features. The main characteristics of the proposed model could be summarised as follows: a.) The images are searched by their general image description model representation [1];
General Image Database Model
33
b.) The model is based on similarity retrieval. Let a query be converted through the general image data model in an image description Q(q1, q2, …, qn) and an image in the image database has the description I(x1, x2, …, xn). Then the retrieval value (RV) between Q and I is defined as: RVQ(I) = Σi = 1, …,n (wi * sim(qi, xi)), where wi (i = 1,2, …, n) is the weight specifying the importance of the ith parameter in the image description and sim(qi, xi) is the similarity between the ith parameter of the query image and database image and is calculated in different way according to the qi, and xi values. They can be: symbol, numerical or linguistic values, histograms, attribute relational graphs, pictures or spatial representations characters. 2.4 IDBS Languages We try to develop the IDBS languages for the GIDB model following an analogy with the conventional database systems languages. Those languages can be divided into three classes: image definition, image storage and image retrieval languages. 2.4.1 Image Definition Language The image definition language consists of two parts: the Logical Image Definition Language (LIDL) and the Physical Image Definition Language (PIDL). The Logical Image Definition Language. One physical image has different logical interpretations. The Global Image Data Model is used to create a logical representation of a physical image. The process of the logical representation is described in Fig. 3. Functions for similarity calculations have been included in the chosen methods. STEP
FORMAT
PROCESS
1.Entering schema
(IDB name, entering media, file format)
reading from outside source into the memory
2. Editing schema
(general parameters, method1, method2, ..., methode)
manipulation, transform, spatial filter, histograms, and morphological filter
3. Global view obtaining
(meta attributes = name1: type1, name2: type2, …, namema: typema; semantic attributes = name1: type1, name2: type2, …, namesa: typesa)
procedure for meta and semantic attribute definition
4. General purpose view obtaining
(colour = method1, method2, ..., methodk1, texture = method1, method2, ..., methodk2,)
procedure for colour and texture definition
5. Segmentation & object definition
(method1, method2, ..., methods)
procedure for image segmentation and object definition
6. Relation definition schema
(method1, method2, ..., methodr)
procedure for relation definition
Fig. 3. The steps in the Logical Image Definition Language
34
Peter L. Stanchev
Physical Image Definition Language. The functions for physical image storage are given in Fig. 4. STEP
FORMAT
PROCESS
1. Physical image storage schema
(method1, method2, ..., method n1)
procedure for physical image storage
2. Logical image storage schema
(method1, method2, ..., method n2)
procedure for logical image storage
3. Indexing mechanism schema
(method1, method2, ..., method n3)
procedure for indexing creation
4. „Thumbnail“ image storage schema
(method1, method2, ..., method n4)
procedure for „thumbnail“ image storage
Fig. 4. The steps in the Physical Image Definition Language
2.4.2 Image Storage Language The ISL contains three parts: (1) image entering language, (2) image updating language and (3) image deletion language. Image updating and deletion are seldom used and are not typical for image databases. For all these operations a specific interactive environment has to be created. Image processing and measurement functions are available in this language to assist the user. 2.4.3 Image Manipulation Language The Image Manipulation Language includes retrieval by attribute value, shape, colour, texture, example image or spatial constrain. The query is translated to a GID model representation and then the GIR model is used to retrieve the desired images. The retrieval method is described in more details in [2].
3 An Example for Applying the GIDB Model The GIDB model manipulation capabilities are illustrated on drawings and pictures of plan image database realised in the Sofia Image Database Management System. 3.1 Image Definition Language The first level of interaction is the domain definition. The definition of the image application area is given in Fig. 5.
General Image Database Model
35
Fig 5. An example of the Logical Image Definition Language
3.2 Image Storage Language and Image Manipulation Language Let a plant image be entered in the image database. The GID description and the image itself are stored in the image database. The Image Manipulation language is described in [2]. An example for a query result is given in Fig. 6.
Fig 6. An example for a query result
36
Peter L. Stanchev
4 Conclusions The main advantages of the proposed model could be summarised as follows: (1) Its generality. The image representation is done trough the general image data model. The image retrieval is based on the general image retrieval model. Therefore the model is applicable to a wide variety of image collections. (2) Its practical applicability. There are numerous methods for the decomposition of the image into objects and for image indexing. According to the application domain the appropriate method could be used. (3) Its flexibility. The model could be customised when used with a specific application. The presented GIDBS model could be extended for distributed IDBS and multimedia database containing text, video and speech signals. At present software realisation of the model for Windows NT is considered in the Sofia Image Database Management System.
Acknowledgement This work is partially supported by a project VRP I 1/99 of the National Foundation for Science Research of Bulgaria and European Commission INCO Copernicus project INTELLECT (PL961099).
References 1.
Stanchev, P.: General Image Data Model. In: 22 intentional conference: Information Technologies and Programming, Sofia (1997) 130-140 2. Stanchev P.: General Image Retrieval Model, In: Proceeding of the Conference of the Union of Bulgarian Math, Pleven (1998) 63-71 3. Tamura, H. Yokoya N.: Image Database Systems: a Survey. Pattern Recognition, Vol. 17, No. 1 (1984) 29-43 4. Grosky W., Mehrotra R.: Image Database Management. IEEE Computer, 22 (1989) 7-8 5. Grosky W.: Multimedia Information Systems. IEEE MultiMedia, Spring (1994) 12-23 6. Stanchev P., Rabitti F.: GRIM_DBMS: a GRaphical IMage DataBase Management System. In: Kunii, T.: Visual Database Systems, North-Holland (1989) 415-430 7. Aref, W., Samet H.: Extending a DBMS with Spatial Operations. In: Second Symposium on Large Spatial Databases, Zurich, Switzerland, ETH Zurich (1991) 299-318 8. Stanchev P., Smeulders A., Groen F.: Retrieval from a Geographical Information System. In: Computing Science in the Netherlands, Amsterdam, The Netherlands, Stichting Mathematish Centrum (1991) 528-539 9. Gudivada V., Raghavan V.: Picture Retrieval Systems: A Unified Perspective and Research Issues. Tech Report, CS-95-03 (1995) 10. Elmasri R., Navathe S.: Fundamentals of Database Systems. The Benjamin/Cummings Publishing Company (1994)
System for Medical Image Retrieval The MIMS Model Richard Chbeir, Youssef Amghar, and Andre Flory LISI - INSA 20 Avenue A. Einstein, F-69621 Villeurbanne - France Tel: (+33) 4 72438595, Fax: (+33) 4 72438597 {rchbeir,amghar,flory}@lisiflory.insa-lyon.fr
Abstract. The multifaceted description of image data presents a couple of problems for traditional information systems designed for textual data. Image databases require efficient and direct spatial search based on image objects and their relationships instead of some cumbersome alphanumeric encodings of the images. Current systems generally approach the problem from different views. Thus, they describe one image facet dependently on application domain. This paper presents a model for Medical Image Management System (MIMS) that allows physicians to retrieve images and required information by combining the strengths of several approaches. Via a convivial iconic interface, our system assign to each image object a graphical representation to describe image objects and their attributes. It allows also to automatically calculate their interrelations. Key-Words : Database management, image data, image indexing and interrogation, multimedia, hypermedia, thesaurus, spatial relation.
1
Introduction
The use of multimedia information in the areas of defense and civilian satellites, spatial data, CAD data, home entertainment and medical systems has witnessed its utility. Manipulating these medias such as text, graphics, image and voice requires an adequate and special management in terms of storage, transmission and manipulation. Traditional systems have been conceived to deal with textual documents; they are not adapted for handling complex objects such as ones of multimedia applications. Current efforts have been focused on retrieving medias using only traditional artificial keys, such as identification number [6]. This obviously limits the consulting power for any domain research. Several approaches have been proposed to solve the induced problem but each one of them tries to manipulate one or more media aspects such as graphical ones ([4]) or semantic one ([2,1,7]). This paper presents a model for medical information system, using medical image as media, whose main objective has been to conceive a general approach and optimize the retrieval process.
Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 37–42, 1999. c Springer-Verlag Berlin Heidelberg 1999
38
Richard Chbeir et al.
In the following section, the description of our data model follows a more detailed description for image representation problem and the medical information problem. Section 3 concludes our actual work and drafts our future one.
2
Medical Image Description
Image description is so person-depended. It differs from person to another and is related to context placed in. Several parameters are taken into consideration: image objects, image context, person culture, application domain, etc. It partially reflects the difficulties encountered in each of indexing and/or retrieving process. For our concern, medical domain is particular and requires various parameters. An example of medical searches would make easier our clarification in this section: ”Select all images for patients older than 50 years, having tumor on lungs”. It’s obvious that one image facet description is so restrictive and unable to satisfy physicians’ needs. In fact, search criterias in medical information system lay not only on image context (patients older than 50 years), but also on semantic content (tumor on lungs). So, it’s primordial to mix approaches to have hybrid system in order to satisfy medical desires. Figure 1 describes the general conceptual schema explained in next sessions. 2.1
Image Context
The medical history of each patient is represented by an Electronic Medical Folder (EMF) [5]. To verify his diagnostic, the physician needs to retrieve all images of a specific patient. ”Select all images of Mr.Robin” could be an obvious example. Thus, each medical image is related to a specific domain (radiology, dermatology, etc.). In addition, we take into consideration several image attributes such as: incidence (axial, coronal, saggital, etc.), type (JPEG, GIF, etc.), device (scanner, MRI, radio, etc.), date. 2.2
Image Description
To specify the medical content of the image, three kinds of semantic objects must be described: – Anatomic Organ (AO 1 ): corresponds to the medical organ that the image presents such as brain, hand, etc. – Pathologic Sign (PS) : identified and detected by physician, it represents a medical anomaly. For instance tumor, fracture, etc. – Anatomic Region (AR): presents the internal structure of AO. For example, the left ventricle, the right lobe, etc. Each medical image corresponds to an AO in which a set of anomalies are positioned on anatomic regions. Each anatomic region and anomaly are represented by a set of descriptors in order to outline their states. A descriptor may be a text, numeric or date. A dangerous tumor, a pumped up lung and a dehydrated lobe could be some examples of what it’s called medical states. 1
These abbreviations will be used later.
System for Medical Image Retrieval: The MIMS Model
39
Fig. 1. The general conceptual schema 2.3 Incertitude Description Incertitude is inherent to image because of description process, which might be automatic or manual. It can be situated at all description levels (AO, AR and PS). Because semantic aspect is so important in medical information system, it’s important to take into consideration such possibility. It’s the case, for example, when the physician is not sure of his diagnostic and hesitates between two anomalies. Our approach provides the possibility of describing each noncertain anomaly by one or several PSs associated to an incertitude degree for each. The incertitude degree is a decimal number majored by the interval [0,1]. Whenever the physician is sure of anomaly, the incertitude degree is evaluated as 1. 2.4
The Thesaurus
This component supports all domain-specific information that must be maintained for each specific application. It represents a dictionary in which terms
40
Richard Chbeir et al.
are interrelated. As we mentioned before, the medical image represents a set of Pathologic Signs (PS) in Anatomic Regions (AR) of the concerned Anatomic Organ (AO). We propose a hierarchical thesaurus 2 built upon these concepts (PS, AR and AO) to normalize indexing process and make more coherent, easier and efficiency both indexing and querying processes. The use of the thesaurus offers the opportunity to combine the two traditional methods: the keywords and the legend methods. 1. The thesaurus relations : The thesaurus terms, in MIMS, are interrelated by a couple of semantic relations: – the synonymy relation: relates a non-descriptor term 3 to a descriptor one. The medical vocabulary is so complex that some equivalent concepts may be different from domain to another. For instance, a tumor is a synonym to a cancer. – The hierarchy relation: defines partial order relation between descriptors in order to classify medical terms and extend queries. It groups generalization / specialization relation (anomaly and tumor) and/or composition relation (head and brain). – The compatibility relation: connects AR to PS. This relation is used essentially to sort PS with regard to compatible AR. For instance, at the left ventricle, only tumor could be found. 2. Thesaurus graphical representation: As it’s mentioned in the first section, it’s so important, when dealing with non-computer specialist, to use convivial interfaces. A graphical iconic interface is proposed in this framework to index and query the medical image database, and to manipulate thesaurus components (Figure 2). – Each AO is attached to a pattern called Organ that physician has to choose in the beginning to present the analyzed image. – The internal structure of the selected organ presenting anatomic regions is presented by a set of polygons. – Each anomaly (or PS) is attached to an icon that could be placed by physician to indicate the anomaly location. 2.5
Spatial Relations
To precise the position of anomaly in anatomic region, the notion of spatial relations is utilized. Euclidean space is used to calculate different spatial relations in both directional and topologic space. We exclude metric space for two reasons: it has to be done manually, and risks to be inappropriate in certain medical domains where measuring variability becomes important [3]. Only validated relations are stored in database and not icon coordinates. This means that scale modification can be done easily because it affects only coordinates and not relations. Two kinds of spatial relations are identified: spatial relations between AR and its PS (AR/PS), and spatial relations between PS of one AR (PS/PS): 2 3
The thesaurus is a knowledge base used to help physicians to formulate their queries. A term semantically near to another one.
System for Medical Image Retrieval: The MIMS Model
41
Fig. 2. The MIMS Indexing Interface
1. AR/PS relations : Concerning the topologic space, only inclusion relation (”IN”) is considered. The creation of such relation is implicitly attached to the creation of AR/PS descriptor whenever the user puts an icon inside the AR. On the other hand, it’s also significant to precise the anomaly direction inside the damaged region in order to locate it. Via barycenters comparison of both damaged region and anomaly, we can locate the direction of the last one. Several eventualities are then possible: left, right, low-right, high, left-high, etc. For that reason, only four relations have been taken into account: ”high, low, left, right” but the possibility that each anomaly may have two directional relations with its damaged region is always maintained. 2. PS/PS relations : Four topologic relations are taken into consideration in our approach: touch, cover, disjoint, mix. To calculate these topologic relations, the rectangular form of each icon associated to PS is utilized. Concerning the directional relations between anomalies, there are only two: High and Left. In fact, equivalence between directional relations allow us to neglect the implicit ones, i.e. if X is higher than Y, then Y is lower than X and vice versa. The main element to achieve this task is always the barycenter.
42
3
Richard Chbeir et al.
Conclusion
This paper describes the current status of our project on medical image management system. A new data model has been introduced, able to use image features and represents all medical image objects aspects. Another contribution exists, it’s the independence aspect between model components that allows us to use our approach in several domains only by changing the knowledge module. Our actual occupation is the amelioration of our model that integrates the knowledge base through an hypermedia thesaurus, a very convivial interface where the user is able to formulate his query in a very flexible manner and access image databases via the Web. The image content evolution seems to be an attractive major. In fact, the patient images describe the medical curriculum evolution . To carry out his diagnostic, the physician needs sometimes to know the chronological sequence of images. Maturity development, disease progression, medical therapeutic, traumatic event and other examples need to integrate the image content evolution in database. Our future work will be devoted to this light in order to provide the possibility in consulting database using medical image content evolution. Also, we aim to construct an online system able to guide the physician during his insertion and interrogation processes. In fact, the construction of another intelligent database based on images description will be explored. The implementation of our approach in several other domains looks like obvious. To prove our believes, we envisage our approach integration in pictorial databases in order to demonstrate the independence aspect of such approach.
References 1. Abad-Mota S., Kulikowski C., Semantic Queries on Image Databases : The IMTKAS model, Proc. Of the Basque international Workshop on Information, BIWIT 95, IEEE, Computer Society Press, P. 20-28. 37 2. Ashley J., Flickner M., Hafner J., Lee D., Niblack W., Petkovic D, The Query By Image Content System. Proceedings Of the 1995 ACM SIGMOD International Conference on management of data, SIGMOD 95, P. 475. 37 3. Clementini E., Di Felice P., Van Oostrom P., A small set of formal topological relationships suitable for end-user interaction, Lecture notes in computer science, 692, Advances in Spatial Databases. Proceedings Of the 3rd international symposium, SSD93, Singapore, Juin 1993, P. 277-295. 40 4. Jeffrey R. Bach, Santanu P., Ramesh J., A Visual Information Management System for the Interactive Retrieval of Faces, IEEE transactions on Knowledge and data engineering, Vol. 5, N 4, August 1993. 37 5. Laforest L., Frnot S., Flory A., A new approach for Hypermedia Medical Records Management, 13th International Congress Medical Informatics Europe (MIE’96), Copenhagen, 1996. 38 6. Mascarini Ch., Ratib O., Trayser G., Ligier Y., Appel R.D. In-house access to PACS images and related data through World Wide Web, European Journal of Radiology, Vol. 22, 1996, P. 218-220. 37 7. Mourad Mechkour EMIR2. An Extended Model for Image Representation and Retrieval., in DEXA’95. Database and EXpert system Applications, London, September, pp395-404, 1995. ISBN 3-540-60303-4. 37
An Agent-Based Visualisation Architecture Jonathan Meddes and Eric McKenzie School of Computer Science, Division of Informatics The University of Edinburgh, James Clerk Maxwell Building The King’s Buildings, Mayfield Road, Edinburgh EH9 3JZ Tel: +44 131 650 5129 Fax: +44 131 667 7209 {jmx,ram}@dcs.ed.ac.uk
Abstract. Advances in storage technology have led to a dramatic increase in the volume of data stored on computer systems. Using a generic visualisation framework, we have developed a data classification scheme that creates an object-orientated data model. This data model is used in an agent-based visualisation architecture to address the problems of data management and visual representations. As part of an ongoing research project we have implemented DIME (Distributed Information in a Multi-agent Environment), a visualisation system based on this approach.
1
Introduction
Computer science is catching up with visualisation, and there is a substantial motivation for computer science to make the effort. We strive to make data more interpretable via a visual representation but visualisation did not start with computers. Bertin’s seminal work with graphical representations [1] is still a widely acknowledged guide to good visualisation practice and predates the widespread introduction of bitmapped displays. The increased storage capacity of computer systems has allowed application developers to retain more detailed data. Managers have realised that they can use this data to have more control of their organisation. From a managers perspective, this has the potential to be an information revolution, and can satisfy their need for enhanced control. Unfortunately, this revolution cannot take place until the information is readily accessible. An effective visualisation uses computer graphics that allows the user to understand data. This paper presents an architecture for a novel approach to visualisation and provides an extensible visualisation system with which to develop our ideas. We have based our research around a generic visualisation framework in which we describe a data classification scheme used to create a data model that accurately classifies the data. The organisation of the data model directly influences the internal operation of our visualisation architecture which is developed using a technique allowing distributed communication orientated visualisation. Finally, we describe the implementation of the visualisation system.
This research is supported by BT Laboratories.
Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 43–50, 1999. c Springer-Verlag Berlin Heidelberg 1999
44
2
Jonathan Meddes and Eric McKenzie
Data Classification
Data classification is the process of identifying data items, structures and relationships within a data series. The result is a data model that can be used by the remaining stages of the visualisation pipeline. Automated systems including SAGE [7] and ANDD [6] emphasised that automated visualisation can only be successful if the system understands the data. The accurate classification of data and relationships using a suitably expressive scheme is crucial to a visualisation system capable of creating effective visualisations. Anomalies introduced at this stage are propagated down the visualisation pipeline; this has the potential to introduce spurious visual cues into a visual representation. Before we describe the main features of our data classification scheme, we shall identify the desirable properties it should exhibit. The data classification scheme attempts to create a model that captures the syntax and semantics of the data. The syntax of data is its raw features that define its structural properties, whereas the semantics of data is the implied meaning projected by the organisation of individual data items. The data classification scheme has two goals: (1) consistently encode the data using a set of well-defined data types so that subsequent stages of the pipeline receive the data model in a familiar format; and (2) capture relationships between data items and allow easy extensions to the model by introducing additional data items and relationships. For illustration purposes, we present a subset of our full data classification scheme that is specifically intended for the classification of geographic and topological based data. An application that generates this type of data is the analysis of wide-area-network traffic; cities have a geographic location, network links have a topological structure and traffic can be measured using a quantitative value. The data model created by our scheme uses an object-orientated paradigm; this is a convenient method of organising the data that simplifies extending the data classes. Every data item is represented in our data model by a data object that is an instantiation of a data class. The basic data classes that are available to the data model are nominal, quantitative, spatial and relational. In a nominal data class, each member is a data object that represents a textual item, e.g., name. A quantitative class contains members representing numerical values, e.g., age. Constrained subclasses provide support for more specialist values such as currency and percentage. The spatial data class has members that represent a location in space. In its basic form the spatial data class is abstract and cannot be instantiated, but the creation of subclasses provides support for two-dimensional, three-dimensional and longitudinal/latitudinal co-ordinates. The relational data class provides the data model with a versatile method of representing relationships. Instantiation of this class creates a data object containing references to its associated objects; the data class also contains references to all the classes of objects which are represented in its relations. Subclasses such as the Complex Data Class (CDC) provide increased specialisation by restricting the types of relations which can be made. The CDC defines a signature consisting of a collection of data classes. Each instantiation creates a Complex
An Agent-Based Visualisation Architecture
45
CDC Domain
Classes Objects
Data item
CDO CDO Tuple CDO
CDO
(a)
(b)
Fig. 1. (a) relational table and (b) its data model Data Object (CDO) and must have exactly the same combination of data classes defining their relationships. Using these basic data classes, a data model that accurately represents the input data is constructed using the data classification scheme. Figure 1 shows an external relational database table (a) translated into the classes and objects of the internal data model (b). In the table, every cell contains a data item, every item from a column is from a similar domain and the data items from a row constitute a tuple. In the data model, a data class captures the domain which is instantiated to describe its data items. A CDC is defined to represent the association between columns in the table. Instantiations create a CDO that captures the association between the individual data items of a tuple. The data contained in the data model is merely a collection of data values and relations. In the absence of further detail describing the contents of the data model, it is meaningless as a self-contained information source. Data is transferred into information by the introduction of context. Typically, context is introduced by the user when they extract data from a model by introducing environmental influences from their domain knowledge. To move the data model from a simple collection of data values and relationships we introduce domain functions to provide context and replicate the environmental influences that dictate how a user changes data to information. In the data model, a domain function provides a basic service that can manipulate the data by mirroring the simple processing and reasoning a user performs when they interpret some data. For example, if we consider the use and capacity of a network link, a typical user of that data can deduce that the average load of that link is represented by use/capacity; a domain function would introduce new load objects to the data model.
3
Visualisation Architecture
The visualisation framework allows us to develop visualisation ideas in an environment where we are not constrained by a specific visualisation architecture.
46
Jonathan Meddes and Eric McKenzie
If we want to create tangible visual representations, we require a visualisation architecture to realise the visualisation system. In the remainder of this paper, we describe a visualisation architecture that demonstrates a novel approach to implementing the visualisation pipeline. A visualisation architecture must address (1) how data is stored or represented within the system, (2) how visual representations of the data are created, and (3) how the operational requirements of (1) and (2) are co-ordinated. The visualisation architecture we have adopted is an agent based-system inspired by behaviours observed in biological systems. 3.1
Principles of an Agent Based Architecture
The simplest way of illustrating the underlying principles behind an agent-based system is to use a biological example. If we consider a room with no windows or doors, the room is essentially a box. Although the room is sparse, it does contain some objects, a telephone, a chair, a blackboard and a stick of chalk. The room is populated by five people; each with a personality that characterises their behaviour. For example, one could posses leadership qualities that leads them to command others, or have organisational skills that makes them request actions from others, or might pace around the room. The person exhibiting the latter behaviour is physically constrained by the boundaries of the room. If they do not rely on the walls to restrict their movement, a cognitive process must be present that not only makes them move but also controls their movement. The inhabitants of the room can communicate using a commonly understood language. The person making the utterance could decide to communicate with another individual, a group of people (e.g., all males or some other commonly accepted group), or broadcast their utterance to everyone. This communication could transfer information, issue instructions (or orders) or ask and respond to questions. It would be fair to assume the people in the room would observe Grice’s maxims of communication [4]. The adoption of these rules means that the people in the room communicate effectively and will not attempt to mislead one another, maliciously or otherwise. A person’s environment is defined by the room and the objects within it. The room is a static boundary that restricts the movement of people and objects. The chair is a passive object that may be used by the people in the room but offers the bear minimum of interaction. Similarly, the blackboard provides a service to the people in the room, but could also be used for storage or as a device to communicate with other people. The telephone is an interesting object; it provides a service allowing people to communicate outside their immediate environment and can affect the people within the environment when it receives an incoming call. In this case the telephone is no longer a passive object that provides a service, it can cause the people in the room to change their behaviour. A simple example will provide an illustration of the types of communication that could take place in the room. It demonstrates the principle of co-operation that is essential for social groups to achieve a common goal. In this example, we want the names of the people in the room to be written on the blackboard. The
An Agent-Based Visualisation Architecture
47
task is initiated by a telephone call that is answered by a person who knows how to use a receiver; the caller issues an unambiguous instruction to “write every persons name on the blackboard”. The person who answered the call is now aware of the task that must be achieved and they take a controlling position in the environment. Luckily, the person who answers the call also has the ability to write on the blackboard and broadcasts a message to all the other people in the room asking them to reply with their name. One by one, the other people in the room reply to the question and as the answers are received they are written on the blackboard. Finally, the task is completed by adding their own name to the bottom of the list. This example demonstrates that some tasks can only be achieved by interaction and co-operation. This distributed computation, co-operation and communication which is evident in the biological example above demonstrates exactly the principles which underpin an agent-based system. In such a system, every person and object is represented as an agent. Agents in the environment have a similar existence to the people in the room; they are autonomous entities but are capable of communication. Through communication, agents can arrange co-operation to achieve common goals. Agents can co-exist in an environment that has a similar constraining effect to the room. The environment also provides a transport mechanism for inter-agent communication; this is analogous to the longitudinal sound waves that are transmitted through the air of the room. We have used this distributed model of computation to create DIME (Distributed Information in a Multi-agent Environment), a visualisation system that moves away from a heavyweight constraint based optimisation algorithm towards a lightweight distributed system that empowers individual data items. The agent environment is populated directly from the data model created by the data classification scheme. The object-orientated nature of the data model easily translates into the agent environment by allowing every data object and class to be represented as an agent. Using this approach provides a convenient method of organising the data within the visualisation system; it also allows the responsibility for creating a visual representation of the data to be devolved to the individual data items and their associates.
vote(property)
P1
P2
Pn action()
Property communication
Agent communication interface
Property communication Information
Instruction
Enivornment communication system
Question Answer
Fig. 2. Architecture of an individual DIME agent with properties (P1 to Pn ) and the communication interface with the agent environment.
48
3.2
Jonathan Meddes and Eric McKenzie
DIME Agent Environment
Using an agent-based philosophy, DIME supports agents operating within an agent environment. We first describe the construction of an agent before describing the operation of multiple agents within the environment; the discussion of our system is presented with a minimum of implementation detail. DIME agent: Figure 2 shows the structure of an agent which is characterised by the properties it holds (P1 to Pn ). Typically an agent is assigned numerous properties and the interaction between the properties dictates its overall behaviour. Properties can be broadly cast into two types, (1) a value property stores details about the agent, and (2) a behaviour property directly describes how the agent behaves within the environment. The internal operation of an agent is controlled by its property communication interface. This is responsible for controlling the individual properties of an agent, organising the updates of each property via a voting mechanism, inter-property communication, and providing access to the agent communication interface for inter-agent communication. When an agent is introduced into the environment, it is activated to perform its tasks. Once activated, an agent uses a roundrobin algorithm allowing each property to optionally perform some action. The combined effect of these actions allows agents to complete their goals. The action of each property is defined within its action method which is invoked each time it is the property’s turn to do some work. This method allows the property to make changes to the character (or state) of the agent by updating or adding properties. Properties in control of the agent are co-operative and operate in a fair and reasonable manner; this is particularly important when a property needs to update the values of other properties in the agent. As a safeguard to an agent being dominated by a minority of properties, we use a voting mechanism in an attempt to reach a consensus on the evolution of the agent. The voting mechanism allows all other properties to cast a vote on their support for the agent to adopt the new property value. The support a property can express for new values ranges from definitely reject to definitely accept, with a neutral position being taken by a property with no interest in the vote. If the outcome of the vote is marginal, a random function makes the casting vote. The operation of an agent is clearly illustrated by a simple example; we limit our consideration to a single agent defined to randomly move around a limited area and only consider its directly relevant properties. This behaviour is expressed using three properties, (1) the position property stores the agents current position in its environment; (2) the boundary property represents the limits of the agents movement; and (3) the random movement property defines the agents behaviour in its environment. Only the random movement property has an action method, the other two characterise the agent but have a passive role in its behaviour. When the random movement property wishes to move the agent, it uses the inter-property communication interface to request the current position of the agent; this is the value of the position property. It can then make a change to the position to represent a random movement in the environment. Before this change can be adopted, a vote must take place involving the position
An Agent-Based Visualisation Architecture
49
and the boundary properties. The position property has no preference in the outcome of the vote and returns a neutral verdict. The boundary property must investigate the proposed change in detail. If the proposed position is outside the boundary, it rejects the change; otherwise, it votes to accept the change and the property communication interface will permit the new value to be adopted. DIME environment: Agents do not directly have access to other agents internal properties and all contact must be channelled through the agent communication interface. This interface provides the necessary functionality to direct messages to individual agents, groups of agents (e.g., all the agents in a class or associated by a relationship agent) or all agents within the environment. Using this communication interface, agents can pass properties to one another using the following styles of communication, (1) instruction messages pass instructions to an agent which must be followed; (2) information messages inform other agents of a property value; and (3) question messages request information from an agent which are responded to using an answer message. Messages are routed by the environment communication system and stored until required by the agents. The agent environment supports the introduction of domain knowledge via domain functions. These specialist agents have two roles, (1) to search for combinations of data agents that can be used to create new agents; and (2) to co-ordinate groups of agents. In the latter role, domain function agents act as a proxy for other agents in the environment. In this guise, the agents for whom it is acting as a proxy are assigned a proxy property; this refers all action and voting to the proxy agent. This mechanism is available to an agent that co-ordinates groups of agents when a substantial influence on their conduct is required. The structure of the data model provides a natural organisation of the data within the visualisation system. Agents have a more complete understanding of the data than any other entity and can create visual representations of themselves using a special agent providing an agent display window. Agents that wish to be represented must posses the visible property; this is responsible for communicating visual information about the agent to the agent display window and encapsulates all the visual information about how to render the agent. This information is received from other properties and can be influenced directly or indirectly by other agents in the environment. Specialist properties can provide the agent with additional knowledge of a suitable visual representation for the data it represents. For example, a Gestalt property gives an agent knowledge about Gestalt principles [5] of appropriate organisation. Another property could provide the agent with knowledge of colour perception [2]. Using such properties, the agents have an improved knowledge of a suitable representation for the data. Agents can collaborate to represent themselves as one visual item in the agent display window. The data model provides several inherent organisational features that are retained within the agent environment. The class agents and their associated data members provide two extreme levels of abstraction. At the most detailed level, agents can provide a visual representation in the agent display window or alternatively the data agents can provide a high level abstraction of the data. A CDO provides an orthogonal dimension of abstraction where
50
Jonathan Meddes and Eric McKenzie
related data items provide an associated concept. In a data model created by the data classification scheme the highest level of abstraction is a complex data class. A CDC agent is an abstract view derived from a table representing all the class agents, data agents and CDO’s. The autonomous but collaborative nature of agents allows them to negotiate suitable visual representations for the natural structures present in the data they represent.
4
Current Work and Future Plans
The agent visualisation architecture has formed the basis for DIME, an ongoing research project implemented using Java and the Java 3D API which has allowed the rapid development of prototype agents and provides a stable platform to introduce additional functionality. DIME provides an interface to the agent environment allowing a user to browse the agent class hierarchy and expand individual classes to study its data agent instantiations. Properties of agents can be investigated and where necessary the characteristics of an agent can be changed by removing, adding or changing its properties. Visible agents are rendered in the agent environment window by the agent rendering system. At present visualisations in DIME use graphical style sheets (GSS) [3]. A GSS specifies the properties that should be assigned to the agents for an effective visual representation of the data they represent. A future enhancement will introduce dynamic property allocation by introducing user profile agents that characterise a user. Such agents are assigned the task of identifying data that is relevant to the user; the agents representing this data are given greater prominence in the visual representation. A long term goal for the development of DIME is to create a system capable of acting as a data repository. Our ambition is for the data management to be controlled by agents encouraging a long-term data management strategy rather than a system for short-term visualisation.
References 1. Bertin, J.: Graphics and graphic information processing. Walter de Gruyter, Berlin, 1981. 43 2. Healey, C.: Choosing effective colours for data visualisation. IEEE Visualisation 1996. 49 3. Felciano, R., Altman, R.: Graphical Style Sheets: Towards Reusable Representations of Biomedical Graphics. Conference on Human Factors in Computer Systems 1998 (to appear). 50 4. Grice, H.: Logic and Conversation. In Cole P and Morgan J eds. Syntax and Semantics: Speech Acts. 3 (1975) 41-58. New York: Academic Press. 46 5. Koffka, K.: Principles of Gestalt Psychology. London New York: Kegan Paul, Trench, Trubner, : Harcourt, Brace. 1935. 49 6. Marks, J.: A Formal Specification Scheme for Network Diagrams That facilitates Automated Design. Journal of Visual Languages and Computing. 2 (1991) 395-414 44 7. Roth, S., Mattis, J.: Data Characterization For Intelligent Graphics Presentation. CHI’90 Proceedings. 44
Error-Tolerant Database for Structured Images
A. Ferro, G. Gallo, and R. Giugno Dipartimento di Matematica Università di Catania, Viale A. Doria 6, 95125, Catania, Italy Phone +39095733051, Fax +39095330091 {ferro, gallo, giugno}@dipmat.unict.it
Abstract. This paper reports the application of an error-tolerant retrieval technique introduced in 1997 by Oflazer [IEEE Transactions on P.A.M.I., vol. 19, No. 12, December 1997] for databases of trees coding pictures of similar objects. The technique is of great interest for the classification and maintenance of historical and archeological pictorial data. We demonstrate the approach on a small postal stamp collection and describe the same application on a large set of pictures.
1
Introduction
Some classes of images can be considered as "statements" in a complex, non-linear, language. Such is the case of icons, coin images, coats and arms, postal stamps etc. All these objects have their own structure and a precise syntax. Although not fully general [3] the "syntactic" approach to image understanding can be usefully applied to machine recognition. The database structure proposed in this paper is one of such applications. A high level approach to classify pictures according to their basic features can be produce great advantages. These features can be the responses to linear and/or nonlinear filters applied to the images or can be features with a precise semantic for the human observer. Features can be recovered and analyzed using object recognition algorithms in a completely automated approach or, in a semiautomatic approach require the intervention of a human operator. Unfortunately there is no simple way, be it automated or dependent on human intervention, of organizing the elements of a generic picture in an unambiguous structure. "Content based" image databases ([1,4,6,7]) are still in an experimental stage although the initial efforts are very promising. Better results are possible when the database includes only very specific kind of images: in particular we propose a semantic approach to build data bases for homogeneous highly structured images. More precisely, a collection of homogeneous and highly structured images is a set of pictures where all elements share the same "layout" but differs for details. Furthermore we require that details range over a finite set of possibilities and that they can be unambiguously classified. Examples of such collections are sets of images of similar objects like those that frequently occur in the cataloging of cultural or artistic artifacts: database of coins, data base of postal stamps, etc. For demonstration sake in the rest of this paper we will describe our Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 51-59, 1999. Springer-Verlag Berlin Heidelberg 1999
52
A. Ferro et al.
innovative approach referring to a collection of postal stamps. The good results obtained in our toy example has been easily replicated on larger collections (about 100 items) and there is no real obstacle to exclude the same approach on much larger collections (more than 1000 items). We report also experiments with a database built in a complete automated way using the relative distribution of pixels in an image. Although this last approach focuses on issues that are not paramount for human perception promising performance has been obtained also in this case.
(a)
(d)
(g)
(b)
(e)
(h)
(c)
(f)
(i)
Fig. 1. The collection of the stamps used to illustrate our proposal referred in the text as “toy example”. The following ids have been used to refer to each stamp: (a) Italia650; (b) Usa60; (c) Berlin20; (d) Deutschland100; (e) Deutschland80; (f) Italia750; (g) Helvetia90; (h) Helvetia180; (i) Republique Francaise.
This paper is organized as follows: Section 2 describes how to code a picture into a tree, both using an interactive procedure based on the semantic of the image and using a totally automated procedure based on the spatial distribution of the pixels. Section 3 introduces a “metric” for the distance between the coding trees and shows how to organize a collection of trees into a compact, space efficient trie. Section 4 reports the results obtained in the experiments and section 5 draws some conclusions and indicates further research directions.
2
Building Tree from Images
This Section describes how to code an image drawn from a collection of structured and homogeneous images into a tree. The basic idea is to treat a structured image as a sentence in a natural language in order to be able to apply the technique described in [5]. Sentences in a natural language are structured in a general pattern. For example statements expressing the action of a subject on an object can be divided into subparts: subject, verb, object. Each subpart can be further decomposed in noun, article, attribute and so on. This, naturally, suggests describing and storing such sentences with a hierarchical data structure like a tree. The choice of the features to organize in a tree for each
Error-Tolerant Database for Structured Images
53
image/object and the structure of the tree is perhaps the most sensitive parameter of the proposed approach. We experimented with two strategies: one semantically based and one based on the distribution of the pixels. 2.1
Semantically Based Tree Construction
The strategy based on the semantic of images depends on the knowledge of an expert: an accurate analysis has to determine which details and features are helpful in assessing the identity of each data base item. It is important to avoid features that introduce only wasteful noise. To illustrate in a concrete way how the image coding is realized we report one possible scheme for some of the features of postal stamps. A typical "stamp tree" can be seen in Fig. 2 and in Fig. 3. Observe that not all of the many possible details and features have been introduced in our scheme. This has been an intentional choice to reduce the complexity of our example and grants a greater clarity in exposition. As a counter effect if the details mentioned in Fig. 2 are the only ones to be considered, even different stamps (for example (g) and (i) in our collection, shown in Fig. 1) have very similar trees. LAYOUT
COLOR
SUBJECT
BACK GROUND
SOUTH
N
T
EAST
N
WEST
NORTH
T
N
T
N
T
Fig. 2. General tree of a stamp, when the semantic approach is adopted. LAYOUT ∈ {Horizontal, Vertical}; COLOR ∈ {1, 2, 3, More}; SUBJECT ∈ {Monument, Face, Drawing}; BACKGROUND ∈ {Empty, Full}; N ≡ Number; T ≡ Text. V
MORE
M
F
S
T
N
N
T
Fig. 3. The tree of the stamp Italia650.
2.2
Pixels Based Tree Construction
The semantic tree construction gives impressive results but it is, in some way, inadequate because it requires the intervention of a human operator/expert. As an
54
A. Ferro et al.
alternative we tested also a completely automated coding of a picture into a tree based on the pixels distribution. The idea is to decompose a picture A recursively as follows: Step1 Divide A with median horizontal cut into two sub regions: Ah1 and Ah2. Step2 Check if the two resulting sub pictures are close under some prescribed similarity measure (for example they have close mean values for the RGB channels). If Ah1 and Ah2 are close go to Step3 else repeat the subdivision recursively on Ah1 and Ah2. Step3 Divide A with a median vertical cut into two sub regions Av1 and Av2. Step4 Check if the two resulting sub pictures are close under some prescribed similarity measure. If Av1 and Av2 are not close repeat the subdivision recursively on Av1 and Av2 else STOP subdivision.
The decomposition obtained following the above algorithm can be stored in a tree whose leaves bear a label with mean RGB values of the non-decomposable regions. For an example Fig. 4 shows the tree associated to stamp “Italia650”. H
H H
H H
V
V V
H V
V
H V
V
V q
a
b
c d e f g
h
i
l m n
o
p
Fig. 4. The decomposition tree of the stamp Italia650, when the pixels approach is adopted. The label H means "horizontal cut" and V means "vertical cut". The values of the leaves are the mean RGB and in this case they are: a = (227, 225, 192); b = (237, 233, 202); c = (192, 129, 92); d = (182, 128, 96); e = (231, 174, 115); f = (184, 153, 112); g = (154, 147, 119); h = (161, 158, 132); i = (166, 169, 146); l = (145, 160, 141); m = (117, 119, 98); n = (153, 155, 135); o = (163, 87, 68); p = (121, 109, 89); q = (233, 202, 178).
3
Finding Trees in a Forest
The constructions in section 2 show how to turn a collection of pictures into a forest of trees. Oflazer [5] has recently reported efficient ways to retrieve in this forest all the trees "similar" up to some degree to a given query. In particular, given an input
Error-Tolerant Database for Structured Images
55
tree and a similarity threshold, an algorithm is known that efficiently retrieves all the trees in the database whose similarity with the input exceeds the threshold. 3.1
Similarity Measures between Trees
A similarity measure is essential if a flexible, error tolerant, retrieval of database items has to be performed. Following [5] the distance between two trees is defined taking into account structural differences and label differences. Two trees can be different because there are two different labels in corresponding nodes or because some branch in one has no correspondence in the other. Let C be the cost in case of different labels and S the cost for a structural difference. The distance between the trees, is the minimum cost of leaves or branches insertions, deletions or leaf labels changes necessary to change one tree into the other. To be more precise, we adopt the data structure “vertex list sequence” to represent a tree T. This structure is a sequence of lists. There are as many lists in this sequence than leaves in the tree. Each list contains the ordered sequence of vertices in the unique path from the root to the corresponding leaf. For example the tree in the Fig. 3 is represented as follows: ((V, MORE), (V, M), (V, F), (V, S, T), (V, N, N), (V, N, T)) Using this formalism we can introduce the following definition. Definition. Let Z=Z1,…, Zp, denote a generic vertex list sequence of p vertex lists. Z[j] denotes the initial subsequence of Z up to and including the j-th vertex list. We will use X (of length m) to denote the query vertex list sequence, and Y (of length j) to denote the sequence that is a (possibly partial) candidate vertex list sequence (from the database of trees). i=0 ìdist ( X [0], Y [ j ]) = j * S ï j=0 ïdist ( X [i], Y [0]) = i * S ïdist ( X [i − 1], Y [ j − 1]) if X i = Y j i.e., last vertex lists are same ï ïmin(dist ( X [i − 1], Y [ j − 1]) + C , dist ( X [i], Y [ j ]) = í dist ( X [i − 1, Y [ j ]) + S , ï ï dist ( X [i], Y [ j − 1]) + S ) if X i and Y j differ only at the leaf label ï ïmin(dist ( X [i − 1], Y [ j ]), ï dist ( X [i], Y [ j − 1])) + S otherwise î
If the semantic tree coding is adopted Table 1 reports the "similarity values" between all the stamps in our toy collection assuming that C=1 and S=2. This is also the choice we made in the experiments reported in Section 4. If the pixels based tree coding is adopted very different results are obtained. Table 2 reports the "similarity values" between all the stamps in our toy collection assuming that C=1 and S=2 in this case. Fig. 5 is a graphical representation of the distances between the stamp Italia650 and the other stamps stored in our collection obtained following both the semantic and the pixel approach.
56
A. Ferro et al. DIST.
A
B
C
D
E
F
G
H
I
A B C D E F G H I
0 5 11 12 4 3 22 24 22
5 0 11 10 7 8 22 24 22
11 11 0 4 7 9 22 24 22
12 10 4 0 8 13 22 24 22
4 7 7 8 0 7 22 24 22
3 8 9 13 7 0 20 22 20
22 22 22 22 22 20 0 3 2
24 24 24 24 24 22 3 0 4
22 22 22 22 22 20 2 4 0
Table 1. Distances between the stamps in the database with C =1 and S = 2 when the semantic tree coding is adopted. A=Italia650, B=Usa60; C=Berlin20; D=Deutschland100; E=Deutschland80; F=Italia750; G=Helvetia90; H=Helvetia180; I=Republique Francaise. DIST.
A
B
C
D
E
F
G
H
I
A B C D E F G H I
0 20 22 20 114 146 89 90 149
20 0 24 16 116 148 91 92 151
22 24 0 24 112 144 87 88 147
20 16 24 0 116 148 91 92 151
114 116 112 116 0 122 98 96 206
146 148 144 148 122 0 178 176 238
89 91 87 91 98 178 0 50 181
90 92 88 92 96 176 50 0 182
149 151 147 151 206 238 181 182 0
Table 2. Distances between the stamps in the database with C=1 and S=2 when the pixel based tree coding is adopted. A=Italia650, B=Usa60; C=Berlin20; D=Deutschland100; E=Deutschland80; F=Italia750; G=Helvetia90; H=Helvetia180; I=Republique Francaise.
144
22
12
11
22
5
3
20
4
89 24
22
(a)
20
146
90 114
(b)
Fig. 5. The similarity between the stamp Italia650 and the other stamps when (a) the semantic tree coding, (b) the pixels based tree is adopted.
Error-Tolerant Database for Structured Images
3.2
57
Organizing a Forest in a Tree
The trees associated to the elements in our collection are stored in the database with their vertex list sequence representation. The set of vertex list sequences is, in turn, converted into a trie structure [2]. The tree will compress any possible redundancies in the prefixes of the vertex list sequences to achieve a compact data structure. For instance, the trees associated to the stamps of the collection in Fig. 1, when coded account the semantic approach, can be represented as a trie as shown in Fig. 6.
(V, MORE) (H, 1)
(H, D)
(H, MORE)
(V, M)
(H, D)
(H, F)
(V, 2)
(V, 3)
(H, F)
(V, F)
(V, E)
(V, M)
(V, F)
(V, F) (V, F)
(V, F)
(V, F)
(V, E, T) (H, S, T)
(H, S, N)
(V, N, N)
(V, S, T)
(H, S, T) (H, N, T)
(i)
(H, N, N) (H, N, N)
(g)
(V, S, N)
(V, S, N)
(O, N, T)
(h)
(V, N, T)
(f)
(V, E, T)
(V, N, N)
(V, W, T)
(V, W, T)
(d)
(c)
(V, N, N)
(V, S, T)
(V, N, T)
(b)
(V, E, T) ( V, N, N)
(e)
(V, N, T)
(a)
Fig. 6. The tree associated to the trees of stamps of our database. 3.3
Error Tolerant Retrieval of Trees
Our goal is the retrieval of trees that match a query up to some degree of approximation. Standard searching within a trie corresponds to traversing a path starting from the start node, to one of the leaves, so that the concatenation of the labels on the arcs along this path matches the input vertex list sequence. For error-tolerant searching, one has to find all paths from the start node to leaves, such that, the corresponding vertex list sequences are within a given distance threshold t of the query vertex list sequence. To perform efficiently this search, paths in the trie that lead to no solutions have to be pruned early so that the search is bound to a very small portion of data structure. The detailed description of on efficient search strategy is outside the scope of this paper; here we report a sketch of the idea developed in [5]. To perform an efficient depth first probing of the trie one has to compute the similarity distance between subsequences of the query and sequences obtained chaining together the labels of the visited nodes of the trie. This can be done recursively, in a dynamic programming fashion, maintaining a suitable array H of distances where the entry H(i, j) represents the distance between the subsequence of the first i vertex lists in the query and the
58
A. Ferro et al.
sequence of j vertex list obtained visiting a node of depth j of the trie. The array H(i, j) is locally updated during the visit of the trie and during the possible backtracking steps. A backtracking step occurs whenever a similarity threshold t (also known as cutoff distance) is exceeded. The leaves that have been reached during the search, within the cutoff distance bounds are the output of the error tolerant matching procedure. Oflazer has shown that this search can be realized in O(L2log L k1/lceiling(t/S)) where L is the number of leaves in each tree, k is number of the tree in the forest, t is the cutoff distance and S is the cost of adding or deleting a leaf in a certain tree. 4
Results
In this section we report some of the results obtained using our algorithm. A first experiment has been to query extensively the database of the stamps in Fig. 1. We have performed searches of queries already in the database and of queries new to the database. If the query is already in the database we have always obtained an exact match. In Fig. 7 the portion of the trie visited with threshold t = 2 with a query outside of stored collection is represented. The output is the stamp “Deutschland100” that belongs to the very some emission of the query and hence is very similar to it. If the threshold t is set to 4 more outputs are found. In particular the algorithm retrieves Deutschland100 and in addition Berlin20. Although Berlin20 may appear very different from Deutschland300 it shares with it the vertical layout, the woman portrait and few other details that in our coding scheme are important. This shows that it is important to have a coding scheme that closely maps the perceptively and semantically important information of each item in the database.
(i)
(g)
(f) (h)
(d)
(c)
(b)
(e)
(a)
Fig. 7. Search resulting from query the stamp “Deutschland300” and threshold t = 2 as a query. The solid branches are the only branches visited from the search.
The good results obtained in these first simple experiments have been replicated on a larger database regarding 100 stamps coded using a semantic approach. We ran the following quantitative test: a human expert have been asked to retrieve form the collection a set of 4 stamps that are, according to his judgment, close to a "query stamps". We compared the expert selections with the output obtained from the same query using our algorithm. In all cases, both expert and algorithm retrieved the very same copy of the query whenever present. The set of the other two stamps retrieved
Error-Tolerant Database for Structured Images
59
from the expert included some of the output obtained from our algorithm in 42% cases with similarity threshold t = 2. This percentage goes to 84% with similarity threshold t = 4. Similar results have been obtained when the query does not have an exact match in the database. As a final experiment we run the same test on the same 100 stamps database using the alternative coding described in section 2.2. Although the performance decreased because the pixel based coding gives more relevance to features that are clearly not taken into account from a human expert, we believe that some combination of the two coding could produce very reliable results. 5
Conclusion and Research Perspectives
In this paper we have demonstrated an application of the error tolerantly retrieval technique proposed in [5] according two coding strategies. We have conducted experiments using a database of . images with similar layout and different details from a finite set of possibilities. The database has been queried and the answers have been statistically compared with the findings of an expert. The observed performance proves the suitability of the method for database of visual information. Future work following this line of research concerns both applications and more refined theoretical analysis of the problem. It is particularly interesting to characterize the class of images that are best for our technique in more precise terms using the syntactic tools provided from visual languages. For the applications we have undertook the construction of a database for a collection of ancient Greek coins in order to validate the proposed method on a real life situation of moderate size. We intend also to prove the applicability of the proposed technique to Web search, and as a support for electronic commerce. References 1. J. R. Back, C. Fuller, A. Gupta, A. Hampapur, B. Horowitz, R. Humphrey, and R. Jain, “The Virage Image Search Engine: An Open framework for Image Management”, Proc. SPIE, Storage and Retrieval for Still Image and Video Databases, vol. 2670, pp. 76-87, 1996. 2. T. H. Cormen, C. E. Leiserson, R. L. Rivest, “Introduction to Algorithms”, MIT press 1990. 3. D. Marr, "Vision", Freeman and Co., New York, 1982. 4. W. Niblack, R. Barber, W. Equitz, M. Flickner, E. Glasman, D. Petkovic, P. Yanker, C. Faloutsos, G. Taubin, “The QBIC Project: Querying Images by Content Using Color Texture, and Shape”, Proc. SPIE, vol. 1908, Storage Retrieval for Image and Video Databases, 1993, pp. 173-187. 5. K. Oflazer, “Error-Tolerant Retrieval of Trees”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, No. 12, December 1997. 6. A. Pentland, R. W. Picard, S. Sclaroff, “Photobook: Tools for Content-Based Manipulation of Image Database”, Proc. SPIE, vol. 2185, Storage and Retrieval for Image and Video Databases, Feb. 1994, pp. 34-47. 7. J. R. Smith and S. F. Chang, “VisualSeek: A fully automated content-based image query system”, Proc. International Conference on Image Processing, Lausanne, Switzerland, 1996.
Query Processing and Optimization for Pictorial Query Trees Aya Soffer1,2 and Hanan Samet2 1
2
Computer Science Department, Technion City, Haifa 32000, Israel [email protected] Computer Science Department, Institute for Advanced Computer Science University of Maryland at College Park, College Park, Maryland 20742 [email protected]
Abstract. Methods for processing of pictorial queries specified by pictorial query trees are presented. Leaves of a pictorial query tree correspond to individual pictorial queries while internal nodes correspond to logical operations on them. Algorithms for processing individual pictorial queries and for parsing and computing the overall result of a pictorial query tree are presented. Issues involved in optimizing query processing of pictorial query trees are outlined and some initial solutions are suggested. Keywords: image databases, query specification, query optimization, retrieval by content, spatial databases, image indexing
1
Introduction
Image databases must be capable of being queried pictorially. The most common method of doing this is via an example image. The problem with this method is that in an image database we are usually not looking for an exact match. Instead, we want to find images similar to a given query image. One of the main issues is how to determine if two images are similar and whether the similarity criteria that are used by the database system match the user’s notion of similarity. Another difficulty with pictorial queries is that they are usually not very expressive in terms of specifying combinations of conditions and negative conditions. A good pictorial query specification method should leverage on the expressiveness of pictorial queries in terms of describing what objects the target images should contain and their desired spatial configuration, resolve the ambiguities inherent to pictorial queries, and enable specifying combinations of conditions. In our previous work [8], we devised a pictorial query specification technique for formulating queries that specify which objects should appear in a target image
The support of the Lady Davis Foundation and the National Science Foundation under Grant CDA-950-3994 is gratefully acknowledged. The support of the National Science Foundation under Grant IRI-97-12715 is gratefully acknowledged.
Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 60–68, 1999. c Springer-Verlag Berlin Heidelberg 1999
Query Processing and Optimization for Pictorial Query Trees
61
as well as how many occurrences of each object are required. Moreover, the minimum matching certainty between query-image and database-image objects can be specified, and spatial constraints that specify bounds on the distance between objects and the relative direction between objects can be imposed. Expressive power is achieved by allowing a pictorial query specification to be composed of one or more query images and by allowing a query image to be negated. In [9] we extended the pictorial query specification technique to allow the formulation of complex pictorial queries via pictorial query trees where leaves correspond to individual pictorial queries while internal nodes represent logical operations on the set of pictorial queries (or subtrees) represented by its children. In this paper we describe in detail the algorithm for processing individual pictorial queries. It can handle multiple instances of each symbol in the query image as well as in the database image and can also handle wild cards. In addition, we present an algorithm for parsing and computing the overall result of a pictorial query tree. We also discuss some issues that are involved in query optimization of pictorial query trees (i.e., their simplification).
2
Related Work
Most image database research deals either with global image matching based on color and texture features [4,5,10] or with the ambiguity associated with matching one query-image object to another [3]. These methods do not address the case of images that are composed of several objects and their desired spatial configuration. There has been some work on the specification of topological and directional relations among query objects [1,2]. These studies only deal with tagged images. Furthermore, it is always assumed that the goal is to match as many query-image objects to database-image objects as possible. A limited form of spatial ambiguity is allowed in pictorial queries based on the 2D-string and its variants [2]. The spatial logic described in [1] also allows specification of query images in terms of spatial relations between objects and permits users to select the level of spatial similarity. In some cases (e.g., [6]), the images are segmented into regions either automatically or semi-automatically and some queries involving spatial constraints specifying the desired arrangement of these regions can be performed. However, the issue of the distance between objects is not addressed by these or any other method. Furthermore, none of these methods provide Boolean combinations or negations of query images.
3
Building Pictorial Query Trees
An individual pictorial query is specified by selecting the required query objects and positioning them according to the desired spatial configuration. Next, the similarity level in terms of three parameters is specified. The matching similarity level msl is a number between 0 and 1 that specifies a lower bound on the certainty that two symbols are from the same class and thus match. Contextual similarity specifies how well the content of database image DI matches that
62
Aya Soffer and Hanan Samet
Fig. 1. Contextual similarity levels (csl).
Fig. 2. Spatial similarity levels (ssl).
of query image QI (e.g., do all of the symbols in QI appear in DI ?). We use four levels of contextual similarity (see Figure 1). Spatial similarity specifies how good a match is required in terms of the relative locations and orientation of the matching symbols between the query and database image. We use five levels of spatial similarity (see Figure 2). For more details and examples, see [8]. Complex pictorial queries are composed of combinations of individual pictorial queries and are specified via pictorial query trees. Leaves of a pictorial query tree correspond to individual pictorial queries. A negated leaf node (NOT) yields the set of all images that do not satisfy the pictorial query. Internal nodes in the tree represent logical operations (AND, OR, XOR) and their negations on the set of images that satisfy the pictorial query (or query subtree) represented by its children.
AND
AND
2
OR 7
OR
csl = 2 ssl = 2
csl = 2 ssl = 4
(a)
csl = 2 ssl = 4
10
5
10
5
csl = 2 ssl = 4
csl = 2 ssl = 4
csl = 2 ssl = 4
(b)
Fig. 3. Images with a (a) camping site within 5 miles of a fishing site OR a hotel within 10 miles of a fishing site AND an airport northeast of and within 7 miles of the fishing site . (b) camping site within 5 miles OR with a hotel within 10 miles of a fishing site of a fishing site AND with no airport within 2 miles of the fishing site (the line above the query denotes negation). The root of the tree is either a pictorial query or a logical operator, while an internal node corresponds to a logical operator and can have one or more
Query Processing and Optimization for Pictorial Query Trees
63
children. For a conjunction of query images where the same symbol appears in both query images, the user may specify whether the two query-symbols must match the same instance of the symbol in the database image, or whether two different instances are allowed. This is termed object binding.
AND
OR
NOR
10
5
csl = 2 ssl = 4
csl = 2 ssl = 4
csl = 2 ssl = 5
csl = 2 ssl = 5
Fig. 4. Images with a camping site within 5 miles OR with a hotel within 10 miles of a fishing site AND with neither a restaurant nor a cafe . Figures 3 and 4 are examples of pictorial query trees. Figure 3a specifies more than one acceptable spatial constraint (i.e., a camping site within 5 of a or a hotel within 10 of a fishing site ) with an OR. We also fishing site specify that there is an airport northeast of and within 7 miles of the fishing site with an AND. Figure 3b shows negation of a pictorial query specifying within 2 miles of the fishing site . Figure 4 shows that there is no airport negation of logical operators in internal nodes. Here we are seeking images with a camping site within 5 miles of a fishing site OR a hotel within 10 but without either a restaurant or a cafe . miles of a fishing site
4
Pictorial Query Processing
In this section, we present an algorithm for retrieving all database images that conform to a given pictorial query tree specification. 4.1
Processing individual pictorial queries
The first step in the algorithm processes each pictorial query image (i.e., each leaf) individually using function GetSimilarImagesM that takes as input a query image (QI ), the matching similarity level (msl ), the contextual similarity level (csl ), and the spatial similarity level (ssl ) associated with QI . It returns the set of database images RI such that each image DI ∈ RI satisfies the pictorial query. Figure 5 summarizes this algorithm. GetSimilarImagesM handles wildcards as well as multiple instances of each class in the query image and in the database image.
64
Aya Soffer and Hanan Samet
GetSimilarImagesM(logical image QI, msl, csl, ssl)
m←0 /* check matching similarity */ foreach el ∈ QI if (el = wildcard) then rm ← set of all images stored in the database else rm ← set of all images containing C(el) with certainty ≥ msl (use index on class) m←m+1 /* check contextual similarity */ if (csl = 1) ∨ (csl = 2) then n−1 RI ← i=0 ri elseif (csl = 3) ∨ (csl = 4) n−1 RI ← i=0 ri if (csl = 1) ∨ (csl = 3) then RI ← RI − {I s.t. I includes symbols not in QI} (use index on image id) /* check multiple symbol instance conditions */ if (csl = 1) ∨ (csl = 2) then I RI ← RI − {I s.t. ∃k : nQI k > nk }, i.e. some (k-th) symbol of QI is underrepresented in I /* check spatial similarity */ foreach I ∈ RI for every possible matching of symbols between QI and I check feasibility of this matching w.r.t. spatial constraints if all matchings are infeasible RI ← RI − I return RI ordered by average certainties
Fig. 5. Algorithm to retrieve all database images similar to a query image (QI) conforming to constraints dictated by msl, csl, and ssl. nIk denotes the number of occurrences of the k th symbol in image I
First, for each symbol in the query image it finds all database images, DI, that contain this symbol with certainty ≥ msl. Next, it handles sthe contextual constraints. If csl is 1 or 2 (images should contain all symbols in QI), then it intersects the set of result images from the first step. If csl is 3 or 4 (any one symbol from QI is enough), then it takes the union of the result images. If the contextual similarity level is 1 or 3, then it avoids including images containing symbols that are not present in QI . Next, it checks the case of multiple instances of query symbols in the query image. If csl is 1 or 2, then for every instance of each symbol in QI, it checks whether there exists an instance of the symbol in DI.
Query Processing and Optimization for Pictorial Query Trees
65
Finally, it checks whether the spatial constraints are satisfied for each candidate image I in the candidate image list RI. Since multiple instances of symbols are allowed in QI and in I, this step needs to check many possible matchings. It can be that some mappings between QI symbols and I symbols create feasible configurations while others do not. For each QI symbol create a set of possible matches in I. Selecting one element from each of these sets generates one possible matching. If none of the possible matchings pass the spatial constraints test, then remove the image from the candidate result set. The spatial similarity between any two matchings is calculated using algorithm CheckSsl [8] which determines whether the spatial constraints dictated by a query image QI and spatial similarity level ssl hold in a logical image DI . Images that pass all of the tests are ordered by the average matching certainty of all matching symbols and returned as the result of the query. 4.2
Parsing and evaluating pictorial query trees
ProcessQueryTree(query tree node: N )
S ← set of all images in the database (global variable) if (isLeaf (N )) N R ← GetSimilarImagesM (QI(N ), msl(N ), csl(N ), ssl(N )) if (hasN egationF lag(N )) NR ← S − NR else n←0 foreach M ∈ sons(N ) rn ← P rocessQueryT ree(M ) n ← n+1 , , or , possibly inverted) N R ← OP (N )n−1 i=0 ri (OP (N ) can be return N R
Fig. 6. Algorithm to retrieve all images satisfying the query represented by node N of a pictorial query tree. Procedure P rocessQueryT ree parses and evaluates the result of a pictorial query tree. Figure 6 summarizes the algorithm. ProcessQueryTree takes as input a node N in the query tree, and returns the set of images that satisfy the query tree rooted at N . If N is a leaf node, then it checks whether the results of this query are cached from earlier invocations. If they are not, then algorithm GetSimilarImagesM is invoked. If the leaf node is negated in the tree, then the complement of the result images set returned by GetSimilarImagesM is taken. The final result image set is returned. If N is an internal node in the query
66
Aya Soffer and Hanan Samet
tree, then ProcessQueryTree is called recursively on each child of N , followed by applying the appropriate logical operation on the results of these calls. The whole query tree is evaluated in this recursive manner by invoking algorithm ProcessQueryTree with the root of the query tree as an argument. Recall, that users can specify object binding. That is, whether the same instance of an object is to be used when it appears in more than one of the pictorial query images that make up the pictorial query tree. The following is an outline of the additions to our algorithms that are necessary for handling object binding. Algorithm ProcessQueryTree receives as additional input a global set of constraints that stipulates the bindings that were specified as part of the query. This set consists of groups of symbols, where all of the symbols in the same group should be matched to the same symbol instance in the database image. To filter out database images that are incompatible with respect to the binding conditions, we combine these binding constraints with information that is provided by the algorithm GetSimilarImagesM, which is augmented to return for each database image that was found similar to the query image, the mapping between query symbols and matched database symbols.
5
Query Optimization Issues
Several optimization techniques can be applied to improve the efficiency of processing pictorial query trees. These include methods designed for optimization of individual pictorial query processing and optimization of query tree processing. Individual pictorial query processing may be made more efficient by handling spatial and contextual constraints simultaneously rather than one followed by the other as we do now. We addressed this issue in [7]. Two optimizations are possible for computing the result of the pictorial query tree. The first optimization is to change the order of processing individual query images in order to execute the parts that are more selective (i.e., result in fewer images) first. The selectivity of a pictorial query is based on three factors. Matching selectivity estimates how many images satisfy the matching constraint as specified by msl. Contextual selectivity estimates how many images satisfy the contextual constraint as specified by the query image and csl. Spatial selectivity estimates how many images satisfy the spatial constraint as specified by ssl. Depending on ssl, either distance, direction, both, or neither are constrained. Matching and contextual selectivity factors are computed based on statistics stored as histograms in the database which indicate the distribution of classifications and certainty levels in the images. These histograms are constructed when populating the database. Computing spatial selectivity is much more complex. One approach to measuring the distance aspect of the spatial selectivity calculates some approximation of the area spanned by the symbols in the query image. This can be estimated, for example. using an approximation of the convex hull of the symbols in the query image. Details of this method are beyond the scope of this paper. Selectivity of an individual pictorial query (leaf) is computed by combining these three selectivity factors.
Query Processing and Optimization for Pictorial Query Trees
67
The query tree selectivity is computed using a recursive algorithm similar to the one executing the query. If an individual pictorial query is negated in the tree, the selectivity is 1 - the selectivity of the query. The selectivity of a subtree is as follows. For OR or XOR, take the sum of the selectivities of the subtrees minus the probability that a combination of cases occured. For AND, take the product of the selectivities of the subtrees. To illustrate the general use of this optimization method, consider the query trees in Figure 3. In both queries the left side of the tree requests images with within 5 miles of a fishing site OR a hotel within 10 a camping site miles of a fishing site . In query (a), we add the constraint that there exists an airport northeast of and within 7 of fishing site . In our database, we have very few airfields and thus the right side is more selective and it will be processed first. On the other hand in query (b), we add the constraint that within 2 miles of the fishing site . Clearly, in most there is no airport cases there will be no such airport , and thus in this case the right side is not selective and the left side should be processed first. The second form of optimization is to combine individual query images and to process them together. To see its usefulness, we study how the query in Figure 4 is processed using the current algorithm. First, find {CF} all images with a within 5 of a fishing site . Next, find {HF} all images with a camping site hotel within 10 of a fishing site . Then, take the union of these two sets: and {LS} = {CF } ∪ {HF }. Now, find the set {R}: images with a restaurant the set {C}: images with a cafe and compute the set RS = I − (R ∪ C). The final result is the intersection of the two sets: LS ∩ RS. A more sensible way , find the nearest to compute this query is as follows. For each fishing site neighbors up to distance 5 in incremental order. If the next nearest neighbor is a camping site or a hotel , then add this image to the candidate list. Continue retrieving nearest neighbors in incremental order up to distance 10. If , then add this image to the candidate the next nearest neighbor is a hotel list. For each image I in the candidate list, examine all of the objects in I. If or a cafe in I, then remove I from the candidate list. there is a restaurant
References 1. A. Del Bimbo, E. Vicario, and D. Zingoni. A spatial logic for symbolic description of image contents. Jour. of Vis. Lang. and Comp., 5(3):267–286, Sept. 1994. 61 2. S. K. Chang, Q. Y. Shi, and C. Y. Yan. Iconic indexing by 2-D strings. IEEE Trans. on Patt. Anal. and Mach. Intel., 9(3):413–428, May 1987. 61 3. W. I. Grosky, P. Neo, and R. Mehrotra. A pictorial index mechanism for modelbased matching. Data & Know. Engin., 8(4):309–327, Sept. 1992. 61 4. W. Niblack, R. Barber, W. Equitz, M. Flickner, E. Glasman, D. Petkovic, and P. Yanker. The QBIC project: Querying images by content using color, texture, and shape. In Proc. of the SPIE, Storage and Retrieval of Image and Video Databases, vol. 1908, pp. 173–187, San Jose, CA, Feb. 1993. 61 5. A. Pentland, R. W. Picard, and S. Sclaroff. Photobook: Content-based manipulation of image databases. In Proc. of the SPIE, Storage and Retrieval of Image and Video Databases II, vol. 2185, pp. 34–47, San Jose, CA, Feb. 1994. 61
68
Aya Soffer and Hanan Samet
6. J. R. Smith and S.-F. Chang. VisualSEEk: a fully automated content-based image query system. In ACM Int. Conf. on Multimedia, pp. 87–98, Boston, Nov. 1996. 61 7. A. Soffer and H. Samet. Pictorial queries by image similarity. In 13th Int. Conf. on Patt. Recog., vol. III, pp. 114–119, Vienna, Austria, Aug. 1996. 66 8. A. Soffer and H. Samet. Pictorial query specification for browsing through spatially-referenced image databases. Jour. of Vis. Lang. and Comp., 9(6):567– 596, Dec. 1998. 60, 62, 65 9. A. Soffer, H. Samet, and D. Zotkin. Pictorial query trees for query specification in image databases. In 14th Int. Conf. on Patt. Recog., vol. I, pp. 919–921, Brisbane, Australia, Aug 1998. 61 10. M. Swain. Interactive indexing into image databases. In Proc. of the SPIE, Storage and Retrieval for Image and Video Databases, vol. 1908, pp. 95–103, San Jose, CA, Feb. 1993. 61
Similarity Search Using Multiple Examples in MARS Kriengkrai Porkaew1, Sharad Mehrotra2 , Michael Ortega1 , and Kaushik Chakrabarti1 1
Department of Computer Science University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA 2 Department of Information and Computer Science University of Cailifornia at Irvine, Irvine, CA 92697, USA {nid,sharad,miki,kaushik}@ics.uci.edu
Abstract. Unlike traditional database management systems, in multimedia databases that support content-based retrieval over multimedia objects, it is difficult for users to express their exact information need directly in the form of a precise query. A typical interface supported by content-based retrieval systems allows users to express their query in the form of examples of objects similar to the ones they wish to retrieve. Such a user interface, however, requires mechanisms to learn the query representation from the examples provided by the user. In our previous work, we proposed a query refinement mechanism in which a query representation is modified by adding new relevant examples based on user feedback. In this paper, we describe query processing mechanisms that can efficiently support query expansion using multidimensional index structures.
1
Introduction
In a content-based multimedia retrieval system, it is difficult for users to specify their information need in a query over the feature sets used to represent the multimedia objects [10, 7, 12]. Motivated by this, recently, many content-based multimedia retrieval systems have explored a query by example (QBE) framework for formulating similarity queries over multimedia objects (e.g., QBIC [4], VIRAGE [1], Photobook [9], MARS [6]). In QBE, a user formulates a query by providing examples of objects similar to the one s/he wishes to retrieve. The system converts this into an internal representation based on the features extracted from the input images. However, a user may not initially be able to provide the system with “good” examples of objects that exactly capture their information needs. Furthermore, a user may also not be able to exactly specify the relative
This work was supported by NSF awards IIS-9734300, and CDA-9624396; in part by the Army Research Laboratory under Cooperative Agreement No. DAAL01-962-0003. Michael Ortega is supported in part by CONACYT grant 89061 and MAVIS fellowship.
Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 68–75, 1999. c Springer-Verlag Berlin Heidelberg 1999
Similarity Search Using Multiple Examples in MARS
69
importance of the different features used to represent the multimedia objects to the query. To overcome the above limitations, in the Multimedia Analysis and Retrieval (MARS) project, we explored techniques that allow users to refine the initial query during the retrieval process using relevance feedback [10]. Given an initial query, the system retrieves objects that are most similar to the query. The feedback from the user about the relevance of the retrieved objects is then used to adjust the query representation. Relevance feedback in MARS serves two purposes as follows. Query Reweighting adjusts the relative importance of the different components to the query. It allows the system to learn the user’s interpretation of similarity between objects. Query Modification changes the underlying representation of the query to incorporate new relevant information from the user’s feedback. It overcomes the deficiency of having started from examples that only partially capture the user’s information need. In [11, 12, 7, 10], various models for query reweighting and query modification were explored and compared over diverse multimedia collections. Specifically, two different strategies for query modification have emerged. The first, referred to as query point movement (QPM) [7, 11], attempts to move the query representation in the direction where relevant objects are located. At any instance, a query is represented using a single point in each of the feature spaces associated with the multimedia object. In contrast to QPM, in [10] we proposed a query expansion model (QEM) in which the query representation is changed by selectively adding new relevant objects (as well as deleting old and less relevant objects). In QEM, the query may consist of multiple points in each feature space. Our experiments over large image collections illustrated that QEM outperforms QPM in retrieval effectiveness (based on precision/recall measures) [10]. However, in QEM, its potential drawback is that the cost of evaluating the query grows linearly with the number of objects in the query if done naively. In this paper, we explore efficient strategies to implement QEM that overcome the above overhead. The key is to traverse a multidimensional index structure (e.g., X-tree [2], hybridtree [3], SS-tree [15], etc.) such that best N objects are retrieved from the data collection without having to explicitly execute N nearest neighbor queries for each object in the query representation. We conduct an experimental evaluation of our developed strategies over a large image collection. Our results show that the developed algorithms make QEM an attractive strategy for query modification in content-based multimedia retrieval since it provides better retrieval effectiveness without extensive overhead. The rest of the paper is developed as follows, Sect. 2 describes the contentbased retrieval in MARS. Section 3 describes the proposed approaches to implementing QEM. Section 4 compares the approaches and shows experimental results. Conclusions are given in Sect. 5.
70
2
Kriengkrai Porkaew et al.
Content-Based Retrieval in MARS
This section briefly describes the content-based retrieval mechanism supported in MARS which is characterized by the following models: Multimedia Object Model: a multimedia object is a collection of features and the functions used to compute the similarity between two objects for each of those features. Query Model: A query is also a collection of features. In QEM, a query may be represented by more than one instance (point) in each feature space. Furthermore, weights are associated with each feature, as well as, with each instance in the feature representation. These weights signify the relative importance of the component to the query. Figure 1 illustrates the query structure which consists of multiple features fi and each feature consists of multiple feature instances rij .
w1 w11
F1
R 11
Query
w2
w12
w21
R 12
R 21
F2
w22 R 22
F i = Feature i w i = Importance of Feature i with respect to the other features w ij = Importance of Feature i of Object j with respect to Feature i of the other objects R ij= Representation of Feature i of Object j
Fig. 1. Query Model
Retrieval model: The retrieval model defines how similarity Sim between a query Q and an object O is computed. n Similarity is computed n hierarchically over the query tree. That is Sim = i=1 wi Simi , where i=1 wi = 1, n is the number of features used in the queries, and Simi is the similarity between the object and the query mbased on feature i which is computed as: Simi = m j=1 wij Simij , where j=1 wij = 1, m is the number of feature instances in the feature i in the query, and Simij is the similarity between instance j and the object based on feature i. Simij is computed using the similarity function determined by the object model. The retrieval process begins with some initial weights associated with nodes at each level of the query tree. For simplicity, initially weights associated with nodes of the same parent are equal. Refinement Model: The refinement model adjusts the query tree and the similarity functions used at different levels of the tree based on the user’s feedback. As discussed in the introduction, the refinement process consists of query reweighting and query modification using query expansion model. The details of the reweighting models, and the query modification models are not critical for the discussion of implementation techniques in this paper and hence omitted due to space restrictions. Details can be found in [10].
Similarity Search Using Multiple Examples in MARS
3
71
Query Processing
At each iteration of query refinement, the system returns to the user N objects from the database that have the highest similarity to the current query representation. Instead of ranking each object in the database and then selecting the best N answers, the query is evaluated in a hierarchical bottom up fashion. First, the best few objects based on each feature individually are retrieved. The similarity values of these objects on individual features are then combined (using the weighted summation model) to generate a ranked list of objects based on the entire query. The process continues until the best N matching objects have been retrieved. We next discuss how feature nodes of the query are evaluated, and the answers are combined to obtain the best N answers for the query. 3.1
Evaluating Feature Nodes
In a query tree, let f be a feature node and r1 , . . . , rm be the instances (points) under the feature space F . The objective of evaluating the feature node is to retrieve N objects from the database that best match f . We will use the notion of distance instead of similarity since the evaluation of the feature node will use multidimensional indexing mechanisms that are organized based on distances. a point x in F and Let drj ,x be the distance between rj and m m Df,x be the distance between f and x in F where Df,x = j=1 wj drj ,x and j=1 wj = 1. Thus, the best N matches to f correspond to objects which are closest to f based on the above definition of distance. In the following two subsections, we describe two different strategies of evaluating the best N objects for a given feature node. Both strategies assume that the feature space is indexed using a multidimensional data structure that supports range and k-nearest neighbor queries. Centroid Expansion Search (CES): The idea is to iteratively retrieve next nearest neighbors of some point c (close to r1 , . . . , rm ) in the feature space F using the feature index until the N best matches to f are found. Let x and y be two objects in the feature space F . x is a better match to f compared to y if and only if Df,x ≤ Df,y , or equivalently m
wj drj ,x ≤
j=1
m
wi drj ,y
(1)
j=1
Since distance functions are metric, the triangle inequality dictates that drj ,x ≤ dc,x + dc,rj and drj ,y ≥ |dc,y − dc,rj |. Substituting drj ,x , drj ,y in (1): m
m wj dc,x + dc,rj ≤ wj |dc,y − dc,rj |
j=1
Since
m j=1
wj = 1, we get: dc,x +
j=1 m
m
j=1
j=1
wj dc,rj ≤
wj |dc,y − dc,rj |
(2)
(3)
72
Kriengkrai Porkaew et al.
Thus, if (3) holds, then (1) also holds. To remove the absolute value from (3), let R = { r1 , . . . , rm }, R1 = { rj ∈ R | dc,rj ≤ dc,y }, and R2 = R − R1 = { rj ∈ R | dc,rj > dc,y }. Replace R1 and R2 in (3), dc,x + ≤
rj ∈R1
wj dc,rj +
rj ∈R1
dc,x ≤ dc,y − 2
wj dc,rj
rj ∈R2
wj (dc,y − dc,rj ) +
wj (dc,rj − dc,y )
rj ∈R2
wj dc,y +
rj ∈R2
dc,x ≤ dc,y − 2
m
(4)
wj dc,rj
(5)
rj ∈R1
wj min(dc,y , dc,rj )
(6)
j=1
Equation (6) provides the strategy to retrieve the best N answers based on the match to f . The strategy works as follows. We find the nearest neighbors to c incrementally. Let x1 , . . . , xP be the objects seen mso far. We determine the target M, 1 ≤ M ≤ P such that Dc,xM ≤ Dc,xP − 2 j=1 wj min(dc,xP , dc,rj ). By (6), Df,xM ≤ Df,xP +k , k = 1, 2, . . .. Let α = max{Df,xi |i = 1, . . . , M }. We then determine the set {xi |i = 1, . . . , P ∧ Df,xi ≤ α}. All such xi are better matches to f than any object xP +k , k = 1, 2, . . . and are hence returned. If N objects have not yet been returned, the process continues iteratively by retrieving the next closest object to c (i.e., xP +1 ) and repeating the above algorithm. mNotice that c can be any point. However, the optimal choice of c minimizes j=1 wj dc,rj ; i.e. c should be the weighted centroid of r1 , . . . , rm . This approach does not require any change to the incremental nearest neighbor search algorithm associated with the original multidimensional data structure. However, it does not perform well when query changes dramatically due to the relevance feedback process since the starting centroid is optimal for the original query. Multiple Expansion Search (MES): In this approach, N nearest neighbor for a feature node f is determined by iteratively retrieving next nearest neighbors for each instance r1 , . . . , rm associated with f . Let Rj be the set of ranked results for the instance rj , j = 1, . . . , m. That is, for all x ∈ Rj and y ∈ Rj , drj ,x ≤ drj ,y . Furthermore, let αj be the maximum distance between rj and any object in Rj in the feature space; that is, αj = max{drj ,x |x ∈ Rj }. nRj contains all objects that are in the range of αj from rj . Note that if y ∈ j=1 Rj , then drj ,y > αj m m m for all j. So j=1 wj drj ,y > j=1 wj αj , that is, Df,y > j=1 wj αj . As a result, m m y ∈ j=1 Rj if Df,y ≤ j=1 wj αj . m Note that if j=1 Rj contains at least N objects x1 , . . . , xN such that for m all xk , Df,xk ≤ j=1 wi αj , then it is guaranteed that N best matches to the feature node f are contained in m j=1 Rj . Thus, in order to evaluate the best
Similarity Search Using Multiple Examples in MARS
73
N matches to f , MES incrementally evaluates the nearest neighbor for each of the instances r1 , . . . , rn thereby increasing the value of at least m one αj in each step, j = 1, . . . , m until there are at least N objects within j=1 Rj for which m Df,xk ≤ j=1 wi αj . Many different strategies can be used to expand αj s. The optimal strategy n determines αj that minimize i=1 Ri since then the least number of objects are explored to retrieve the best N objects based on the match to the feature. We try different strategies for determining αj s and compare them in Sect. 4. 3.2
Evaluating the Query Node
Given the best matching answers for each of the feature nodes f1 , . . . , fn , the objective in evaluating the query node is to combine the results to determine the best N objects to the overall query. That is, we need to determine the N objects with the least distance to thequery, where the distance n between object n and the query is defined as DQ,x = i=1 wi Df,x where i=1 wi = 1. MES discussed for the feature node evaluation can also be used for this purpose and is hence not discussed any further.
4
Experiments
To explore the effectiveness of the algorithms, we performed experiments over a large image dataset (65,000 images) obtained from the Corel collection. Images features used to test the query processing are color histogram [14], color histogram layout [8], color moments [13], and co-occurrence texture [5]. Manhattan distance is used for the first two features and Euclidean distance is used for the last two features [8]. The purposes of this experiment are to compare various approaches we proposed, and to show that QEM can be implemented efficiently. The effectiveness is measured by the number of objects seen before the best N answers are found. A good approach should not need to explore so many objects to guarantee the best N answers and it should not degrade significantly when multiple objects are added to the query. We performed experiments on CES and MES with various parameters. Specifically, CES searches from the centroid of the query point set. In MES, we explored 4 expansion options as follows. Single Search searches only in one of the query points. Balanced Search searches on all query points with equal ranges. Weighted Search searches on all query points with the ranges proportional to the weights of the query points. Inverse Weighted Search searches on all query points with the ranges proportional to the inverse of the weights of the query points. In the experiments, we do not use any index structure in order to avoid hidden effects caused by the specific index structure. Instead, we simulate a k-nearest neighbor search by scanning the dataset and ranking the answers.
74
Kriengkrai Porkaew et al.
The experimental result shows that single search performs the worst. Intuitively, one may expect the weighted search to perform the best among the four approaches. However, surprisingly, even though the weights are not balanced, the balanced search performed better than any search techniques including the centroid expansion search.
2000
10 Centroid Balance Weighted 1/Weight Single
Centroid Balance Weighted 1/Weight 8 Number of objects seen (x Top N)
top N
1500
1000
6
4
500 2
0
0 0
500
1000 Seen
1500
(a) Seen vs best N
2000
1
2
3
4 5 Number of points in a query
6
7
(b) Query size vs Seen/best N ratio
Fig. 2. Experimental Result
Figure 2 compares the different approaches and shows that the number of objects in the query representation has very little impact on the balanced search and the weighted search which are the best searches. The reason is simply because the feature space is sparse and the multiple query points are close together due to the query expansion model which selectively adds relevant query points and removes less relevant ones. Other approaches do not perform well since they may have seen best answers but they cannot guarantee that those answers are among the best ones unless they explore further.
5
Conclusions
Content-based multimedia retrieval and multidimensional indexing are among the most active research areas in the past few years. The two research areas are closely related. The supporting index structure has a big impact on the efficiency of the retrieval. In this paper, we proposed algorithms to extend index structures to support complex queries efficiently in the MARS weighted summation retrieval model. We focussed on an efficient implementation to support QEM proposed in [10]. QEM modifies the query by selectively adding new relevant objects to the query (as well as deleting old and less relevant objects). In contrast, QPM modifies the query by moving the query point in the direction of the relevant objects.
Similarity Search Using Multiple Examples in MARS
75
Our previous work showed that QEM outperforms QPM in retrieval effectiveness. This paper further illustrates that QEM can be efficiently implemented using multidimensional index structures. As a result, we believe that QEM is a viable approach for query refinement in multimedia content based retrieval.
References [1] Jeffrey R. Bach, Charles Fuller, Amarnath Gupta, Arun Hampapur, Bradley Horowitz, Rich Humphrey, Ramesh Jain, and Chiao fe Shu. The Virage image search engine: An open framework for image management. In SPIE Conf. on Vis. Commun. and Image Proc., 1996. 68 [2] S. Berchtold, D. A. Keim, and H. P. Kriegel. The x-tree: An index structure for high-dimensional data. In VLDB, 1996. 69 [3] Kaushik Chakrabarti and Sharad Mehrotra. High dimensional feature indexing using hybrid trees. In ICDE, 1999. 69 [4] M. Flickner, Harpreet Sawhney, Wayne Niblack, Jonathan Ashley, Q. Huang, Byron Dom, Monika Gorkani, Jim Hafine, Denis Lee, Dragutin Petkovic, David Steele, and Peter Yanker. Query by image and video content: The QBIC system. IEEE Computer, Sep 1995. 68 [5] Robert M. Haralick, K. Shanmugam, and Its’hak Dinstein. Texture features for image classification. IEEE Trans. on Sys., Man, and Cyb., SMC-3(6), 1973. 73 [6] Thomas S. Huang, Sharad Mehrotra, and Kannan Ramchandran. Multimedia analysis and retrieval system (MARS) project. In Annual Clinic on Library Application of Data Processing - Digital Image Access and Retrieval, 1996. 68 [7] Yoshiharu Ishikawa, Ravishankar Subramanya, and Christos Faloutsos. Mindreader: Querying databases through multiple examples. In VLDB, 1998. 68, 69 [8] Michael Ortega, Yong Rui, Kaushik Chakrabarti, Sharad Mehrotra, and Thomas S. Huang. Supporting similarity queries in MARS. In ACM Multimedia, 1997. 73 [9] A. Pentland, R.W. Picard, and S. Sclaroff. Photobook: Content-based manipulation of image databases. Int’l Journal of Computer Vision, 18(3), 1996. 68 [10] Kriengkrai Porkaew, Sharad Mehrotra, and Michael Ortega. Query reformulation for content based multimedia retrieval in MARS. In IEEE Int’l Conf. on Multimedia Computing and Systems, 1999. 68, 69, 70, 74 [11] Yong Rui, Thomas S. Huang, and Sharad Mehrotra. Content-based image retrieval with relevance feedback in MARS. In IEEE Int’l Conf. on Image Proc., 1997. 69 [12] Yong Rui, Thomas S. Huang, Michael Ortega, and Sharad Mehrotra. Relevance feedback: A power tool for interactive content-based image retrieval. IEEE Trans. on Circuits and Video Technology, Sep 1998. 68, 69 [13] Markus Stricker and Markus Orengo. Similarity of color images. In SPIE Conf. on Vis. Commun. and Image Proc., 1995. 73 [14] Michael Swain and Dana Ballard. Color indexing. Int’l Journal of Computer Vision, 7(1), 1991. 73 [15] D. White and R. Jain. Similarity indexing with the ss-tree. In ICDE, 1995. 69
Excluding Specified Colors from Image Queries Using a Multidimensional Query Space Dimitrios Androutsos1 , Kostas N. Plataniotis2, and Anastasios N. Venetsanopoulos1 1
University of Toronto Department of Electrical & Computer Engineering Digital Signal & Image Processing Lab 10 King’s College Road, Toronto, Ontario, M5S 3G4, CANADA {zeus,anv}@dsp.toronto.edu WWW:http://www.dsp.toronto.edu 2 Ryerson Polytechnic University Department of Math, Physics & Computer Science 350 Victoria Street, Toronto, Ontario, M5B 2K3, CANADA [email protected]
Abstract. Retrieving images in a database based on user specified colors is a popular low-level retrieval technique. However, the available systems today do not easily allow for a user or a specified query to tag certain colors as unwanted in the query result to ultimately be excluded in the query. Specifically, color histogram techniques do not allow for a direct approach to excluding colors and would require a separate query stage to filter out images containing unwanted colors. In this paper we present our vector-based scheme to image retrieval using a multidimensional query space which naturally accepts the exclusion of specified colors in the overall similarity measure.
1
Introduction
Color image retrieval has received increasing attention lately as the field of image database retrieval grows. It’s importance stems from the fact that color is a lowlevel image feature which is essential to the early stages of human vision. Color is easily recalled and identified and a natural attribute for describing objects and scenes. For these reasons, image retrieval researchers have been trying to find efficient and effective ways to retrieve color images from large databases using color in the query definition [1]. To this end, color indices are created using color histograms to capture the color representation of all the database images [2,3]. Using these indices, a user can retrieve images from the database by building a query by specifying certain colors which they want the retrieved images to contain or by specifying an example image which the retrieved images should match. There are a number of image retrieval systems which employ these techniques and there is much ongoing research in the area [4,5]. Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 76–82, 1999. c Springer-Verlag Berlin Heidelberg 1999
Excluding Specified Colors from Image Queries
77
However, these systems do not address a very important issue in color retrieval namely, color exclusion. It is important for users to be able to tag a certain color or group of colors as unwanted so that they do not appear in the retrieval results. With present systems, specifying which colors to exclude would require an additional stage to filter retrieved images and modify their ranking according to whether or not an exclusion color is present. In this paper we describe how our system addresses this issue by virtue of using a Multidimensional Query Space which incorporates the exclusion of any unwanted colors directly into the image similarity measure, without requiring an extra processing stage.
2
System Description
Utilizing color histograms for indexing and retrieval has gained much popularity. However, there are inherent problems with this technique, which reduce the flexibility and accuracy of the query process and results. In particular, color histograms capture global color activity. Attempts to include spatial information by image partitioning has had some success but storage and computational requirements increase accordingly. In addition, the similarity metrics which are commonly accepted and utilized allow little flexibility and have no valid perceptual basis. We have developed a system which is color vector-based. We do not use histograms to build indices. Instead, we store representative RGB color vectors from extracted color regions, along with spatial color information to build an index of smaller dimension and with more information than a simple color histogram. In this section, we give a brief overview of our system, specifically how the feature extraction is done via segmentation and we also present the distance measure which we use to perform similarity matching. 2.1
Feature Extraction & Indexing
Our feature extraction is based on unsupervised recursive color segmentation. Specifically, we perform HSV-space segmentation while taking into consideration certain perceptual attributes of human color perception and recall. The HSVspace classifies similar colors under similar hue orientations and thus provides a more natural grouping. In addition, it allows for automated segmentation since it allows for fast and efficient automation. It is not dependent on variables, such as seed pixels or number of extracted colors, such as in clustering techniques. The details of our segmentation technique can be found in [8]. However, it is important to note that we: – extract bright colors first – extract and classify white and black regions – treat the saturation histogram as multi-modal instead of bi-modal
78
Dimitrios Androutsos et al.
For each image we extract c colors, which is an image dependent quantity. We calculate the average color of each of the c colors and use that RGB value as each region’s representative vector. These c colors, along with spatial information such as size and location of each region, are used to build each image index. 2.2
Similarity Measure
Since our color indices are actual 3-dimensional color vectors which span the RGB space, a number of vector distance measures can be implemented for retrieval. However, we implement a measure which is based on the angle of a color vector. Angular measures are chromaticity-based, which means that they operate primarily on the orientation of the color vector in the RGB space and therefore are more resistant to intensity changes and it has been found that they provide much more accurate retrieval results than other measures [7]. Specifically, our similarity measure is a perceptually-tuned combination of the angle between two vectors and a magnitude difference part, defined as [9]: · x − x | x |x 2 i j i j ) 1− √ ) , (1) β(xi , xj ) = exp −α(1 − 1 − cos−1 ( π |xi ||xj | 3 · 2552 angle magnitude where xi and xj are 3-dimensional color vectors, α is a design parameter and √ and 3 · 2552 are normalization factors.
3
2 π
Image Query
During the query process, for each user-specified query color, a similarity measure is calculated using (1), to each representative color vector in a given database index. For each query color, the minimum distance is kept and a multidimensional measure is created which consists of the minimum distances of the query colors to the indexed representative vectors in the given index:
D(d , . . . , d ) = I − (min(β(q , i ), . . . , β(q , i min(β(q , i ), . . . , β(q , i ))), 1
n
1
n
1
1
1
n
m
m )), . . . ,
(2)
where I is a vector of size n with all entries of value 1, q n are the n query colors and im are the m indexed representative color vectors for a given image. 3.1
Multidimensional Query Space
The vector D in (3) exists in a vector space defined by the similarity measure of the specified query colors to the indexed colors. The dimensionality of this space changes and is dependent on the number of query colors. We refer to this space as the multidimensional query space.
Excluding Specified Colors from Image Queries
79
The database image that is the closest match to all the given query colors q1 , q2 , . . . , qn is the one which is closest to the origin of the multidimensional query space. Within this query space, there is a line on which all components of D are equal. We refer to this line as the equidistant line. A distance vector D that is most centrally located, i.e, is collinear with the equidistant line and at the same time has the smallest magnitude, corresponds to the image which contains the best match to all the query colors, as depicted in Figure 1(a). For each query, each database image exists at a point in this multidimensional query space. It’s location and relation to the origin and equidistant line determines it’s retrieval ranking which we quantify by taking a weighted sum of the magnitude of D and the angle, D to the equidistant line: R = w1 |D| + w2 D,
(3)
where lower rank values R imply images with a closer match to all the query colors. The weights w1 and w2 can be adjusted to control which of the two parameters, i.e., magnitude or angle, are to dominate. We have found that values of w1 = 0.8 and w2 = 0.2 give the most robust results. This is to be expected since collinearity with the equidistant line does not necessarily imply a match with any query color. It implies that each query color is equally close (or far) to the indexed colors. However, as |D| → 0, implies closer matches to one or more colors. Thus, a greater emphasis must be placed on the magnitude component.
q1
equidistant line q1
equidistant line
∆
D
Ξ D
x1
q
2 q2
(a)
(b)
Fig. 1. (a) Vector representation of 2 query colors q1 &q2 , their multidimensional distance vector D and the corresponding equidistant line. (b) the same 2 query colors,1 exclusion color, x1 and the resulting multidimensional distance vector ∆.
80
4
Dimitrios Androutsos et al.
Color Exclusion
Our proposed vector approach provides a framework which easily accepts exclusion in the query process. It allows for image queries containing any number of colors to be excluded in addition to including colors in the retrieval results. From the discussion in Section 3.1 above, we are interested in distance vectors D which are collinear with the equidistant line and which have small magnitude. The exclusion of a certain color should thus affect D accordingly and it’s relation to the equidistant line and the origin. For example, if it is found that an image contains an indexed color which is close to an exclusion color, the distance between the two can be used to either pull or push D closer or further to the ideal and accordingly affect the retrieval ranking of the given image, as shown in Figure 1(b). To this end, we determine the minimum distances of each exclusion color with the indexed representative colors, using (1), to quantify how close the indexed colors are to the exclusion colors:
X (x , . . . , x ) = (min(β( , i ), . . . , δ( , i 1
n
1
1
1
i
i
m )), . . . min(β( n , 1 ), . . . , δ( n , m )))
(4)
where ξn are the n exclusion colors and im are the m indexed representative colors of each database image. Equation (4) quantifies how similar any indexed colors are to the exclusion colors. To quantify dissimilarity, a transformation of each vector component of X is required, and then this is merged with D to give a new overall multidimensional distance vector:
= [D I − X ],
(5)
where I is a vector of size n with all entries of value 1. The dimensionality of ∆ is equal to the number of query colors + number of exclusion colors. The final retrieval rankings are then determined from |∆| and the angle which D in (5) makes with the equidistant line of the query color space (i.e., the space without excluded colors). We performed an example query from our database of 1850 natural images, both with exclusion and without. Figure 2(a) depicts the query result when R,G,B colors (26, 153, 33) (green) and (200, 7, 25) (red) were desired and the color (255, 240, 20) (yellow) was excluded. It can be seen that images which contained colors close to yellow were removed from the top ranking results, as compared to Figure 2(b), where yellow was not excluded. We further investigated these exclusion results by determining by how much the retrieval ranking of the images which contained yellow changed. A trained user was asked to look at the top 40 retrieval results for the query of red and green, and determine which of these images contained yellow. This first step resulted with a set of 25 images, which we refer to as X , that contained the exclusion color. The retrieval ranking of each of the images in X was then calculated when the same query also excluded yellow. It was found that none of the images in X remained among the top 40 retrieval results. Furthermore, their ranking decreased significantly and all 25 images were now ranked among the bottom 27% of the entire 1850 image database, i.e., among the 500 least similar images.
Excluding Specified Colors from Image Queries
81
(a)
(b)
Fig. 2. Query result for images with (a) red & green, excluding yellow and (b) red & green, not excluding yellow.
8 images remained in the top 40 retrieval results, which contained red and green and 7 images had their ranking slightly decreased for containing colors that were perceptually close to yellow. The flexibility of this technique allows any number of colors to be excluded in a given color query and can also be incorporated in query-by-example, where a seed image is fed as a query. Furthermore, the amount by which X of (4) affects D can be varied by a simple weighting to tune the strictness of the exclusion.
5
Conclusions
We have shown how easily and effectively our system addresses the concept of color exclusion in a color image query. It is incorporated into the overall similarity calculation of each candidate image in a given query and does not require a post-processing stage to filter out images which contain a color to be
82
Dimitrios Androutsos et al.
excluded. This is accomplished by virtue of the multidimensional query space which the distance measures of the query vectors span and their relation to the equidistant line. The similarity of specified exclusion colors to indexed database colors affects the overall ranking by effectively lowering the rank of a given image which contains color that should be excluded. In our system, any number of colors can be excluded in a given query to provide greater flexibility in how a user query is defined, to ultimately retrieve more valid images from a given database.
References 1. V. N. Gudivada and V. V. Raghavan, “Content-based image retrieval systems,” Computer 28, September 1995. 76 2. M. J. Swain and D. H. Ballard, “Color indexing,” International Journal of Computer Vision 7(1), 1991. 76 3. M. Stricker and M. Orengo, “Similarity of color images,” in Storage and Retrieval for Image and Video Databases III, Proc. SPIE 2420, pp. 381–392, 1995. 76 4. W. Niblack, R. Barber, W. Equitz, M. Flickner, Glasman, D. Petkovic, P. Yanker, C. Faloutsos, and G. Taubin, “The qbic project: Querying images by content using color, texture and shape,” in Storage and Retrieval for Image and Video Databases, M. H. Loew, ed., Proc. SPIE 1908, 1993. 77 5. J. R. Smith and S. F. Chang, “Visualseek: a fully automated content-based image query system,” in ACM Multimedia Conference, November 1996. 77 6. X. Wan and C.-C. J. Kuo, “Color distribution analysis and quantization for image retrieval,” in Storage and Retrieval for Image and Video Databases IV, Proc. SPIE 2670, pp. 8–16, 1995. 7. D. Androutsos, K.N. Plataniotis and A.N. Venetsanopoulos, “Distance Measures for Color Image Retrieval,” International Conference on Image Processing ’98, Chicago, USA, October 1998. 78 8. D. Androutsos, K.N. Plataniotis, A.N. Venetsanopoulos, “A Vector Angular Distance Measure for Indexing and Retrieval of Color,” Storage & Retrieval for Image and Video Databases VII, San Jose, USA, January 26-29, 1998. 77 9. D. Androutsos, K.N. Plataniotis, A.N. Venetsanopoulos, “A Perceptually Motivated Method for Indexing and Retrieval of Color Images,” International Conference on Multimedia Computing Systems 1999, Florence, Italy, June 7-11, 1999. Submitted. 78
Generic Viewer Interaction Semantics for Dynamic Virtual Video Synthesis Craig A. Lindley1 and Anne-Marie Vercoustre2 1
CSIRO Mathematical and Information Sciences Locked Bag 17, North Ryde NSW 2113, Australia Phone: +61-2-9325-3150, Fax: +61-2-9325-3101 [email protected] 2
INRIA-Rocquencourt, France [email protected]
Abstract. The FRAMES project is developing a system for video database search, content-based retrieval, and virtual video program synthesis. For dynamic synthesis applications, a video program is specified at a high level using a virtual video prescription. The prescription is a document specifying the video structure, including specifications for generating associative chains of video components. Association specifications are sent to an association engine during video synthesis. User selection of a virtual video prescription together with the default behavior of the prescription interpreter and the association engine define a tree structured search of specifications, queries, and video data components. This tree structure supports generic user interaction functions that either modify the traversal path across this tree structure, or modify the actual tree structure dynamically during video synthesis.
Introduction
The FRAMES project is developing a system for video database search, content-based retrieval, and virtual video program synthesis. The FRAMES project has been carried out within the Cooperative Research Centre for Advanced Computational Systems established under the Australian Government's Cooperative Research Centres Program. Video components within the FRAMES database are described in terms of a multi-layered model of film semantics, derived from film semiotics. For dynamic video program synthesis applications, a program is specified at a high level using a virtual video prescription (Lindley and Vercoustre, 1998a). Coherent sequences of video are required, rather than just lists of material satisfying a common description. To meet this requirement, the FRAMES system uses an engine for generating associative chains of video sequences, initiated by an initial specification embedded within a virtual video prescription. Once a virtual video prescription has been Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp.83-90, 1999. Springer-Verlag Berlin Heidelberg 1999
84
Craig A. Lindley and Anne-Marie Vercoustre
selected, the prescription interpreter and associated instruction processing functions can be allowed to generate a virtual video with no further interaction from the viewer. In this case the resulting presentation has the form of a traditional linear film or video. However, depending upon the viewer’s overall purpose, it may be desirable to steer the ongoing presentation in various ways. For example, the user may wish to steer the presentation towards subjects of interest and away from those of less interest, gain an overview of the area, go into detail, or follow a particular mood or emotion. This paper defines generic user interaction semantics for dynamic virtual video synthesis based upon the data structures and sequencing functions of the FRAMES system. The semantics provide run-time interactions for the viewers of a virtual video; the interactions do not result in any permanent changes to the data structures involved, but affect the way those data structures are used to generate a particular video presentation. We begin with a summary of FRAMES system users and user tasks, provide an overview of the FRAMES system, and summarise the processes that are used to select video components during the generation of a synthesised video sequence. The high level algorithm used within the FRAMES association engine is described, and is seen to define a tree-structured search through the available video components. User interaction semantics are then analysed in terms of generic user interaction strategies, the default data structure that models the selection action of the synthesis engine, and generic interaction operations that can be defined in terms of their effect upon the implied data structure.
FRAMES System Users and User Tasks The FRAMES video synthesis process implies four different author/system user roles that may be involved in the production and use of a virtual video. Within the FRAMES system, video data is a primitive (atomic) data input, organised as a set of discrete video sequences. The video maker may use a variety of software tools and products to create these digital video clips. Interactive video systems that support interaction within a complete video program represent a new medium requiring customised development of video data. The FRAMES video synthesis engine operates upon descriptions associated with raw video data. Hence once the video data is available, a description author must develop a descriptor set and associate descriptors with appropriate video data sequences. The FRAMES environment includes data modeling interfaces to support this authoring process. The interfaces and underlying database are based upon the semiotic model described by Lindley and Srinivasan (1998). Once the descriptions have been created, they are stored in the FRAMES database for use by the video synthesis engine.
Generic Viewer Interaction Semantics for Dynamic Virtual Video Synthesis
85
The FRAMES system can be used with these semantic descriptions to provide basic semantic search and retrieval services, where a user can directly interrogate the database using relational parametric queries, or interrogate the database via the FRAMES association engine either to conduct fuzzy parametric searches, or to generate an associative chain of video components. However, for many users and applications a specific high level program structure may be required. Such a structure can be defined using a virtual video prescription. A prescription, defined by a virtual video prescription author, contains a sequence of embedded queries for generating the low level video content, where the particular order, form, and content of the queries implements a specific type, genre and style of video production. The final end user/viewer community is the audience for whom the virtual video production is created. Such a user will typically select a virtual video prescription according to their current tasks and needs, and use the FRAMES virtual video synthesis engine to generate a virtual video presentation. For dynamic virtual video synthesis, there are a number of ways and points in the process where viewer interaction is meaningful. All viewer interaction functions may be available to the authors of the interaction system, to provide feedback to authors about the appropriateness and effectiveness of descriptions and prescriptions as they are being developed. The authoring process for interactive virtual videos is highly complex, and requires careful coordination between the video makers, description authors, and prescription authors to ensure that these three levels of content are compatible and function correctly to produce coherent viewer sequences. Understanding the principles for doing this effectively is an important topic of ongoing research.
The FRAMES Video Synthesis System
The FRAMES system consists of three primary elements: a virtual video prescription interpreter, a database containing semantic descriptions of individual video components, and the instruction engines for generating sequences of video data. A virtual video prescription represents a high level structure of, or template for, a video program of a particular type, containing a list of instructions for generating a virtual video production (Lindley and Vercoustre, 1998a). The virtual video interpreter reads virtual video prescriptions. A user may select a prescription, which may have values assigned to various embedded parameters to reflect the particular requirements and interests of that user before being forwarded to the interpreter. The interpreter reads the instructions within a prescription sequentially, routing each instruction in turn to an appropriate processor. Three types of instructions may occur within a prescription: direct references to explicitly identified video components, parametric database queries, and specifications for generating an associative chain of video components (Lindley, 1998). Access by direct reference uses an explicit, hard-coded reference to a video data file plus start and end offsets of the required segment (eg. using the referencing syntax of SMIL, Hoschka 1998). Parametric database queries may
86
Craig A. Lindley and Anne-Marie Vercoustre
include complex logical conditions or descriptor patterns. In parametric search, the initial query may form a hard constraint upon the material that is returned, such that all of its conditions must be satisfied. Alternatively, a ranked parametric search can return a list of items ranked in decreasing order of match to the initial query, down to some specified threshold. Access by associative chaining is a less constrained way of accessing video data, where material may be incorporated on the basis of its degree of match to an initial search specification, and then incrementally to successive component descriptions in the associative chain. Associative chaining starts with specific parameters that are progressively substituted as the chain develops. At each step of associative chaining, the video component selected for presentation at the next step is the component having descriptors that most match the association specification when parameterised using values from the descriptors attached to the video segment presented at the current step. The high-level algorithm for associative chaining is: 1. initialise the current state description according to the associative chaining specification. The current state description includes: • the specification of object, attribute, and entity types that will be matched in the chaining process, • current values for those types (including NULL values when initial values are not explicitly given or components of the next instantiation are NULL), • conditions and constraints upon the types and values of a condition, and • weights indicating the significance of particular statements in a specification 2. Generate a ranked list of video sequences matching the current state description. 3. Replace the current state description with the most highly ranked matching description: this becomes the new current state description. 4. Output the associated video sequence identification for the new current state description to the media server. 5. If further matches can be made and the termination condition (specified as a play length, number of items, or associative weight threshold) is not yet satisfied, go back to step 2. 6. End. Since association is conducted progressively against descriptors associated with each successive video component, paths may evolve significantly away from the content descriptions that match the initial specification. This algorithm (described in detail in Lindley and Vercoustre, 1998b) has been implemented in the current FRAMES demonstrator. Specific filmic structures and forms can be generated in FRAMES by using particular description structures, association criteria and constraints. In this way the sequencing mechanisms remain generic, with emphasis shifting to the authoring of metamodels, interpretations, and specifications for the creation of specific types of dynamic virtual video productions.
Generic Viewer Interaction Semantics for Dynamic Virtual Video Synthesis
87
Generic Interaction Strategies User interaction in the context of dynamic virtual video synthesis can take place at several levels, and in relation to several broad types of user task. Canter et al (described in McAleese, 1989) distinguish five discernible strategies that users may use in moving through an information space: 1. 2. 3. 4. 5.
scanning: covering a large area without depth browsing: following a path until a goal is achieved searching: striving to find an explicit goal exploring: finding out the extent of the information given wandering: purposeless and unstructured globetrotting
These strategies are all relevant to interaction with dynamic virtual video synthesis, and the interactive presentation system for virtual videos should support each strategy. To these five strategies we can also add: 6. viewing: allowing the algorithm to generate a video sequence without further direction from a user (ie. the viewer is passively watching a video) Dynamic virtual video syntheses in the FRAMES project uses the viewing model as the default behavior of the system. That is, once a virtual video prescription has been selected, the synthesiser generates the video display based upon that prescription and the semantics defined by the underlying algorithms. The virtual video prescription may define a video program amounting to a scan, browse, search, exploration of, or wander through the underlying video database, depending upon the applicationspecific purpose of the prescription. To provide interactive viewing functions, suitable interfaces must be provided allowing viewers to modify the behavior of the video synthesis engine away from this default behavior within the form defined by the original virtual video prescription.
User Interaction Semantics A prescription can be customised for a particular user by setting its parameter values. Parametric search may be an exact search mechanism (eg. if a traditional relational database is used), or may involve a fuzzy search process that returns identifiers of video component having descriptors that approximately match the search query, ranked in decreasing order of match to the query. A video synthesis system incorporating ranked search can include interfaces allowing users to select from the ranked list of returned results. Associative chaining can be modified in several ways by user interactions, by using user interactions to effectively modify the chaining specification dynamically as chaining proceeds. Users can modify the entity types used to associate the current component with the next component, modify the current
88
Craig A. Lindley and Anne-Marie Vercoustre
entity values, set or reset constraints upon entity values, or modify the weightings upon entity types. Users can also interrupt the default selection of the most highly associated video component by selecting another ranked element as the current element, which will re-parameterise the associative chaining specification at the current point in the chain.
S1
Prescription P1 instr 1 instr 2 S2 . . . Prescription
Fig. 1.
Associative Chain Components: C1 C2 . . . Cp
Ranked components not selected:
S3
C1,1 C1,2 . . . C1,q
S4
Prescription
The semantics of these user interactions can be modeled by regarding the operation of the association engine as a tree search behaviour, as shown on Figure 1. In considering choices that can be made by users, it is useful to regard the starting point as the decision about which virtual prescription to execute, this being the root node of the search tree. Each prescription contains a list of instructions that constitute its child nodes. The algorithm that interprets prescriptions will execute each instructions in sequential order. An instruction (specifically, an instruction that is an association specification) generates a series of video components that are its child nodes in turn, each component being presented for display in the sequential order in which it is returned. Finally, for each selected video component in a series, there is a list of other candidate components that have not been selected, ranked in decreasing order of associative strength (to the previous component in the selected list); this ranked list may be considered to be a set of child nodes for a selected component. Hence the video synthesis process defines an ordered, depth-first traversal of the system data structures and the dynamically generated association structure of video components. The default behavior of the synthesis engine without user interaction satisfies the user interaction strategy identified above as viewing. However, to support scanning, browsing, searching, exploring, and wandering strategies, specific and generic interaction functions can be provided. These are divided into two classes. The first class of interaction functions are those that determine the path taken by the user in traversing the default synthesis tree amount to functions that interrupt or modify the default depth-first traversal behavior of the algorithm. These functions include: • control of whether the process should stop, loop back to some point (eg. as identified on a history list), or proceed to the next default item
Generic Viewer Interaction Semantics for Dynamic Virtual Video Synthesis
89
• jump to a position on the tree other than the next position defined by the depth-first algorithm • display a set of video components in parallel The second class of interaction functions are those that dynamically alter the structure of the default tree during video synthesis are functions that effectively produce an alteration in the specification that is driving the generation of a virtual video production. This can include: • functions that dynamically modify virtual video prescriptions (eg. changing the values of variables used within a prescription during execution) • functions that dynamically modify queries prior to their execution, or as they are being executed. Examples include adding or removing descriptor types that associative matching is taking place against, and modifying the weightings attached to descriptor types.
Related Work Interactive video covers a broad range of technologies and interests, including interactive video editing systems, model-based video image generation, and interactive search and browsing of video data in archives or databases. The FRAMES project is addressing the interactive use of predefined video sequences. Dynamic access to predefined video using content-based retrieval techniques has generally been based upon an information retrieval model in which data is generated in response to a single query (eg. the IBM QBIC system, http:// wwwqbic.almaden.ibm.com/ stage/ index.html); sequencing from this perspective is a contextual task within which content-based retrieval may take place. The MOVI project has incorporated some automated video analysis techniques into an interactive video environment that then uses hard-coded links between video elements (see http:// www.inrialpes.fr/ movi/ Demos/ DemoPascal/ videoclic.html). Unlike these approaches, FRAMES generates links between video sequences dynamically using an associative chaining approach similar to that of the Automatist storytelling system developed at MIT (Davenport and Murtaugh, 1995, and Murtaugh, 1996). The Automatist system uses simple keyword descriptors specified by authors and associated with relatively self-contained video segments. In Automatist, users can interact with the associative chaining process either by explicitly modifying the influence of specific keyword descriptors arranged around the periphery of the interface, or by selecting a less strongly associated video component to become the current displayed component determining the ongoing associative chain. The FRAMES system extends this associative chaining approach by using a highly structured semantic model (described in Lindley and Srinivasan, 1998), which allows greater discrimination on descriptor types, and more types of relationship between sequenced video components. Flexible and modifiable association specifications in FRAMES and the incorporation of direct references and parametric queries in high
90
Craig A. Lindley and Anne-Marie Vercoustre
level prescriptions create opportunities for interaction beyond the simple selection of keywords and ranked components.
Conclusion This paper has presented an analysis of the underlying semantics of user interaction in the context of the FRAMES dynamic virtual video sequence synthesis algorithms. Ongoing research is addressing the presentation of interaction options to users, and the problem of disorientation within the unfolding interactive video.
References Aigrain P., Zhang H., and Petkovic D. 1996 “Content-Based Representation and Retrieval of Visual Media: A State-of-the-Art Review”, Multimedia Tools and Applications 3, 179-202, Klewer Academic Publishers, The Netherlands. Davenport G. and Murtaugh M. 1995 “ConText: Towards the Evolving Documentary” Proceedings, ACM Multimedia, San Francisco, California, Nov. 5-11. Hoschka P.(ed) 1998, “Synchronised Multimedia Integration Language (SMIL) 1.0 Specification” W3C Recommendation 15 June 1998. Lindley C. A. 1998 “The FRAMES Processing Model for the Synthesis of Dynamic Virtual Video Sequences”, Second International Workshop on Query Processing in Multimedia Information Systems (QPMIDS) August 26-27th 1998 in conjunction with 9th International Conference DEXA98 Vienna, Austria. Lindley C. A. and Srinivasan U. 1998 “Query Semantics for Content-Based Retrieval of Video Data: An Empirical Investigation”, Storage and Retrieval Issues in Image- and Multimedia Databases, August 24-28, in conjunction with 9th International Conference DEXA98 Vienna, Austria. Lindley C. A. & Vercoustre A. M. 1998a “Intelligent Video Synthesis Using Virtual Video Prescriptions”, Proceedings, International Conference on Computational Intelligence and Multimedia Applications, Churchill, Victoria, 9-11 Feb., 661-666. Lindley C. A. & Vercoustre A. M. 1998b “A Specification Language for Dynamic Virtual Video Sequence Generation”, International Symposium on Audio, Video, Image Processing and Intelligent Applications, 17-21 August, Baden-Baden, Germany. McAleese R. 1989 “Navigation and Browsing in Hypertext” in Hypertext theory into practice, R. McAleese ed., Ablex Publishing Corp., 6-44. Murtaugh M. 1996 The Automatist Storytelling System, Masters Thesis, MIT Media Lab, Massachusetts Institute of Technology.
Category Oriented Analysis for Visual Data Mining H. Shiohara, Y. Iizuka, T. Maruyama, and S. Isobe NTT, Cyber Solutions Laboratories 1-1 Hikarinooka, Yokosuka-shi, Kanagawa, 239 JAPAN TEL: +81 468 59 3701, FAX: +81 468 59 2332 {shiohara,iizuka,maruyama,isobe}@dq.isl.ntt.co.jp
Abstract. Enterprises are now storing large amount of a data and data warehousing and data mining are gaining a great deal of attention for identifying effective business strategies. Data mining extracts effective patterns and rules from data warehouses automatically. Although various approaches have been attempted, we focus on visual data mining support to harness the perceptual and cognitive capabilities of the human user. The proposed visual data mining support system visualizes data using the rules or information induced by data mining algorithms. It helps users to acquire information. Although existing systems can extract data characteristics only from the complete data set, this paper proposes a category oriented analysis approach that can detect the features of the data of associated with one or more particular categories.
1
Introduction
The great evolution in computing power has enabled businesses to collect and store copious amounts of data. As competition between enterprises intensifies, it is more important for the business strategy to be based on real data. Data mining has thus attracted attention as a way of obtaining such knowledge. Data mining can extract rules from copious amounts of data or classify data by using algorithms established in the field of artificial intelligence. Although it’s suitable for handling copious amounts of data, its algorithms are very difficult to use if the user is not familiar with data analysis. We developed a visual data mining support system that combines data mining algorithms with visualization for better usability. Because effective visualization is needed to help users to discover rules/patterns, the selection of the attribute(s) to be the visualization target is very important. In our system, attribute selection is performed automatically by utilizing data mining methods. The 3 main functions of our system are as follows. 1. extract data characteristics by applying data mining 2. select the effective attributes to be visualized based on the extracted characteristics Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 91–98, 1999. c Springer-Verlag Berlin Heidelberg 1999
92
H. Shiohara et al.
3. create visual representations effectively by mapping the selected attributes to parameters of a plot profile The resulting visualization makes it easier for the user to put forward hypotheses. Existing systems apply data mining only to all data which can lead to the significant characteristics of partial data sets being overlooked. Accordingly, we add category analysis to the existing method to create a more powerful visual data mining support system. This article overviews the visual data mining support method in section 2, and section 3 introduces the new attribute selection method, the category oriented selection method. The remaining sections describe effective visualization offered by this method and some application examples.
2
Visual Data Mining Support Method
The visual data mining support method should help the user in discovering rules/patterns in voluminous data sets easily by combining powerful data mining algorithms with user-friendly visualization techniques. Human beings can well grasp data trends if they are well visualized. Furthermore, when they have background knowledge, they can understand what the trends mean in the real world. If the data set is too large, it is not easy even to guess which attributes are correlated which degrades analysis performance. Therefore, we focused on the selection of the attributes to be visualized and adapted data mining algorithm to support visual analysis. 2.1
Multi-dimensional Visualization System
As the visualization environment, we developed INFOVISER, a multi-dimensional visual analysis tool. INFOVISER transfers character-based data into graphical information and visualizes one record of data as one element such as a node or line. Each attribute of data is mapped to one parameter of a plot profile such as axes, size, color, shape, and so on. This yields multi-dimensional and powerful visualization that is impossible with ordinary bar or circular charts. The degree of multi-dimensionality that can be visualized at once is about 10, and business data have many more attributes. This makes it difficult to manually find the attributes that are correlated. 2.2
Visualized Attribute Selection Method
As described before, the key to visual data mining support is how to find the effective target to be visualized. Selection of a group of attributes that are strongly correlated can lead to the discovery of new rules/patterns. For this purpose, the support system extracts characteristics from data, and selects target attributes based on these characteristics. The user evaluates the visualization results and finds a clue to the next analysis step or, may change
Category Oriented Analysis for Visual Data Mining
93
the extraction process to generate another visualization result. This integration of machine and human analysis is the key to our proposal. Existing attribute selection methods use decision trees or correlation coefficients for extracting the data characteristics of all data. The decision tree method is effective especially when there is an analysis target attribute. The attributes, listed in the tree created by the decision tree algorithm, are selected as visualization targets in order of hierarchy from the root node. When there is no analysis target or no clue, a correlation coefficient matrix is effective. The pairs of attributes having higher correlation coefficients are selected as visualization targets. These methods can overcome the difficulty of setting hypotheses due to excessive attribute number. The system configuration is depicted in Fig.1
Fig. 1. System Configuration of Visual Data Mining Support System
3
Category Oriented Attribute Selection Method
3.1
Requirements
The following are examples of visualization results manually generated using INFOVISER ’s GUI. sumo wrestler a high rank wrestler have good balance of height and weight.(Fig.2) medical checkup 1 a heavy drinker has high γGTP value and high blood sugar level.(Fig.3)
Fig.2. sumo wrestler
Fig.3. medical checkup 1
Fig.4. medical checkup 2
94
H. Shiohara et al.
medical checkup 2 in the case of a light drinker, the person’s obesity is inversely proportional to how much he smokes.(Fig.4) In these examples, the attributes that seem to influence the attribute that attracted the user’s interest like rank of wrestler and obesity, are visualized using particular patterns. However, these attributes were not selected by the existing selection method. Fig.5 depicts the decision tree for medical checkup data whose target is “Drinking”. Because attributes like γGTP or blood sugar or smoke or obesity don’t have high rank, they aren’t visualized. Even the correlation coefficient method didn’t select these attributes because of their small absolute values. ( Table 1) The reason seems to be that these characteristics are seen only Fig. 5. Result of Decision Tree as part of the data and are lost when extraction is applied uniformly to the whole set. Human beings can detect distinguishing patterns in the visualized figures by their pattern recognition ability and induce rules from their background knowledge. For selecting attributes like above examples automatically, it is necessary to detect data subset including prominent patterns. In order achieve this, we propose to combine data categorization with the attribute selection process. That is, grouping data according to some guideline, characterizing each data group, and extracting attributes that have remarkable patterns compare to the complete data set. This should satisfy the following requirements. TREE Root |--TREE NODE 1 | if Sex in { 2 } | |--IF Systolic Pressure <= 157 | | THEN Drinking = 2.375 | +--IF Systolic Pressure > 157 | THEN Drinking = 1 +--if Sex not in { 2 } |--if Cholesterol <= 222.5 | |--if Diastolic Pressure <= 68 | | |--IF Age <= 45 | | | THEN Drinking = 3 | | +--IF Age > 45 | | THEN Drinking = 1 | +--IF Diastolic Pressure > 68 | THEN Drinking = 3.04762 +-IF Cholesterol > 222.5 THEN Drinking = 3.95238
1. able to reflect user’s interest in data categorization 2. able to evaluate partial characteristics for attribute selection 3.2
Data Categorization
The direct way to reflect the user’s interest is to make the analysis target attribute a guideline for categorization. The attributes that have special values within in a certain category, can be considered as those that are correlated to the user’s interest. As for the user’s interest, there are several cases. · there is only one target attribute Table 1. Correlation Coefficient Attribute Correlation Coefficient
Age Height Weight Systolic pressure 0.47 0.32 0.07
-1.37
Diastolic Choles pressure -terol 0.18 0.16
Blood sugar 0.13
COT GPT γGTP Obesity degree Smoke 0.21 0.20 0.24 0.20 0.47
Category Oriented Analysis for Visual Data Mining
95
· there are multiple target attributes · there is no target attribute ( target attribute is not clear) With just one analysis target attribute, categorization is achieved by dividing the data into groups by setting discrimination levels or simply into an equal number. When there are multiple target attributes, such a naive method is not desirable. Categorization that takes account of the correlation of the attributes like multi-dimensional clustering is more suitable. How then can we categorize data if the user does not select any target attributes ? In this case, we use factor analysis ( a statistical technique ) to divide the data attributes into target attributes and explanation attributes (or dependent variables and independent variables) at first. 3.3
Attribute Selection
This section discusses how to evaluate, for attribute selection, the characteristics extracted from each category. As for the correlation coefficient method, a pair of attributes that shows a different trend from the remaining data is taken as the characteristic, rather than just considering high values within one category. That is, a pair of attributes that have low correlation value in the whole data set may show high correlation in a certain category, or show an inverse correlation. We formalize these behaviors into the following expression to evaluate how characteristic the category is. 1 f1 (rA , rp , np ) = (1 − √ )rp (rp − rA ) np np is the number of partial data, rA is the correlation of all data, rp is the correlation of partial data set. In the same way, data distribution can be used for characterizing categories. That is, locating attributes whose distribution is much different from those of the whole data set using basic statistical values such as average and standard deviation. We use the following expression to compare and evaluate the strength of the characteristics created by the categories and attributes. 1 mp − mA f2 (np , mp , sp , mA ) = (1 − √ ) np sp np is the number of partial data, mp is the average of partial data set, mA is average of all data, sp is standard deviation of partial data. Attributes are selected in order of the evaluation of characteristics.
4 4.1
Visualization Method Scatter Chart Representation
In this category oriented analysis method, the visualization target attributes are classified into two groups that are mapped as follows.
96
H. Shiohara et al.
Fig.6 Visualization Result 1
Fig.7 Visualization Result 2
· attributes that categorize data: targeted attributes → color, shape · attributes that characterize data: explanative attributes → X, Y-axis In this method, from which category the characteristics are extracted is significant information, as is the attributes themselves, so we represent the categorizing attributes by color, category by shape, and the extracted attributes as the X-axis and Y-axis. 4.2
Aggregation Representation
When a large amount of data is visualized, figures overlap because the display area is not infinite. This overlapping causes not only visual confusion but also a loss of information. In order to avoid this problem, the number of figures visualized on screen is reduced. One way is to thin the number of data records, but it is possible the remaining data doesn’t retain the original trend. Another way is to summarize neighboring data and this is more desirable from the viewpoint of analysis. It is common to combine figures that are close in ordinary scatter charts. In INFOVISER, however, profiles such as color, shape, size, have meaning. So, if these profiles are ignored and only position is considered when figures are summarized, the data profile information is lost and can’t be evaluated. Therefore, we suppose a virtual space where in all profiles are treated equally, and summarize by distance in this space. This enables visualization by fewer figures without losing the significance of the trends of all data.
5
Application Examples
This section shows results of applying the category oriented methods and the visualization method to test data consisting of medical checkup data. (3600 records with 24 attributes). In this article, only cases of one target attribute and no attribute are shown.
Category Oriented Analysis for Visual Data Mining
97
Case of one target attribute We selected “smoking” as the target attribute. The result of categorization by equal division of value into 6 groups was evaluated using equation f1 and f2 . The following features were extracted. Correlation coefficient matrix: results · as smoking rate increases, the correlation of drinking and uric acid becomes stronger. · as smoking rate increases, the inverse correlation of obesity and HDL cholesterol(good cholesterol) increases. Basic statistics: results · heavy smokers have lower HDL cholesterol and more neutral fat. · light smokers have lower γGTP value ( means healthy hepatic function). Medically it is said that HDL cholesterol reduces as the smoking rate increases, or uric acid increases as the rate of drinking increases. In this result, when the degree of smoking or obesity is large, this phenomenon is seen strongly. Fig.6 and 7 show visualization results. ( Count as size, more smoke as more dark color, and category as shape) Case of no target attribute The top 4 results of factor analysis of the test data are shown in Table 2. By performing categorization using multi-dimensional clustering, and using attributes highly correlated with the first factor, the following features were extracted. Correlation coefficient matrix: results · in the highest factor scoring group (cluster 6), the correlation of total cholesterol and GPT(hepatic index) is high and meal frequency is inversely proportional to the drinking rate. · in the high factor scoring group (cluster 1), the rate of eating before sleep is inversely correlated to meal frequency. Basic statistics: results · in the highest factor scoring group(cluster 6) heavy drinking is common, smoking rate is high, and exercise and sleeping hours are small. · in a high factor scoring group (cluster 1), smoking rate is high. In this case, the height of factor scoring is interpreted as an index of poor healthy.
6
Discussion
We ascertained the effective attributes and visualization results were obtained by applying the proposed method to test data with enough records. A shortcoming is that the visualization result may not be so comprehensible even if the numerical value is significant. That is due to the relative low value of the correlation coefficient or a small difference in distribution. The existing method can generate very plain visualizations. One of the examples generated by the
98
H. Shiohara et al.
existing method indicates that systolic pressure and diastolic pressure are almost proportional and both are highly correlated with obesity as shown in Fig8. However, the proposed method can identify conspicuous characteristics.
7
Conclusion
This article proposed a category oriented analysis method that can detect the strength of the characteristics of different categories, and confirmed that it effectively supports visual data mining. In the future, we will examine a user interpretation support function and other characterization methods.
References 1. K.Kurokawa, S.Isobe, H.Shiohara, “Information Visualization Environment for Character-based Database Systems” VISUAL ’96, pages 38-47, Feb. 1996. 2. Y.Iizuka, et al., “Automatic Visualization Method for Visual Data Mining”, Lecture Notes in Artificial Intelligence Vol.1394, PAKDD-98, pp.174-185, Apr. 1998. 3. B.H.MacCormik, T. A. DeFanti and M.D.Brown, eds., “Visualization in Scientific Computing,” Computer Graphics, Vol.21, No.6, ACM Siggraph, Nov. 1987. 4. A.S.Jacobson, A.L.Berkin and M.N.Orton, “Linkwinds: Interactive Scientific Data Analysis and Visualization”, Communications of the ACM, Vol.37, No.4, Apr.1994. 5. U. M. Fayyad and E. Simoudis, “Knowledge Discovery in Databases”, Tutorial Notes, 14th International Joint Conference on Artificial Intelligence (IJCAI-95), 1995. 6. U. M. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, “Advances in Knowledge Discovery and Data Mining”, AAAI/MIT Press, 1995. 7. D. A. Keim, “Database and Visualization”, Tutorial Notes, ACM-SIGMOD’96, 1996.
User Interaction in Region-Based Color Image Segmentation Nicolaos Ikonomakis1, Kostas N. Plataniotis2 , and Anastasios N. Venetsanopoulos1 1
Department of Electrical & Computer Engineering Digital Signal & Image Processing Lab University of Toronto 10 King’s College Road, Toronto, Ontario, M5S 3G4, Canada {minoas,anv}@dsp.toronto.edu WWW:http://www.dsp.toronto.edu 2 School of Computer Science Ryerson Polytechnic University 350 Victoria Street, Toronto, Ontario, M5B 2K3, Canada [email protected]
Abstract. An interactive color image segmentation technique is presented for use in applications where the segmented regions correspond to meaningful objects, such as for image retrieval. The proposed technique utilizes the perceptual HSI (hue,saturation,intensity) color space. The scheme incorporates user interaction so that the best possible results can be achieved. Interaction with the user allows the segmentation algorithm to start efficiently and to refine the results. Interaction is performed for the key images (usually the first image or those where new ojects enter the scene) of the video-telephony sequence. The user is allowed to identify on the screen the relevant regions by marking their seeds. The user guidance can be given by the sender, by the receiver, or by both. The effectiveness of the algorithm is found to be much improved over techniques used in the past.
1
Introduction
Image segmentation refers to partitioning an image into different regions that are homogeneous or ”similar” in some image characteristic. It is an important first task of any image analysis process, because all subsequent tasks, such as feature extraction and object recognition, rely heavily on the quality of the segmentation. Image segmentation has taken a central place in numerous applications, including, but not limited to, multimedia databases, color image and video transmission over the Internet, digital broadcasting, interactive TV, video-ondemand, computer-based training, distance education, video-conferencing and
Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 99–106, 1999. c Springer-Verlag Berlin Heidelberg 1999
100
Nicolaos Ikonomakis et al.
tele-medicine. For some specific applications, it is possible to find an automated analysis process that performs segmentation as desired. This may be the case of a surveillance application, where the automated analysis system provides the video encoder detailed information about the object of importance, allowing the selective coding of the scene. A simple, real-time, and automated analysis process based on the detection of moving objects may be used. However, for many applications (multimedia databases, web-based search engines, video-conferencing, tele-medicine, etc.), fully automated analysis schemes provide only part of the desired analysis results [1,2]. The performance of an automatic segmentation scheme on complex video scenes (i.e. lots of background objects) may not produce the desired results. For these applications, user interaction is imperative so that the achieved results can have a meaningful and powerful semantic value. For this reason, more recent research is given to interactive and “human in the loop” systems [1,2,3,4,5]. The QBIC (Query By Image Content) team [5] uses interactive region segmentation for image retrieval purposes. In this paper, an interactive color image segmentation scheme is proposed that employs the perceptual HSI (Hue, Saturation, Intensity) color space to segment color images. More specifically, the proposed scheme is developed for implementation in applications where the segmented regions should correspond to meaningful objects, such as image retrieval or video-telephony type sequences. The region-based segmentation scheme first employs an initial user interaction seed determination technique to find seed pixels to be used in a region growing algorithm. Initial user interaction also includes selecting several values for threshold parameters used in the region growing algorithm. Following the automatic growing of the regions a supervised region merging algorithm is employed to refine the results of the segmentation. The next section explains the segmentation scheme. This is followed by the results and conclusions.
2
Color Image Segmentation
The segmentation scheme presented utilizes the HSI color space, and thus, the color values of the pixel are first converted from the standard RGB (red, green, blue) color values to the HSI color values using well known transformation formulas [6]. The scheme can be split into four general steps: 1. The pixels in the image are classified as chromatic or achromatic pixels by examining their HSI color values. 2. User classifies seed pixels in image. 3. The region growing algorithm is employed on the chromatic and achromatic pixels separately starting from the seed pixels. 4. Regions are merged through user interaction. The region growing algorithm has been presented in the past [7], but with arbitrary unsupervised seed determination. The automatic method gave good
User Interaction in Region-Based Color Image Segmentation
101
results but still needed improvement. Automatic seed determination is one of the most difficult problems in color image segmentation [6]. A good seed pixel is the pixel with the most dominant color and is usually the center pixel of the region. Thus, to determine them, an initial segmentation of the image is needed to find the regions. The new seed determination method presented in this paper constitutes an initial user interaction process. Because the human visual system can segment an image automatically with little or no hesitation [8], the user can achieve the initial segmentation and, thus, determine the best starting pixels. Each step in the above generalization is explained in the following sections. Due to limitation in space, experimental results for the Claire video-telephony type image will only be discussed. 2.1
Chromatic/Achromatic Separation of Pixels
The HSI color model corresponds closely to the human perception of color [6]. The hue value of the pixel has the greatest discrimination power among the three values because it is independent of any intensity attribute. Even though hue is the most useful attribute, there are two problems in using this color value: hue is meaningless when the intensity is very low or very high; and hue is unstable when the saturation is very low [6]. Because of these attributes, in the proposed scheme, the image is first divided into chromatic and achromatic regions by defining effective ranges of hue, saturation, and intensity values. Since the hue value of a pixel is meaningless when the intensity is very low or very high the achromatic pixels in the image are defined as the pixels that have low or high intensity values. Pixels can also be categorized as achromatic if their saturation value is very low, since hue is unstable for low saturation values. From the concepts discussed above, the achromatic pixels in the HSI color space are defined as follows: achromatic pixels:(intensity>90) or (intensity<10) or (saturation<10),
(1)
where the saturation and intensity values are normalized from 0 to 100. Only the intensity values of the achromatic pixels are considered when segmenting the achromatic pixels into regions. Pixels that do not fall into this category are categorized as chromatic pixels. For chromatic pixels all three HSI color values are considered in the algorithm. 2.2
Seed Determination
The region growing algorithm starts with a set of seed pixels and from these grows regions by appending to each seed pixel those neighboring pixels that satisfy a certain homogeneity criterion, which will be described later. An interactive algorithm is used to find the ’best’ seed pixels in the image. The ’best’ seed pixels are defined as the pixels that are in the center of their respective object. A benefit of having an initial user interaction is that different levels of
102
Nicolaos Ikonomakis et al.
segmentation can be employed. More specifically, the user can choose two different sets of seed pixels. One for regions that require a low degree of segmentation (i.e. facial areas), and one for regions that require a high degree of segmentation (i.e. background, clothing, etc.). This is beneficial, because areas where some detail is important can be segmented separately from areas where detail is not essential. The user is first asked to select low level seed pixels from the original image. After this, the user is asked to select the high level seeds. It was found that having only two levels makes it quicker and easier for the user to choose the seed pixels. For classical unsupervised seed determination algorithms, this two level seed determination option is not an easy benefit to implement. 2.3
Region Growing
The region growing algorithm starts with the set of seed pixels determined from the user and from these grows regions by appending to each seed pixel those neighboring pixels that satisfy a certain homogeneity criterion. The algorithm is summarized in the Figure 1. The first seed pixel is compared to its 8-connected neighbors: eight neighbors of the seed pixel. Any of the neighboring pixels that satisfy a homogeneity criterion are assigned to the first region. This neighbor comparison step is repeated for every new pixel assigned to the first region until the region is completely bounded by the edge of the image or by pixels that do not satisfy the criterion. The color of each pixel in the first region is changed to the average color of the all the pixels in the region. The process is repeated for the next and each of the remaining seed pixels. If all of the seed pixels are grown and there are still unassigned pixels in the image further region growing segmentation is employed by considering random seed pixels amongst the unassigned pixels. For the chromatic pixels, the homogeneity criterion used for comparing the seed pixel and the unassigned pixel is that if the value of the distance metric used to compare the unassigned pixel and the seed pixel is less than a threshold value Tchrom than the pixel is assigned to the region. Varying the value of Tchrom controls the degree of segmentation with a low value resorting to an over-segmented image and a high value to an under-segmented image. With the two different levels of seeds found, two different values of Tchrom are used. The seeds that require a low degree of segmentation have a threshold lower than the seeds that require a high degree of segmentation. Besides setting the seed pixels, the user initially sets the values of the low and high level thresholds. If the region growing segmentation results with these thresholds are not perfect the user can refine the threshold values, during the segmentation process, until optimal results are obtained. The distance measure used for the chromatic pixels will be referred to as the cylindrical metric. It computes the distance between the projections of the pixel points on a chromatic plane. It is defined as follows[9]: 1 (2) dcyl (s, i) = (dI )2 + (dC )2 2 , with dI = |Is − Ii |
(3)
User Interaction in Region-Based Color Image Segmentation
103
Select next seed pixel
neighbors are 8 neighbors of seed pixel
compare neighbors to the seed pixel with homogeneity criterion
if any neighbor pixels compared satisfy the homogeneity condition
satisfying pixels are assigned to the region and are the new neighbors
TRUE
FALSE
Fig. 1. The region growing algorithm. and
1 dC = (Ss )2 + (Si )2 − 2Ss Si cos θ 2 ,
where
θ=
|Hs − Hi | if 360◦ − |Hs − Hi | if
|Hs − Hi | < 180◦ , |Hs − Hi | > 180◦ .
(4)
(5)
The value of dC is the distance between the 2-dimensional (hue and saturation) vectors, on the chromatic plane, of the seed pixel and the pixel under consideration, as shown in Figure 2. Henceforth, dC combines both the hue and saturation (chromatic) components of the color. An examination of the metric equation (Equation (2)) shows that it can be considered as a form of the popular Euclidean distance (L2 norm) metric. A pixel is assigned to a region if the value of the metric dcyl is less than a threshold Tchrom . In the case of the achromatic pixels, the homogeneity criterion used is that if the difference in the intensity values between an unassigned pixel and the seed pixel is less than a threshold value Tachrom than the pixel is assigned to the seed pixel’s region. That is, if (6) |Is − Ii | < Tachrom, then pixel i would be assign to the region of seed pixel s. 2.4
Region Merging
Following the region growing algorithm for image segmentation a user refinement process is employed. The user supervises the results of the region growing
104
Nicolaos Ikonomakis et al.
R
Si
G
pixel under consideration θ Ss
seed pixel
B
Fig. 2. The chromatic plane of the HSI color model. algorithm, correcting the undesired deviations when needed. The user does this by telling the algorithm which regions in the segmented image to merge into one. This further segments the image into less regions. The user specifies how many regions to be merged and then specifies the regions. The color of the regions that are merged is then changed to the average color of the regions. This is defined as: r k=1 ck Nk (7) average color = r k=1 Nk where r, ck , and Nk are the number of regions being merged, the color vector of region k, and the number of pixels in region k, respectively. This ensures that if a small region is merged with a larger one the average color will be heavily weighted by the larger region.
3
Results and Conclusions
The performance of the interactive color image segmentation scheme was tested with the video-telephony type image Claire. For this type of image, segmentation can be used for the coding and/or compressions of the sequences before transmission. The original Claire image is displayed in Figure 3(a). Interaction with the user allows the automatic analysis process to start efficiently, and to refine the produced results. Interaction is performed for the key images (usually the first image or those where new ojects enter the scene) of the video-telephony sequence. The user is allowed to identify on the screen the relevant regions by marking their seeds. The user guidance can be given by the sender, by the receiver, or by both. Considering the achromatic pixels, it was found that the best results were obtained with a Tachrom threshold value of 15. This is 15% of the maximum achromatic distance. This was the the value used throughout the experimental analysis. Figure 3(b) shows an example of the seed pixels a user may choose for the Claire image. The white pixels refer to the high threshold level seed pixels and the black pixels refer to the low threshold level seed pixels.
User Interaction in Region-Based Color Image Segmentation
(a)
(b)
(c)
(d)
105
(e)
Fig. 3. Claire Image. (a) Original (b) Image showing seeds (white:high level, black:low level) (c) Segmented image before interactive merging (d) Classical segmented image with random seed determination (e) Segmented image after interactive merging.
106
Nicolaos Ikonomakis et al.
Figure 3(c) shows the segmented images, before merging, of Claire with Tchromlow = 10 (5 % of maximum distance) and Tchromhigh = 30 (15% of maximum distance). It was found that these threshold values give the best results for a varied set of images. As can be seen, the facial areas which incorporate a lot of detail are grown through several low level seeds. The background, the jacket, and the hair in the Claire image are segmented using the high level seeds. User refinement is incorporated to merge together similar regions. Figure 3(e) shows the segmented images after merging. As can be seen, the images have less regions. Figure 3(d) shows the segmented images of Claire after segmenting them with the classical method of segmentation with random arbitrary seed determination and no merging. These results are clearly not as good as the one segmented with the interactive segmentation scheme. A color image segmentation scheme using the HSI color space was used to segment video-telephony type images. It incorporated an interactive seed determination algorithm The technique was found to be robust and relatively computationally inexpensive. The effectiveness of the algorithm was found to be much improved over the technique used in the past [7]. Other than video-telephony type images, the proposed interactive scheme can be employed in other applications. These include: web-based search engines, multimedia databases, digital broadcasting, interactive TV, video-on-demand, computer-based training, distance education, and tele-medicine.
References 1. Soltanian, Z.H., Windham, J.P., Robbins, L.,: Semi-supervised segmentation of MRI stroke studies. Proc. of the SPIE vol. 3034 (1997) 437–448. 100 2. Olabarriaga, S.D., Smeulders, W.M.,: Setting the Mind for Intelligent Interactive Segmentation: Overview, Requirements, and Framework. Proc. of the 15th Int. Conf. on Information Processing in Medical Imaging (1997) 417–422. 100 3. Castagno, R., Ebrahimi, T., Kunt, M.,: Video Segmentation Based on Multiple Features for Interactive Multimedia Applications. IEEE Transactions on Circuits and Systems for Video Technology (Sept 1998) vol. 8, no. 5 562–571. 100 4. Correia, P., Pereira, F.,: User Interaction in Content-based Video Coding and Indexing. EUSIPCO III (1998) 2293–2296 . 100 5. Niblack, W., Barber, R., and et al.: The QBIC project: Querying images by content using color, texture and shape. Proc. SPIE Storage and Retrieval for Image and Video Databases (1996). 100 6. Gonzales, R.C., Wood, R.E.,: Digital Image Processing. Addison-Wesley, Massachusetts (1992). 100, 101 7. Ikonomakis, N., Plataniotis, K.N., Venetsanopoulos, A.N.,: A Region-based Color Image Segmentation Scheme. Proc. of the SPIE: Visual Communications and Image Processing vol. 3653 (1999) 1202–1209. 100, 105 8. Marr, D.,: Vision. Freeman, San Fransisco, California (1982). 101 9. Tseng, D.C., Chang, C.H.,: Color Segmentation Using Perceptual Attributes. Proc. 11th Int. Conf. on Pattern Recognition III:Conf c. (1992) 228–231. 102
Using a Relevance Feedback Mechanism to Improve Content-Based Image Retrieval G. Ciocca and R. Schettini Istituto Tecnologie Informatiche Multimediali Consiglio Nazionale delle Ricerche Via Ampere 56, 20131 Milano, Italy {ciocca,centaura}@itim.mi.cnr.it
Abstract. the paper describes a new relevance feedback mechanism that evaluates the distribution of the features of images judged relevant or not relevant by the user, and dynamically updates both the similarity measure and query in order to accurately represent the user's particular information needs. Experimental results are reported to demonstrate the effectiveness of this mechanism.
1. Introduction Content-based retrieval systems operate on image databases to extract relevant images in response to a visual query. Unfortunately, the concept of "relevance" is frequently associated with the semantics of the image, and the encoding and exploitation of semantic information in a general purpose retrieval system is still an unsolved issue. However in practice low-level visual features are often correlated with the semantic contents [1, 8], and image retrieval techniques exploiting these have become a promising research issue [3, 9, 13]. General purpose systems of this type are already available [6, 16]. These systems usually make it possible to extract image descriptions in terms of color, texture, shape and layout features from the images and define the relative search/matching functions that can be used to retrieve those of interest. The performance of an image retrieval system is, consequently, closely related to the nature and quality of the features used to represent image content, but it is not limited to this. Another important issue is that the measure adopted to quantify image similarity is user- and task- dependent [3, 10], and this dependence is not in general understood well enough to permit careful, a-priori selection of the optimal measure. In this paper we present a mechanism that allows the user to query the database and progressively refine the system’s response by indicating the relevance, or irrelevance of the retrieved items. Our approach differs from other recently presented studies [11, 15, 18] in both the strategy for learning the query representing the information needs and the strategy for learning the similarity function. The relevance feedback mechanism presented is a part of a Visual Information Retrieval system currently under development. A full description of all the functionalities of this system is beyond the scope of the paper. We should like to provide here only Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp.107 -114, 1999. Springer-Verlag Berlin Heidelberg 1999
108
G. Ciocca and R. Schettini
some experimental evidence that our retrieval mechanism makes it possible to approximate the user's information needs in a great variety of applications.
2. Image Indexing The absolute performance of an image retrieval system is closely linked with to the nature and the quality of the features used to represent the image content. The features used to index the images in the experiments reported here were calculated on the global image and on 5 sub-images and are: • A Color Coherence Vector (CCV) in the CIELAB color space quantized in 64 colors (pixels in regions whose size exceeds 1% of the image size are counted as coherent pixels) [13]; • A histogram of the transition in colors (CIELAB color space quantized in 11 colors) [7]; • Moments of inertia of the distribution of colors in the unquantized CIELAB color space [17]; • A histogram of contour directions opportunely filtered (only high gradient pixels are considered) and using box widths of 15 [4, 9]; • The mean and variance of the absolute values of the coefficients of the subimages of the first three levels of the multi-resolution wavelet transform of the luminance image [4]; • The neighborhood Gray-Tone Difference Matrix (NGTDM), i.e. coarseness, contrast, busyness, complexity, and strength, as proposed by Amadasum et al. [2]; • The spatial composition of the color regions identified by the process of quantization in 11 colors [4].
3. Relevance Feedback The sub-vectors of features are indicated by Xihs, where i is the vector index, h is the index of the feature, and s is the index of the sub-image to which that feature refers. We indicate with Dhs the distance associated with the feature h-th of region s. In our experiment all the features are compared with the L1 distance measure, as it is statistically more robust than the L2 distance measure [14]. The global metric used to evaluate the similarity between two images of the database is, in general, a linear combination of the distances between the individual features: q
p
i j Sim(X i , X j ) = ∑∑ w hs D hs ( X hs , X hs )
(1)
s =1 h =1
in which the whs are weights. There are two problems in this formulation of image similarity. First, since the single distances may be defined on intervals of widely varying values, they must be normalized to a common interval to place equal
Using a Relevance Feedback Mechanism
109
emphasis on every feature score. Second, the weights must often be heuristically set by the user, and this may be rather difficult to do, as there may be no clear relationship between the features used to index the image database and those evaluated by the user in a subjective image similarity evaluation. To cope with the first problem we use Guassian normalization as follows [12, 15]: i j D ( X i , X j ) − µ11 D hs ( X hs D ( X i , X j ) − µ pq , X hs ) − µ hs D ( X , X ) = 11 11 11 , ,..., pq pq pq K σ 11 K σ hs K σ pq i
T
j
(2)
Assuming that the features' distance distributions have a Gaussian distribution, it can be shown that there is a 68% probability that the feature values will lie within the range of [-1, 1] if K=1, and a 99% probability if K=3 [12]. As we can not assume apriori that the distances have will have Gaussian distributions, the following general relationship holds: D − µ hs 1 ≤ 1 ≥ 1 − 2 P − 1 ≤ hs K σ K hs
(3)
According to this relationship the probability that the distance will lie within the range of [-1,1] is of 89% if we set K at 3, and 94% if k is set at 4 [13]. The latter is the default value used in all our experiments. A simple additional shift moves the distances into the [0,1] range. Out-of-range values are mapped to the extreme values, so that they do not bias further processing. At this point our similarity function has the following form: q
p
Sim(X i , X j ) = ∑∑ w hs s =1 h =1
i Dhs ( X hs , X hsj ) + 1 q p j = ∑∑ w hs d hs ( X ihs , X hs ) 2 s =1 h =1
(4)
The algorithm must now determine the weights for the individual distances by a statistical analysis of the distance feature values of the images of the relevance-set, and these weights are used to accentuate, or diminish, the influence of a given feature in the overall evaluation of similarity [18]. In our experience in content-based retrieval relevant images are sometimes selected because they resemble the query image in just some pictorial features. Consequently, after an initial query, one retrieved image may be considered relevant because it has the same color as the query, and another be selected for its similarity in shape, although the two are actually quite different from each other. Letting the relevant set R+, d+hs be the set of normalized distances among the elements of R+, and σ hs+ be the corresponding variance, we similarly, let R- be the set of non-relevant images, and d-hs , the set of normalized distances among the elements of R-. We when consider the union between R+ and R-, and compute the corresponding distance sets d+-hs, letting d*hs be d+-hs \d+hs, * , the corresponding variance. The weight terms are defined as: and σ hs
110
G. Ciocca and R. Schettini
1 ε w +hs = 1 + ε + σ hs
if R + < 3
(5)
otherwise
0 w = 1 * ε + σ hs
if R + + R − < 3 or R − = 0
0 w hs = + w hs − w *hs
+ < w *hs if w hs
* hs
(6)
otherwise
(7)
otherwise
where ε is a positive constant (set at 0.01 in our experiments). When at least three examples are given, we can take into account negative examples in tuning the similarity weights. For any given feature the first term of whs is high when there is some form of agreement among the feature values of the relevant (positive) set, while the second term is high when there is a similarity between positive and negative examples.
4. Query Processing Query processing consists in modifying the feature vector of the query by taking into account the feature vectors of the images judged relevant by the user. Our approach is to let R+ be the set of relevant images the user has selected (including the original query), Q the average query and σ the corresponding standard deviation, we proceed as follows:
{
Yhs (j) = X ihs (j)
}
X ihs (j) − Qhs (j) ≤ 3σ hs (j)
~ Q hs (j) =
1 Yhs (j)
∑X
i hs Xihs (j)∈Yhs (j)
(j)
∀ h, s, i, and j
(8) (9)
~ that can better represent the The query processing formulates a new query Q hs images of interest to the user, taking into account the features of the relevant images, without allowing one different feature value to bias query computation. The query process could similarly be applied to compute a query representing nonrelevant examples. But this seems of little practical interest: non-relevant examples are usually not similar to each other, and are, consequently, scattered throughout the feature space.
Using a Relevance Feedback Mechanism
111
5. Query Classification At the first iteration, when the user has selected just one image to be searched, all the weights in the similarity function (4) are set at the equal value of 1/ε. For faster tuning of the similarity function we can exploit previous query sessions performed by the user on the same database. To this end the user is allowed to have satisfactory queries memorized together with the corresponding weights in the similarity measure. When the user has already formulated a query "similar" to the new one, the algorithm sets the initial weights of the similarity function at the value of the former query reducing the time and effort needed to adapt ~the similarity measure by means of the k k relevance feedback algorithm. We let Qˆ = Q , w hs be the set of queries and the
{
}
corresponding weights. When a new query Qn is submitted, the system first evaluates its similarity with respect previous queries, using the corresponding weights in the similarity function; it then selects the closest one as follows: ~ Qk , w khs = argmin Q~ k ,w k
hs
q
ˆ ∈Q
p
∑∑ w
k hs
~ d hs (Qn , Qk )
(10)
s=1 h =1
The initial weights selected in Equation (17) are now set as follows:
w k(initial) hs
q p ~ w khs d hs (Q n , Q k ) ∑∑ k s=1 h =1 w hs if ≤T 1 = pq ε 1 otherwise ε
0 < T ≤1
(11)
Parameter T allows the user to tune the sensitivity. In our implementation the default is set at 0.1, that is, the new query is considered similar to an old one when the two differ less than the 10% of 1 ε ( pq) , the maximum distance value allowed.
6. Experimental Results and Discussion Users do not find it difficult to provide interactively examples of similar and dissimilar images. However if the image database queried is large and heterogeneous, or the retrieval task particularly complex, the user may find not enough examples of images that are actually very similar to the query in the first screens, and, to avoid the time-consuming visual browsing of the database, may mark as relevant images that are only partially similar. The user's information needs may also be rather vague, such as: find all the images containing people. In both cases images judged relevant may differ widely. Treating all these images in the same way - for example, averaging the features of the relevant images to compute a new query vector or updating the similarity measure [18] - may consequently produce very poor results, while
112
G. Ciocca and R. Schettini
processing all the relevant images as single queries and then combining the retrieval outputs may create an unacceptable computational burden when the database is large. Last but not least, relevant images may have some features, color for example, that are only casually similar. If the system is not able to identify these features and treat them differently, subsequent retrieval iterations will be biased. The key concept of the relevance feedback we have proposed is the statistical analysis of the feature distributions of the images the user has judged relevant, or not relevant, in order to understand what features have been taken into account (and to what extent) by the user in formulating this judgment, so that we can then accentuate the influence of these features in the overall evaluation of image similarity as well as in the formulation of a new query. The archives on which the algorithms were tested contain some 5,000 images: photographs of real landscapes, people and animals; reproductions of antique textiles of the Poldi Pezzoli Museum of Milan; a collection of paintings of the Accademia Carrara of Bergamo; and a collection of ceramics belonging to the Museo Internazionale della Ceramica of Faenza. To evaluate the effectiveness of our retrieval method we applied a measure proposed by Methre et al.[5]. We let S be the number of relevant items the user wanted to retrieve when posing a query; R Iq , the set of relevant images, and R qE , the set of relevant images retrieved in the short list. The effectiveness measure was defined as: R qI ∩ R qE if R qI ≤ S I (12) R q ηS = I E Rq ∩ Rq if R qI > S E R q If R qI ≤S, the effectiveness was reduced to the traditional recall measure, while if
R Iq >S, the effectiveness corresponded to precision (In our implementation S was set at 24). The effectiveness of the algorithms was tested on the individual databases, and on a combination of them, to evaluate the system's capacity for adaptation. In Table I we have summarized the experimental results for twenty queries for each of the different databases considered. The queries have not been classified: the first iteration corresponds to a similarity measure in which all the features have the same importance. Each bin corresponds to a different database, and reports the average effectiveness value at each of the first three retrieval iterations. The results show that relevance feedback improves the effectiveness of the retrieval considerably for all the databases, and, in general, the second iteration (that is the first relevance feedback iteration) corresponds to the largest single improvement. There is, instead, little benefit in repeating the relevance feedback more than five or six times. It can reasonably be argued that this is due to the limited capability of the low-level features used to exhaustively describe image content, and not to the relevance feedback itself. The integration of other features and the exploitation of unsupervised image segmentation to focus automatically on the significant parts of the database images to
Using a Relevance Feedback Mechanism
113
be indexed should improve the results. This integration will be straightforward for us, as the structure of the relevance feedback mechanism is actually descriptionindependent, that is, the index can be modified, or extended to include other features, without requiring any change in the algorithm. However, as already pointed out by several authors, perceptually similar images are not necessarily similar in terms of low-level features. Therefore, we believe that only the integration of text-based image annotation will make it possible to further increase retrieval effectiveness in a significant way. Figure 1 represents an example of the system's application. Table I. Retrieval effectiveness 1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0,1 0
First iterarion Second iteration T hird iteration
Photos (1745 images)
Paintings Ceramic and (1768 images) ancient textiles (942 images)
Figure 1. The retrieval results after the second iteration of relevance feedback.
114
G. Ciocca and R. Schettini
Interested readers may find additional examples at the following address: http://www.test.itim.mi.cnr.it/sitoitim/schettini/relfeme.htm.
References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18.
Aigrain O., Zhang H., Petkovic D.: Content-Based Representation and Retrieval of Visual Media: A State-of-the-Art Review. Multimedia Tools and Applications, 3 (1996) 179-182 Amadasun M., King R.: Textural features corresponding to textural properties, IEEE Transaction on System, Man and Cybernetics, 19 (1989) 1264-1274 Binaghi E., Gagliardi I., Schettini R.: Image retrieval using fuzzy evaluation of color similarity. International Journal of Pattern Recognition and Artificial Intelligence, 8 (1994) 945-968 Ciocca G., Gagliardi I., Schettini R.: Retrieving color images by content. In: Del Bimbo A., Schettini R. (eds.) Proc. of the Image and Video Content-Based Retrieval Workshop (1998) Desai Narasimhalu A., Kankanhalli M.S.: Benchmarking multimedia databases. Multimedia Tools and Applications, 4 (1997) 333-356. Faloutsos C., Barber R., Flickner M., Hafner J., Niblack W., Petrovic D. Efficient and effective querying by image content. Journal of Intelligent Systems, 3 (1994) 231-262 Gagliardi I., Schettini R.: A method for the automatic indexing of color images for effective image retrieval. The New Review of Hypermedia and Multimedia, 3 (1997) 201224 Gudivada V.N, Rahavan V.V.: Modeling and retrieving images by content. Information Processing and Management, 33 (1997) 427-452 Jain A.K., Vailaya A.: Image retrieval using color and shape. Pattern Recognition, 29 (1996) 1233-1244. Minka T., Picard R.W.: Interactive learning with a “Society of Models”. Pattern Recognition, 30 (1997) 565-581. Mitra M., Huang J., Kumar S.R.: Combining Supervised Learning with Color Correlograms for Content-Based Image Retrieval, Proc. of the Fifth ACM Multimedia 97 Conference (1997). Mood A.M, Graybill F.A., & Boes D.C: Introduzione alla statistica. McGraw-Hill (1988) Pass G., Zabih R., Miller J.: Comparing Images Using Color Coherence Vectors. Proc. Fourth ACM Multimedia 96 Conference (1996) P.J. Rousseeuw P.J., Leroy A.M.: Robust regression and outlier detection, John Wiley & Sons (1987). Rui Y, Huang T.S., Mehrotra S., Ortega M.: A Relevance feedback architecture in content-based multimedia information retrieval systems. Proc. of the IEEE Workshop on Content-based Access of Image and Video Libraries (1997) Smith J.R., Chang S-F.: VisualSEEK: a fully automated content-based image query. Proc. of the Fourth ACM Multimedia 96 Conference (1996) Stricker M., Orengo M.: Similarity of Color Images. Proc. of the SPIE Storage and Retrieval for Image and Video Databases III Conference (1995) Taycher L., La Cascia M., Sclaroff S.: Image Digestion and Relevance Feedback in the ImageRover WWW Search Engine, Proc. of the Visual1997 Conference (1997)
Region Queries without Segmentation for Image Retrieval by Content Jamal Malki, Nozha Boujemaa, Chahab Nastar, and Alexandre Winter INRIA Rocquencourt, BP 105, 78153 Le Chesnay, France [email protected]
Abstract. Content-based image retrieval is today ubiquitous in computer vision. Most systems use the query-by-example approach, performing queries such as "show me more images that look like this one''. Most often, the user is more specifically interested in specifying an object (or region) and in retrieving more images with similar objects (or regions), as opposed to similar images as a whole. This paper deals with that problem, called region querying. We suggest a method that uses a multiresolution quadtree representation of the images and thus avoids the hard problem of region segmentation. Several experimental results are presented in real-world databases.
1
Introduction
Surfimage is a Content-Based Image Retrieval (CBIR) system developed at INRIA since 1996. Its specificity is the capacity to deal with both categories of image databases: • For image databases with ground truth, the system should be as efficient as possible on the specific application. Examples include face recognition or medical image retrieval. A quantitative evaluation of the system can then be reported in terms of recognition rate, precision/recall graph, etc. • For image databases where no ground truth is available, the system should be flexible, since the notion of perceptual similarity is subjective and contextdependent. Smart browsing, query refinement, multiple queries, and partial search on user-defined regions are among the desirable features of the system. Applications include stock photography and the World Wide Web. Surfimage uses the query-by-example approach for retrieving images and integrates advanced features such as image signature combination, multiple queries, query refinement, and partial queries [1,2]. We focus on the latter problem in this paper. Indeed, the user is most often interested in performing a query on an object (or region), rather than on the whole image. The goal of the system is then to retrieve those images in the database that contain similar objects. This observation motivates recent research on spatially-localized features and region matching. Methods range Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp.115 -122, 1999. Springer-Verlag Berlin Heidelberg 1999
116
Jamal Malki et al.
from histogram computation without segmentation [3-5], approximate region segmentation [6-9], graphical and spatial structure of the image [10-12]. In this paper, we suggest a multiresolution quadtree approach for performing region queries. It represents a simple and efficient way to select image parts for retreival. Endeed, when manual object segmentation through all database images is too hard or when images are too complex for performing automatic segmentation, quadtrees image subdivision is an appropriate solution for partial queries. Image signatures are then computed systematically on each subimage. A dedicated similarity measure is computed, allowing to retrieve images with similar regions. The similarity measure defines the invariance properties of the query (e.g. "find similar objects anywhere in the image" vs. "find similar objects in the same image location"). Section 2 details feature computation in the multiresolution quadtrees. Section 3 discusses user queries and the associated inducted invariance. In section 4 we present comparative experimental results. We draw the conclusions in section 5.
2. Multiresolution Quadtrees Several problems occur with region (or partial) matching. The first one is segmentation: accurate segmentation of an image into regions in the general case is quite difficult. Another problem is invariance: is the user interested in finding other occurrences the object (or region) with the same position, orientation, and scale, or do they require invariance against these transformations? In order to deal with partial queries, we use a multiresolution quadtree representation (similar to [13]) that localizes image features in structured regions (fig. 1). This approach avoids image segmentation and offers effective alternatives for the invariance problem. A1 A2 A3 Fig. 1. Computation of image signatures on the multiresolution quadtree representation of the original image for 3 levels
Let A the original image and p the number of levels: For a given level i ( 1 ≤ i ≤ p ) A is divided into ni parts such that, for all levels, A is represented by a total number of N subimages:
( )
A = A ni i
ni
p
,
∑n i =1
i
= N,
n i = 4 i −1 , 1 ≤ i ≤ p
(1)
Region Queries without Segmentation for Image Retrieval by Content
117
2.1. Feature Computation
We use a large selection of features offered by the Surfimage system [1]. Examples include color, shape and texture, mostly captured via histograms. The presentation of these commonly used features is not the scope of this paper -for further details, see [14, 15] or [1]. The features are computed on each subimage of the quad tree representation, yielding higher dimensional feature vectors. If we consider the p-level representation, the following feature vector is computed: (2) h = hi , 1 ≤ i ≤ p , n = 4i −1
( ) ni
features
i
( )is defined as the feature h computed on the sub-image A . The set of (h )for a fixed i corresponds to the features of the i-th level of the
where each h
i ni
i ni
i ni
representation. For instance, if the original single feature h is a 64-bin histogram, the dimensionality of the 3-level quad-tree feature is (1+4+16)*64 = 1344. Eventually, note that, when using histograms, we normalize the feature vectors so that they sum up to 1 -they thus represent distributions of that feature in the corresponding sub-image. 2.2. Feature Combination
Combination of different features has been a recent focus of image retrieval [2]. The main problem is how do we combine ``apples and oranges'', i.e. features that have different number of components, different scales etc.? We have experimented with several combination methods: voting, gymnastics-rule, etc… [16]. Among these methods, one seems to be the most appropriate and is described in the following. Under Gaussian assumption, the normalized linear combination method uses the estimated mean µi and standard deviation σi of the distance measure d for each feature i, providing the normalized distance: d ' ( x (i ) , y (i) ) =
d ( x ( i ) , y ( i ) ) − ( µ i − 3σ i ) 6σ i
(3)
Where x(i) and y(i) are the vector signatures of images X and Y within feature i. The new distance measure d' will essentially have its values in [0...1]. The combined distance between X and Y is then: D (Q , X ) =
∑ ρ (d ' ( x i =1
where α = µ i − 3σ i 6σ i
3.
i
(i )
, y ( i ) )) −
∑ ρ ( −α
i
)
(4)
i
, ρ is an increasing function (e.g. ρ (x)=x, ρ (x)=x3 etc.).
User Queries
3.1. Performing a Partial Query
For performing a region query, the user has to specify the regions of interest (RoI) at each level. Note that the RoIs do not have to be connex (fig. 2). For a given level,
118
Jamal Malki et al.
we build the corresponding bounding box, defined as the smallest subimage containing the RoIs (see fig.2). This bounding box (Bb) is able to catch the relative geometric positions of the different RoIs, and will be used for retrieving images . Partial queries composed of RoIs Bounding box Image query Fig. 2. Bounding box of partial query at the third level In connection to the RoIs, the user has to specify which metric (fixed, location invariant) they are using. Note that most of the image signatures that we use are histograms, yielding most often rotation, translation and scale invariance within a subimage. The main invariances that the user can specify are thus: no location invariance (e.g. a face on top left, a hand on bottom left), no relative location encoding (a face and a hand anywhere in the image), location invariance (a face and a hand with specified relative positions). 3.2. Similarity Metric and Invariance Issues
The previous invariances have to be translated into similarity metrics (or equivalently, distances). We have dealt with the ”no invariance'' case in our previous work [16]. The ”loose relative location” case is an alternative which can be easily derived. We describe the location invariance hereafter; the method encodes the relative locations of the RoIs and is translation invariant (fig. 3).
Fig. 3. Authorized translations of partial queries bounding box at level 3 Location invariance is built in as follows. Let RQ the set of Nr partial queries selected
by the user at the level l: R Q =
Nr − 1
UQ
β =0
Nr −1
Let M the Bb of
U {Q } , i
i=0
lβ
{
= Q l 0 , K Q l NR − 1
}
Nt is the number of authorized translations. The
similarity measure between a query image Q and any image X in the database is given by:
d (Q , X ) =
Nt − 1
∑ d (M (Q ), M j=0
M j (X ) =
j
( X ))
Nr −1
UQ α
α j =0
l
is the sub-image obtained after a translation j. Note that: j
(5)
Region Queries without Segmentation for Image Retrieval by Content
d (M ( Q ), M j ( X ) ) =
Nr −1
Tm
∑ α∑ δ α β β =0
j
=0
j
(
d Q l β , X lα j
)
119
(6)
Where δ is the Kronecker symbol. 3.3. Redundancy
We note that the quadtree representation of image features is redundant, especially with respect to histogram computation. More precisely, a histogram feature computed at any node of the tree is equal to the sum of the histogram features of its 4 sons (regardless of the normalization procedure). The quadtree representation is thus redundant. However, experimentally, this redundancy is effective and useful: indeed, the lower levels tend to specify the holistic features of the image, thus allowing to restrict the search (e.g. I am looking for red roses, but in landscape scenes only). Note also that the contribution of each subimage to the global metric being equivalent, the lower levels participate less to the overall metric. Examples are shown in section 4. The user selects the first level to focus on an image category (e.g. a landscape), and specifies the query regions in a higher level.
4. Retrieval Results We present results on our homebrew bigdatabase (Fig.4), which was built by merging the MIT Vistex database of textures, the BTphoto database of city and country scenes, a homebrew paintings database, and the homeface database of people in the lab. The total number of images in bigdatabase is 3670. We use a kd-tree structure to optimize the search process. Note that when spatial relations between partial queries are not preserved, retrieval results are different from those displayed in figure 5, 6 and 7.
Fig4. A sample of bigdatabase consisting in 3670 heterogeneous images
120
Jamal Malki et al.
Fig.5 Retrieval of the top left image. Retrieved images are from top left to bottom right in order of best match. Query image at level 1 and partial queries at level 3 are also presented. We search for selections every where in the database images
Fig6: Second example of partial queries on bigdatabase
Region Queries without Segmentation for Image Retrieval by Content
121
Fig 7 Retrieved face images from bigdatabase with both level 1 and 3
Fig 8 Retrieved face images from bigdatabase with only level 1. System returns just face images among texture, landscape images and painting
5.
Conclusion
To address the issue of region matching, we avoided the difficult problem of image segmentation using a multiresolution quadtree representation of the image. Our experience shows that we need two levels: level 1 to pick image category and level 3
122
Jamal Malki et al.
to specify more precise details in a same category. This approach provides flexibility for general and heterogeneous image databases retrieval without using specialized features such for face retrieval as shown in fig.7 and fig.8. Various general features are computed and combined together on each subimage. Dedicated similarity metrics allow for inferring invariance properties -in particular location invariance. Experimental results show that the method is effective for local queries in image databases. They also emphasize the importance of preserving spatial organization of local features to achieve good retrieval results.
6.
References
1. Nastar C., Mitschke M., Meilhac C. and Boujemaa N.: Surfimage: a Flexible Content-Based Image Retrieval System", ACM Multimedia'98", Bristol, September, 1998 2. Nastar C., Mitschke M., Meilhac C.: Efficient Query Refinement for Image Retrieval, Computer Vision and Pattern Recognition (CVPR'98), Santa Barbara, June, 1998 3. Michaeal J. Swain and D. H. Ballard: Color indexing, IJCV,vol.7, n°1, pp.11-32, 1991 4. John R. Smith and Shih-Fu Chang,Tools and Techniques for Color Image Retrieval,Proceedings of Storage & Retrieval for Image and VideoDatabases I, San Jose, CA, USA, February, pp. 426-437, 1996, 5. Jing Huang and Ravi Kumar and Mandar Mitra and Wei-Jing Zhuand Ramin Zabih,Image Indexing Using Color Correlograms,CVPR, Puerto Rico, June, pp.762-768, 1997 6. Sergio D. Servetto and Yong Rui and Kannan Ramchandran and Thomas S. Huang,A: Region -Based Representation of Images in MARS, Special Issue on Multimedia Signal Processing, Journal on VLSI Signal Processing, Oct, 1998 7. Belongie S. and Carson C. and Greenspan H. and Malik J.: Color- and Texture-Based Image Segmentation Using EM and Its Application to Content-Based Image Retrieval", Proceedings of the Sixth International Conference on Computer Vision(ICCV'98), Bombay, January,1998 8.Cinque L., Lecca F., Levialdi S., and Tanimoto S. : Retrieval of images using rich image descriptions, Proceedings of the ICPR, 1998 9. Neill W. Campbell, William P. J. Mackeown and Barry T. Thomas and Tom Troscianko: Interpreting image databases by region classification,Pattern Recognition (Special Edition on Image Databases), vol. 30,n° 4, Apr,pp. 555--563, 1997 10. Simone Santini and Ramesh Jain: The Graphical Specification of Similarity Queries,Journal of Visual Languages and Computing,1997 11. Del Bimbo and Vicario E. : Using Weighted Spatial Relationships in Retrieval by Visual Contents, IEEE workshop on Image and Video Libraries",Santa Barbara,June,1998 12. Soffer A., Samet H. and Zotkin D. : Pictorial Query trees for query specification in image databases,Proceedings of the ICPR,1998 13. Vellaikal A. and Kuo C.: Joint Spatial-Spectral Indexing of JPEG Compressed Data for Image Retrieval, Int'l Conf. on Image Proc.", Lausanne, 1996 14. Pentland A., Picard R. and Sclaroff S. : Photobook: Tools for Content-Based Manipulation of Image Databases", Storage and Retrieval of Image and Video Databases II,vol. 2185, San Jose, 1994 15. Jain A. and Vailaya A. : Image Retrieval Using Color and Shape: Pattern Recognition,vol.29,n°8, 1996 16. Nastar C., Mitschke M., Meilhac C., Boujemaa N., Bernard H. and Mautref M.: Retrieving Images by Content: the Surfimage System, Multimedia Informations Systems'98,Istanbul, September,1998
Content-Based Image Retrieval over the Web Using Query by Sketch and Relevance Feedback E. Di Sciascio, G. Mingolla, and M. Mongiello Dip. Elettrotecnica ed Elettronica, Politecnico di Bari, Via E. Orabona 4, I-70125, Bari Italy Tel.+39(0)805460641 Fax +39(0)805460410 [email protected]
Abstract. This paper investigates the combined use of query by sketch and relevance feedback as techniques to ease user interaction and improve retrieval effectiveness in content-based image retrieval over the World Wide Web. To substantiate our ideas we implemented DrawSearch, a prototype image retrieval by content system that uses color, shape and texture to index and retrieve images. The system avails of Java applets for query by sketch and uses relevance feedback to allow users dynamically refine queries.
1 Introduction Content-based image retrieval (CBIR) [1-3] information systems use information extracted from the content of images for retrieval, and help the user retrieve images relevant to the contents of the query. A number of methodologies, techniques and tools, related to image content processing, have been studied for identification and comparison of image features in order to develop classification and retrieval systems based on (almost) automatic interpretation of image content. Complete image classification, indexing and retrieval based on the content interpretation require semantic interpretation and cannot be afforded with current technology. A surrogate of semantic interpretation is the computation of visual features that can be used as quantitative parameters for the identification of similar images. Thus, the problem of retrieving images with homogeneous content is substituted with the problem of retrieving images visually close to a target one. Several systems have been proposed in recent years in the framework of content-based retrieval [5-20]. With particular reference to papers related to query by sketch, the interested reader is also referred to QBIC [10,11], QVE [18] and a recent work on the use of elastic deformation of user sketches [19]. It must anyway be noticed that these works emphasize more the pattern matching problem than the retrieval by similarity one. Also noteworthy is the work in [20], which uses wavelet-based indexing and query by sketch for color images retrieval. Here the emphasis is in avoiding any user specification but the submitted query sketch. Approaches proposed in [5-9] introduce relevance feedback [4] as a distinguished aspect that can allow improving retrieval results using feedback provided by the user. Porting CBIR systems to the WWW is not a straightforward task. Largest part of current systems basically allow query by example letting the user Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 123-130, 1999. Springer-Verlag Berlin Heidelberg 1999
124
E. Di Sciascio et al.
browse into the collection until he/she finds an image visually similar to searched ones Once such an image has been found the user can pose a query and retrieve similar images. This process is cumbersome. As image collections grow in size this process may take a lot of time, and eventually reduce the query-retrieval process to trivial browsing. Furthermore the well-experienced problem of current Internet low speed makes the all process much tedious. We have investigated the possibility to increase user's interaction with image retrieval systems over the Internet using query by sketch and giving the user the ability to interact easily with the system by browsing the retrieved images and tuning the response through relevance feedback analysis. As a matter of fact, rather than putting much processing effort into answering a query in the sharpest way, in an image retrieval system it is reasonable to provide a fast reply with a rough retrieval method (yet offering non-trivial discrimination performance). Differently from text-based documents, browsing of a small retrieved set of images is fast and easy since the relevance of the content can be stated at a glance. To address these and other issues related to image information systems, we have started the DrawSearch [5] project with our group at Politecnico di Bari.
query by texture interface (Java applet)
user
query by shape and color interface (Java applet)
feature data archive
WWW
query server
feature extraction
retrieved set viewer (html) image archive
DrawSearch Client side
DrawSearch Server
side
Figure 1. DrawSearch system overview
2 System Overview In order to substantiate our ideas on the use of query by sketch and relevance feedback to increase user's interactivity and retrieval effectiveness in a web based environment, we designed and implemented a prototype system, which is accessible over the Internet at: http://deecom03.poliba.it/DrawSearch/DrawSearch-home.html. Figure 1 shows its main components. Our system extracts as relevant features, as other systems do, color, shape and texture. Two separate user interfaces are provided. The first one allows posing queries by sketch combining shape and color distribution; the second one allows queries by texture content. Reasons for having separate interfaces are various, most relevant for us was to avoid increasing the computational burden on the client side. We are
Content-Based Image Retrieval over the Web
125
anyway integrating relevant texture information in the retrieval stage of the color and shape system. Features extraction is a two-fold problem in our approach. Basically the features described here are extracted off-line for images during the database population stage. Hence, though always relevant, time performance is not critical when referred to this stage. On the other hand, when the features have to be extracted from the user's sketch, time, and hence algorithmic complexity, becomes a primary issue. Furthermore, while features computation on the server side can be done with efficient standard compiled languages, the client side has to rely on the much less efficient Java language. All extracted features share a representation based on the vector space model [4]. This model is based on the association of term vectors to documents, each vector representing a specific document by holding information about the index terms or keywords associated to it. Such information may appear simply as a set of present/not present flags, but more often it is a measure (weight) of the ability of each index term to discriminate the document within the collection. In our image retrieval model, feature vectors play almost the same role that term vectors play in text retrieval, holding normalized values of the image features as indexing information. However, differently from the vector space model, we do not weight the features against the whole image collection. The reason for this difference comes from the different user perception of image similarity with respect to text similarity, and to the different level at which the retrieved items are evaluated: visual for the images, semantic for the text. Image similarity is evaluated on visual properties, and may be verified at a glance. It is therefore independent from the image collection size and variety. The features are used to find similarities rather than to discriminate differences. On the other hand, text reading to verify the adequacy of the retrieved documents is a long process, therefore a text retrieval system must be provided with good discrimination capabilities, relying on the use of words to describe the document meaning and not its appearance. We believe that this model has an intrinsic strength due to its widespread use and its reliability. It also allows an extremely simple integration with text-based information systems. Hence we considered of primary importance feature evaluation schemes that allow a straightforward implementation in this model. Color feature is extracted computing the average values within 16 predefined blocks (in a 4 by 4 arrangement) the image is divided into. Computing the color feature in this way is obviously not very precise; but it allows, with limited computational effort, to represent the color distribution of the most immediately visible components like large objects or background, hence also providing a limited degree of spatial information. The adopted color model is RGB (Red, Green, Blue). Resulting data are normalized to a sum of 1 and arranged in a vector of 48 components. We are currently changing our color model to the HSB one. Anyway results reported in this paper still refer to color distribution computed using RGB. Shape feature extraction in real images requires image segmentation. Although a number of segmentation algorithms have been proposed in the literature, this issue is far from being solved. Most problems come from the ill-posed nature of the edge detection problem. The problem becomes harder when we need segmentation into semantically coherent regions, i.e. objects. We adopted a contour-based segmentation algorithm, which uses as ancillary information color and texture.
126
E. Di Sciascio et al.
a)
b)
c) Figure 2. Query and retrieval results for a by color and shape query. a) sketch drawn on the canvas; b) retrieval results; c) retrieval results after two rounds of relevance feedback.
Shape characteristics extraction has been performed using a modified version of the Fourier descriptors approach proposed in [21]. Our shapes are hence assumed simply connected with borders represented by a closed curve. We currently use the real part
Content-Based Image Retrieval over the Web
127
of the lower 100 Fourier coefficients for shape description, which are arranged in a single vector. As we discard the continuous component, we obtain scale, rotation and traslation invariant coefficients. Retrieval based on texture analysis in real images also requires image segmentation. Our approach uses Gaussian Markovian Random Fields (GMRF) for texture-based segmentation and texture extraction [8]. Two vectors storing mean values and variance values characterize each segmented area, obtained applying the GMRF algorithm.
3 User Interface and Query Processing The basic prerequisite a user interface for visual, content-based, image retrieval system should have is to let the user simply express "what" is his/her information need. Figure 2.a shows the color and shape query interface with a query sketched on the canvas. Here the user simply has to roughly draw a sketch, in terms of color and shape, of the images he/she would like to retrieve. A message window is provided to inform the user of the actions. A control panel shows the values computed when features are extracted. It is worth noticing that separate layers exist for shape and color distribution. A further "border" layer is provided to let the user check if the shape he/she has drawn is closed. This has drawbacks, as it may require drawing twice an object within the shape. The first one on the color layer and the second on the shape layer. This strategy was determined by the need to avoid the further computational burden of extracting the shape information from a color image. Also to reduce the computational burden on the client side, the retrieval interface uses a simple html page to show results. The user interface for texture processing is also a Java applet. Figure 3.a shows a query obtained selecting a user defined area within a database image. Differently from most systems, which basically allow posing a query using pre-selected texture images, our system allows the user to dynamically select a texture area within images in the database in order to increase user's interactivity with the system. Also this interface is endowed of a text area for messages from the system and a control panel to allow inspection of extracted parameters. Retrieval is performed by measuring the distance, in the n-dimensional space defined by the index terms, between the term vector of the query and the term vectors of the documents. In the retrieval by shape and color sub-system, two similarity functions are computed: simC(R,Q) and simS(R,Q), respectively accounting for color and shape. Each function simX(R,Q) representing the similarity between a database image feature, defined by the tuple R = (r0, r1, ..., rn), and the query image feature, also defined by a tuple Q = (q0, q1, ..., qn) is computed using the cosine similarity coefficient, defined as:
sim( R , Q) =
∑ ri q i ∑ ri2 × ∑ q i2
(1)
The resulting coefficients are merged to form the final similarity function as a linear combination:
128
E. Di Sciascio et al.
sim(R,Q) = α × simC(R,Q) + β × simS(R,Q)
(2)
where α and β are weighting coefficients. The weights defined by the coefficients obviously lead to increase or decrease the contribution of a feature with respect to the others. To better characterize the user's information need, these coefficients are dynamically modified during the relevance feedback stage. Figure 2.b shows an example of retrieval results.
a)
b) Figure 3. Query and retrieval results for a query by texture. a) selected query area; b) retrieval results after relevance feedback. Relevant texture areas are the highlighted ones.
In the retrieval by texture sub-system, a single similarity function is computed between the selected query area, represented by its feature vector Q, and textured areas of images in the database, represented by their feature vectors Ri is performed computing the Euclidean distance between vectors storing associated mean values. Such distance is normalised with associated variance values providing the following expression: simT (Q, Ri ) =
4
∑ j =1
(f
(Q ) − f j ( Ri ) )
2
j
σ 2j ( Ri )
(3)
Content-Based Image Retrieval over the Web
129
Resulting retrieved images are indexed according to a growing distance score. Uncertainty is obviously present in image retrieval, due to the weak correspondence between the computed features and the image content perceived by the user. In other words, the system may not match the user perception of similarity. For this reason there is a growing interest in the image database community towards methodologies and techniques for relevance feedback. In our model the user can improve retrieval results by selecting, among the topmost ranked retrieved images, the ones he/she considers relevant. This is known as "positive feedback". Based on experiments, this type of feedback is used in the retrieval by texture subsystem. The other sub-system also implements negative feedback for images considered not relevant. Leaving an image unselected marks it as "don't care" and it does not contribute to the relevance feedback process. A new query is computed by combining the feature vectors of the original query with the ones of the relevant and not relevant images. In practice, the modified query is computed by adding to the feature vector Q associated to the query image the feature vectors X of relevant images and subtracting the not relevant ones Y, respectively weighted with suitable δ and ε coefficients:
Q ( k +1) = Q ( k ) + δ
N rel
∑
i =1
Xi − ε
M notrel
∑
Yi
(4)
i =1
with δ=0 in the retrieval by texture sub-system. The modification is performed separately on the features. The modified query is then used in a new retrieval step. It is worth noticing that the shape and color retrieval utilizes a 3-layer relevance feedback strategy. The first one has been just described and it operates at the feature level. The other two operate on the combined features and on the images domain, i.e. if the user tends to concentrate on a single feature, the systems strengthens the contribution of that feature in the similarity measure. Furthermore if the user concentrates his/her interest on images of a single domain, i.e. all relevant images belong to a single category, the system retrieves in the following relevance feedback stages only images belonging to that domain. Figures 2.c and 3.b show results after relevance feedback processing. Table I shows quantitative results in terms of precision and recall for a first set of tests performed. Please notice mp1=mp2 as in our test we always retrieved the same number of images regardless of any threshold.
4 Conclusions Content-based access appears to be a promising direction to increase the efficiency and accuracy of unstructured data retrieval, yet strategies for specifically web-based CBIR systems have to be devised. We have investigated techniques to increase user's interactivity in such a scenario, proposing query by sketch as an effective way to start a significant query, and relying on relevance feedback to refine retrieval results. We are still integrating and refining our model; first experiments carried out with the aid of some volunteers have shown anyway fair precision and recall scores, and noteworthy improvements obtained by relevance feedback.
130
E. Di Sciascio et al.
TableI. Precision and recall scores Queries results without relevance feedback results with single layer relevance feedback results with two-layer relevance feedback
mp1≡mp2 0,495 0,593 0,637
mr1 0,649 0,778 0,836
mr2 0,659 0,787 0,842
References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21]
Communications of the ACM, 40, 12, 1997. ACM Multimedia Systems, special issue on Multimedia Databases, 3, 5/6, 1995. IEEE Computer, special issue on Content Based Image Retrieval, 28, 9, 1995. G. Salton, Automatic Text Processing, Addison Wesley, 1989. E. Di Sciascio, M. Mongiello, DrawSearch: a Tool for Interactive Content-Based Image Retrieval over the Net, in Proc. of SPIE vol. 3656,.561-572, 1999. Y. Rui, T.S.Huang, S. Mehrotra, Content based image retrieval with relevance feedback in MARS", Proc. of IEEE ICIP'97, 1997. A. Celentano, E. Di Sciascio, Features Integration and Relevance Feedback Analysis in Image Similarity Evaluation, Journal of Electronic Imaging, 7, 2, 1998. E. Di Sciascio, G. Piscitelli, A. Celentano, Textural Features and Relevance Feedback in Image Retrieval, in Visual Database Systems 4, Chapman and Hall, 1998. H.G. Stark, On image retrieval with wavelets, Journal of Imaging Systems and Technology, 7, 200-210, 1996. W. Niblak et al., The QBIC project: Querying images by content using color, texture, and shape, in Proc. of SPIE, vol. 1908, 173-182, 1993. M. Flickner et al., Query by Image and Video Content: The QBIC System, IEEE Computer, 28, 9, 23-31, 1995. P. M. Kelly, T. M. Cannon, D. R. Rush, Query by image example: the CANDID approach, in Proc. of SPIE, vol. 2420, 238-248, 1995. V. E. Ogle, M. Stonebrakes, Chabot: retrieval from a relational database of images, IEEE Computer, 28, 9, 40-56, 1995 R. Bach et al., The Virage Image Search Engine: An open framework for image management, in Proc. of SPIE, vol. 2670, 76-87, 1996. J.R. Smith, S.F. Chang, VisualSEEK: a fully automated content-based image query system, Proc. of ACM Multimedia'96, 1996. W.Y. Ma, B.S. Manjunath, NETRA: A toolbox for navigating large image databases, Proc. IEEE ICIP '97, 1997. R.W. Picard, T.Kabir, Finding similar patterns in large image databases, Proc. ICASSP, 1993. K. Hirata, T. Kato, Query by visual example, content based image retrieval, Lecture Notes in Computer Science, vol. 580, 1992. A Del Bimbo, P. Pala, Visual image retrieval by elastic matching of user sketches, IEEE Trans. PAMI, 19, 2, 1997. C. E. Jacobs, A. Finkelstein, D. H. Salesin. Fast Multiresolution Image Querying. Proc. of SIGGAPH 95, 1995. Y. Rui, A.C. She, T.S. Huang, Modified Fourier descriptors for shape representation-a practical approach, Proc. of 1st workshop on image databases and multimedia search, 1996.
Visual Learning of Simple Semantics in ImageScape Jean Marie Buijs and Michael S. Lew Leiden Institute for Advanced Computer Science Leiden University, Postbus 9512, 2300 RA Leiden, The Netherlands {buijs,mlew}@cs.leidenuniv.nl Abstract Learning visual concepts is an important tool for automatic annotation and visual querying of networked multimedia databases. It allows the user to express queries in his own vocabulary instead of the computer's vocabulary. This paper gives an overview of our current research directions in learning visual concepts for use in our online visual webcrawler, ImageScape. We discuss using the Kullback relative information for finding the most informative features in the case of human faces and generalize the method to other objects/concepts.
1 Introduction In many content based retrieval systems, the user is asked to understand how the computer sees the world. An emerging trend is to try to have the computer understand how people see the world. However, understanding the world is a fundamental computer vision problem which has withstood decades of research. The critical aspect to these emerging methods is that they have modest ambitions. Petkovic[1997] has called this finding "simple semantics." From recent literature, this generally means finding computable image features which are correlated with visual concepts. The key distinction is that we are not trying to fully understand how human intelligence works. This would imply creating a general model for understanding all visual concepts. Instead, we are satisfied to find features which describe some small, but useful domains of visual concepts.
1.1 Visual Search Paradigms Content based search researchers are constantly looking for methods which are usable by nonexperts. The typical method for this intuitive search is by finding a similar image. In this paradigm, the user clicks on an image, and then the search engine ranks the database images by similarity with respect to color, texture, and shape. In sketch search methods, the user draws a rough sketch of the goal image. The assumption is that the sketch corresponds to the object edges and contours. The database images which have the most similar shapes to the user sketch are returned. Sketches represent Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 131-138, 1999. Springer-Verlag Berlin Heidelberg 1999
132
Jean Marie Buijs and Michael S. Lew
an abstract level of reasoning of the image. Another abstract method uses icons to represent objects and/or concepts in the image. The user places the icons on a canvas in the position where they should appear in the goal image. In this context, the database images must have been preprocessed for the locations of the available objects/concepts. The database images which are most similar by the content of objects/concepts to the iconic user query are returned. For an overview, see Gudivada and Raghavan[1995] and especially, Flickner, et al. [1995]. We designed a system for searching networked multimedia databases called ImageScape. One of the principal query types in ImageScape is semantic icons. Semantic icons are essentially drag-and-drop icons which represent concepts from the user's vocabulary. These can be objects such as human faces, textures such as sand or wood, or even colors. The importance of the method is it does not require the user to learn how the computer understands the image. Instead, the computer learns how humans perceive images.
1.2 Imagescape System Overview When an image is brought to the server, it is analyzed for features pertaining to the semantic icons (i.e. faces, sand, water, etc.) and for extraction of the computer sketches. Then a thumbnail of the image and the features are stored in a compressed database. When a user sends an image query from a WWW Java browser/client program, the query is sent to the server, and matched against the database. The user drawn sketch is compared to the computer sketches and the semantic icons are compared to the automatically extracted features. The best matches are then sent back to the WWW browser/client program to be displayed to the user. In summary, the ImageScape system consists of the following modules: • • • • • •
collection of text, images, audio, and video from the WWW compression of the image database semantic object detection in images computer sketching of images matching between the icons/sketches with the database images Java client connecting to host server for the visual query input and processing
There are other interesting WWW image search engines which have been described in the research literature. The WebSeek[Smith and Chang 1997] system from Columbia University finds similar images and performs automatic text based category classification. The Webseer[Frankel, Swain, and Athitsos 1996] system from the University of Chicago lets users search by the number of faces and by text queries. Taylor, Cascia, and Sclaroff[1997] designed the ImageRover system to primarily use
Visual Learning of Simple Semantics in ImageScape
133
relevance feedback for the search process, and in the PicToSeek system, Gevers and Smeulders[1997] search through the images using similar images and image features. In a previous paper [Lew, et al. 1997], we introduced an early version of this system. In this work, our focus is entirely on the object/concept detection.
2 Learning Simple Semantics In this paper we discuss learning simple semantics, or in another way, visual learning of concepts. This brings into mind the question raised by readers and referees, which is “What is visual learning?” Such a general term could refer to anything having to do with artificial or human intelligence in all its sophistication and complexity. Rather than a vague description, we seek to define it clearly at least within the boundaries of this paper as either (1) feature tuning; (2) feature selection; or (3) feature construction. Feature tuning refers to determining the parameters which optimize the use of the feature. This is often called parameter estimation. Feature selection means choosing one or more features from a given initial set of features. The chosen features typically optimize the discriminatory power regarding the ground truth which consists of positive and negative examples. Feature construction is defined as creating new features from a base set of atomic features and integration rules. In this paper, we focus on feature selection and in the section on future work, we reveal preliminary results toward feature construction. What is an object/concept? For our purposes, an object/concept is anything which we can apply a label or recognize visually. These could be clearly defined objects like faces or more difficult concepts such as textures. Most textures do not have corresponding labels in common language. Object/Concept detection is essential to the usage of the WWW image search engine because it gives the computer the ability to understand our notion of an object or concept. Instead of requiring all users to understand low level feature queries, we are asking the computer to understand the high level queries posed by humans. For instance, if we want to find an image with a beach under a blue sky, most systems require the user to translate the concept of beach to a particular color and texture. In our system, the user can pose the query visually as a beach under a blue sky using icons to represent beach and blue sky, respectively. Giving a complete discussion of visual concept learning would not fit within the scope of a conference paper. In fact, it would require several books to do it justice. Furthermore, we suggest that what is necessary in the field now is a thorough survey on visual concept learning. For the scope of this paper, we give a brief overview of recent visual learning techniques in the research literature. We turn to an example of feature selection in the domain of human face detection, and then observe that it is straightforward to generalize the face detection method to other objects.
2.1 Background Picard[1996] reported promising results in classifying blocks in an image into "at a glance" categories. What this means is that she investigated multiple model methods
134
Jean Marie Buijs and Michael S. Lew
to classify an NxN block into categories which humans could classify without logically analyzing the content. Forsyth, et al. [1996] found objects from feature blobs. More recently, Vailaya, Jain, and Zhang [1998] have reported success in classifying images as city vs. landscape. They found that the edge direction features have sufficient discriminatory power for accurate classification of their test set. Buijs[1998] reported promising results in learning primary colors and textures using the Kullback relative information. The commonality between these methods was using multiple features for object/concept detection. Regarding object detection, the recent surge of research toward face recognition has motivated robust methods for face detection in complex scenery. Representative results have been reported by Sung and Poggio[1998], Rowley and Kanade[1998], Lew and Huijsmans [1996], and Lew and Huang[1996].
2.2 Feature Selection We begin by describing a method of finding human faces in grayscale images with complex backgrounds and then show that the method is easily extensible to other objects/concepts. The Kullback relative information[1959] is generally regarded as one of the canonical methods of measuring discriminatory power. Specifically, we formulated the problem as discriminating between the classes of face and nonface, and used the Kullback relative information as a measurement of the class separation, which is the distance between the classes in feature space. As the class separation increases, the overlap between the classes decreases making the confidence in the class decision increase. The detection algorithm can be stated concisely as follows: (1) Create a ground truth set of positive (face) and negative(nonface) examples (2) Compute the histograms from the ground truth set for the classes face and nonface. (3) Find the N most informative features by maximizing the Kullback relative information combined with a Markov random field. (4) Arrange the N most informative features in a vector, and apply a minimum distance classifier and is shown in Figure 2.1.
Visual Learning of Simple Semantics in ImageScape Eyes
Eyes/nose Templates
,
135
, ...
Kullback Relative Information
NonFace Templates Nose Area ,
, ...
Kullback Relative Information Image
(a) (b) Figure 2.1. (a) Result of using Kullback relative information on face and nonface examples; and (b) the 256 pixels which have the greatest discriminatory power.
2.3 Generalizing to Multiple Models In the previous explanation covering face detection, we used the 256 pixels which had the greatest discriminatory power. For lack of a better word, we define this set of features as a discriminatory model. This discriminatory model has the advantage that for N features in the model, it minimizes the misdetection rate. The question arises then of how to generalize the method to more features such as color, texture, and shape. For each object we wish to detect, a large set of positive and negative examples is collected. We measured a variety of texture, color, and shape features, and for each one we calculate the Kullback discriminant. The candidate features for our system included the color, gradient, Laplacian, and texture information from every pixel as shown in Figure 2.2. For the texture models, we used Trigrams[Huijsmans, Poles, and Lew 1996], LBP [Wang and He 1990], LBP/C, Grad. Mag., XDiff, YDiff, [Ojala, Pietikainen, and Harwood 1996]. For shape comparison, we used the features derived from elastic contours[Del Bimbo and Pala 1997], invariant moments[Hu 1962], and Fourier Descriptors [ Gonzalez and Woods 1993]. Best Model
Positive Examples
Texture Distributions and Projections Trigrams, LBP/C, Grad. Mag, Xdiff, Ydiff
Measure Discriminatory Power
Color RGB, HSV
Negative Example
Shape Invariant Moments, Snakes, Fourier Desc.
Figure 2.2. Selecting the best discriminatory model of N features from texture, color, and shape features.
136
Jean Marie Buijs and Michael S. Lew
3 Feature Construction In the previous discussion, we proposed using the Kullback relative information for feature selection. The next logical step was to consider methods for feature construction. In recent work[Buijs 1998], we presented a rule based method for combining the atomic features of color, texture, and shape toward representing more sophisticated visual concepts. Recall that in feature construction there are atomic features and rules for integrating them. Our atomic features were instances of color, texture, and shape. The atomic colors were red, yellow, purple, green, blue, brown, orange, white, gray, and black. From the Kullback relative information, the color model with the greatest discriminatory power was the HSV. The atomic textural features were coarse, semi-coarse, semi fine, fine, nonlinear, semi-linear, linear, and texture features based on examples: marble, wood, water, herringbone, etc. LBP/C had the greatest discriminatory power regarding the Kullback relative information. For the atomic shape features, we created a basic set of geometric primitives: circular, elliptical square, triangular, rectangular, and pentagonal. We also tuned a variety of shape examples. Shape features were detected using template matching and active contour energy [del Bimbo and Pala 1997]. Simple concepts were represented as AND conjoined boolean expressions: If (color is orange) AND (texture is coarse) AND (shape is circular) Then object is an orange More complex concepts were represented using AND/OR expressions such as: If ((color is yellow OR color is white) AND (texture is fine) AND (texture is nonlinear)) Then object is sand Rules were automatically generated from positive and negative example training sets using decision trees. We ranked the features used in the decision trees by the Kullback relative information, and created the tree using the features with greater discriminatory power first. Results for five outdoor categories are shown in Table 1. Table 1. Probability Of Misdetection Misdetection
forest 0.23
mountain 0.14
sand 0.27
sky 0.09
water 0.15
Visual Learning of Simple Semantics in ImageScape
137
4 Conclusions Visual concept learning has the potential of bringing intuitive searching for visual media to the general public. Regarding the World Wide Web, we can bring image and video search to anyone with a Web browser if these visual learning technologies mature. At Leiden University, we are currently creating a large library of visual feature training databases and detectors. In this paper, we gave an overview of the visual concept learning methods being used for the ImageScape project. From the perspective of visual learning as feature tuning, feature selection, and/or feature construction, we have shown a progression of techniques for learning simple concept domains. Regarding future work, we think that the methods for combining the results of multiple classifiers[Kittler 1998] have the most versatility and potential for improvement of simple semantic detection.
Acknowledgements This research was funded by the Dutch National Science Foundation and the Leiden Institute for Advanced Computer Science
References Buijs, J. M., “Toward Semantic Based Multimedia Search,” Master’s Thesis, , Leiden Institute for Advanced Computer Science, August 13, 1998. Del Bimbo, A., and P. Pala, “Visual Image Retrieval by Elastic Matching of User Sketches,” IEEE Trans. Pattern Analysis and Machine Intelligence, February, pp. 121-132, 1997. Flickner, M., H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic, D. Steele, and P. Yanker, “Query by Image and Video Content: The QBIC System,” Computer, IEEE Computer Society, pp. 23-32, Sept. 1995. Forsyth, D., J. Malik, M. Fleck, T. Leung, C. Bregler, C. Carson, and H. Greenspan, "Finding Pictures of Objects in Large Collections of Images," Proceedings, International Workshop on Object Recognition, Cambridge, April 1996 Frankel, C., M. Swain and V. Athitsos, “WebSeer: An Image Search Engine for the World Wide Web,” Technical Report 96-14, University of Chicago, August 1996. Gevers, T. and A. Smeulders, “PicToSeek: A Content-Based Image Search System for the World Wide Web,” VISUAL’97, San Diego, December, pp. 93-100. Gonzalez, R. and R. E. Woods, “Digital Image Processing”, Addison Wesley, 1993. Gudivada, V. N., and V. V. Raghavan, “Finding the Right Image, Content-Based Image Retrieval Systems,” Computer, IEEE Computer Society, pp. 18-62, Sept. 1995. Hu, M., “Visual Pattern Recognition by Moment Invariants”, IRA Trans. on Information Theory, vol. 17-8, no. 2, pp. 179-187, Feb. 1962.
138
Jean Marie Buijs and Michael S. Lew
Huijsmans, D. P., M. Lew, and D. Denteneer, “Quality Measures for Interactive Image Retrieval with a Performance Evaluation of Two 3x3 Texel-based Methods,” International Conference on Image Analysis and Processing, Florence, Italy, September, 1997. Kittler, J., M. Hatef, R. Duin, and J. Matas, “On Combining Classifiers,” IEEE Trans. Patt. Anal. and Mach. Intel., vol. 20, no. 3, March 1998. Kullback, S. “Information Theory and Statistics,” Wiley, New York, 1959. Lew, M., K. Lempinen, and N. Huijsmans, "Webcrawling Using Sketches," VISUAL'97, San Diego, December, 1997, pp. 77-84. Lew, M. and N. Huijsmans, “Information Theory and Face Detection,” Proceedings of the International Conference on Pattern Recognition, Vienna, Austria, August 25-30, 1996, pp.601-605. Lew, M. and T. Huang, “Optimal Supports for Image Matching,” Proc. of the IEEE Digital Signal Processing Workshop, Loen, Norway, Sept. 1-4, 1996, pp. 251254. Ojala, T., M. Pietikainen and D. Harwood, “A Comparative Study of Texture Measures with Classification Based on Feature Distributions,” vol. 29, no. 1, pp. 51-59, 1996. Petkovic, D., "Challenges and Opportunities for Pattern Recognition and Computer Vision Research in Year 2000 and Beyond, "Proc. of the Int. Conf. on Image Analysis and Processing, September, Florence, vol. 2, pp. 1-5, 1997. Picard, R.. "A Society of Models for Video and Image Libraries." IBM Systems Journal. 1996. Rowley, H, and T. Kanade, Neural Network Based Face Detection, IEEE Trans. Patt. Anal. and Mach. Intell., vol. 20, no. 1, pp. 23-38, 1998. Smith, J. R. and S.F. Chang, “Visually Searching the Web for Content,” IEEE Multimedia, 1997, pg. 12-20. Sung, K. K., and T. Poggio, Example-Based Learning for View-Based Human Face Detection, IEEE Trans. on Patt. Anal. and Mach. Intell, vol. 20, no. 1, pp. 39-51, 1998. Taycher, L., M Cascia, and S. Sclaroff, “Image Digestion and Relevance Feedback in the ImageRover WWW Search Engine,” VISUAL’97, December, San Diego, pp. 85-91. Tekalp, A. M., Digital Video Processing, Prentice Hall, New Jersey, 1995. Vailaya, A., A. Jain and H. Zhang, "On Image Classification: City vs. Landscape," IEEE Workshop on Content-Based Access of Image and Video Libraries, Santa Barbara, June 21, 1998. Wang, L. and D. C. He, “Texture Classification Using Texture Spectrum,” Pattern Recognition 23, pp. 905-910, 1990.
Task Analysis for Information Visualization Stacie L. Hibino Bell Labs, Lucent Technologies 263 Shuman Boulevard, Naperville, IL 60566 USA [email protected] http://www.bell-labs.com/~hibino/
Abstract. Previous research in information visualization has primarily focused on providing novel views and frameworks to aid users in exploring or accessing data; very little work has been done to support users through the full analysis processfrom the raw data to the final results. But what tasks do users perform when analyzing data using an information visualization (infoVis) environment? A task analysis of experts’ use of an existing infoVis system was conducted to examine this question. Results indicate that users work on various tasks outside of data explorationtasks such as conditioning and preparing data, collecting results, and gathering evidence for a presentation. This pilot study identifies key data analysis tasks that expert users perform when using an infoVis environment to analyze some real-life data.
1 Introduction Recent advances in information visualization (infoVis) have led to novel visualizations and new paradigms to support users in accessing and exploring a wide variety of data. General infoVis frameworks [1,13] and applications to fields such as temporal [8], software [9] and medical [10] data are just a few examples of the growing work in infoVis. Most of this work, however, has focused on enhancing data access and exploration; little, if any work has examined the larger problem of supporting users’ data analysis processthe processes they use in transforming raw data into a presentation of results based on using infoVis as their key analysis tool. The continual introduction of more powerful computers and the recent explosion in large data repositories offers new opportunities while posing new challenges to the infoVis community. In our research lab, where we have already developed a suite of infoVis tools capable of handling moderately sized data sets (easily accommodating data sets containing 100,000 data records), we are beginning to see evidence of these challenges as users complain about the effort required to prepare data for exploration or abstract key results from an infoVis analysis of a complex data set. Previous work in information workspaces [3,7] offers glimpses of an organizing framework, but much of this work focuses on organizing data and objects (e.g., documents) rather than a user’s process.
Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL'99, LNCS 1614, pp. 139-146, 1999. Springer-Verlag Berlin Heidelberg 1999
140
Stacie L. Hibino
In order to understand how to better support users in the infoVis analysis process, we set out to identify the types of tasks they conduct during data analysis with an infoVis tool. In this paper, we describe the task analysis conducted to accomplish this goal. We had three driving questions in this endeavor: • Do users conduct other tasks besides data exploration during data analysis? • If so, what are these tasks? • How important are these tasks to the data analysis process? The task analysis was conducted on five infoVis experts using an existing infoVis environment (EDV: the Exploratory Data Visualizer [13]). Users were asked to analyze a disease data set used as part of an American Statistical Association (ASA) Data Exposition. Their goal at the end of their analysis sessions was to present key results of their analysis in their typical target presentation format. Overview. This paper is divided into four additional sections. In the next section, I describe the experimental method. I then present and discuss the results, summarize some related work and finally, I provide conclusions and describe some future work.
2 Experimental Method The goal of the task analysis was to study expert users analyzing real-life data with an existing infoVis system (EDV [13]) as their primary analysis toolwhere users started with the raw data, analyzed it, and worked towards a presentation of results. In order to examine how users cope with a complex data analysis problem requiring multiple sessions, the users were asked to accomplish this over two separate sessions. Participants. Five users (all male) participated in the study. They were invited to participate based on their expertise in using EDV as well as their experience in using information visualization (infoVis) environments for data analysis. Two users have a statistics background, while the other three hold advanced computer science degrees. EDV. The Exploratory Data Visualizer (EDV) is an information visualization framework which provides linked data views [13]. EDV allows users to create and interact with any number of dynamically linked, user-specified data views including univariate (e.g., bar charts and histograms), bivariate (e.g., scatterplots and table views), multivariate, network, and text-based views (e.g., similar to a spreadsheet). Views are dynamically linked to one another so that a selection made in one view results in automatic highlighting of the corresponding selections in all other views. Data Set. Users analyzed real-life tuberculosis (TB) disease data from the 1991 American Statistical Association Data Exposition [5]. The TB data included one primary file of over 11,000 records and five other supporting data files on information such as census data. Procedure. The users’ goal was to analyze the TB data and create a presentation of key results. They were given two 1-hour sessions, separated in time by at least one week, to accomplish this goal. They used EDV as their primary tool for analyzing the data, and were told that they could use any other tools that they typically use in conjunction with EDV to accomplish the analysis task. In the first session, users read a sheet describing this task analysis study and their goals, and a hard copy of the readme
Task Analysis for Information Visualization
141
file included with the TB data set. They were asked to provide think-aloud verbal protocols of what they were doing as they were doing it, and then proceeded with the analysis task at hand. At the end of the second session, if extra time was available, a short informal post-interview was conducted. Information about the post-questionnaire was sent to users after they had completed their second session. Data Collected. The following data was collected: observational data, files generated during the users’ sessions (e.g., scripts written to transform the data, new files of transformed data, etc.), post-interview notes, and results of the post questionnaire. In addition, output of the users’ screen was captured directly onto video, along with audio of their verbal protocols.
3 Results and Discussion While users did take different approaches to their analyses, their core set of tasks and findings were very similar. However, none of them created an actual presentation of their results. Although they analyzed a real-life data set, they were not motivated to create a presentation and in most cases did not feel that they had enough time to do so. Instead, they either captured their key findings on paper or articulated them aloud during their analysis. Information about presentation-related tasks was gathered through informal post-interviews and through the post-questionnaire. 3.1 High-Level Analysis Tasks During the task analysis, we observed several low-level user tasks. Through grouping similar tasks together, we identified seven categories of high-level tasks: • prepare: data background and preparation tasks, • plan: analysis planning and/or strategizing tasks, • explore: data exploration tasks, • present: presentation-related tasks, • overlay: overlay and assessment tasks, • re-orient: re-orientation tasks (when analysis requires more than one session), and • other: statistics-based tasks. The low-level tasks for each of the task categories are described in Section 3.2. When asked, users did not present an alternative list or any additional categories of highlevel tasks. Users rated the importance of each of the low-level tasks to the analysis process based on a scale of 1 to 5. Figure 1 shows the average importance ratings for each of the high-level task categories, based on an average of all of the low-level task ratings in each category. Overall, the ratings are fairly high, with averages ranging from 3.4 to 4.5 on a 5-point scale. However, taking individual users into account, an analysis of variance indicates that the differences in importance ratings between categories is significant, leading to the following order of importance based on user ratings: plan > explore > prepare > present > statistics > overlay > re-orient tasks.
142
Stacie L. Hibino
Fig. 1. Average importance ratings for task categories (1=unimportant, 5=very important; error bars indicate standard deviation)
3.2 Low-Level Analysis Tasks The low-level tasks were identified through observational data, previous infoVis taxonomies [12], informal interviews and through the post-questionnaire. Tables 1 to 6 summarize the low-level tasks by category and include user examples. While there is some overlap between tasks and task categories, these summaries provide a working taxonomy of data analysis through information visualization. Table 1. Prepare: data background and preparation tasks Task Description Gather background information about data set at hand Understand data sources Get clarification on data ambiguities Collect additional data from other external sources Reformat data for suitable input Check data for potential data errors Check for missing data Transform the data
Example Review TB readme file Note TB data file names, sizes, formats How is “race” different from “ethnicity?” Can I get my almanac? Add header information to a data file for importing into EDV Spot check raw data file Is there any missing data? Split variables, rollup/aggregate data
Table 2. Plan: analysis planning and strategizing tasks Task Description Hypothesize Make a strategy or plan for all or a part of your analysis Identify data formats and variables required for desired views
Example There was a hypothesis that TB incidence increased in HIV infected groups Decide what, how, and how much to investigate or explore We need sums of census data by state…
Task Analysis for Information Visualization
143
Table 3. Explore: data exploration tasks (incorporating tasks from [12]) Task Description Get an overview of the data Investigate data to test hypotheses (top down approach) Explore data in search of trends or exceptions (bottom-up approach) “Query” or filter the database Identify curiosities to investigate further Zoom in on items of interest Remove uninteresting items Identify data clusters Identify relationships between variables Explain view/visualization Identify a trend or exception Verify a trend or exception Drill-down for more details
Example I always like to use something to get some idea of the whole data set… There was a hypothesis that…. We can look at that actually Now let’s look at [the] race [variable] We go to race=2 [African American], we see that they get TB around … So that’s sort of interesting… I wonder, let’s… Let’s look at [just] that peak of youngsters I’m just going to eliminate those early years; concentrate on data where there at least seems to be stable reporting going on Alright, let’s try clustering…cluster view So in terms of age, whites seem to get TB more when they’re older in comparison to the other races… There are two possible answers [explanations] here. One is that… An interesting gap in the [age] data here… There’s a gap in around 12 year olds. Examine alternative view to verify a trend [looking at text records] You can actually count the number of 15 year olds...
Table 4. Present: presentation-related tasks Task Description Gather evidence to answer a hypothesis or driving question Record or keep track of trends and results tested and found Articulate importance of a result (rank it or identify it as “interesting”) Articulate/summarize all and/or key results Decide what to include in presentation of results Create presentation of results Give presentation of results
Example Well, in terms of the urban area hypothesis, it looks like it might be reasonably, likely; District of Columbia, which is an urban area … No significant effect per time-of-year How does this result rank in comparison to the others? I’m going to present a summary of my results in written form What are the top 2-3 interesting results I want to show? Paste screen dumps into an electronic presentation and annotate Communicate results to others
144
Stacie L. Hibino
Table 5. Overlay: overlay and assessment tasks Task Description Take notes Window management Assess your strategy Assess your observations Assess your assumptions about data formats Assess your progress Estimate cost-benefit ratio of additional data collection or conditioning
Example Write down code info: 1=male; 2=female Move and resize windows Is this the right strategy to take? Does this observation make sense? Is my data in the right format to accomplish this part of the analysis? I’m wondering if there’s anything else I haven’t considered which I should look at I should really recode those… but I couldn’t be bothered with that
Table 6. Re-Orient: re-orientation tasks Task Description Review goal(s), Review data and formats, Review notes Review progress Identify starting point for current session
Example Review TB readme file head tb.txt Flip through written notes What I remember doing last time was… So what I wanted to try doing today was…
Due to space limitations, we cannot discuss each of the low-level tasks in detail, but we do highlight some of the more interesting observations here. A couple of interesting data exploration tasks include explaining a visualization and verifying a trend or exception. These tasks, which have not previously been reported in other infoVis taxonomies, indicate that expert users do not just stop at the identification of a trend or exception, and that they consider their analysis to be as much of an investigation as an exploration. Different users have different types of target presentations. Users in this study listed a variety of typical target presentations ranging from static screen dumps to interactive web pages [6] and live EDV demonstrations. For small data sets and in situations where only a few results are identified, presentations are much easier to create and require fewer tasks. Large complex data sets, however, require additional work to keep track of, rank, and decide on presentation contents of results. This is especially the case when users may be sorting through a series of 20 to 30 results. The take notes and window management tasks are overlay tasks that cut across the other task categories. For example, users could take notes while preparing data, planning their analysis, or exploring the data. The assessment tasks identify the metacognitive activity exhibited by users during their analysis sessions. That is, users asked themselves the types of sample questions listed with each task in Table°5. Statistics-Based Tasks. Although users did not conduct any statistical tests during their analysis of the TB data, several users mentioned situations where they either would follow-up with a statistical test or where they thought it would be nice to have system support to conduct a particular statistical test.
Task Analysis for Information Visualization
145
Additional Tasks. Three users listed five tasks between them that they felt were not included in the task list presented in the post-questionnaire. Three of the five tasks listed were very similar to existing tasks while the other two included: • given a relationship or feature of interest, explore what other factors may contribute to it, and • sort and order results so that related results and their impact on each other can easily be accomplished.
4 Related Work Several infoVis taxonomies have been proposed (e.g., [12, 2]), but these typically focus on categorizing aspects of infoVis limited to accessing and exploring dataaspects such as data types, visualization types and exploration tasks. The results of the task analysis presented in this paper indicates that such taxonomies only address a part of the problem, especially when considering the use of infoVis for data analysis rather than only data access. While no infoVis environment currently addresses all of the types of tasks identified through this task analysis, some work has touched on some of the issues identified here. For example, the SAGE system [11] is a knowledge-based presentation system that potentially reduces the users’ task load on analysis planning as well as presentation-related tasks. A second example is an infoVis spreadsheet [4] that provides a framework for organizing data conditioning as well as data exploration based on graphical transformations. Information workspaces (e.g., [3, 7]) have typically focused on organizing data rather than processes. However, one can imagine using a rooms [7] or book metaphor [3] for organizing an infoVis analysis. For example, rooms or books could be used as logical separators for the different types of tasks (e.g., data preparation, analysis planning, data exploration, etc.) or they could be used to separate the analysis along themes or threads (e.g., separate rooms could be dedicated to investigations of different hypotheses). The challenge in using either of these metaphors, however, is in understanding how, if, and when process support can be bridged across rooms.
5 Conclusion and Future Work Users do perform many other tasks beyond data exploration when using an infoVis environment for analyzing data. In this pilot study, I identified six other categories of analysis tasks besides data exploration: data background and preparation, analysis planning and strategizing, presentation-related, overlay and assessment, re-orientation, and statistics-based tasks. Moreover, not only do users conduct these other tasks, they also rate them highly in terms of their importance to the analysis process. In particular, they rate planning and strategizing tasks significantly higher than exploration tasks. We are currently in the process of performing a detailed analysis of the video data and verbal protocols to identify how often users performed the various tasks as well as how much time they spent on each of them. In the mean time, we note that users
146
Stacie L. Hibino
indicated on the post-questionnaire that they typically spend, on average, about 25% of their analysis time on data exploration and at most 40% of their time; thereby spending over half of their analysis time on tasks other than data exploration. In the future, we plan to investigate and prioritize the importance of system support of each of the tasks. Our long-term goal is to work towards a more integrated infoVis framework, one that provides better support to users through the full data analysis process.
Acknowledgments Special thanks to expert users who participated in the study, to Graham Wills for EDV, and to Beki Grinter and Ken Cox for reviewing earlier drafts of this paper.
References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.
Ahlberg, C., & Shneiderman, B. (1994). Visual Information Seeking: Tight Coupling of Dynamic Query Filters with Starfield Displays. CHI'94 Conf. Proc. ACM Press, 313-317. Card, S. and J. Mackinlay. (1997). The Structure of the Information Visualization Design Space. IEEE Proceedings of Information Visualization’97, 92-99. Card, S., Robertson, G., and W. York. (1996). The WebBook and the Web Forager: an information workspace for the World-Wide Web. CHI '96. Conf. Proceedings, 111-119. Chi, E.H., Riedl, J., Barry, P., and J. Konstan. (1998). Principles for Information Visualization Spreadsheets. IEEE Computer Graphics & Applications, 18(4), 30-38. 1991 ASA Data Exposition, Disease Data. Available at: http://www.stat.cmu.edu/disease/. Eick, S., Mockus, A., Graves, T. and Karr, A. (1998). A Web Laboratory for Software Data Analysis. World Wide Web Journal, 12, 55-60. Henderson, J. & S. Card. (1986). Rooms: The use of multiple virtual workspaces to reduce space contention in window-based graphical user interfaces. ACM Transactions on Graphics, 5(3), 211-241. Hibino, S. and Rundensteiner, E. (1996). MMVIS: Design and Implementation of a Multimedia Visual Information Seeking Environment. ACM Multimedia’96 Conf. Proc. NY:ACM Press, 75-86. Jerding, D.F., Stasko, J.T. and Ball, T. (1997). Visualizing interactions in program executions. ICSE’97 Conference Proceedings. NY:ACM Press, 360-370. North, C., Shneiderman, B. and Plaisant, C. (1997). Visual Information Seeking in Digital Image Libraries: The Visible Human Explorer. Information in Images (G. Becker, Ed.), Thomson Technology Labs (http://www.thomtech.com/mmedia/tmr97/chap4.htm). Roth, S., Kolojejchick, Mattis, J. and J. Goldstein. Interactive graphic design using automatic presentation knowledge. CHI’94 Conference Proceedings, 318-322. Shneiderman, B. (1996). The eyes have it: A task by data type taxonomy for information visualizations. IEEE Proceedings of Visual Languages 1996, 336-343. Wills, G. (1995). Visual Exploration of Large Structured Datasets. New Techniques and Trends in Statistics. IOS Press, 237-246.
Filter Image Browsing Exploiting Interaction in Image Retrieval J. Vendrig, M. Worring, and A.W.M. Smeulders Intelligent Sensory Information Systems, Department of Computer Science University of Amsterdam Kruislaan 403, 1098 SJ Amsterdam, The Netherlands Tel/fax: ++31-20-525.7463/7490 {vendrig, worring, smeulders}@wins.uva.nl
Abstract. In current image retrieval systems the user refines his query by selecting example images from a relevance ranking. Since the top ranked images are all similar, user feedback often results in rearrangement of the presented images only. The Filter Image Browsing method provides better incorporation of user interaction in the retrieval process, because it is based on differences between images rather than similarities. Filter Image Browsing presents overviews of the database to users and lets them iteratively zoom in on parts of the image collection. In contrast to many papers where a new system is just introduced, we performed an extensive evaluation of the methods presented using a user simulation. Results for a database containing 10,000 images show that Filter Image Browsing requires less effort from the user. The implementation of Filter Image Browsing in the ImageRETRO system is accessible via the Web.
1
Introduction
An important future research direction in image retrieval is the introduction of the ”human in the loop” [4]. User interaction can be helpful to reduce large diverse domains to small specific domains, exploiting human knowledge about the query and context. In our opinion, there are only three image retrieval methods that are truly interactive, i.e. the input and output of the methods accommodate further interaction with the user. The first two methods are subtypes of Query by Navigation, meaning the user input consists of a choice of one of the navigation controls provided by the system. The subtypes are distinguished based on the relation of the navigation controls to the content of the image database: – Visual Inspection. Controls are content-independent, e.g. ”next” buttons. – Query by Association. Controls are content-dependent, e.g. hyperlinks. The third interactive image retrieval method is a special case of the wellknown Query by Pictorial Example method, viz. Query by internal Pictorial Example (QiPE). In a QiPE system user input consists of one or more images Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 147–155, 1999. c Springer-Verlag Berlin Heidelberg 1999
148
J. Vendrig et al.
being selected from the database. The system returns a relevance ranking, i.e. a list containing the distances from the user input images to images in the database. In practice, systems make use of more than one retrieval method. Visual Inspection is always involved in showing the output images of all other methods. Although trivial, the use of Visual Inspection alone is too time consuming. Therefore we concentrate on the characteristics of the other two interactive methods, viz. Query by Association and QiPE. The advantage of Query by Association is the use of structure, which can easily be visualized. However, it requires construction of the relations between the items in the database, which is usually done manually. QiPE is very flexible and dynamic. The search path is determined run-time by computing relevance rankings. However, there are two important drawbacks to the QiPE method. Firstly, users can get stuck at a local optimum. Secondly, the selection of the initial set of images to show to the user is non-trivial. It is generally neglected in literature though. While traditional systems do not help users after answering a query in further specification of their information need, interactive systems do provide the user more overviews than just an initial one. Recent research does take advantage of interaction by providing relevance feedback, e.g. by extracting knowledge from the browse path [5] or giving feedback by visualizing distances between images [3,7]. The Filter Image Browsing method we developed combines the powerful concepts for interaction found in Query by internal Pictorial Example and Query by Association. Furthermore, our method makes use of the browse path of a user when interacting. Filter Image Browsing is described in section 2. In section 3 evaluation criteria and the simulation environment are described. The results of experiments are presented in section 4. Conclusions are given in section 5.
2
Filter Image Browsing
In Filter Image Browsing (FilIB) a user recursively selects dynamically generated clusters of images. By choosing the cluster most similar to the information need, the user zooms in on a small collection of relevant images. The scatter/gather method [2] uses a comparable approach for the retrieval of textual documents, but focuses on the visualization of features of document clusters in the form of words.. The goal of FilIB is to assist in the quick retrieval of images by facilitating interaction between system and user with database overviews. FilIB can be seen as a structuring overlay over QiPE. The structuring overlay handles the lack of overview in traditional QiPE systems. Alternatively, FilIB can be viewed as the addition of a dynamic zoom function for image databases to Query by Association. In this section, the method and its consequences are discussed. Furthermore, a detailed description of the basics of Filter Image Browsing is given.
Filter Image Browsing
149
Fig. 1. Filter Image Browsing retrieval process.
2.1
The Retrieval Process
A Filter Image Browsing retrieval session (Fig. 1) first presents an initial overview of the content of the database in the form of images. The user inspects the images shown and selects the one most similar to the images he is looking for. Then the system performs a similarity ranking for the selected image. Next, the filter step, which characterizes this method, is performed. Only images most alike the query image are used in the remainder of the retrieval session. The three steps overview, selection and reduction are repeated until the set of remaining images is small enough to switch to Visual Inspection. Each overview is based on the subset of images in that state of the retrieval process so that the user zooms in on the image database. However, the reduction filter potentially results in the loss of relevant images during the retrieval process. Once an image is excluded, it cannot be retrieved anymore during that particular session. The amount of reduction and the number of required selections should be balanced to minimize loss of relevant images and time spent searching. For the sake of brevity, we do not consider navigational options such as “back (to previous state)” here. More formally explained, let Is be the active set with Ns images in state s of the retrieval process. In the initial state s=0, I0 is the entire image database of size N0 . In state s=0, the system presents images in the overview Is ⊆Is . Then the user selects seed images i∈ Is . For the sake of simplicity we limit i to one seed image. The system performs a filter operation ϕ resulting in a new active set of images: Is+1 =ϕ(Is ,i). The user ends the session when the desired images are found or when he thinks the images cannot be found. The latter case indicates that either the interactive system failed or the desired images are not present in the database. In the following paragraphs the overview and filter operations are explored in more detail. As shown later, the overview operation is influenced by the filter operation. Therefore, the filter operation is described first. Filter The appliance of a filter results in a change of scope of the active image set. Since the purpose of FilIB is to zoom in on a set of images that suits the user’s information need, reduction is the only filter operation considered here. The goal of reduction is getting a smaller set of images still containing the images desired. Similarity ranking of the active set according to user selected
150
J. Vendrig et al.
images is an appropriate technique to select images for the new (reduced) set. Is+1 then contains the images from Is that are most similar to query images i. A reduction factor ρ is used. E.g., if ρ=0.25, Is+1 is 4 times as small as Is . Thus the reduction operation targets at a fixed size for the new active set so that the outcome is predictable. When all parameters are known, it is easy to compute the number of steps necessary to reduce I0 to an image set suitable for Visual Inspection. Since Ns =ρ·N0 , s can be computed for a fixed end size Nend : send =
log
Nend N0
log ρ
.
(1)
Overview The goal of the overview function is to present to the user a number of images representative for the active image set. The number of presented images must be small, since the user looks at the images by way of Visual Inspection. The images in Is are the only access to other images in Is . Combined with the reduction operation this results in orphan images in Is , i.e. images i for which it is not possible to choose an image from Is resulting in an Is+1 that contains i. Orphan images are always lost in the next state. To guarantee Is is fully covered by Is , we introduce the cover constraint, which says every image in Is must be part of at least one of the possible new active sets Is+1 . Thus the overview function must produce a presentation set Is that complies to the following constraint: ϕ(i, Is ) = Is . (2) i∈Is
The selection of a presentation set that complies to the cover constraint consists of three stages: Is = postSelection(coverConstraint(preSelection)). In stage 1 a preselection is made, i.e. a set of input images are determined, e.g. randomly or by a user. In stage 2 the preselection is extended so that the total set of images complies to the cover constraint. If the number of images required for compliance to the cover constraint exceeds the maximum number of images shown to the user, the reduction factor should be adapted by user or system. In the optional stage 3 a postselection is made to extend the set to a predefined size. Postselection ensures a predictable amount of output to the user. We have constructed a brute force algorithm that guarantees compliance to the cover constraint for a presentation set Is . The preselection is one seed image, either randomly chosen from I0 , or selected by the user from Is−1 . The image is the first member of Is . Then it is used to perform a reduction on a copy of Is . Images in the (virtually) reduced set are checked as being a child of a member of Is . The algorithm then uses the image least similar to the last seed image to find the new seed image. Again, it is added to Is . The process is repeated until all images in Is are known to be a child of at least one of the images in Is . In a best case scenario, 1/ρ seed images are necessary. In practice the brute force approach resulted in an average of 12 images necessary to cover our data sets for ρ=0.25. In our opinion Is typically should contain about 20 images. The
Filter Image Browsing
151
brute force algorithm does not exceed this maximum. We have implemented a scenario for the postselection of images that consists of the random selection of images from Is . The results for this scenario are discussed in section 4.
3
Evaluation
In order to evaluate the Filter Image Browsing concept an experiment on the ImageRETRO system1 was set up. In the following sections we describe the evaluation criteria and the experimental environment. 3.1
Criteria
For evaluation of the effectiveness of Filter Image Browsing, separate criteria are used for each of the two functions, viz. reduction and overview. It is assumed that the information need of the user, expressed by the set of target images T⊂I0 , is static during the entire session. Reduction Evaluation To measure the effect of discarding images during the filter process, the Recall in each state of the retrieval session is computed: |Is T | . (3) Recalls = |T | In the best case, the reduction operation discards irrelevant images only, and Recall is 1 (maximum) in every state. The reduction criterion measures to what degree the best case scenario is approached in practice. By definition, Recall is 1 in initial state 0. To prevent forced loss of relevant images, in every state of the retrieval session Ns has to be equal to or greater than the size of T. Overview Evaluation For comparison of presentation set generators, Sought Recall (SR) [6] in each state of the retrieval session is measured. SRs expresses the amount of relevant images the user has actually seen on screen during the current and previous states. (Ij T ) j=0..s . (4) SRs = |T | The values for SR of FilIB can be predicted by using Recall values, assuming all images in Is have an equal chance of being selected for Is : SRs ≈ 1
|Is | · Recalls . Ns
http://carol.wins.uva.nl/˜vendrig/imageretro/
(5)
152
J. Vendrig et al.
Since SR is known to the user, the prediction can be used to derive Recall. Subsequently conclusions about continuing or restarting the session can be made. In our opinion the use of representative images in FilIB should lead to better results than in the case of the presentation of a relevance ranking, albeit a local optimum. In order to test this hypothesis, SR of both methods is compared. FilIB has to outperform QiPE (relevance ranking system) within a reasonable amount of user interactions. 3.2
Experiment
Domain A large and diverse domain was chosen to populate the image database, viz. the World Wide Web. The collection of 10,000 images2 is representative for the domain. For every image, index values were computed for 9 simple image features [8], primarily based on the distribution of hue, saturation and intensity. Ranking The relevance of an image is computed by averaging similarity scores of all individual features. Since features cannot be expected to have similar distributions of their values, scores cannot be averaged to a meaningful overall score directly. Therefore a normalization function is added to the feature scoring function. For each feature a priori a histogram containing the frequency of a similarity score is made. When the feature scoring function is invoked run-time, it looks up the similarity score in the histogram and returns the percentile, e.g. stating that the similarity between the two objects is in the top 5%. The percentiles of the various features can be compared and averaged with one another because they are independent of the distribution and similarity metric used. The use of the normalization via frequencies does not only allow the use of different types of image features and similarity metrics, e.g. histograms and metric values, but also the use of features of other media than images. Simulation In the evaluation experiment users were simulated by a user model as introduced in [1]. In the user model it is assumed that all users are the same and that the decisions they make are based on image features, so that the modeled users are consistent and unbiased. Target sets defining the simulated information need comply with 3 conditions: – Small distance in feature space, so that clustering the target images is possible. – Same style (visual similarity). Style is defined by objective meta-data (common original site) and subjective evaluation of the images. – Size bandwidth. There is a minimum and maximum for the number of images to focus on finding medium sized groups of images. For the experiment seven target sets3 were selected from the image collection. 2 3
publicly available at http://carol.wins.uva.nl/˜vendrig/icons/ http://carol.wins.uva.nl/˜vendrig/imageretro/target/
Filter Image Browsing
1
153
FilIB (0.2) FilIB (0.3) FilIB (0.4) FilIB (0.5) QiPE
Recall
0.9
0.8
0.7 10000
Sought Recall
1
0.75
0.5
0.25
0
1000
100
Size (logarithmic scale)
10
1
2
3
4
5
6
7
8
# interactions
9
10
11
Fig. 2. Evaluation results. Effect of reduction for various reduction factors (left) and retrieval performance of FilIB and QiPE (right).
The algorithms used for simulation of FilIB and QiPE make use of the predefined target sets and a given presentation size for Is (the desired number of pictures to be shown) fixed on 20 in our experiments. The reduction factor ρ is given as a constant for each retrieval session. Both simulations use the choose seed function, which computes which image in the presentation set is most similar to the entire target set. If one of the images in the presentation set is a member of the target set, it is chosen by default. For QiPE I0 is given, i.c. I0 =overview(I0). The presentation set function returns overview (Is ,i) for FilIB, and the top ranked images from Is for QiPE. The convergence criterion is “Ns equals presentation size” for Filter Image Browsing, and “high similarity of Is and Is+1 (>80% overlap)” for QiPE. Finally, the PresentationSetsimilarity ranking function results in ϕ(Is ,i) for Filter Image Browsing, and a plain similarity ranking relative to seed image i for QiPE.
4
Results & Discussion
Simulations were run for the 7 target sets, with 5 different randomized seeds. In Filter Image Browsing simulations, for each of the reduction factors used, the average of 5 slightly varying reduction factors was used. The reduction factors mentioned are the medians. The results for each Filter Image Browsing simulation are based on 175 runs in total. In Fig. 2 the graphs for the evaluation of the reduction operation are shown. Since the loss of relevant images depends on the reduction factor and the resulting size Ns , Recall is expressed as a function of Ns in the reduction effect graph. The reduction evaluation shows that even though Filter Image Browsing does cause the loss of desired images, far more irrelevant than relevant images are discarded. Low reduction factors of about 0.3, as well as high reduction factors of about 0.5 result in good performance. For comparison of Filter Image Browsing to a QiPE system a cluster based overview was evaluated for both systems. In the case of QiPE this means the initial presentation set only is constructed by
154
J. Vendrig et al.
way of clustering techniques. For Filter Image Browsing four different reduction factors were used. The graphs for Filter Image Browsing simulations stop when converged. The number of states necessary to converge can be computed from equation 1. The maximum number of interactions for both FilIB and QiPE is 11. The retrieval performance graph shows that even though both FilIB and QiPE find approximately the same amount of relevant images, the former method requires less user interactions to reach that result.
5
Conclusions
The concept of Filter Image Browsing shows that incorporating user interaction in an image retrieval method pays off. Subsequent cycles of database overview and reduction lead the user in few steps to a small collection of similar images. The consequences of the inherent ambiguity in the selection of representative images based on a combination of feature similarities is left for future research. The simulations used to evaluate the performance of Filter Image Browsing show satisfying results for all determined criteria. We conclude that more elaborate use of user interaction does result in quicker retrieval of images. Furthermore, the results of Filter Image Browsing are more predictable, as the number of user interactions can be computed a priori. This indicates that the method is helpful as well when desired images are not present in the image collection, since a user does not have to search indefinitely. Thus the combination of Query by internal Pictorial Example and Query by Association into Filter Image Browsing results in a powerful method for browsing through image databases.
References 1. I. Cox, M. Miller, S. Omohundro, and P. Yianilos. Target testing and the PicHunter bayesian multimedia retrieval system. In Proceedings of the Advanced Digital Libraries (ADL’96) Forum, pages 66–75, Washington D.C., 1996. 152 2. D. Cutting, D. Karger, J. Pedersen, and J. Tukey. Scatter/gather: A cluster-based approach to browsing large document collections. In Proceedings of SIGIR’92, Copenhagen, Denmark, 1992. 148 3. Y. Rubner, C. Tomasi, and L. Guibas. Adaptive color-image embeddings for database navigation. In Proceedings of ACCV, pages 104–111, Hongkong, 1998. 148 4. Y. Rui, T. Huang, and S.-F. Chang. Image retrieval: Current techniques, promising directions and open issues. Journal of Visual Communication and Image Representation, 10:1–23, 1999. 147 5. Y. Rui, T. Huang, M. Ortega, and S. Mehrotra. Relevance feedback: A power tool in interactive content-based image retrieval. IEEE Trans on Circuits and Systems for Video Technology, 8(5):644–655, 1998. 148 6. G. Salton and M. McGill. Introduction to Modern Information Retrieval. McGrawHill, New York, 1983. 151
Filter Image Browsing
155
7. S. Santini and R. Jain. Beyond query by example. In Proceedings of the Sixth ACM International Multimedia Conference, pages 345–350, Bristol, England, 1998. 148 8. J. Vendrig, M. Worring, and A. Smeulders. Filter image browsing. Technical Report 5, Intelligent Sensory Information Systems, Faculty WINS, Universiteit van Amsterdam, 1998. 152
Visualization of Information Spaces to Retrieve and Browse Image Data Atsushi Hiroike1 , Yoshinori Musha1 , Akihiro Sugimoto1 , and Yasuhide Mori2 1
2
Information-Base Functions Hitachi Laboratory, RWCP c/o Advanced Research Laboratory, Hitachi Ltd., Hatoyama, Saitama 350-03, JAPAN. Tel. +81-492-96-6111, Fax. +81-492-96-6006 {he,sha,sugimoto}@harl.hitachi.co.jp Information-Base Functions Tsukuba Laboratory, RWCP [email protected]
Abstract. We have developed a user interface for similarity-based image retrieval, where the distribution of retrieved data in a high-dimensional feature space is represented as a dynamical scatter diagram of thumbnail images in a 3-dimensional visualization space and similarities between data are represented as sizes in the 3-dimensional space. Coordinate systems in the visualization space are obtained by statistical calculations on the distribution of feature vectors of retrieved images. Our system provides some different transformations from a high-dimensional feature space to a 3-dimensional space that give different coordinate systems to the visualization space. By changing the coordinates automatically at some intervals, a spatial-temporal pattern of the distribution of images is generated. Furthermore a hierarchical coordinate system that consists of some local coordinate systems based on key images can be defined in the visualization space. These methods can represent a large number of retrieved results in a way that users can grasp intuitively.
1
Introduction
In recent years, image retrieval systems based on similarity have been reported by many researchers and companies [1], [2], [3]. We have been studying this issue from two points of view, i.e. “metrization” and “visualization.” For metrization, we are developing pattern recognition technologies applicable to a large-scale database containing various types of images. For visualization, our purpose is to find a suitable user interface for image retrieval systems, which is the main topic in this report. The importance of visualization in similarity-based image retrieval systems has been discussed by several researchers [4]. We will report our prototype system that uses the dynamical 3-dimensional representation.
2 2.1
Basic Concepts Retrieval System as an Extension of Human Perception
In conventional retrieval systems the conditions of queries are well-defined in an algorithm, e.g. “documents including a keyword”, “data created in 1998”, . . . . Dionysius P. Huijsmans, Arnold W.M. Smeulders (Eds.): VISUAL’99, LNCS 1614, pp. 155–163, 1999. c Springer-Verlag Berlin Heidelberg 1999
156
Atsushi Hiroike et al.
In similarity-based systems, the definitions of conditions are not clear, because similarity is a subjective or psychological criterion. It changes among individuals and can change within a single individual. A user may not have a definite criterion; that is, the condition of a query may be obscure. Unfortunately, we have not yet developed enough technology to give a general definition of the similarity applicable to an arbitrary image. Therefore it is not guaranteed that a user will be satisfied with a decision of the retrieval system. Furthermore, it is not certain that a user would be satisfied even if the system were as intelligent as a human being, since he or she might not want to accept the decision of another person. Our main objective is to develop a similarity-based retrieval system which allows users to encounter as many data as possible. This is in contrast to the concept of conventional retrieval systems where it is important to reduce the number of retrieved results appropriately. A computer system is inferior to a human being at pattern recognition, but superior in processing a large amount of data. Our system should present an understandable representation of a mass of data to users. It will enable a user to grasp a large information space that cannot be directly perceived. That is, the system will extend the user’s perception system. The system should assist users to make their queries clear and also be adaptable to changes of their intentions. It should not discourage users but should enhance their imagination. 2.2
Representation of Metric Information
In our system, as in other similarity-based retrieval systems, each image is represented as a vector or a set of vectors, where each vector is a feature characterizing a specific property of the image. In many cases, the dimensionality of the vector is high. Our basic idea for the user interface is to use visualization of the metric space that consists of feature vectors. The 2- or 3-dimensional representation of a data distribution in a highdimensional space is not a new topic, but a classical issue in multivariate analysis. However, whereas the conventional usage applies multivariate analysis to represent the static structure of data, we use it to represent the status of the system changed by a user’s operation. In our system, the resultant representation is interactive and dynamic. A user can give a requirement to the system through the visualization space; the requirement induces a change in the internal state of the system; and this change is reported to the user in the visualization space. The internal state of the system consists of abstract quantities, e.g. similarities defined in a high-dimensional space, which are difficult for users to understand. The visualization transforms these quantities into visible ones that can be perceived directly by users. In the desktop-metaphor where logical or symbolic relationships among objects in file systems are visualized, the intuitive operations of files and directories are available. In our system where metric relationships among pattern information are visualized, the intuitive operations in retrieving and browsing image data will be available.
Visualization of Information Spaces to Retrieve and Browse Image Data
157
Most similarity-based retrieval systems present retrieved images as a list sorted according to the similarity. This is quasi-1-dimensional visualization, which represents the order of the similarity between the key image and the results. In the previous report, we proposed a system using 2-dimensional visualization [5], [6]. In that system, retrieved results are displayed as a scatter diagram of thumbnail images in a 2-dimensional space constructed from features used in the similarity calculation. The coordinates of the space are eigenvectors given by applying principal component analysis (PCA) to the feature vectors of the retrieved results. Therefore the space is optimal in the second order to represent a distribution of retrieved data. List-representation may gather similar images in the higher ranks. However, it loses the impression of orderliness in the lower ranks because adjacency among the retrieved images has no meaning. So a user will be unhappy if given a long list. In the 2-dimensional representation, the configuration of data represents the similarity among retrieved data, and gives a well-ordered impression to users. Consequently the system can report more than 100 data without boring a user.
3 3.1
Model of the Metrization Features
Color-based feature A histogram of color distribution in an image is calculated by dividing the color space into Nr × Ng × Nb boxes. If compositional information is needed, an image is divided into Nx ×Ny rectangular areas and multiple histograms are calculated. In this case, the number of dimension is Nr × Ng × Nb × Nx × Ny . Gradient-based feature This feature is based on a direction-distribution of gradient vectors in a gray-scale image. Suppose that the directions of vectors with a range −π/2 ≤ θ < π/2 are quantized into Nθ levels. Let v = {vk |k = 0, . . . Nθ −1} be a vector in which accumulation results are stored and (fx , fy ) be a gradient vector at a pixel. vk is updated as vk → vk + fx2 + fy2 , where k is determined bythe direction of (fx , fy ). Finally, each element of v is normalized as vk → vk /S, where S is the number of pixels used in the updating. In the same way as for the color-based feature, features can be extracted in Nx × Ny separated areas of an image. Furthermore it is well known that the resolution of an image is important in this type of feature. If Nl -level pyramidal data consisting of gray-scale images with different resolutions are given, the number of dimensions of the feature vector is Nθ × Nx × Ny × Nl . Usually, we reduce the dimensions of these features by using PCA on the whole of database. For example, a 1024-dimensional color-feature (Nr = Ng = Nb = Nx = Ny = 4) and a 512-dimensional gradient-feature (Nθ = 8,Nx = Ny = 4, Nl = 4) are transformed into 100-dimensional feature vectors respectively. This reduction of dimensions is necessary to perform statistical calculations depending on the retrieval.
158
3.2
Atsushi Hiroike et al.
Similarity
The definition of similarity between two images is based on the squared distance. Let Nf be the number of feature types, and {xi } and {yi } (i = 0, . . . , Nf − 1) be sets of feature vectors of an image X and an image Y respectively. The similarity between X and Y is defined as 2 wi xi − yi , (1) s(X, Y ) = exp − i
where each squared distance is assumed to be normalized appropriately, xi − yi 2 = 1 for all i, and w = {wi } is a non-negative weight vector. Our system allows a user to retrieve images using multiple keys which we call “reference images.” A user can also arbitrarily divide reference images into some groups, each of which we call a “cluster.” The similarity between an image X and a cluster C is defined as s(X, C) = min s(X, Yi ) . Yi ∈C
(2)
A weight vector w is assigned to a cluster. Increasing the total value of w decreases the similarities within a cluster. This is counterintuitive for users. Therefore we transform a weight vector W specified on the user level into w so that an increase in the mean value of W increases the similarities in the cluster. The −2 definition of w is wi = Nf W Wi , where W = i Wi . In addition, each cluster has an attribute of “positive” or “negative.” While in the normal case clusters are positive, an image in a negative cluster is what a user does not want included in retrieved data. Let the similarity of an image X given by the system be s(X), and let C be the nearest cluster to X. s(X) = −s(X, C) if C is negative; otherwise, s(X) = s(X, C). The retrieved results that the system finally reports are limited by the threshold of s(X) and the maximum number of retrieved data. 3.3
Statistical Calculation
To obtain appropriate visualization, we perform statistical calculations depending on the retrieved results. The modified statistics based on retrieved data are calculated within each cluster. Let {Xi } be a set of retrieved images whose nearest cluster is C, and let {xij } (j = 0, . . . , Nf ) be feature vectors of Xi , which are not necessarily identical to the features used in eq. 1. The statistics of the j-th feature in C are defined as p s(Xi , C) (3) N = i
µj = N −1
p
s(Xi , C) xij
(4)
s(Xi , C)p (xij − µj ) (xij − µj )t .
(5)
i
Σj = N −1
i
For p = 0, this is identical to the statistics of retrieved data used in our previous model. For p → ∞, this is the statistics of the reference data.
Visualization of Information Spaces to Retrieve and Browse Image Data
159
a. View of the user interface.
b. Spatial-temporal pattern of a distribution of images.
c Fig. 1. The user interface of our system (sample images 1996 PhotoDisc, Inc).
4
Main features of the user interface
In the prototype system, a user selects key images from previously retrieved results (the system usually provides some sample keys at the first step). Fig. 1-a shows a view of the user interface. The background is the visualization space, where the dynamical 3-dimensional scatter diagram of thumbnail images is displayed. In this case, up to 1,000 images are retrieved from among about 10,000. There are some panels at the front. The top one is a “collection panel” where a user keeps his favorite images, which has no effect on the retrieval. The panel displayed at the bottom is a “control panel” equipped with GUI components for controlling the view of the visualization space. The other panels are cluster panels. If a user selects a reference image by clicking the mouse,
160
Atsushi Hiroike et al.
the image is displayed on a cluster panel specified by the user. In this example, the front one is a positive cluster panel, and the back one is a negative cluster panel. If the user considers a retrieved image to be an error or noise, he can add it to the negative cluster panel. The sliders on the cluster panels are for setting the weights of features. The main features of the user interface are as follows. Representation as a spatial-temporal pattern Many transformations from a high-dimensional feature space to a 2- or 3-dimensional visualization space are possible. In the previous system using 2-dimensional representation, a user selects a combination of coordinates by himself. In the present system, the coordinate system changes automatically at some intervals (Fig. 1-b). The system provides a series of 3-dimensional coordinate systems. An element of the series determines the position of each image at a time step. The positions between two adjacent steps are defined by linear interpolation between the positions in the two different coordinate systems. This makes the independent motion of each image and a 3-dimensional spatial-temporal pattern of an image distribution. The buttons in the top left-hand corner of the “control panel” show the current step, and clicking them causes the view to jump to an arbitrary step. The upper one of the two sliders below the buttons controls of the speed of motion, and the lower one is for setting the view at an arbitrary point between two adjacent steps while pausing the motion. Representation of similarity The similarity of each retrieved image is represented as the size of the image in the 3-dimensional space. The function of the transformation from a similarity to a size has two parameters S = βsα , where α determines the intensity of the effect of the similarity and β is a general scaling parameter. The GUI component on the left side of the “control panel,” which is the rectangular area with the small black circle, is the controller for this function. The x-y coordinates of the area correspond to the parameters of α and β. Local coordinate systems and hierarchical representation Fig. 2 illustrates the local coordinate systems based on reference images. Reference images are located on a global coordinate system, and each of the retrieved results is located on a local coordinate system around the most similar reference. In the case of Fig. 1-a, the statistics of retrieved data were calculated (eq. 5), and the eigenspaces of color-based and gradient-based features were used as local and global coordinates respectively. The local coordinates enable intuitive representation of the relationship between a key image and each retrieved image, and more structured representation in displaying a large number of images. The coordinates used in visualizing are defined as a cluster-attribute. When some clusters are defined, the center of each cluster (the mean vector defined in eq. 4) is located on the root coordinate system, which is also constructed from the feature vectors. Walk-through functions The arrow-shaped buttons on the right of the control panel are GUI components for the walk-through: x or y-translations,
Visualization of Information Spaces to Retrieve and Browse Image Data
161
z-translations, and rotations. Our system provides two