Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max Planck Institute for Informatics, Saarbruecken, Germany
6493
Ron Kimmel Reinhard Klette Akihiro Sugimoto (Eds.)
Computer Vision – ACCV 2010 10th Asian Conference on Computer Vision Queenstown, New Zealand, November 8-12, 2010 Revised Selected Papers, Part II
13
Volume Editors Ron Kimmel Department of Computer Science Technion – Israel Institute of Technology Haifa 32000, Israel E-mail:
[email protected] Reinhard Klette The University of Auckland Private Bag 92019, Auckland 1142, New Zealand E-mail:
[email protected] Akihiro Sugimoto National Institute of Informatics Chiyoda, Tokyo 1018430, Japan E-mail:
[email protected]
ISSN 0302-9743 ISBN 978-3-642-19308-8 DOI 10.1007/978-3-642-19309-5
e-ISSN 1611-3349 e-ISBN 978-3-642-19309-5
Springer Heidelberg Dordrecht London New York Library of Congress Control Number: 2011921594 CR Subject Classification (1998): I.4, I.5, I.2.10, I.2.6, I.3.5, F.2.2 LNCS Sublibrary: SL 6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics
© Springer-Verlag Berlin Heidelberg 2011 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Coverpicture: Lake Wakatipu and the The Remarkables, from ‘Skyline Queenstown’ where the conference dinner took place. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)
Preface
The 2010 Asian Conference on Computer Vision took place in the southern hemisphere, in “The Land of the Long White Cloud” in Maori language, also known as New Zealand, in the beautiful town of Queenstown. If we try to segment the world we realize that New Zealand does not belong officially to any continent. Similarly, in computer vision we often try to define outliers while attempting to segment images, separate them to well-defined “continents” we refer to as objects. Thus, the ACCV Steering Committee consciously chose this remote and pretty island as a perfect location for ACCV2010, to host the computer vision conference of the most populated and largest continent, Asia. Here, on South Island we studied and exchanged ideas about the most recent advances in image understanding and processing sciences. Scientists from all well-defined continents (as well as ill-defined ones) submitted high-quality papers on subjects ranging from algorithms that attempt to automatically understand the content of images, optical methods coupled with computational techniques that enhance and improve images, and capturing and analyzing the world’s geometry while preparing for higher-level image and shape understanding. Novel geometry techniques, statistical-learning methods, and modern algebraic procedures rapidly propagate their way into this fascinating field as we witness in many of the papers one can find in this collection. For this 2010 issue of ACCV, we had to select a relatively small part of all the submissions and did our best to solve the impossible ranking problem in the process. We had three keynote speakers (Sing Bing Kang lecturing on modeling of plants and trees, Sebastian Sylwan talking about computer vision in production of visual effects, and Tim Cootes lecturing about modelling deformable object), eight workshops (Computational Photography and Esthetics, Computer Vision in Vehicle Technology, e-Heritage, Gaze Sensing and Interactions, Subspace, Video Event Categorization, Tagging and Retrieval, Visual Surveillance, and Application of Computer Vision for Mixed and Augmented Reality), and four tutorials. Three Program Chairs and 38 Area Chairs finalized the decision about the selection of 35 oral presentations and 171 posters that were voted for out of 739, so far the highest number of ACCV, submissions. During the reviewing process we made sure that each paper was reviewed by at least three reviewers, we added a rebuttal phase for the first time in ACCV, and held a three-day AC meeting in Tokyo to finalize the non-trivial acceptance decision-making process. Our sponsors were the Asian Federation of Computer Vision Societies (AFCV), NextWindow–Touch-Screen Technology, NICTA–Australia’s Information and Communications Technology (ICT), Microsoft Research Asia, Areograph–Interactive Computer Graphics, Adept Electronic Solutions, and 4D View Solutions.
VI
Preface
Finally, the International Journal of Computer Vision (IJCV) sponsored the Best Student Paper Award. We wish to acknowledge a number of people for their invaluable help in putting this conference together. Many thanks to the Organizing Committee for their excellent logistical management, the Area Chairs for their rigorous evaluation of papers, the Program Committee members as well as external reviewers for their considerable time and effort, and the authors for their outstanding contributions. We also wish to acknowledge the following individuals for their tremendous service: Yoshihiko Mochizuki for support in Tokyo (especially also for the Area Chair meeting), Gisela Klette, Konstantin Schauwecker, and Simon Hermann for processing the 200+ Latex submissions for these proceedings, Kaye Saunders for running the conference office at Otago University, and the volunteer students during the conference from Otago University and the .enpeda.. group at The University of Auckland. We also thank all the colleagues listed on the following pages who contributed to this conference in their specified roles, led by Brendan McCane who took the main responsibilities. ACCV2010 was a very enjoyable conference. We hope that the next ACCV meetings will attract even more high-quality submissions. November 2010
Ron Kimmel Reinhard Klette Akihiro Sugimoto
Organization
Steering Committee Katsushi Ikeuchi Tieniu Tan Chil-Woo Lee Yasushi Yagi
University of Tokyo, Japan Institute of Automation, Chinese Academy of Science, China Chonnam National University, Korea Osaka University, Japan
Honorary Chairs P. Anandan Richard Hartley
Microsoft Research India Australian National University, NICTA
General Chairs Brendan McCane Hongbin Zha
University of Otago, New Zealand Peking University, China
Program Chairs Ron Kimmel Reinhard Klette Akihiro Sugimoto
Israel Institute of Technology University of Auckland, New Zealand National Institute of Informatics, Japan
Local Organization Chairs Brendan McCane John Morris
University of Otago, New Zealand University of Auckland, New Zealand
Workshop Chairs Fay Huang Reinhard Koch
Ilan University, Yi-Lan, Taiwan University of Kiel, Germany
Tutorial Chair Terrence Sim
National University of Singapore
VIII
Organization
Demo Chairs Kenji Irie Alan McKinnon
Lincoln Ventures, New Zealand Lincoln University, New Zealand
Publication Chairs Michael Cree Keith Unsworth
University of Waikato, New Zealand Lincoln University, New Zealand
Publicity Chairs John Barron Domingo Mery Ioannis Pitas
University of Western Ontario, Canada Pontificia Universidad Cat´ olica de Chile Aristotle University of Thessaloniki, Greece
Area Chairs Donald G. Bailey Horst Bischof Alex Bronstein Michael S. Brown Chu-Song Chen Hui Chen Laurent Cohen Daniel Cremers Eduardo Destefanis Hamid Krim Chil-Woo Lee Facundo Memoli Kyoung Mu Lee Stephen Lin Kai-Kuang Ma Niloy J. Mitra P.J. Narayanan Nassir Navab Takayuki Okatani Tomas Pajdla Nikos Paragios Robert Pless Marc Pollefeys Mariano Rivera Antonio Robles-Kelly Hideo Saito
Massey University, Palmerston North, New Zealand TU Graz, Austria Technion, Haifa, Israel National University of Singapore Academia Sinica, Taipei, Taiwan Shandong University, Jinan, China University Paris Dauphine, France Bonn University, Germany Technical University Cordoba, Argentina North Carolina State University, Raleigh, USA Chonnam National University, Gwangju, Korea Stanford University, USA Seoul National University, Korea Microsoft Research Asia, Beijing, China Nanyang Technological University, Singapore Indian Institute of Technology, New Delhi, India International Institute of Information Technology, Hyderabad, India TU Munich, Germany Tohoku University, Sendai City, Japan Czech Technical University, Prague, Czech Republic Ecole Centrale de Paris, France Washington University, St. Louis, USA ETH Z¨ urich, Switzerland CIMAT Guanajuato, Mexico National ICT, Canberra, Australia Keio University, Yokohama, Japan
Organization
Yoichi Sato Nicu Sebe Stefano Soatto Nir Sochen Peter Sturm David Suter Robby T. Tan Toshikazu Wada Yaser Yacoob Ming-Hsuan Yang Hong Zhang Mengjie Zhang
The University of Tokyo, Japan University of Trento, Italy University of California, Los Angeles, USA Tel Aviv University, Israel INRIA Grenoble, France University of Adelaide, Australia University of Utrecht, The Netherlands Wakayama University, Japan University of Maryland, College Park, USA University of California, Merced, USA University of Alberta, Edmonton, Canada Victoria University of Wellington, New Zealand
Program Committee Members Abdenour, Hadid Achard, Catherine Ai, Haizhou Aiger, Dror Alahari, Karteek Araguas, Gaston Arica, Nafiz Ariki, Yasuo Arslan, Abdullah Astroem, Kalle August, Jonas Aura Vese, Luminita Azevedo-Marques, Paulo Bagdanov, Andy Bagon, Shai Bai, Xiang Baloch, Sajjad Baltes, Jacky Bao, Yufang Bar, Leah Barbu, Adrian Barnes, Nick Barron, John Bartoli, Adrien Baust, Maximilian Ben Hamza, Abdessamad BenAbdelkader, Chiraz Ben-ari, Rami Beng-Jin, AndrewTeoh
Benosman, Ryad Berkels, Benjamin Berthier, Michel Bhattacharya, Bhargab Biswas, Prabir Bo, Liefeng Boerdgen, Markus Bors, Adrian Boshra, Michael Bouguila, Nizar Boyer, Edmond Bronstein, Michael Bruhn, Andres Buckley, Michael Cai, Jinhai Cai, Zhenjiang Calder´ on, Jes´ us Camastra, Francesco Canavesio, Luisa Cao, Xun Carlo, Colombo Carlsson, Stefan Caspi, Yaron Castellani, Umberto Celik, Turgay Cham, Tat-Jen Chan, Antoni Chandran, Sharat Charvillat, Vincent
IX
X
Organization
Chellappa, Rama Chen, Bing-Yu Chen, Chia-Yen Chen, Chi-Fa Chen, Haifeng Chen, Hwann-Tzong Chen, Jie Chen, Jiun-Hung Chen, Ling Chen, Xiaowu Chen, Xilin Chen, Yong-Sheng Cheng, Shyi-Chyi Chia, Liang-Tien Chien, Shao-Yi Chin, Tat-Jun Chuang, Yung-Yu Chung, Albert Chunhong, Pan Civera, Javier Coleman, Sonya Cootes, Tim Costeira, JoaoPaulo Cristani, Marco Csaba, Beleznai Cui, Jinshi Daniilidis, Kostas Daras, Petros Davis, Larry De Campos, Teofilo Demirci, Fatih Deng, D. Jeremiah Deng, Hongli Denzler, Joachim Derrode, Stephane Diana, Mateus Didas, Stephan Dong, Qiulei Donoser, Michael Doretto, Gianfranco Dorst, Leo Duan, Fuqing Dueck, Delbert Duric, Zoran Dutta Roy, Sumantra
Ebner, Marc Einhauser, Wolfgang Engels, Christopher Eroglu-Erdem, Cigdem Escolano, Francisco Esteves, Claudia Evans, Adrian Fang, Wen-Pinn Feigin, Micha Feng, Jianjiang Ferri, Francesc Fite Georgel, Pierre Flitti, Farid Frahm, Jan-Michael Francisco Giro Mart´ın, Juan Fraundorfer, Friedrich Frosini, Patrizio Fu, Chi-Wing Fuh, Chiou-Shann Fujiyoshi, Hironobu Fukui, Kazuhiro Fumera, Giorgio Furst, Jacob Fusiello, Andrea Gall, Juergen Gallup, David Gang, Li Gasparini, Simone Geiger, Andreas Gertych, Arkadiusz Gevers, Theo Glocker, Ben Godin, Guy Goecke, Roland Goldluecke, Bastian Goras, Bogdan Gross, Ralph Gu, I Guerrero, Josechu Guest, Richard Guo, Guodong Gupta, Abhinav Gur, Yaniv Hajebi, Kiana Hall, Peter
Organization
Hamsici, Onur Han, Bohyung Hanbury, Allan Harit, Gaurav Hartley, Richard HassabElgawi, Osman Havlena, Michal Hayes, Michael Hayet, Jean-Bernard He, Junfeng Hee Han, Joon Hiura, Shinsaku Ho, Jeffrey Ho, Yo-Sung Ho Seo, Yung Hollitt, Christopher Hong, Hyunki Hotta, Kazuhiro Hotta, Seiji Hou, Zujun Hsu, Pai-Hui Hua, Gang Hua, Xian-Sheng Huang, Chun-Rong Huang, Fay Huang, Kaiqi Huang, Peter Huang, Xiangsheng Huang, Xiaolei Hudelot, Celine Hugo Sauchelli, V´ıctor Hung, Yi-Ping Hussein, Mohamed Huynh, Cong Phuoc Hyung Kim, Soo Ichimura, Naoyuki Ik Cho, Nam Ikizler-Cinbis, Nazli Il Park, Jong Ilic, Slobodan Imiya, Atsushi Ishikawa, Hiroshi Ishiyama, Rui Iwai, Yoshio Iwashita, Yumi
Jacobs, Nathan Jafari-Khouzani, Kourosh Jain, Arpit Jannin, Pierre Jawahar, C.V. Jenkin, Michael Jia, Jiaya Jia, JinYuan Jia, Yunde Jiang, Shuqiang Jiang, Xiaoyi Jin Chung, Myung Jo, Kang-Hyun Johnson, Taylor Joshi, Manjunath Jurie, Frederic Kagami, Shingo Kakadiaris, Ioannis Kale, Amit Kamberov, George Kanatani, Kenichi Kankanhalli, Mohan Kato, Zoltan Katti, Harish Kawakami, Rei Kawasaki, Hiroshi Keun Lee, Sang Khan, Saad-Masood Kim, Hansung Kim, Kyungnam Kim, Seon Joo Kim, TaeHoon Kita, Yasuyo Kitahara, Itaru Koepfler, Georges Koeppen, Mario Koeser, Kevin Kokiopoulou, Effrosyni Kokkinos, Iasonas Kolesnikov, Alexander Koschan, Andreas Kotsiantis, Sotiris Kown, Junghyun Kruger, Norbert Kuijper, Arjan
XI
XII
Organization
Kukenys, Ignas Kuno, Yoshinori Kuthirummal, Sujit Kwolek, Bogdan Kwon, Junseok Kybic, Jan Kyu Park, In Ladikos, Alexander Lai, Po-Hsiang Lai, Shang-Hong Lane, Richard Langs, Georg Lao, Shihong Lao, Zhiqiang Lauze, Francois Le, Duy-Dinh Le, Triet Lee, Jae-Ho Lee, Soochahn Leistner, Christian Leonardo, Bocchi Leow, Wee-Kheng Lepri, Bruno Lerasle, Frederic Li, Chunming Li, Hao Li, Hongdong Li, Stan Li, Yongmin Liao, T.Warren Lie, Wen-Nung Lien, Jenn-Jier Lim, Jongwoo Lim, Joo-Hwee Lin, Huei-Yung Lin, Weisi Lin, Wen-Chieh(Steve) Ling, Haibin Lipman, Yaron Liu, Cheng-Lin Liu, Jingen Liu, Ligang Liu, Qingshan Liu, Qingzhong Liu, Tianming
Liu, Tyng-Luh Liu, Xiaoming Liu, Yuncai Loog, Marco Lu, Huchuan Lu, Juwei Lu, Le Lucey, Simon Luo, Jiebo Macaire, Ludovic Maccormick, John Madabhushi, Anant Makris, Dimitrios Manabe, Yoshitsugu Marsland, Stephen Martinec, Daniel Martinet, Jean Martinez, Aleix Masuda, Takeshi Matsushita, Yasuyuki Mauthner, Thomas Maybank, Stephen McHenry, Kenton McNeill, Stephen Medioni, Gerard Mery, Domingo Mio, Washington Mittal, Anurag Miyazaki, Daisuke Mobahi, Hossein Moeslund, Thomas Mordohai, Philippos Moreno, Francesc Mori, Greg Mori, Kensaku Morris, John Mueller, Henning Mukaigawa, Yasuhiro Mukhopadhyay, Jayanta Muse, Pablo Nagahara, Hajime Nakajima, Shin-ichi Nanni, Loris Neshatian, Kourosh Newsam, Shawn
Organization
Niethammer, Marc Nieuwenhuis, Claudia Nikos, Komodakis Nobuhara, Shohei Norimichi, Ukita Nozick, Vincent Ofek, Eyal Ohnishi, Naoya Oishi, Takeshi Okabe, Takahiro Okuma, Kenji Olague, Gustavo Omachi, Shinichiro Ovsjanikov, Maks Pankanti, Sharath Paquet, Thierry Paternak, Ofer Patras, Ioannis Pauly, Olivier Pavlovic, Vladimir Peers, Pieter Peng, Yigang Penman, David Pernici, Federico Petrou, Maria Ping, Wong Ya Prasad Mukherjee, Dipti Prati, Andrea Qian, Zhen Qin, Xueyin Raducanu, Bogdan Rafael Canali, Luis Rajashekar, Umesh Ramalingam, Srikumar Ray, Nilanjan Real, Pedro Remondino, Fabio Reulke, Ralf Reyes, EdelGarcia Ribeiro, Eraldo Riklin Raviv, Tammy Roberto, Tron Rosenhahn, Bodo Rosman, Guy Roth, Peter
Roy Chowdhury, Amit Rugis, John Ruiz Shulcloper, Jose Ruiz-Correa, Salvador Rusinkiewicz, Szymon Rustamov, Raif Sadri, Javad Saffari, Amir Saga, Satoshi Sagawa, Ryusuke Salzmann, Mathieu Sanchez, Jorge Sang, Nong Sang Hong, Ki Sang Lee, Guee Sappa, Angel Sarkis, Michel Sato, Imari Sato, Jun Sato, Tomokazu Schiele, Bernt Schikora, Marek Schoenemann, Thomas Scotney, Bryan Shan, Shiguang Sheikh, Yaser Shen, Chunhua Shi, Qinfeng Shih, Sheng-Wen Shimizu, Ikuko Shimshoni, Ilan Shin Park, You Sigal, Leonid Sinha, Sudipta So Kweon, In Sommerlade, Eric Song, Andy Souvenir, Richard Srivastava, Anuj Staiano, Jacopo Stein, Gideon Stottinge, Julian Strecha, Christoph Strekalovskiy, Evgeny Subramanian, Ramanathan
XIII
XIV
Organization
Sugaya, Noriyuki Sumi, Yasushi Sun, Weidong Swaminathan, Rahul Tai, Yu-Wing Takamatsu, Jun Talbot, Hugues Tamaki, Toru Tan, Ping Tanaka, Masayuki Tang, Chi-Keung Tang, Jinshan Tang, Ming Taniguchi, Rinichiro Tao, Dacheng Tavares, Jo˜ao Manuel R.S. Teboul, Olivier Terauchi, Mutsuhiro Tian, Jing Tian, Taipeng Tobias, Reichl Toews, Matt Tominaga, Shoji Torii, Akihiko Tsin, Yanghai Turaga, Pavan Uchida, Seiichi Ueshiba, Toshio Unger, Markus Urtasun, Raquel van de Weijer, Joost Van Horebeek, Johan Vassallo, Raquel Vasseur, Pascal Vaswani, Namrata Wachinger, Christian Wang, Chen Wang, Cheng Wang, Hongcheng Wang, Jue Wang, Yu-Chiang Wang, Yunhong Wang, Zhi-Heng
Wang, Zhijie Wolf, Christian Wolf, Lior Wong, Kwan-Yee Woo, Young Wook Lee, Byung Wu, Jianxin Xue, Jianru Yagi, Yasushi Yan, Pingkun Yan, Shuicheng Yanai, Keiji Yang, Herbert Yang, Jie Yang, Yongliang Yi, June-Ho Yilmaz, Alper You, Suya Yu, Jin Yu, Tianli Yuan, Junsong Yun, Il Dong Zach, Christopher Zelek, John Zha, Zheng-Jun Zhang, Cha Zhang, Changshui Zhang, Guofeng Zhang, Hongbin Zhang, Li Zhang, Liqing Zhang, Xiaoqin Zheng, Lu Zheng, Wenming Zhong, Baojiang Zhou, Cathy Zhou, Changyin Zhou, Feng Zhou, Jun Zhou, S. Zhu, Feng Zou, Danping Zucker, Steve
Organization
Additional Reviewers Bai, Xiang Collins, Toby Compte, Benot Cong, Yang Das, Samarjit Duan, Lixing Fihl, Preben Garro, Valeria Geng, Bo Gherardi, Riccardo Giusti, Alessandro Guo, Jing-Ming Gupta, Vipin Han, Long Korchev, Dmitriy Kulkarni, Kaustubh Lewandowski, Michal Li, Xin Li, Zhu Lin, Guo-Shiang Lin, Wei-Yang
Liu, Damon Shing-Min Liu, Dong Luo, Ye Magerand, Ludovic Molineros, Jose Rao, Shankar Samir, Chafik Sanchez-Riera, Jordy Suryanarayana, Venkata Tang, Sheng Thota, Rahul Toldo, Roberto Tran, Du Wang, Jingdong Wu, Jun Yang, Jianchao Yang, Linjun Yang, Kuiyuan Yuan, Fei Zhang, Guofeng Zhuang, Jinfeng
ACCV2010 Best Paper Award Committee Alfred M. Bruckstein Larry S. Davis Richard Hartley Long Quan
Technion, Israel Institute of Techonlogy, Israel University of Maryland, USA Australian National University, Australia The Hong Kong University of Science and Technology, Hong Kong
XV
XVI
Organization
Sponsors of ACCV2010 Main Sponsor
The Asian Federation of Computer Vision Societies (AFCV)
Gold Sponsor
NextWindow – Touch-Screen Technology
Silver Sponsors
Areograph – Interactive Computer Graphics Microsoft Research Asia Australia’s Information and Communications Technology (NICTA) Adept Electronic Solutions
Bronze Sponsor
4D View Solutions
Best Student Paper Sponsor
The International Journal of Computer Vision (IJCV)
Best Paper Prize ACCV 2010 Context-Based Support Vector Machines for Interconnected Image Annotation Hichem Sahbi, Xi Li.
Best Student Paper ACCV 2010 Fast Spectral Reflectance Recovery Using DLP Projector Shuai Han, Imari Sato, Takahiro Okabe, Yoichi Sato
Best Application Paper ACCV 2010 Network Connectivity via Inference Over Curvature-Regularizing Line Graphs Maxwell Collins, Vikas Singh, Andrew Alexander
Honorable Mention ACCV 2010 Image-Based 3D Modeling via Cheeger Sets Eno Toeppe, Martin Oswald, Daniel Cremers, Carsten Rother
Outstanding Reviewers ACCV 2010 Philippos Mordohai Peter Roth Matt Toews Andres Bruhn Sudipta Sinha Benjamin Berkels Mathieu Salzmann
Table of Contents – Part II
Posters on Day 1 of ACCV 2010 Generic Object Class Detection Using Boosted Configurations of Oriented Edges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Oscar Danielsson and Stefan Carlsson
1
Unsupervised Feature Selection for Salient Object Detection . . . . . . . . . . . Viswanath Gopalakrishnan, Yiqun Hu, and Deepu Rajan
15
MRF Labeling for Multi-view Range Image Integration . . . . . . . . . . . . . . . Ran Song, Yonghuai Liu, Ralph R. Martin, and Paul L. Rosin
27
Wave Interference for Pattern Description . . . . . . . . . . . . . . . . . . . . . . . . . . . Selen Atasoy, Diana Mateus, Andreas Georgiou, Nassir Navab, and Guang-Zhong Yang
41
Colour Dynamic Photometric Stereo for Textured Surfaces . . . . . . . . . . . . Zsolt Jank´ o, Ama¨el Delaunoy, and Emmanuel Prados
55
Multi-Target Tracking by Learning Class-Specific and Instance-Specific Cues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Min Li, Wei Chen, Kaiqi Huang, and Tieniu Tan
67
Modeling Complex Scenes for Accurate Moving Objects Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jianwei Ding, Min Li, Kaiqi Huang, and Tieniu Tan
82
Online Learning for PLSA-Based Visual Recognition . . . . . . . . . . . . . . . . . Jie Xu, Getian Ye, Yang Wang, Wei Wang, and Jun Yang Emphasizing 3D Structure Visually Using Coded Projection from Multiple Projectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ryo Nakamura, Fumihiko Sakaue, and Jun Sato Object Class Segmentation Using Reliable Regions . . . . . . . . . . . . . . . . . . . Vida Vakili and Olga Veksler Specular Surface Recovery from Reflections of a Planar Pattern Undergoing an Unknown Pure Translation . . . . . . . . . . . . . . . . . . . . . . . . . . Miaomiao Liu, Kwan-Yee K. Wong, Zhenwen Dai, and Zhihu Chen Medical Image Segmentation Based on Novel Local Order Energy . . . . . . LingFeng Wang, Zeyun Yu, and ChunHong Pan
95
109 123
137 148
XVIII
Table of Contents – Part II
Geometries on Spaces of Treelike Shapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . Aasa Feragen, Francois Lauze, Pechin Lo, Marleen de Bruijne, and Mads Nielsen
160
Human Pose Estimation Using Exemplars and Part Based Refinement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yanchao Su, Haizhou Ai, Takayoshi Yamashita, and Shihong Lao
174
Full-Resolution Depth Map Estimation from an Aliased Plenoptic Light Field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tom E. Bishop and Paolo Favaro
186
Indoor Scene Classification Using Combined 3D and Gist Features . . . . . Agnes Swadzba and Sven Wachsmuth
201
Closed-Form Solutions to Minimal Absolute Pose Problems with Known Vertical Direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zuzana Kukelova, Martin Bujnak, and Tomas Pajdla
216
Level Set with Embedded Conditional Random Fields and Shape Priors for Segmentation of Overlapping Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xuqing Wu and Shishir K. Shah
230
Optimal Two-View Planar Scene Triangulation . . . . . . . . . . . . . . . . . . . . . . Kenichi Kanatani and Hirotaka Niitsuma
242
Pursuing Atomic Video Words by Information Projection . . . . . . . . . . . . . Youdong Zhao, Haifeng Gong, and Yunde Jia
254
A Direct Method for Estimating Planar Projective Transform . . . . . . . . . Yu-Tseh Chi, Jeffrey Ho, and Ming-Hsuan Yang
268
Spatial-Temporal Motion Compensation Based Video Super Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yaozu An, Yao Lu, and Ziye Yan Learning Rare Behaviours . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jian Li, Timothy M. Hospedales, Shaogang Gong, and Tao Xiang Character Energy and Link Energy-Based Text Extraction in Scene Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jing Zhang and Rangachar Kasturi
282
293
308
A Novel Representation of Palm-Print for Recognition . . . . . . . . . . . . . . . . G.S. Badrinath and Phalguni Gupta
321
Real-Time Robust Image Feature Description and Matching . . . . . . . . . . . Stephen J. Thomas, Bruce A. MacDonald, and Karl A. Stol
334
Table of Contents – Part II
XIX
A Biologically-Inspired Theory for Non-axiomatic Parametric Curve Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Guy Ben-Yosef and Ohad Ben-Shahar
346
Geotagged Image Recognition by Combining Three Different Kinds of Geolocation Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Keita Yaegashi and Keiji Yanai
360
Modeling Urban Scenes in the Spatial-Temporal Space . . . . . . . . . . . . . . . . Jiong Xu, Qing Wang, and Jie Yang
374
Family Facial Patch Resemblance Extraction . . . . . . . . . . . . . . . . . . . . . . . . M. Ghahramani, W.Y. Yau, and E.K. Teoh
388
3D Line Segment Detection for Unorganized Point Clouds from Multi-view Stereo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tingwang Chen and Qing Wang
400
Multi-View Stereo Reconstruction with High Dynamic Range Texture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Feng Lu, Xiangyang Ji, Qionghai Dai, and Guihua Er
412
Feature Quarrels: The Dempster-Shafer Evidence Theory for Image Segmentation Using a Variational Framework . . . . . . . . . . . . . . . . . . . . . . . . Bj¨ orn Scheuermann and Bodo Rosenhahn
426
Gait Analysis of Gender and Age Using a Large-Scale Multi-view Gait Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yasushi Makihara, Hidetoshi Mannami, and Yasushi Yagi
440
A System for Colorectal Tumor Classification in Magnifying Endoscopic NBI Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Toru Tamaki, Junki Yoshimuta, Takahishi Takeda, Bisser Raytchev, Kazufumi Kaneda, Shigeto Yoshida, Yoshito Takemura, and Shinji Tanaka A Linear Solution to 1-Dimensional Subspace Fitting under Incomplete Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hanno Ackermann and Bodo Rosenhahn
452
464
Efficient Clustering Earth Mover’s Distance . . . . . . . . . . . . . . . . . . . . . . . . . Jenny Wagner and Bj¨ orn Ommer
477
One-Class Classification with Gaussian Processes . . . . . . . . . . . . . . . . . . . . Michael Kemmler, Erik Rodner, and Joachim Denzler
489
A Fast Semi-inverse Approach to Detect and Remove the Haze from a Single Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Codruta O. Ancuti, Cosmin Ancuti, Chris Hermans, and Philippe Bekaert
501
XX
Table of Contents – Part II
Salient Region Detection by Jointly Modeling Distinctness and Redundancy of Image Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yiqun Hu, Zhixiang Ren, Deepu Rajan, and Liang-Tien Chia
515
Unsupervised Selective Transfer Learning for Object Recognition . . . . . . . Wei-Shi Zheng, Shaogang Gong, and Tao Xiang
527
A Heuristic Deformable Pedestrian Detection Method . . . . . . . . . . . . . . . . Yongzhen Huang, Kaiqi Huang, and Tieniu Tan
542
Gradual Sampling and Mutual Information Maximisation for Markerless Motion Capture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yifan Lu, Lei Wang, Richard Hartley, Hongdong Li, and Dan Xu Temporal Feature Weighting for Prototype-Based Action Recognition . . . Thomas Mauthner, Peter M. Roth, and Horst Bischof PTZ Camera Modeling and Panoramic View Generation via Focal Plane Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Karthik Sankaranarayanan and James W. Davis Horror Image Recognition Based on Emotional Attention . . . . . . . . . . . . . Bing Li, Weiming Hu, Weihua Xiong, Ou Wu, and Wei Li Spatial-Temporal Affinity Propagation for Feature Clustering with Application to Traffic Video Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jun Yang, Yang Wang, Arcot Sowmya, Jie Xu, Zhidong Li, and Bang Zhang Minimal Representations for Uncertainty and Estimation in Projective Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wolfgang F¨ orstner Personalized 3D-Aided 2D Facial Landmark Localization . . . . . . . . . . . . . . Zhihong Zeng, Tianhong Fang, Shishir K. Shah, and Ioannis A. Kakadiaris
554 566
580 594
606
619 633
A Theoretical and Numerical Study of a Phase Field Higher-Order Active Contour Model of Directed Networks . . . . . . . . . . . . . . . . . . . . . . . . . Aymen El Ghoul, Ian H. Jermyn, and Josiane Zerubia
647
Sparse Coding on Local Spatial-Temporal Volumes for Human Action Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yan Zhu, Xu Zhao, Yun Fu, and Yuncai Liu
660
Multi-illumination Face Recognition from a Single Training Image per Person with Sparse Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Die Hu, Li Song, and Cheng Zhi
672
Table of Contents – Part II
Human Detection in Video over Large Viewpoint Changes . . . . . . . . . . . . Genquan Duan, Haizhou Ai, and Shihong Lao Adaptive Parameter Selection for Image Segmentation Based on Similarity Estimation of Multiple Segmenters . . . . . . . . . . . . . . . . . . . . . . . . Lucas Franek and Xiaoyi Jiang
XXI
683
697
Cosine Similarity Metric Learning for Face Verification . . . . . . . . . . . . . . . Hieu V. Nguyen and Li Bai
709
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
721
Generic Object Class Detection Using Boosted Configurations of Oriented Edges Oscar Danielsson and Stefan Carlsson School of Computer Science and Communications Royal Inst. of Technology, Stockholm, Sweden {osda02,stefanc}@csc.kth.se
Abstract. In this paper we introduce a new representation for shapebased object class detection. This representation is based on very sparse and slightly flexible configurations of oriented edges. An ensemble of such configurations is learnt in a boosting framework. Each edge configuration can capture some local or global shape property of the target class and the representation is thus not limited to representing and detecting visual classes that have distinctive local structures. The representation is also able to handle significant intra-class variation. The representation allows for very efficient detection and can be learnt automatically from weakly labelled training images of the target class. The main drawback of the method is that, since its inductive bias is rather weak, it needs a comparatively large training set. We evaluate on a standard database [1] and when using a slightly extended training set, our method outperforms state of the art [2] on four out of five classes.
1
Introduction
Generic shape-based detection of object classes is a difficult problem because there is often significant intra-class variation. This means that the class cannot be well described by a simple representation, such as a template or some sort of mean shape. If templates are used, typically a very large number of them is required [3]. Another way of handling intra-class variation is to use more meaningful distance measures than the Euclidean or Chamfer distances typically used to match templates to images. Then, the category can potentially be represented by a substantially smaller number of templates/prototypes [4]. A meaningful distance measure, for example based on thin-plate spline deformation, generally requires correspondences between the model and the image [4–6]. Although many good papers have been devoted to computing shape based correspondences [4,7], this is still a very difficult problem under realistic conditions. Existing methods are known to be sensitive to clutter and are typically computed using an iterative process of high computational complexity. Another common way of achieving a compact description of an object category is to decompose the class into a set of parts and then model each part independently. For shape classes, the parts are typically contour segments [5,8,9], making R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 1–14, 2011. c Springer-Verlag Berlin Heidelberg 2011
2
O. Danielsson and S. Carlsson
such methods practical mainly for object classes with locally discriminative contours. However, many object classes lack such contour features. These classes might be better described by global contour properties or by image gradient structures that are not on the contour at all, as illustrated in figure 1.
Fig. 1. Intra-class shape variation too large for efficient template-based representation. No discriminative local features are shared among the exemplars of the class. However, exemplars share discriminative edge configurations.
In this paper we present an object representation that achieves a compact description of the target class without assuming the presence of discriminative local shape or appearance features. The main components of this representation are sparse and slightly flexible configurations of oriented edge features, built incrementally in a greedy error minimization scheme. We will refer to these configurations as weak classifiers. The final strong classifier is a linear combination of weak classifiers learnt using AdaBoost [10]. We also stress that the method can be trivially extended to include other types of features (for example corners or appearance features) when applicable. We use the attentional cascade of Viola and Jones for selection of relevant negative training examples and for fast rejection of negative test examples [11]. For detection we evaluate the slidingwindow approach and we also point out the possibility of a hierarchical search strategy. Feature values are in both cases computed efficiently using the distance transform [12]. We achieve detection performance comparable to state-of-the-art methods. Our main contributions are: (i) a generic, shape-based object class representation that can be learnt automatically in a discriminative framework from weakly labelled training examples, (ii) an efficient detection algorithm that combines an attentional cascade with efficient feature value computation and (iii) an evaluation showing very promising results on a standard dataset. In the following section we review other methods that use similar object representations. The rest of the paper is organized as follows. In section 3 we describe how our method represents the object category. In section 4 we describe the edge features and their computation. In sections 5 and 6 we describe the algorithms for learning and detection in detail. In sections 7 and 8 we present an experimental evaluation. Finally, in sections 9, 10 and 11, we analyze the results, suggest directions for future study and draw conclusions.
Object Class Detection Using Boosted Configurations of Oriented Edges
2
3
Related Work
In the proposed approach the target object category is represented by sparse, flexible configurations of oriented edges. Several other works have relied on similar representations. For example, Fleuret and Geman learn a disjunction of oriented edge configurations, where all oriented edges of at least one configuration are required to be present within a user-specified tolerance in order for detection to occur [13]. Wu et al. learn a configuration of Gabor basis functions, where each basis function is allowed to move within a user-specified distance from its preferred position in order to maximize its response [14]. The most similar representation is probably from Danielsson et al. [15], who use a combination of sparse configurations of oriented edges in a voting scheme for object detection. In their work, however, the number of configurations used to represent a class, the number of edges in each configuration and the flexibility tolerance of edges are all defined by the user and not learned automatically. They also use different methods for learning and detection. We argue that our representation essentially generalizes these representations. For example, by constraining the weak classifiers to use only a single edge feature we get a representation similar to that of Wu et al [14]. The proposed approach also extends the mentioned methods because the tolerance in edge location is learnt rather than defined by the user. Furthermore, being discriminative rather than generative, it can naturally learn tolerance thresholds with negative as well as positive parity, ie. it can also encode that an edge of a certain orientation is unlikely to be within a certain region. The use of a cascade of increasingly complex classifiers to quickly reject obvious negatives was pioneered by Viola and Jones [11,16]. The cascade also provides a means of focusing the negative training set on relevant examples during training. Each stage in the cascade contains an AdaBoost classifier, which is a linear combination of weak classifiers. Viola and Jones constructed weak classifiers by thresholding single Haar features, whereas our weak classifiers are flexible configurations of oriented edge features.
3
Object Class Representation
The key component of the object representation is the weak classifiers. A weak classifier can be regarded as a conjunction of a set of single feature classifiers, where a single feature classifier is defined by an edge feature (a location and orientation) along with a tolerance threshold and its parity. A single feature classifier returns true if the distance from the specified location to the closest edge with the specified orientation is within tolerance (i.e. it should be sufficiently small if the parity is positive and sufficiently large if the parity is negative). A weak classifier returns true if all its constituent single feature classifiers return true. A strong classifier is then formed by boosting the weak classifiers. Figure 2 illustrates how the output of the strong classifier is computed for two different examples of the target class. The output of the strong classifier is thresholded
4
O. Danielsson and S. Carlsson
w2
w1 +
w4 +
+
w1
= w1+w2+w4
w3 +
+
w4 +
= w1+w3+w4
Fig. 2. An object class is represented by a strong classifier, which is a linear combination of weak classifiers. Each weak classifier is a conjunction of single feature classifiers. Each single feature classifier is described by a feature (bars), a distance threshold (circles) and a parity of the threshold (green = pos. parity, red = neg. parity). The output of the strong classifier is the sum of the weights corresponding to the “active” weak classifiers, as illustrated for two different examples in the figure.
to determine class membership. The edge features will be described in the next section and the weak classifiers in section 5.1.
4
Edge Features
An edge feature defines a single feature classifier together with a threshold and its parity. In this section we define the edge feature and the feature value. We also describe how to compute feature values efficiently. An edge feature Fk = (xk , θk ) is defined by a location xk and orientation θk in a normalized coordinate system. The feature value, fk , is the (normalized) distance to the closest image edge with similar orientation (two orientations are defined as similar if they differ by less than a threshold, tθ ). In order to emphasize that feature values are computed by aligning the normalized feature coordinate system with an image, I, using translation t and scaling s, we write fk (I, t, s). See figure 3 for an illustration.
ε(I) s . f3 F1 F3 F2
s . f1 t
s
s . f2
Fig. 3. Feature values fk (I, t, s) are computed by translating (t) and scaling (s) features Fk = (xk , θk ) and taking the (normalized) distances to the closest edges with similar orientations in the image
Object Class Detection Using Boosted Configurations of Oriented Edges
εθ(I)
1
dθ(I,p) 3
ε(I)
I
5
2 3
Fig. 4. Images are preprocessed as follows: (1) extract a set of oriented edges E (I), (2) split E (I) into (overlapping) subsets Eθ (I) and (3) compute the distance transforms dθ (I, p).
In order to define fk (I, t, s), let E(I) = {. . . , (p , θ ) , . . .} be the set of oriented edge elements (or edgels) in image I (E(I) is simply computed by taking the output of any edge detector and appending the edge orientation at each edge point) and let Eθ (I) = (p , θ ) ∈ E(I) |cos(θ − θ)| ≥ cos(tθ ) be the set of edgels with orientation similar to θ. We can then define fk (I, t, s) as: fk (I, t, s) =
1 · min ||s · xk + t − p || s (p ,θ )∈Eθk (I)
(1)
Typically we define a set of features F by constructing a uniformly spaced grid in the normalized coordinate system, i.e. F = X × Y × Θ, where X , Y and Θ are uniformly spaced points. 4.1
Efficient Computation of Feature Values
The feature value fk (I, t, s) defined in the previous section can be computed using only a single array reference and a few elementary operations. The required preprocessing is as follows. Start by extracting edges E(I) from the input image I. This involves running an edge detector on the image and then computing the orientation of each edge point. The orientation is orthogonal to the image gradient and unsigned, i.e. defined modulo π. For edge points belonging to an edge segment, we compute the orientation by fitting a line to nearby points on the segment. We call the elements of E(I) edgels (edge elements). The second step of preprocessing involves splitting the edgels into several overlapping subsets Eθ (I), one subset for each orientation θ ∈ Θ (as defined in the previous section). Then compute the distance transform dθ (I, p) on each subset [12]: dθ (I, p) =
min
(p ,θ )∈Eθ (I)
||p − p ||
(2)
6
O. Danielsson and S. Carlsson
This step is illustrated in figure 4; in the first column of the figure the subsets corresponding to horizontal and vertical orientations are displayed as feature maps and in the last column the corresponding distance transforms dθ (I, p) are shown. Feature values can be efficiently computed as: fk (I, t, s) = dθk (I, s · xk + t)/s. This requires only one array reference and a few elementary operations.
5
Learning
The basic component of our object representation is the weak classifier (section 5.1). It is boosted to form a strong classifier (section 5.2). Finally, a cascade of strong classifiers is built (section 5.3). 5.1
Learning the Weak Classifiers
In this section we describe how to learn a weak classifier given a weighted set of training examples. The goal of the learner is to find a weak classifier with minimal (weighted) classification error on the training set. The input to the training algorithm is a set of training examples {Ij |j ∈ J } with target class cj ∈ {0, 1} and a weight distribution, {dj |j ∈ J }, j∈J dj = 1, over these examples. Each training image is annotated with the centroid tj and scale sj of the object in the image. A weak classifier is the conjunction of a set of single feature classifiers. A single feature classifier is defined by a feature, Fk , a threshold, t, and a parity, p ∈ {−1, 1}. The single feature classifier has the classification function g (I, t, s) = p · fk (I, t, s) ≤ p · t and the weak classifier thus has the classiN N fication function h (I, t, s) = i=1 gi (I, t, s) = i=1 (pi · fki (I, t, s) ≤ pi · ti ). The number of single feature classifiers, N , is learnt automatically. We want to find the weak classifier that minimizes the classification error, e, wrt. the current importance distribution: dj (3) e = Pj∼d (h (Ij , tj , sj ) = cj ) = {j∈J |h(Ij ,tj ,sj )=cj }
We use a greedy algorithm to incrementally nconstruct a weak classifier. In order to define this algorithm, let hn (I, t, s) = i=1 gi (I, t, s) be the conjunction of the first n single feature classifiers. Then let Jn = {j ∈ J |hn (Ij , tj , sj ) = 1} be the indices of all training examples classified as positive by hn and let en be the classification error of hn : en = dj (4) {j∈J |hn (Ij ,tj ,sj )=cj }
We can write en in terms of en−1 as: en = en−1 + dj {j∈Jn−1 |gn (Ij ,tj ,sj )=cj }
−
dj
{j∈Jn−1 |cj =0}
(5)
Object Class Detection Using Boosted Configurations of Oriented Edges
7
This gives us a recipe for learning the nth single feature classifier: gn = arg min dj g
(6)
{j∈Jn−1 |g(Ij ,tj ,sj )=cj }
We keep gn if en < en−1 , which can be evaluated easily using equation 5, otherwise we let N = n − 1 and stop. The process is illustrated in figure 5: in a) the weak classifier is not sufficiently specific to delineate the class manifold well, in b) the the weak classifier is too specific and thus inefficient and prone to overfitting, whereas in c) the specificity of the weak classifier is tuned to the task at hand.
-n-1 (a) Too generic
(b) Too specific
-n
-N
(c) Optimal
Fig. 5. The number of single feature classifiers controls the specificity of a weak classifier. a) Using too few yields weak classifiers that are too generic to delineate the class manifold (gray). b) Using too many yields weak classifiers that are prone to overfitting and that are inefficient at representing the class. c) We learn the optimal number of single feature classifiers for each weak classifier (the notation is defined in the text).
Learning a Single Feature Classifier. Learning a single feature classifier involves selecting a feature Fk from a “dictionary” F , a threshold t from R+ and a parity p from {−1, 1}. An optimal single feature classifier, that minimizes equation (6), can be found by evaluating a finite set of candidate classifiers [17]. However, typically a large number of candidate classifiers have to be evaluated, leading to a time consuming learning process. Selecting the unconstrained optimum might also lead to overfitting. We have observed empirically that our feature values (being nonnegative) tend to be exponentially distributed. This suggests selecting the threshold t for a particular feature as the intersection of two exponential pdfs, where μ+ is the (weighted) average of the feature values from the positive examples and μ− is the (weighted) average from the negative examples. − μ− μ+ μ (7) · t = ln μ+ μ− − μ+ The parity is 1 if μ+ ≤ μ− and -1 otherwise. We thus have only one possible classifier for each feature and can simply loop over all features and select the classifier yielding the smallest value for equation (6). This method for single feature classifier selection yielded good results in evaluation and we used it in our experiments.
8
5.2
O. Danielsson and S. Carlsson
Learning the Strong Classifier
We use asymmetric AdaBoost for learning the strong classifiers at each stage of the cascade [16]. This requires setting a parameter k specifying that false negatives cost k times more than false positives. We empirically found k = 3n− /n+ to be a reasonable choice. Finally the detection threshold of the strong classifier is adjusted to yield a true positive rate (tpr) close to one. 5.3
Learning the Cascade
The cascade is learnt according to Viola and Jones [11]. After the construction of each stage we run the current cascade on a database of randomly selected negative images to acquire a negative training set for the next stage. It is important to use a large set of negative images and in our experiments we used about 300.
6 6.1
Detection Sliding Window
The simplest and most direct search strategy is the sliding window approach in scale-space. Since the feature values are computed quickly and we are using a cascaded classifier, this approach yields an efficient detector. Typically we use a multiplicative step-size of 1.25 in scale, so that si+1 = 1.25 · si , and an additive step-size of 0.01 · s in position, so that xi+1 = xi + 0.01 · s and yi+1 = yi + 0.01 · s. Non-maximum suppression is used to remove overlapping detections. There are some drawbacks with the sliding window approach: (i) all possible windows have to be evaluated, (ii) the user is required to select a step-size for each search space dimension, (iii) the detector has to be made invariant to translations and scalings of at least half a step-size (typically by translating and scaling all training exemplars) and (iv) the output detections have an error of about half a step-size in each dimension of the search space (if the step-sizes are sufficiently large to yield an efficient detector these errors typically translate to significant errors in the estimated bounding box). Since the feature values fk represent distances, we can easily compute bounds (u) (l) (l) fk and fk , given a region S in search space, such that fk ≤ fk (I, t, s) ≤ (u) fk ∀ (t, s) ∈ S. This can be used to for efficient search space culling and should enable detectors with better precision at the same computational cost. We will explore the issue of hierarchical search in future work. 6.2
Estimating the Aspect Ratio
The detector scans the image over position and scale, but in order to produce a good estimate of the bounding box of a detected object we also need the aspect ratio (which typically varies significantly within an object class). We
Object Class Detection Using Boosted Configurations of Oriented Edges
9
estimate the aspect ratio of a detected object by finding one or a few similar training exemplars and taking the average aspect ratio of these exemplars as the estimate. We retrieve similar training exemplars using the weak classifiers that are turned “on” by the detected object. Each weak classifier, hk , is “on” for a subset of the positive training exemplars, Tk = {x|hk (x) = 1} and by requiring several weak classifiers to be “on” we get the intersection of the corresponding subsets: Tk1 ∩Tk2 = {x|hk1 (x) = 1 ∧ hk2 (x) = 1}. Thus we focus on a successively smaller subset of training exemplars by selecting weak classifiers that fire on the detected object, as illustrated in figure 6. We are searching for a non-empty subset of minimal cardinality and the greedy approach at each iteration is to specify the weak classifier that gives maximal reduction in the cardinality of the “active” subset (under the constraint that the new “active” subset is non-empty). h1 a
d b
c f
g
h1
h2
e
h3
h
a
d c
b
f
j i
(a)
g
h1
h2
e
h3
h
a
d b
j
f
i
(b)
h2
e c g
h3
h j i
(c)
Fig. 6. Retrieving similar training exemplars for a detected object (shown as a star). a) Weak classifiers h2 and h3 fire on the detection. b) Weak classifier h3 is selected, since it gives a smaller “active” subset than h2 . c) Weak classifier h2 is selected. Since no more weak classifiers can be selected, training exemplars “g” and “h” are returned.
7
Experiments and Dataset
We have evaluated object detection performance on the ETHZ Shape Classes dataset [1]. This dataset is challenging due to large intra-class variation, clutter and varying scales. Several other authors have evaluated their methods on this dataset: in [5] a deformable shape model learnt from images is evaluated, while hand drawn models are used in [1,18–20]. More recent results are [2,21,22], where [2] reports the best performance. Experiments were performed on all classes in the dataset and results are produced as in [5], using 5-fold cross-validation. We build 5 different detectors for each class by randomly sampling 5 subsets of half the images of that class. All other images in the dataset are used for testing. The dataset contains a total of 255 images and the number of class images varies from 32 to 87. Thus, the number of training images will vary from 16 to 43 and the test set will consist of about 200 background images and 16 to 44 images containing occurrences of the target object.
10
O. Danielsson and S. Carlsson
Bottles
Giraffes 1 0.9
0.8
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
0.5
DR
1 0.9
DR
DR
Applelogos 1 0.9
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0
0 0
0.5
1
1.5
0.1 0
0
0.5
1
1.5
(a) Applelogos
0.5
1
1.5
FPPI
(b) Bottles
(c) Giraffes
Mugs
Swans
1
1
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
DR
DR
0
FPPI
FPPI
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0 0
0.5
1
FPPI
(d) Mugs
1.5
0
0.5
1
1.5
FPPI
(e) Swans
Fig. 7. Detection rate (DR) plotted versus false positives per image (FPPI). The results from the initial experiment are shown by the blue (lower) curves and the results from the experiment using extended training sets are shown by the red (upper) curves.
Quantitative results are presented as the detection rate (the number of true positives divided by the number of occurrences of the target object in the test set) versus the number of false positives per image (FPPI). We prefer using FPPI, instead of precision, as a measure of error rate, since it is not biased by the number of positive and negative examples in the test set. Precision would for example be unsuitable for comparison to methods evaluated using a different cross-validation scheme, since this might affect the number of positive test examples. As in [5], a detection is counted as correct if the detected bounding box overlaps more than 20 % with the ground truth bounding box. Bounding box overlap is defined as the area of intersection divided by the area of union of the bounding boxes.
8
Results
Quantitative results are plotted in figure 7. The results of the initial experiment are shown by the blue (lower) curves. The problem is that there are very few training examples (from 16 to 43). Since the presented method is statistical in nature it is not expected to perform well when training on a too small training set. Therefore we downloaded a set of extra training images from Google Images. These images contained a total of 40 applelogos, 65 bottles, 160 giraffes, 67 mugs and 103 swans, which were used to extend the training sets. The red
Object Class Detection Using Boosted Configurations of Oriented Edges
(a) Applelogos true positives
(c) Bottles true positives
(b) Applelogos false positives
(d) Bottles false positives
(e) Giraffes true positives
(f) Giraffes false positives
(g) Mugs true positives
(h) Mugs false positives
(i) Swans true positives
11
(j) Swans false positives
Fig. 8. Example detections (true and false positives) for each class.
(upper) curves are the results using the extended training set and the detection performance is now almost perfect. In table 1 we compare the detection performance of our system to previously presented methods. Figure 8 shows some sample detections (true and false positives).
12
O. Danielsson and S. Carlsson
Table 1. Comparison of detection performance. We state the average detection rate at 0.3 and 0.4 FPPI and its standard deviation in parentheses. We compare to the systems of [2,5].
[email protected] [email protected] [5]@0.4 [2]@0.3 [2]@0.4
FPPI: FPPI: FPPI: FPPI: FPPI:
A. logos 95.5(3.2) 95.5(3.2) 83.2(1.7) 95.0 95.0
Bottles 91.9(4.1) 92.6(3.7) 83.2(7.5) 92.9 96.4
Giraffes 92.9(1.9) 93.3(1.6) 58.6(14.6) 89.6 89.6
Mugs 96.4(1.4) 97.0(2.1) 83.6(8.6) 93.6 96.7
Swans 98.8(2.8) 100(0) 75.4(13.4) 88.2 88.2
Our currently unoptimized MATLAB/mex implementation of the sliding window detector, running (a single thread) on a 2.8 GHz Pentium D desktop computer, requires on the order of 10 seconds for scanning a 640 x 480 image (excluding the edge detection step). We believe that this could be greatly improved by making a more efficient implementation. This algorithm is also very simple to parallelize, since the computations done for one sliding window position are independent of the computations for other positions.
9
Discussion
Viola and Jones constructed their weak classifiers by thresholding single Haarfeatures in order to minimize the computational cost of weak classifier evaluation [16]. In our case, it is also a good idea to put an upper limit on the number of features used by the weak classifiers in the first few stages of the cascade for two reasons: (1) it reduces the risk of overfitting and (2) it improves the speed of the learnt detector. However, a strong classifier built using only single-feature weak classifiers should only be able to achieve low false positive rates for object classes with small intra-class variability. Therefore later stages of the cascade should not limit the number of features used by each weak classifier (rather, the learner should be unconstrained).
10
Future Work
As mentioned, search space culling and hierarchical search strategies will be explored, ideally with improvements in precision and speed as a result. Another straight forward extension of the current work is to use other features in addition to oriented edges. We only need to be able to produce a binary feature map, indicating the occurrences of the feature in an image. For example we could use corners, blobs or other interest points. We could also use the visual words in a visual code book (bag-of-words) as features. This would allow modeling of object appearance and shape in the same framework, but would only be practical for small code books.
Object Class Detection Using Boosted Configurations of Oriented Edges
13
We will investigate thoroughly how the performance of the algorithm is affected by constraining the number of features per weak classifier and we will also investigate the parameter estimation method presented in section 6.2 and the possibility of using it for estimating other parameters of a detected object, like the pose or viewpoint.
11
Conclusions
In this paper we have presented a novel shape-based object class representation. We have shown experimentally that this representation yields very good detection performance on a commonly used database. The advantages of the presented method compared to its competitors are that (1) it is generic and does not make any strong assumptions about for example the existence of discriminative parts or the detection of connected edge segments, (2) it is fast and (3) it is able to represent categories with significant intra-class variation. The method can also be extended to use other types of features in addition to the oriented edge features. The main drawback of the method is that, since its inductive bias is rather weak, it needs a comparatively large training set (at least about 100 training exemplars for the investigated dataset).
Acknowledgement This work was supported by the Swedish Foundation for Strategic Research (SSF) project VINST.
References 1. Ferrari, V., Tuytelaars, T., Van Gool, L.: Object detection by contour segment networks. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3953, pp. 14–28. Springer, Heidelberg (2006) 2. Maji, S., Malik, J.: Object detection using a max-margin. In: Proc. of the IEEE Computer Vision and Pattern Recognition (2009) 3. Gavrila, D.M.: A bayesian, exemplar-based approach to hierarchical shape matching. IEEE Transactions on Pattern Analysis and Machine Intelligence 29 (2007) 4. Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 509–522 (2002) 5. Ferrari, V., Jurie, F., Schmid, C.: Accurate object detection with deformable shape models learnt from images. In: Proc. of the IEEE Computer Vision and Pattern Recognition (2007) 6. Thuresson, J., Carlsson, S.: Finding object categories in cluttered images using minimal shape prototypes. In: Bigun, J., Gustavsson, T. (eds.) SCIA 2003. LNCS, vol. 2749, pp. 1122–1129. Springer, Heidelberg (2003) 7. Carlsson, S.: Order structure, correspondence, and shape based categories. In: Forsyth, D., Mundy, J.L., Di Ges´ u, V., Cipolla, R. (eds.) Shape, Contour, and Grouping 1999. LNCS, vol. 1681, p. 58. Springer, Heidelberg (1999)
14
O. Danielsson and S. Carlsson
8. Shotton, J., Blake, A., Cipolla, R.: Contour-based learning for object detection. In: Proc. of the International Conference of Computer Vision (2005) 9. Opelt, A., Pinz, A., Zisserman, A.: A boundary-fragment-model for object detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 575–588. Springer, Heidelberg (2006) 10. Freund, Y., Schapire, R.E.: A short introduction to boosting. Journal of Japanese Society for Artificial Intelligence 14, 771–780 (1999) 11. Viola, P.A., Jones, M.J.: Robust real-time face detection. International Journal of Computer Vision 57, 137–154 (2004) 12. Breu, H., Gil, J., Kirkpatrick, D., Werman, M.: Linear time euclidean distance transform algorithms. IEEE Trans. Pattern Analysis and Machine Intelligence 17, 529–533 (1995) 13. Fleuret, F., Geman, D.: Coarse-to-fine face detection. International Journal of Computer Vision 4, 85–107 (2001) 14. Wu, Y.N., Si, Z., Fleming, C., Zhu, S.C.: Deformable template as active basis. In: Proc. of the International Conference on Computer Vision (2007) 15. Danielsson, O., Carlsson, S., Sullivan, J.: Automatic learning and extraction of multi-local features. In: Proc. of the International Conference on Computer Vision (2009) 16. Viola, P.A., Jones, M.J.: Fast and robust classification using asymmetric adaboost and a detector cascade. In: Proc. of Neural Information Processing Systems, pp. 1311–1318 (2001) 17. Fayyad, U.M.: On the Induction of Decision Trees for Multiple Concept Learning. PhD thesis, The University of Michigan (1991) 18. Ravishankar, S., Jain, A., Mittal, A.: Multi-stage contour based detection of deformable objects. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 483–496. Springer, Heidelberg (2008) 19. Zhu, Q., Wang, L., Wu, Y., Shi, J.: Contour context selection for object detection: A set-to-set contour matching approach. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 774–787. Springer, Heidelberg (2008) 20. Schindler, K., Suter, D.: Object detection by global contour shape. Pattern Recognition 41, 3736–3748 (2008) 21. Stark, M., Goesele, M., Schiele, B.: A shape-based object class model for knowledge transfer. In: Proc. of International Conference on Computer Vision (2009) 22. Ommer, B., Malik, J.: Multi-scale object detection by clustering lines. In: Proc. of International Conference on Computer Vision (2009)
Unsupervised Feature Selection for Salient Object Detection Viswanath Gopalakrishnan, Yiqun Hu, and Deepu Rajan School of Computer Engineering Nanyang Technological University, Singapore
Abstract. Feature selection plays a crucial role in deciding the salient regions of an image as in any other pattern recognition problem. However the problem of identifying the relevant features that plays a fundamental role in saliency of an image has not received much attention so far. We introduce an unsupervised feature selection method to improve the accuracy of salient object detection. The noisy irrelevant features in the image are identified by maximizing the mixing rate of a Markov process running on a linear combination of various graphs, each representing a feature. The global optimum of this convex problem is achieved by maximizing the second smallest eigen value of the graph Laplacian via semi-definite programming. The enhanced image graph model, after the removal of irrelevant features, is shown to improve the salient object detection performance on a large image data base with annotated ‘ground truth’.
1
Introduction
The detection of salient objects in images has been a challenging problem in computer vision for more than a decade. The task of identification of salient parts of an image is essentially that of choosing a subset of the input data which is considered as most relevant by the human visual system (HVS). The ‘relevance’ of such a subset is closely linked to two different task processing mechanisms of the HVS - a stimulus driven ‘bottom-up’ processing and a context dependent ‘top-down’ processing. The information regarding the salient parts in an image finds application in many computer vision tasks like content based image retrieval , intelligent image cropping to adapt to small screen devices, content aware image and video resizing [1, 5, 20, 21]. Salient region detection techniques include local contrast based approaches modeling the ‘bottom-up’ processing of HVS [15, 25], information theory based saliency computation [4], salient feature identification using discriminative object class features [7], saliency map computation by modeling color and orientation distributions [9] and using the Fourier spectrum of images [11, 14]. Graph based approaches for salient object detection have not been explored much although they have the advantage of capturing the geometric structure of the image when compared to feature-based clustering techniques. The limited major works in this regard include the random walk based methods in [12] and R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 15–26, 2011. c Springer-Verlag Berlin Heidelberg 2011
16
V. Gopalakrishnan, Y. Hu, and D. Rajan
[8]. In [12] edge strengths are used to represent the dissimilarity between two nodes and the most frequently visited node during the random walk is labeled as the most dissimilar node in a local context. In [8], random walks are performed on a complete and a sparse k-regular graph to capture the global and local properties of the most salient node. In [17] Liu et al. proposed a supervised learning approach in which 1000 annotated images were used to train a salient region detector which was tested on another 4000 images. We use the same database in this work to demonstrate the effectiveness of the proposed method. Feature selection, which is essentially selecting a subset of features from the original feature set (consequently representing an image in lower dimension), is quite important in many classification problems like face detection that involve high dimensional data. There has not been much work reported on unsupervised feature selection in contrast to supervised feature selection as the former is a far more challenging problem due to the lack of any label information. Feature selection methods can be generally classified as filter, embedded and wrapper methods depending on how the feature selection strategy is related to the classification algorithm that follows. He et.al[13] and Mitra et.al [18] discuss unsupervised ‘filter’ methods of feature selection in which the feature selection strategy is independent of the classification algorithm. The problem of simultaneously finding a classification and a feature subset in an unsupervised manner is discussed in the ‘embedded’ feature selection methods in [16] and [26]. Law et.al [16] use a clustering algorithm based on Gaussian Mixture Models along with the feature selection, while Zhao et.al [26] optimize an objective function to find a maximum margin classifier on the given data along with a set of feature weights that favor the features which most ‘respect’ the data manifold. In [24], a ‘wrapper’ strategy for feature selection is discussed in which the features are selected by optimizing the discriminative power of the classification algorithm used. Feature selection plays a vital role in salient object detection as well, like in many other pattern recognition problems. However, choosing the right mixture of features for identifying the salient parts of an image accurately is one problem that has not received much attention so far. For example, graph based methods like [8] give equal weight to all features while the saliency factor may be attributed only to a smaller subset of features. The relevancy of different features in different images is illustrated in Figure 1. The saliency of the image in Figure 1(a) is mainly derived from the color feature, in Figure 1(b) from texture features while the salient parts in Figure 1(c) is due to some combination of color and texture features. Hence it can be concluded that some of the features used in [8] are irrelevant and only add noise to the graph structure of the image thereby reducing the accuracy of the salient object detection. In this context, we propose an unsupervised filter method for identifying irrelevant features to improve the performance of salient region detection algorithms.
Unsupervised Feature Selection for Salient Object Detection
(a)
(b)
17
(c)
Fig. 1. Example images illustrating the relevancy of different features to saliency. The relevant salient features are (a) color (b) texture (c) some combination of color and texture features.
The major contributions of this paper can be summarized as follows: – We relate the problem of identifying irrelevant features for salient object detection to maximizing the mixing rate of a Markov process running on a weighted linear combination of ‘feature graphs’. – A semi-definite programming formulation is used to maximize the second smallest eigen value of the graph Laplacian to maximize the Markov process mixing rate. – The effectiveness of the proposed unsupervised filter method of feature selection (elimination) method is demonstrated by using it in conjunction with a graph based salient object detection method [8] resulting in improved performance on a large annotated dataset. The paper is organized as follows. Section 2 reviews some of the theory related to the graph models, Markov process equilibrium and the mixing rate of Markov processes. In Section 3 we discuss the relation between irrelevant feature identification and maximization of mixing rate of Markov process. The detailed optimization algorithm is also provided in Section 3. Section 4 details the experiments done and the comparative results showing performance improvement. Finally conclusions are given in Section 5.
2
Graph Laplacian and Mixing Rate of Markov Process
In this section we review some of the theory related to graph Laplacians and mixing rate of a Markov process running on the graph. Consider an undirected graph, G(V, E), where V is the set of n vertices and E is the set of edges in the graph. The graph G is characterized by the affinity matrix A where wij , i = j Aij = (1) 0, i = j. wij is the weight between node i and node j based on the affinity of features on the respective nodes.
18
V. Gopalakrishnan, Y. Hu, and D. Rajan
The Laplacian matrix L of the graph G is defined as L=D−A
(2)
D = diag {d1 , d2 ...dn } , di = wij ,
(3)
where
j
and di is the degree of node i and is the total weight connected to node i in the graph G. Now, consider a symmetric Markov process running on the graph G(V, E). The state space of the Markov process is the vertex (node) set V , and the edge set E represent the transition strengths. The edge eij connecting nodes i and j is associated with a transition rate wij which is the respective weight entry in the affinity matrix, A, of the graph G(V, E). This implies that the transition from node i to node j is faster when the affinities between the respective nodes are larger. Consequently, a Markov process entering node i at any specific time t will take more time to move into the nodes that possess lesser affinities with node i. The mixing time of a Markov process is the time taken to reach the equilibrium distribution π starting from some initial distribution π(0). Let π(t) ∈ Rn be the n dimensional vector showing probability distributions on the n states at time t for a symmetric Markov process starting from any given distribution on the graph. The evolution of π(t) is given by [19], dπ(t) = Q.π(t) dt where Q is the transition rate matrix of the Markov process defined as i = j qij , Qij = − j,j=i qij , i = j.
(4)
(5)
qij is the transition rate or rate of going from state i to state j. As explained previously, we equate this rate to the affinity between nodes i and j, wij . Hence the Laplacian matrix L and the transition rate matrix Q is related as Q = −L. Consequently, the solution for the distribution π(t) at time t starting from the initial condition π(0) is given as π(t) = e−tL π(0).
(6)
The Laplacian L is a symmetric and positive semi-definite matrix. The eigen values of L are such that λ1 = 0 ≤ λ2 ≤ .......... ≤ λN . λ2 is also known as the algebraic connectivity of the graph G. If the graph G is connected, i.e. any node of the graph is accessible from any other node through the edge set E, then λ2 > 0 [6]. The probability distribution π(t) will tend to the equilibrium distribution π as t → ∞. The total variation distance (maximum difference
Unsupervised Feature Selection for Salient Object Detection
19
in probabilities assigned by two different probability distributions to the same event) between distributions π(t) and π is upper bounded by λ2 as [22] sup π(t) − πtv ≤ (1/2)n0.5 e−λ2 t
(7)
where .tv is the total variance distance. Equation (7) implies that a larger value of λ2 will result in a tighter bound on the total variation distance between π(t) and π and consequently result in a faster mixing of the Markov process. The second smallest eigen value λ2 can be expressed as a function of some Laplacian matrix L as [3] (8) λ2 (L) = inf xT Lx | x = 1, 1T x = 0 where 1 is the vector with all ones. As L1 = 0, 1 is the eigen vector of the smallest eigen value, λ1 = 0, of L. Equation (8) implies that λ2 is a concave function of L, as it is the point wise infimum of a family of linear functions of L. Hence maximization of λ2 as a function of the Laplacian matrix L is a convex optimization problem for which a global maximum is guaranteed.
3
Feature Graphs and Irrelevant Features
We now relate the maximization of Markov process mixing rate to the identification of irrelevant features for salient object detection. In salient region detection methods like [8, 12] that use multiple features, all features are given equal importance. Our method for unsupervised feature selection aims to identify the important features by representing the image with several ‘feature graphs’, which implies that each feature is encoded into a separate image graph model. The nodes in the ‘feature graphs’ represents 8 × 8 patches in the image similar to [8]. The affinity between nodes in a feature graph depends on the respective features. If there are M features available for every node in the image, we have M different feature graphs in which the affinity between node i and node j for the mth feature is defined as equation (1) in which m
m = e−||(fi wij
−fjm ||2
,
(9)
where fim and fjm are the values of mth feature on node i and j respectively. It is evident that the connectivity in the M feature graphs will be different. The relevant features or the features that contribute to the salient parts of the image will have feature graphs with ‘bottlenecks’. The term ‘bottleneck’ implies a large group of edges which show very less affinity between two groups (clusters) of nodes. Such bottlenecks arise from the fact that the feature values for a certain feature on the salient part of an image will be different from the feature values on a non-salient part like the background. For example, in Figure 2(a), if the feature graph is attributed to the color feature, then the nodes connected by thick lines (indicating strong affinity) fall into two groups, each having a different color value. One group might belong to the salient object and the other to the
20
V. Gopalakrishnan, Y. Hu, and D. Rajan
background and both of them are connected by thin edges (indicating weak affinity). This group of thin edges signifies the presence of a bottleneck. This phenomenon of clustering of nodes does not occur in feature graphs attributed to irrelevant features and hence such graphs will not have bottlenecks. This is illustrated in Figure 2(b) for the same set of nodes as in Figure 2(a), but for a different feature. Here the affinities for the irrelevant feature vary smoothly (for example, consider the shading variations of color in Figure 1(b)) and hence do not result in a bottleneck in its respective feature graph. Therefore, a Markov process running on an irrelevant feature graph mixes faster compared to a Markov process in a relevant feature graph since in the relevant feature graph the transition time between two groups of nodes connected by very weak edge strengths will be always high. Now consider all possible graph realizations resulting as a linear combination of the feature graphs. When we maximize the Markov mixing rate of a linear combination of these feature graphs, what we are essentially doing is to identify the irrelevant features that does not form any kind of bottleneck.
(a)
(b)
Fig. 2. Feature graphs with the same set of nodes showing (a) bottleneck between two groups of nodes (b) no bottleneck. The thicker edges indicate larger affinity between nodes.
3.1
Irrelevant Feature Identification
We adopt a semi-definite programming method similar to [22] for maximizing the second smallest eigen value of the Laplacian. However, in [22] a convex optimization method is proposed to find the fastest mixing Markov process among all possible edge weights of the graph realization G(V, E) to give a generic solution. In our proposed method, searching the entire family of graph realizations that gives the fastest mixing rate will not give us the required specific solution. Hence we restrict our search to all possible linear combinations of the feature graphs. The maximization problem of the second smallest eigen value is formulated as maximize subject to
L=
k
λ2 (L) ak .Lk ,
ak = 1
k
ak ≥ 0, ∀k
.
(10)
Unsupervised Feature Selection for Salient Object Detection
21
where Lk is the Laplacian matrix of the k th feature graph and ak is the weight associated with the respective feature graph. Now we establish that the constraints on matrix L in Equation (10) make it a valid candidate for a Laplacian matrix. matrices of M feature graphs and if Lemma 1. If L1 ,L2 ,. . . ,LM are Laplacian L = k ak .Lk under the conditions k ak = 1 and ak ≥ 0, then L is also a Laplacian matrix. Lij = = Proof. j k ak .Lkij k j ak .Lkij j = a L = 0
k k ij k j Also Lij = k ak .Lkij is negative for i = j as ak ≥ 0 and Lkij ≤ 0 ∀k and for i = j. Similarly all diagonal elements of L are positive proving that L represents the Laplacian matrix of some graph resulting from the linear combination of feature graphs. The optimization problem in Equation (10) can be equivalently expressed as maximize s subject to sI ≺ L + β11T L= ak .Lk , ak = 1, ak ≥ 0, ∀k, β > 0 k
(11)
k
where the symbol ≺ represents matrix inequality. The equivalent optimization problem maximizes a variable s subject to some matrix inequality constraints and some linear equality constraints. The term β11T removes the space of matrices with the eigen vector 1 and eigen value 0 from the space of matrices L [2]. Hence maximizing the second smallest eigen value of L is equivalent to maximizing the smallest eigen value of L + β11T . Lemma 2. The maximum s that satisfies the matrix inequality constraint in Equation (11) is the minimum eigen value of the matrix variable L + β11T Proof. Let P = L + β11T . The smallest eigen value of P can be expressed as , T λmin = inf( xxTPxx ). Assume P ≺ λmin I. This implies that, P − λmin I is negative definite. ⇒ xT (P − λmin I)x < 0 for any x, ⇒ xT P x < λmin .xT x, T ⇒ xxTPxx < λmin T which is false since the minimum value of xxTPxx is λmin .
Hence P λmin I and the maximum s that satisifies the constraint P sI is λmin . The inequalities in Equation (11) can also be expressed as
Since L = as
k
sI − (L + β11T ) ≺ 0.
(12)
ak .Lk , the matrix inequality in Equation (12) can be expressed
sI + a1 (−L1 ) + a2 (−L2 ) + ....aM (−LM ) + β(−11T ) ≺ 0.
(13)
22
V. Gopalakrishnan, Y. Hu, and D. Rajan
which is a standard Linear Matrix Inequality (LMI) constraint in an SDP problem formulation [3] as all the matrices in Equation (13) are symmetric. The Equation (11) can be seen as the minimization of a linear function subject to an LMI and linear equality constraints and can be effectively solved by standard SDP solvers like CVX or SeDuMi. Our proposed algorithm for finding the weights of different features can be summarized as follows: 1) Calculate the feature graph of every feature with the n × n affinity matrix Am for mth feature according to Equations (9) and (1.) 2) Calculate the Laplacian matrices of M features L1 , L2 , ...LM according to Equations (2) and (3) 3) Formulate the SDP problem in Equation (11) with input data as L1 , L2 , ...LM and variables as s, β, L, a1 , a2 , ..., aM and solve for a1 , a2 , ...aM using a standard SDP solver. 4) Find the feature weights as r = 1 − a and normalize the weights.
4
Experiments
The effectiveness of the proposed algorithm for unsupervised feature selection is experimentally verified by using it in conjunction with the random walk based method for salient object detection described in [8], where a random walk is performed on a simple graph model for the image with equal weight for all features. The proposed method will provide the feature weights according to feature relevancy resulting in a much more accurate graph model in the context of salient object detection. The final image graph model is decided by the graph affinity matrix A, which is evaluated as ri .Ai (14) A= i
where r1 , r2 , ...rM are the final feature weights evaluated in Section 3.1 and A1 , A2 , ...AM are the feature affinity matrices. The proposed feature selection method falls in the class of unsupervised filter method of feature selection where the feature selection is independent of the classification algorithm that follows. Hence we compare the proposed method with two unsupervised filter methods of feature selection explained in [13] and [18].The Laplacian score based feature selection method [13] assigns a score to every feature denoting the extent to which each feature ‘respects’ the data manifold. In [18], an unsupervised filter method is proposed to remove redundant features by evaluating a feature correlation measure for every pair of features. The features are clustered according to their similarity and the representative feature of each set is selected starting from the most dissimilar feature set. However, as our proposed method weighs different features according to their importance, we use the feature correlation measure in [18] to weigh the features while comparing with the proposed method. We used the same dataset of 5000 images used in [8] which was originally provided by [17]. The dataset contains user annotations as rectangular boxes
Unsupervised Feature Selection for Salient Object Detection
23
around the most salient region in the image. The performance of the algorithm is objectively evaluated with the ground truth available in the data set. We use the same seven dimensional feature set used in [8]. The features are color (Cb and Cr) and texture (orientation complexity values at five different scales). According to our algorithm, there will be seven different feature graphs (transition matrices), one for each feature. The optimal weights of features are obtained as explained in Section 3.1 with semi-definite programming in Matlab using the cvx package [10]. Using the optimal feature weights, a new graph model for the image is designed according to Equation (14), which is used with [8] to detect the salient regions.
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 3. Comparison of saliency maps for [8] with the proposed feature selection. (a),(d) Original Images. (b),(e) Saliency maps using [8]. (c),(f) Saliency maps of [8] with the proposed feature selection.
Figure 3 shows saliency maps on some example images for which the original algorithm in [8] has resulted in wrong or partially correct saliency maps. The original images are shown in figure 3(a), 3(d) and figures 3(b) and 3(e) show the saliency map generated by [8], in which the image graph model is designed with equal importance to all features, whether relevant or irrelevant . Figure 3(c)and 3(f) shows the saliency maps resulting from the proposed feature selection method used with [8]. It can be seen that the saliency maps are more accurate in Figure 3(c) and 3(f) where the features are weighted according to their relevance. For example, consider the first image in Figure 3(a). The color feature plays a good role in saliency for this image and the proposed algorithm
24
V. Gopalakrishnan, Y. Hu, and D. Rajan Table 1. F-measure comparison for different methods Method
F-Measure
Random walk method [8] [8] with Laplacian score [13] based feature selection [8] with correlation based feature selection [18] [8] with proposed unsupervised feature selection
0.63 0.68 0.68 0.7
detects the salient region more accurately by giving more weightage to color feature. On the contrary, for the eagle image in the second row of 3(d), the color features do not have much role in deciding the salient region and the proposed algorithm gets a more accurate saliency map by giving more priority to the texture feature. In most of the real life images it is some specific mixture of various features ( e.g. a combination of color and orientation entropy feature at some scale) that proves to be relevant in the context of salient object detection. The proposed algorithm intends to model this rightly weighted combination of features as can be testified from the results for other images in Figure 3. It has to be also noted that, though our proposed method assigns soft weights to different features according to their relevance, a hard decision on the features as in a selection of a feature subset can be made by applying some heuristics to the soft weights (for example, select the minimum number of features that satisfies 90% of the total soft weights assigned). The ground truth available for the dataset allows us to objectively compare various algorithms using precision, recall and F-measure. Precision is calculated as ratio of the total saliency in the saliency map captured inside the user annotated rectangle (sum of intensities inside user box) to the total saliency computed for the image (sum of intensities for full image). Recall is calculated as the ratio of the total saliency captured inside the user annotated window to the area of the user annotated window. F-measure, which is the harmonic mean of precision and recall, is the final metric used for comparison. Table 1 lists the F-measure computed for various feature selection methods on the entire dataset. We compare the proposed unsupervised feature selection algorithm with the Laplacian score based feature selection algorithm [13] and the correlation based feature selection algorithm [18]. It can be seen from the table that the proposed method significantly improves the performance of the random walk method and is better than the methods in [13] and [18]. It has to be noted that the methods proposed by [17] and [23] claims better fmeasure than the feature selection combined graph based approach on the same data set. However [17] is a supervised method in which 1000 annotated images were used to train a salient region detector which was tested on the rest of 4000 images. In [23], after obtaining the saliency map, a graph cut based segmentation algorithm is used to segment the image. Some segments with higher saliency are combined to form final saliency map. The accuracy of [23] depends heavily on the accuracy of segmentation algorithm and on the techniques used to combine different segments to obtain the final salient object. Hence as an unsupervised
Unsupervised Feature Selection for Salient Object Detection
25
and independent method (like [25] and [11]) that provides a good f-measure on a large dataset, the graph based approach is still attractive and demands attention for future enhancements.
5
Conclusion
We have proposed an unsupervised feature selection method for salient object detection by identifying the irrelevant features. The problem formulation is convex and is guaranteed to achieve the global optimum. The proposed feature selection method can be used as a filter method to improve the performance of salient object detection algorithms. The power of the proposed method is demonstrated by using it in conjunction with a graph based algorithm for salient object detection and demonstrating improved performance.
References 1. Avidan, S., Shamir, A.: Seam carving for content-aware image resizing. ACM Trans. Graph. 26(3) (2007) ISSN: 0301–0730 2. Boyd, S.: Convex Optimization of Graph Laplacian Eigenvalues. In: Proceedings International Congress of Mathematicians, vol. 3, pp. 1311–1319 (2006) 3. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004) 4. Bruce, N.D., Tsotsos, J.K.: Saliency Based on Information Maximization. In: NIPS, pp. 155–162 (2005) 5. Chen, L.Q., Xie, X., Fan, X., Ma, W.Y., Zhang, H.J., Zhou, H.Q.: A visual attention model for adapting images on small displays. Multimedia Syst. 9, 353–364 (2003) 6. Chung, F.R.K.: Spectral Graph Theory. American Mathematical Society, Providence (1997) 7. Gao, D., Vasconcelos, N.: Discriminant Saliency for Visual Recognition from Cluttered Scenes. In: NIPS, pp. 481–488 (2004) 8. Gopalakrishnan, V., Hu, Y., Rajan, D.: Random walks on graphs to model saliency in images. In: CVPR, pp. 1698–1705 (2009) 9. Gopalakrishnan, V., Hu, Y., Rajan, D.: Salient Region Detection by Modeling Distributions of Color and Orientation. IEEE Transactions on Multimedia 11(5), 892–905 (2009) 10. Grant, M., Boyd, S.: CVX: Matlab software for disciplined convex programming (web page and software) (2009), http://stanford.edu/~ boyd/cvx 11. Guo, C., Ma, Q., Zhang, L.: Spatio-temporal Saliency Detection Using Phase Spectrum of Quaternion Fourier Transform. In: CVPR (2008) 12. Harel, J., Koch, C., Perona, P.: Graph-Based Visual Saliency. In: NIPS, pp. 545– 552 (2006) 13. He, X., Cai, D., Niyogi, P.: Laplacian Score for Feature Selection. In: NIPS (2002) 14. Hou, X., Zhang, L.: Saliency Detection: A Spectral Residual Approach. In: CVPR (2007) 15. Itti, L., Koch, C., Niebur, E.: A Model of Saliency-Based Visual Attention for Rapid Scene Analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)
26
V. Gopalakrishnan, Y. Hu, and D. Rajan
16. Law, M.H.C., Figueiredo, M.A.T., Jain, A.K.: Simultaneous Feature Selection and Clustering Using Mixture Models. IEEE Trans. Pattern Anal. Mach. Intell. (2004) 17. Liu, T., Sun, J., Zheng, N., Tang, X., Shum, H.Y.: Learning to Detect A Salient Object. In: CVPR (2007) 18. Mitra, P., Murthy, C.A., Pal, S.K.: Unsupervised Feature Selection Using Feature Similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 301–312 (2002) 19. Norris, J.: Markov Chains. Cambridge University Press, Cambridge (1997) 20. Santella, A., Agrawala, M., DeCarlo, D., Salesin, D., Cohen, M.F.: Gaze-based interaction for semi-automatic photo cropping. In: CHI, pp. 771–780 (2006) 21. Stentiford, F.: Attention based Auto Image Cropping. In: The 5th International Conference on Computer Vision Systems (2007) 22. Sun, J., Boyd, S., Xiao, L., Diaconis, P.: The Fastest Mixing Markov Process on a Graph and a Connection to a Maximum Variance Unfolding Problem. SIAM Review 48, 681–699 (2004) 23. Valenti, R., Sebe, N., Gevers, T.: Image Saliency by Isocentric Curvedness and Color. In: ICCV (2009) 24. Volker, R., Tilman, L.: Feature selection in clustering problems. In: NIPS (2004) 25. Walther, D., Koch, C.: Modeling attention to salient proto-objects. Neural Network 19, 1395–1407 (2006) 26. Zhao, B., Kwok, J., Wang, F., Zhang, C.: Unsupervised Maximum Margin Feature Selection with Manifold Regularization. In: CVPR (June 2009)
MRF Labeling for Multi-view Range Image Integration Ran Song1 , Yonghuai Liu1 , Ralph R. Martin2 , and Paul L. Rosin2 1 2
Department of Computer Science, Aberystwyth University, UK {res,yyl}@aber.ac.uk School of Computer Science & Informatics, Cardiff University, UK {Ralph.Martin,Paul.Rosin}@cs.cardiff.ac.uk
Abstract. Multi-view range image integration focuses on producing a single reasonable 3D point cloud from multiple 2.5D range images for the reconstruction of a watertight manifold surface. However, registration errors and scanning noise usually lead to a poor integration and, as a result, the reconstructed surface cannot have topology and geometry consistent with the data source. This paper proposes a novel method cast in the framework of Markov random fields (MRF) to address the problem. We define a probabilistic description of a MRF labeling based on all input range images and then employ loopy belief propagation to solve this MRF, leading to a globally optimised integration with accurate local details. Experiments show the advantages and superiority of our MRF-based approach over existing methods.
1
Introduction
3D surface model reconstruction from multi-view 2.5D range image has a wide range of applications in many fields such as reverse engineering, CAD, medical imagery and film industry, etc. Its goal is to estimate a manifold surface that approximates an unknown object surface using multi-view range images, each of which essentially represents a sample of points in 3D Euclidean space, combined with knowledge about the scanning resolution and the measurement confidence. These samples of points are usually described in local, system centred, coordinate systems and cannot offer a full coverage of the object surface. Therefore, to build up a complete 3D surface model, we usually need to register a set of overlapped range images into a common coordinate frame and then integrate them to fuse the redundant data contained in overlapping regions while retain enough data sufficiently representing the correct surface details. However, to achieve both is challenging due to its ad hoc nature. On the one hand, registered range images are actually 3D unstructured point clouds downgraded from original 2.5D images. On the other hand, scanning noise such as unwanted outliers and data loss typically caused by self-occlusion, large registration errors and connectivity relationship loss among sampled points in acquired data often lead to a poor reconstruction. As a result, the reconstructed surface may include holes, false connections, thick and non-smooth or over-smooth patches, and artefacts. R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 27–40, 2011. Springer-Verlag Berlin Heidelberg 2011
28
R. Song et al.
Hence, a good integration should be robust to registration errors and scanning noise introduced in the stages of registration and data acquisition. Once multiple registered range images have been fused into a single reasonable point cloud, many techniques [1–4] can be employed to reconstruct a watertight surface.
2
Related Work
Existing integration methods can be classified into four categories: volumetric method, mesh-based method, point-based method and clustering-based method. The volumetric method [5–8] first divides the space around objects into voxels and then fuse the data in each voxel. But the comparative studies [9, 10] show they are time-consuming, memory-hungry and not robust to registration errors and scanning noise. The mesh-based method [11–14] first employs a step discontinuity constrained triangulation and then detects the overlapping regions between the triangular meshes derived from successive range images. Finally, it reserves the most accurate triangles in the overlapping regions and reconnects all remaining triangles subject to a certain objective function such as maximising the product of the interior angles of triangles. However, since the number of triangles is usually much larger than that of the sampled points, the mesh-based methods are computationally expensive. Thus some of the mesh-based methods just employ a 2D triangulation in the image plane to estimate the local surface connectivity in the first step as computation in a 2D sub-space is more efficient. But projection from 3D to 2D may lead to ambiguities if the projection is not injective. The mesh-based methods are thus highly likely to fail in non-flat regions where no unique projection plane exists. This speeded-up strategy cannot deal with 3D unstructured point clouds either. The point-based method [15, 16] produces a set of new points with optimised locations. Due to the neglect of local surface topology, its integration result is often over-smooth and cannot retain enough surface details. The latest clustering-based method [9, 10] employs classical clustering methods to minimise the objective dissimilarity functions. It surpasses previous methods since it is more robust to scanning noise and registration errors. Nonetheless, the clustering, which measures Euclidean distances to find closest centroids, does not consider local surface details, leading to some severe errors in non-flat areas. For instance, in Fig. 1, although point A is closer to point B and thus the clustering-based method will wrongly group them together, we would rather group A with C or D to maintain the surface topology.
Fig. 1. Local topology has a significant effect on the point clustering in non-flat areas
MRF Labeling for Multi-view Range Image Integration
29
To overcome the drawbacks of the existing integration methods, we propose a novel MRF-based method. We first construct a network by a point shifting strategy in section 3. In section 4, the MRF labeling is then configured on this network where each node is formally labeled with an image number. In section 5, we employ loopy belief propagation (LBP) to solve the MRF and find the optimal label assignment. Each node will just select its closest point from the image with its assigned image number. These selected points, directly from the original data, are used for surface reconstruction. The integration achieves a global optimisation as the MRF labeling considers all image numbers for each node. Also, according to the experimental resutls shown in section 6, it effectively preserves local surface details since in the output point cloud, the points representing a local patch of the reconstructed surface are usually from the same input image due to the neighbourhood consistency of MRF. More importantly, our method can also cope with the more general input—multiple 3D unstructured point clouds.
3
MRF Network Construction for Integration
MRF describes a system based on a network or a graphical model and allows many features of the system of interest to be captured by simply adding appropriate terms representing contextual dependencies into it. Once the network has been constructed, contextual dependencies can be cast in the manipulations within a neighbourhood. In a MRF network, the nodes are usually pixels. For example, in [17–20], the network is the 2D image lattice and 4 neighbouring pixels are selected to form a neighbourhood for each pixel. But it is difficult to define a network for multiple registered range images since the concept ‘pixel’ does not exist. The simplest method is to produce a point set by k-means clustering and the neighbours of a point can be found by Nearest Neighbours (NNs) algorithm. Since both k-means clustering and NNs are based only on Euclidean distance, in such a network neither the delivery of surface topological information nor the compensation for pairwise registration errors can be achieved. In this paper, we propose a novel scheme to produce a MRF network Inet . Given a set of consecutive range images I1 , I2 , . . . , Im , we first employ the pairwise registration method proposed in [21] to obtain a transform H12 mapping I1 into the coordinate system of I2 . The transformed image I1 and the reference image I2 supply not only the redundant surface information (overlapping area) but also the new surface information (non-overlapping area) from their own viewpoints for the fused surface. To integrate I1 and I2 , the overlapping area and non-overlapping area have to be accurately and efficiently detected. In the new scheme, we declare that a point from one viewpoint belongs to the overlapping area if its distance to its corresponding point from the other viewpoint is within a threshold, otherwise it belongs to the non-overlapping area. The corresponding point is found by a closest point search with k-D tree speedup. The threshold is set as 3R where R is the scanning resolution of the input range images. After the overlapping area detection, we initialise Inet as: Inet = Snon−overlap ,
Snon−overlap = S1 + S2
(1)
30
R. Song et al.
where S1 from I1 and S2 from I2 are the points in the non-overlapping area. Then, each point in the overlapping area is shifted by half of its distance to its corresponding point along its normal N towards its corresponding point. → − → − P = P + 0.5d · N, d = P · N and P = Pcor − P (2) where Pcor is the corresponding point of P . After such a shift, the corresponding points are closer to each other. For each point shifted from a point in the overlapping region of the reference image I2 , a sphere with a radius r = M × R (M is a parameter controlling the density of the output point cloud) is defined. If some points fall into this sphere, then their original points, without shifting, are retrieved. Their averages Soverlap are then computed and Inet is updated as: Inet = Snon−overlap + Soverlap
(3)
So, this point shifting (1) compensates the pairwise registration errors, (2) does not change the size of overlapping area as the operation of points is along the normal, and (3) still keeps the surface topology as the shift is along the normal. Now, we consider the third input image I3 . We map the current Inet into the coordinate system of I3 through the transformation H23 . Similarly, we then do the overlapping area detection between Inet transformed from Inet and the current reference image I3 . We still update Inet by Eq. (3). In this update, Snon−overlap contains the points from Inet and I3 in non-overlapping regions and Soverlap are produced by the aforementioned point shifting strategy. We keep running this updating scheme over all input range images. The final Inet will be consistent with the coordinate system of the last input image Im . Although it is not usual, if some points in Inet only appear in Snon−overlap , we delete them from Inet . Fig. 2 shows 18 range images of a bird mapped into a single global coordinate system and the MRF network Inet produced by our new scheme. In this paper, for clarity, we use surfaces instead of point clouds when we show range data. The triangulation method employed here for surface reconstruction is the ball-pivoting algorithm [1]. In Fig. 2, it can be seen that the registered images contain a massive amount of noise such as thick patches, false connections and fragments caused by poor scanning and registration errors. The point shifting scheme compensates the errors a bit whereas the resultant point set Inet is still noisy, leading to an over-smoothed surface without enough local details. But it offers us a roughly correct manifold for estimating the ground truth surface. We also need to define a neighbourhood for each node in Inet , consistent with the MRF theory. Instead of performing a NNs algorithm to search the neighbours for a point i in such a network, we find the neighbouring triangles of i in the triangular mesh. The vertices of these triangles, excluding i, are defined as the neighbours of i. The collection of all neighbours of i is defined as the neighbourhood of i, written as N (i). In this work, to strengthen the neighbourhood consistency, we define a larger neighbourhood N (i) consisting of neighbours and neighbours’ neighbours as shown in Fig. 3. Doing so is allowed since the condition i ∈ N (j) ⇔ j ∈ N (i) is still satisfied. The superiority of this definition over the NNs method is that it reflects the surface topology and is parameter-insensitive.
MRF Labeling for Multi-view Range Image Integration
31
Fig. 2. Left: 18 range images are registered (Please note that a patch from a certain range image is visible if and only if it is the outermost surface); Right: The surface representing point set Inet produced by point shifting is usually noisy and over-smooth
Fig. 3. Left: A neighbourhood and the normalised normal vectors attached to the neighbours. Right: the expanded neighbourhood used in this work. Please note that the number of neighbours of each point in such a network can vary.
4
MAP-MRF Labeling for Integration
We denote s = {1, . . . , n} representing the points in Inet and define the label assignment x = {x1 , . . . , xn } to all points as a realisation of a family of random variables defined on s. We also define the feature space that describes the points and understand it as a random observation field with its realisation y = {y1 , . . . , yn }. Finally, we denote a label set L = {1, . . . , m}, where each label corresponds to an image number. Thus, we have {xi , yj } ∈ L, {i, j} ∈ s. An optimal labeling should satisfy the maximum a posteriori probability (MAP) criterion [22] which requires the maximisation of the posterior probability P (x|y). Considering the Bayes rule, this maximisation can be written as: (4) x∗ = arg max P (x|y) = arg max P (x)p(y|x) x
x
In a MRF, the prior probability of a certain value at any given point depends only upon neighbouring points. This Markov property leads to the HammersleyClifford theorem [22]: the prior probability P (x) satisfies a Gibbs distribution with respect to a neighbourhood N , expressed as P (x) = Z −1 ×e−(U(x)/Q) , where Z is a normalising constant and U (x) is the priori energy. Q, a global control parameter, is set to 1 in most computer vision and graphics problems [22]. Let p(y|x) be expressed in the exponential form p(y|x) = Zy−1 × e−U(y|x) , where U (y|x) is the likelihood enery. Then P (x|y) is a Gibbs distribution
32
R. Song et al.
P (x|y) = ZU−1 × e−(U(x)+U(y|x))
(5)
The MAP-MRF problem is then converted to a minimisation problem for the posterior energy U (x|y): x∗ = arg minx U (x|y) = arg minx U (x) + U (y|x) . Under the MRF assumption that the observations are mutually independent, U (y|x) can be computed by the sum of the likelihood potentials V (yi |xi ) at all points. In this work, we estimate V (yi |xi ) by a truncation function: V (yi |xi ) = min(Di (yi , xi ), F ) (6) U (y|x) = i∈s
i∈s yi ∈L\xi
where F is a constant and L \ xi denotes the labels in L other than xi , and Di (yi , xi ) = Ci (yi ) − Ci (xi )
(7)
is a distance function where Ci (l), l ∈ L denotes i’s closest point in the lth input image. In this way, we convert the estimation for the likelihood potential of different labels at point i (or the single label cost) into the measurement of the distances between i’s closest points from the input images with different labels. The truncation parameter F plays an important role here. It eliminates the effect from the input images which do not cover the area around i. The prior energy U (x) can be expressed as the sum of clique energies: V1 (xi )+ V2 (xi , xj )+ V3 (xi , xj , xk )+· · · (8) U (x) = i
i
j∈N (i)
i
j∈N (i) k∈N (i)
We only consider the unary clique energy and the binary one. Here, single-point clique V1 (xi ) are set to 0 as we have no preference which label should be better. We hope that in the output point cloud, neighbouring points are from the same input image so that the local surface details can be well preserved. Therefore, we utilize the neigbhourhood consistency of MRF to ‘discourage’ the routine from assigning different labels to two neighbouring points: V2 (xi , xj ) = λAij (xi , xj ) (9) U (x) = i
j∈N (i)
i
j∈N (i)
where λ is a weighting parameter and here we use the Potts model [23] to measure the clique energy (or the pairwise label cost): 1, xi = xj , Aij (xi , xj ) = (10) 0, otherwise Once we find the optimal x∗ which assigns a label xi to point i by minimising the posterior energy, i will be replaced with its closest point from the input image with the number xi (xi th image). Please note that in this step we do not need to run a closest point search again as this point Ci (xi ) has already been searched for (see Eq. (7)). The final output of the integration is thus a single point cloud composed of points directly from different registered input range images.
MRF Labeling for Multi-view Range Image Integration
33
Fig. 4. Left: The idea of point shifting to generate Inet . Right: The illustration of a 2-image integration based on the MAP-MRF labeling.
The labeling is actually a segmentation for the point set s. Due to Eq. (9) and (10), neighbouring points in s tend to be labeled with the same image number. The surface reconstructed from the output point cloud will consist of patches directly from different registered input images. Therefore, one advantage of the new integration method is the preservation for surface details, which avoids the error illustrated in Fig. 1. Another significant advantage is its high robustness to possible moving objects. The image patches containing moving objects are less probably selected. On the one hand, the single label cost for labeling the points in the area corresponding to a moving object with the image containing the moving object must be high in terms of Eq. (6). On the other hand, the points on the boundaries among patches always have high pairwise label costs according to Eq. (9) and (10). As a result of minimising the posterior energy, the boundaries usually bypass the areas where the moving objects appear. Here we also add a robustness parameter β in order to make the algorithm more robust to noise. When we compute the single label cost V (yi |xi ), we always carry out the robustness judgment as below: • if V (yi |xi ) < β, Go through the routine; Terminate the calculation of V (yi |xi ) and delete the • if V (yi |xi ) > β, point i from the list of s. And, β = (m − q) × F
(11)
where m is the number of input images (m = max(L)) and q is the number of input images that contain the roughly same noise if known. Generally, we set q as 2 so that random noise can be effectively reduced. We can also adjust q experimentally. If the reconstructed surface still has some undesirable noise, q should be increased. If there are some holes in the surface, we use a smaller q. In summary, Fig. 4 illustrates the idea of point shifting which is the key step to construct the MRF network in section 3 and a 2-image integration based on the MAP-MRF labeling described in details in section 4.
5
Energy Minimisation Using Loopy Belief Propagation
There are several methods which can minimise the posterior energy U (x|y). The comparative study in [24] shows that two methods, Graph-Cuts(GC) [23, 25] and LBP [19, 26], have proven to be efficient and powerful. GC is usually regarded fast
34
R. Song et al.
and has some desirable theoretical guarantee on the optimality of the solution it can find. And since we use the ‘metric’ Potts model to measure the difference between labels, the energy function can be minimised by GC indeed. However, in the future research, we plan to extend our technique by exploring different types of functions for the measurement of pairwise label cost such as the linear model and quadratic model [19]. Thereby, LBP becomes a natural choice for us. Max-product and sum-product are the two variants of LBP. The algorithm proposed in [19] can efficiently reduce their computational complexity from O(nk 2 T ) to O(nkT ) and O(nk log kT ) respectively, where n is the number of points or pixels in the image, k is the number of possible labels and T is the number of iterations. Obviously, max-product is suitable here for the minimisation. It takes O(k) time to compute each message and there are O(n) messages to be computed in each iteration. In our work, each input range image has about 104 points and typically a set of ranges images have 18 images. To reflect the true surface details of the object, the number of points in the output point cloud must be large enough. Thus if we use a point set as the label set, the number of labels would be tens of thousands. In that case LBP would be extremely time-consuming and too memory-hungry to be tractable. A MATLAB implementation would immediately go out of memory because of the huge n × k matrix needed to save the messages passing through the network. But the final output of integration must be a single point cloud. So, our MRF labeling using image numbers as labels makes it feasible to employ LBP for the minimisation. The LBP works by passing messages in a MRF network, briefed as follows: 1. For all point pairs (i, j) ∈ N , initialising message m0ij to zero, where mtij is a vector of dimension given by the number of possible labels k and denotes the message that point i sends to a neighbouring point j at iteration t. 2. For t = 1, 2, . . . , T , updating the messages as t−1 t mij (xj ) = min λAij (xi , xj ) + min(Di (yi , xi ), F ) + mhi (xi ) (12) xi
yi ∈L\xi
h∈N (i)\j
3. After T iterations, computing a belief vector for each point, bj (xj ) = min(Dj (yj , xj ), F ) + mTij (xj ) yj ∈L\xj
(13)
i∈N (j)
and then determining labels as:
x∗j = arg min bj (xj ) xj
(14)
According to Eq. (12), the computational complexity for the calculation of the message vector is O(k 2 ) because we need to minimise over xi for each choice of xj . By a simple analysis, we can reduce it to O(k). The detail is given as below: 1. Rewriting Eq. (12) as, mtij (xj ) = min λAij (xi , xj ) + g(xi ) xi
(15)
MRF Labeling for Multi-view Range Image Integration
where
g(xi ) =
min(Di (yi , xi ), F ) +
yi ∈L\xi
35
mt−1 hi (xi )
h∈N (i)\j
2. Considering two cases: xi = xj and xi = xj . (1) If xi = xj , λAij (xi , xj ) = 0, thus mtij (xj ) = g(xj ). (2) If xi = xj , λAij (xi , xj ) = λ, thus mtij (xj ) = min g(xi ) + λ xi
3. Synthesizing the two cases: mtij (xj ) = min g(xj ), min g(xi ) + λ xi
(16)
According to Eq. (16), the minimisation over xi can be performed only once, independent of the value of xj . In other words, Eq. (12) needs two nested FOR loops to compute the messages but Eq. (16) just needs two independent FOR loops. Therefore, the computational complexity is reduced from O(k 2 ) to O(k) and the memory demand in a typical MATLAB implementaion is reduced from a k × k matrix to a 1 × k vector. As for implementation, we use a multi-scale version to further speed up the LBP algorithm, as suggested in [19], which significantly reduces the total number of iterations needed to reach convergence or an acceptable solution. Briefly speaking, the message updating process is coarse-to-fine, i.e. we start LBP from the coarsest network, after a few iterations transfer the messages to a finer scale and continue LBP there, so on and so forth.
6
Experimental Results and Performance Analysis
The input range images are all downloaded from the OSU Range Image Database (http://sampl.ece.ohio-state.edu/data/3DDB/RID/index.htm). On average, each ‘Bird’ image has 9022 points and each ‘Frog’ image has 9997 points. We employed the algorithm proposed in [21] for pairwise registration. Inevitably, the images are not accurately registered, since we found the average registration error (0.30mm for the ‘Bird’ images and 0.29mm for the ‘Frog’ images) is as high as 1/2 scanning resolution of the input data. Fig. 5 shows that the input range data are registered into the same coordinate system, rendering a noisy 3D surface model with many thick patches, false connections and fragments due to local and global registration errors. A large registration error causes corresponding points in the overlapping region to move away from each other and the overlapping area detection will thus be more difficult. The final integration is likely to be inaccurate accordingly. In our tests, the truncation parameter F is set to 4 and the weighting parameter λ is set to 10. Fig. 6 and 7 show the integration results produced by existing methods and our MRF-based integration, proving that our method is more robust to registration errors and thus performs best. Now we analyse some details of the new algorithm. Fig. 8 shows the labeling result. On the integrated surface, we use different colours to mark the points assigned with
36
R. Song et al.
Fig. 5. Range images (coloured differently) are registered into the same coordinate system as the input of the integration. A patch from some range image is visible if and only if it is the outermost surface. Left: 18 ‘Bird’ images; Right: 18 ‘Frog’ images.
Fig. 6. Integration results of 18 ‘Bird’ images. Top left: volumetric method [6]. Top middle: mesh-based method [14]. Top right: fuzzy-c means clustering [10]. Bottom left: k-means clustering [9]. Bottom right: the new method.
Fig. 7. Integration results of 18 ‘Frog’ images. Top left: volumetric method [6]. Top middle: mesh-based method [14]. Top right: fuzzy-c means clustering [10]. Bottom left: k-means clustering [9]. Bottom right: the new method.
MRF Labeling for Multi-view Range Image Integration
37
Fig. 8. Left: labeling result of ‘Bird’ images. Right: labeling result of ‘Frog’ images.
Fig. 9. Convergence performance of the new method using different input images. Left: The ‘Bird’ image set; Right: The ‘Frog’ image set.
Fig. 10. Integration results of 18 ‘Bird’ images produced by different clustering algorithms. From left to right (including the top row and the bottom row): k-means clustering [9], fuzzy-c means clustering [10], the new algorithm.
Fig. 11. The triangular meshes of integrated surfaces using different methods. From left to right: volumetric [6], mesh-based method [14],point-based method [16], our method.
38
R. Song et al.
different labels, corresponding to different input range images. The boundary areas coloured with red actually represent the triangles where the three vertices are not assigned with the same label (Please note that the surface is essentially a triangular mesh). If a group of points are all assigned with the same label, the surface they define will be marked with the same colour. We find that not all input images have contribution to the final integrated surface. For the ‘Bird’ and the ‘Frog’ images, the reconstructed surfaces are composed of the points from 10 and 11 source images respectively. We plot in Fig. 9 how the LBP algorithm converges. To observe the convergence performance, in the LBP algorithm, we calculate Equ. (13) to determine the label assignment in each iteration of message passing. And then we count the number of points assigned with different labels from what they are assigned in last iteration. When the number is less than 2% of the total number of the initialisation points (Inet ), the iteration is terminated. Fig. 9 shows that the new method achieves a high computational efficiency in terms of iteration number required for convergence. Due to the different objective functions, it is very difficult to define a uniform metric such as the integration error [9, 10] for a comparative study because to maintain local surface topology, we reject the idea that the smaller the average of the distances from the fused points to the closest points on the overlapping surfaces, the more stable and accurate the final fused surface. Thus we cannot use such a metric in this paper. Nonetheless, Fig. 10 highlights the visual difference of the integration results produced by classical clustering methods (according to Fig. 6 and 7, they are superior to other existing methods) and the new method. It can be seen that our algorithm performs better, particularly in the non-flat regions such as the neck and the tail of the bird. Even so, for a fair comparison, we adopt some measurement parameters widely used but not relevant to the objective function: 1) The distribution of interior angles of triangles. The angle distribution shows the global optimal degree of triangles. The closer the interior angles are to 60◦ , the more similar the triangles are to equilateral ones; 2) Average distortion metric [27]: the distortion metric of a triangle is defined as its area divided by the √sum of the squares of the lengths of its edges and then normalised by a factor 2 3. The value of distortion metric is in [0,1]. The higher the average distortion metric value, the higher the quality of
Fig. 12. Different performance measures of integration algorithms. Left: distortion metric. Right: computational time.
MRF Labeling for Multi-view Range Image Integration
39
a surface; 3) The computational time; Fig. 11 and 12 show that the new method performs better in the sense of the distribution of interior angles of triangles, the distortion metric and the computational time. All experiments were done on a Pentium IV 2.40 GHz computer. Additionally, because there is no specific segmentation scheme involved, our method based on MRF labeling saves computational time compared with the techniques using some segmentation algorithms as a preprocessing before the integration [10].
7
Conclusion and Future Work
Clustering-based methods proved superior to other existing methods for integrating multi-view range images. It has, however, been shown that classical clustering methods lead to significant misclassification in non-flat areas as the local surface topology are neglected. In this paper, we develop a MRF model to describe the integration problem and solve it by LBP, producing better results as local surface details are well preserved. Since the novel MRF-based method considers all registered input images for each point belonging to the network Inet , the integration is globally optimised and robust to noise and registration errors. The reconstructed surface is thus geometrically realistic. Also, it is applicable to many other data sources such as 3D unstructured point clouds. However, a couple of works can still be done to improve the integration in the future. As we mentioned above, the Potts model used in this paper is merely the simplest one. So we plan to explore different types of MRF models. We also hope to reduce the accumulated registration errors. One pairwise registration can produce one equation subject to the transform matrix. If we have more locally registered pairs than the total number of images in the sequence, the system of equations will be overdetermined. Solving this linear system of equations in a least squares sense will produce a set of global registrations with minimal deviation from the set of calculated pairwise registrations. In the future work, we will develop a scheme that can make a good balance between the extra cost caused by more pairwise registrations and the reduction of error accumulation. Acknowledgments. Ran Song is supported by HEFCW/WAG on the RIVIC project. This support is gratefully acknowledged.
References 1. Bernardini, F., Mittleman, J., Rushmeier, H., Silva, C., Taubin, G.: The ballpivoting algorithm for surface reconstruction. IEEE Trans. Visual. Comput. Graph. 5, 349–359 (1999) 2. Katulakos, K., Seitz, S.: A theory of shape by space carving. Int. J. Comput. Vision 38, 199–218 (2000) 3. Dey, T., Goswami, S.: Tight cocone: a watertight surface reconstructor. J. Comput. Inf. Sci. Eng. 13, 302–307 (2003) 4. Mederos, B., Velho, L., de Figueiredo, L.: Smooth surface reconstruction from noisy clouds. J. Braz. Comput. Soc. 9, 52–66 (2004) 5. Curless, B., Levoy, M.: A volumetric method for building complex models from range images. In: Proc. SIGGRAPH, pp. 303–312 (1996)
40
R. Song et al.
6. Dorai, C., Wang, G.: Registration and integration of multiple object views for 3d model construction. IEEE Trans. Pattern Anal. Mach. Intell. 20, 83–89 (1998) 7. Rusinkiewicz, S., Hall-Holt, O., Levoy, M.: Real-time 3d model acquisition. In: Proc. SIGGRAPH, pp. 438–446 (2002) 8. Sagawa, R., Nishino, K., Ikeuchi, K.: Adaptively merging large-scale range data with reflectance properties. IEEE Trans. Pattern Anal. Mach. Intell. 27, 392–405 (2005) 9. Zhou, H., Liu, Y.: Accurate integration of multi-view range images using k-means clustering. Pattern Recognition 41, 152–175 (2008) 10. Zhou, H., Liu, Y., Li, L., Wei, B.: A clustering approach to free form surface reconstruction from multi-view range images. Image and Vision Computing 27, 725–747 (2009) 11. Hilton, A.: On reliable surface reconstruction from multiple range images. Technical report (1995) 12. Turk, G., Levoy, M.: Zippered polygon meshes from range images. In: Proc. SIGGRAPH, pp. 311–318 (1994) 13. Rutishauser, M., Stricker, M., Trobina, M.: Merging range images of arbitrarily shaped objects. In: Proc. CVPR, pp. 573–580 (1994) 14. Sun, Y., Paik, J., Koschan, A., Abidi, M.: Surface modeling using multi-view range and color images. Int. J. Comput. Aided Eng. 10, 137–150 (2003) 15. Li, X., Wee, W.: Range image fusion for object reconstruction and modelling. In: Proc. First Canadian Conference on Computer and Robot Vision, pp. 306–314 (2004) 16. Zhou, H., Liu, Y.: Incremental point-based integration of registered multiple range images. In: Proc. IECON, pp. 468–473 (2005) 17. Wang, X., Wang, H.: Markov random field modelled range image segmentation. Pattern Recognition Letters 25, 367–375 (2004) 18. Suliga, M., Deklerck, R., Nyssen, E.: Markov random field-based clustering applied to the segmentation of masses in digital mammograms. Computerized Medical Imaging and Graphics 32, 502–512 (2008) 19. Felzenszwalb, P., Huttenlocher, D.: Efficient belief propagation for early vision. International Journal of Computer Vision 70, 41–54 (2006) 20. Diebel, J., Thrun, S.: An application of markov random fields to range sensing. In: Proc. NIPS (2005) 21. Liu, Y.: Automatic 3d free form shape matching using the graduated assignment algorithm. Pattern Recognition 38, 1615–1631 (2005) 22. Li, S.: Markov random field modelling in computer vision, 2nd edn. Springer, Heidelberg (1995) 23. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23, 1222–1239 (2001) 24. Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., Rother, C.: A comparative study of energy minimization methods for markov random fields with smoothness-based priors. Pattern Analysis and Machine Intelligence. IEEE Trans. Pattern Anal. Mach. Intell. 30, 1068–1080 (2008) 25. Kolmogorov, V., Zabih, R.: What energy functions can be minimized via graph cuts? IEEE Trans. Pattern Anal. Mach. Intell. 26, 147–159 (2004) 26. Yedidia, J., Freeman, W., Weiss, Y.: Understanding belief propagation and its generalizations. Techincal Report TR-2001-22, MERL (2002) 27. Lee, C., Lo, S.: A new scheme for the generation of a graded quadrilateral mesh. Computers and Structures 52, 847–857 (1994)
Wave Interference for Pattern Description Selen Atasoy1,2 , Diana Mateus2 , Andreas Georgiou1 , Nassir Navab2 , and Guang-Zhong Yang1 1
Visual Information Processing Group, Imperial College London, UK {catasoy,a.georgiou,g.z.yang}@imperial.ac.uk 2 Computer Aided Medical Procedures and Augmented Reality, Technical University of Munich, Germany {atasoy,mateus,navab}@in.tum.de
Abstract. This paper presents a novel compact description of a pattern based on the interference of circular waves. The proposed approach, called “interference description”, leads to a representation of the pattern, where the spatial relations of its constituent parts are intrinsically taken into account. Due to the intrinsic characteristics of the interference phenomenon, this description includes more information than a simple sum of individual parts. Therefore it is suitable for representing the interrelations of different pattern components. We illustrate that the proposed description satisfies some of the key Gestalt properties of human perception such as invariance, emergence and reification, which are also desirable for efficient pattern description. We further present a method for matching the proposed interference descriptions of different patterns. In a series of experiments, we demonstrate the effectiveness of our description for several computer vision tasks such as pattern recognition, shape matching and retrieval.
1
Introduction
Many tasks in computer vision require describing a structure by the contextual relations of its consitituents. Recognition of patterns where the dominant structure is due to global layout rather than its individual texture elements such as the triangle images in Figure 1a-c and peace symbols in Figure 10a, patterns with missing contours such as the Kanizsa triangle in Figure 1d and Figure 5b and patterns with large homogeneous regions such as the shape images in Figure 9, necessitate the description of contextual relations within a structure in an efficient yet distinct manner. The captured information has to be discriminative enough to distinguish between structures with small but contextually important differences and be robust enough to missing information. In this paper, we introduce a method based on wave interference that efficiently describes the contextual relations between the constituent parts of a structure. Interference of several waves that originate from different parts of a structure leads to a new wave profile, namely the interference pattern (Figure 2). This interference profile is computed as the superposition of all constituent circular waves and has two relevant properties for describing a structure. Firstly, the R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 41–54, 2011. Springer-Verlag Berlin Heidelberg 2011
42
S. Atasoy et al.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
Fig. 1. Interference patterns created from input images (shown in the first row) with several geometric transformations (a-c) and missing contours (d) leads to the emergence of similar patterns (shown in the second row), whereas shapes with only very small variations yield very distinct interference patterns (e-j)
interference profile depends on the relative spatial arrangement of the sources that create the constituent waves. Secondly, it intrinsically captures all relationships of the constituent waves. Thus, interference phenomenon provides a natural mechanism to describe the structure of a pattern by representing the interrelations of its components. The resulting interference patterns are discriminative enough to separate different structures with only very small variation such as the examples shown in Figure 1e-j but not sensitive to a loss of information due to missing contours or to geometric transformations as illustrated in Figure 1a-d. Recognition of patterns with missing contours is an important property of human perception which has been extensively studied by Gestalt psychology [1]. This constructive property, known as “reification”, is demonstrated in Figures 1d and 5b, where a triangle is perceived although it is not explicitly delineated. This is due to the spatial configuration of the constituent contours. Gestalt theory also points out two other relevant properties of human vision, i.e., invariance and emergence. “Invariance” analyses how objects are recognized invariant of their location, orientation and illumination. It is also an active area of research in computer vision. “Emergence” property states that a pattern is not recognized by first detecting its parts and combining them to a meaningful whole, but rather emerges as a complete object ; i.e. “the whole is more than the sum of its parts”. In fact, the interference phenomenon intrinsically exhibits the emergence property. The interference pattern is produced by the superposition of two or more waves. This leads to the occurrence of new structures which are not present in any of the constituent waves nor in their simple sum but are created by the interrelations of the parts as shown in Figure 2. The contribution of our work presented in this paper is twofold: firstly, we introduce the notion of wave interference for pattern description. To this end, we create interference patterns from an input image. This leads to an emergent description of the structure in the input pattern by representing the interrelations of its constituent parts. We demonstrate that our method, called ”interference description” (ID), exhibits some of the key Gestalt properties including emergence, reification and invariance. Secondly, we use these interference patterns to
Wave Interference for Pattern Description
43
create compact descriptor vectors and show in a series of experiments how the interference patterns can be applied to several computer vision tasks such as pattern recognition, shape matching and retrieval.
2
Related Work
In general, pattern description techniques can be analyzed in three main categories. Local methods extract and describe discriminative local information such as features and patches. The descriptors are then used for object matching or pattern recognition without using their contextual relations (See [2] and references inside). Global methods, such as descriptions based on the spectrum of Laplacian [3, 4] or finite element modal [5], capture distinctive information using the whole content of the pattern. In a third category, contextual methods, the local information is analyzed within the global context. To this end, relevant local information of the parts is combined with the contextual relationships between them. For instance, Sechtman and Irani create a descriptor using patches with a certain degree of self-similarities [6], Belongie et al. compute the shape context descriptors using distance histograms for each point. These descriptors are then usually combined with a suitable matching method such as star-graph [6] or hyper-graph matching [7], or a nearest neighbour matching after aligning two shapes [8]. One of the major challenges for modelling the contextual relations in the image domain is the computational complexity which increases exponentially with the order of relationships considered. Therefore, most methods include relations up to second order (pair-wise) [6, 8] or third order [7]. Furthermore, for applications such as retrieval, shape comparison or pattern recognition, a more compact description of the whole structure is desired as the explicit computation of feature correspondences is actually not required. Another form of global description methods, the frequency analysis techniques, on the other hand, allows the consideration of more complex relations of a pattern by encoding all global information. Fourier descriptors [9], for example, project the shape contour onto a set of orthogonal basis functions. Wavelet based methods such as [10] analyze the response to wave functions that are more localized in space and frequency. Spectral analysis using spherical harmonics [11] and eigenfunctions of the Laplace-Beltrami operator (vibration modes) has been successfully applied also for representation and matching of images [4] and 3D shapes [3]. In this paper, we introduce the notion of wave interference in order to create a compact pattern description. The proposed method intrinsically captures norder relations1 without increasing the computational complexity. The novelty of our method lies in the fact that it analyses the interference, and thus the interaction, of circular waves over a range of frequencies, whereas other frequency techniques ([3–5, 9–11]) are based on the reponses at different frequencies that do not interact with each other. 1
n is the number of all individual parts modelled as sources in our method.
44
3
S. Atasoy et al.
Properties of Interference Description
In this section we introduce and discuss some of the key properties of ID which are relevant for pattern description. Modelling Contextual Relations: In ID, contextual relations emerge automatically due to the relation of individual circular waves. This has two important benefits: Firstly, it eliminates the need for explicit modelling of any order relations or for defining a set of rules. Secondly, all contextual relations of a pattern are considered without increasing the computational cost. The interference is computed by a simple addition of complex wave functions, where the computation of individual wave functions are independent. Therefore, including a new part to the pattern is simply done by adding its wave function to the sum, whereas in other contextual methods, all descriptors need to be re-computed in order to consider their relations to the newly added part. Furthermore, as the contextual relations are already included in the ID, comparison of descriptors for retrieval or recognition can be performed by simple nearest neighbor matching. This avoids the requirement of more sophisticated methods such as graph matching or non-linear optimization. Specificity and Precision: In pattern description there exists a natural tradeoff between the specificity and precision of the method. ID allows to regulate this trade-off by the choice of the number of frequencies. A structure is described by a set of interference patterns computed over a range of frequencies (explained in Section 4.3), where the higher the frequency, the more specific the description becomes. Therefore the number of frequencies, which is the only parameter of the proposed method, allows adjusting the specificity of the description depending on the application. Verification of Gestalt Properties: Gestalt psychology has played a crucial role in understanding human visual perception. It also has been an inspiring model for several computer vision applications such as analyzing shapes [12], learning object boundaries [13], defining mid-level image features [14] and modeling illusionary contour formation [15]. An extensive study of the Gestalt principles and their application in computer vision is presented in [1, 16]. In this work, we point out the analogy between the interference phenomenon and three important Gestalt principles, namely emergence, reification and invariance. In Section 5.1, we demonstrate that due to the particular nature of interference, our method intrinsically satisfies the mentioned Gestalt properties and eliminates the need for a set of defined rules or learning using a training dataset.
4
Methods
In this Section, we first introduce the background for waves and interference phenomenon. Then we explain our approach for pattern description, where an input pattern is first transformed into a field of sources; each creating a circular
Wave Interference for Pattern Description
(a)
(b)
45
(c)
Fig. 2. a) Demonstration of the emergent nature of interference. Each source in the stimulus field creates a (attenuating) circular wave on the medium resulting in the final interference pattern. b) Sum of the individual wave patterns. c) Interference of individual wave patterns. The interference phenomena intrinsically exhibits the emergence property, as the whole (the interference pattern) shown in c) is different than the sum of its constituent wave profiles shown in b).
wave; and resulting jointly in an interference pattern. Finally, we propose a method to perform comparison of the created interference patterns. 4.1
Background
In its most basic form, a wave can be defined as a pattern (or distribution) of disturbance that propagates through a medium as time evolves. Harmonic waves are waves with the simplest wave profile, namely sine or cosine functions. Due to their special waveform they create wave patterns which are not only periodic in time but also in space. Thus, on a 2D medium, harmonic waves give rise to circular wave patterns such as the ones observed on a liquid surface when a particle in dropped into the liquid. Using complex representation, a circular wave is described by the following wave function of any point x on the medium and a time instance t: ψφ (A0 , ω; x, t) = A0 · ei(ω||x−φ||) · eiωt ,
(1)
where i is the imaginary unit, ω is the frequency of the wave and φ is the location of the source (stimulus). A0 denotes the intensity of the source and ||x− φ|| gives the distance of each point x on the medium to the location of the source. As the time dependent term eiωt is not a function of space (x), it does not affect the interference of the waves. Therefore, in the rest of the paper it will be taken as t = 0. So, the instantaneous amplitude of the disturbance at each point on the medium is given by the real part of the wave function Real [ψφ (A0 , ω; x)]. For simplicity, we will denote waves with a fixed frequency ω and a fixed amplitude A0 as ψφ (x) := ψφ (A0 , ω; x). As waves propagate outwards from the source, the amplitude of the wave will gradually decrease with the increased distance from the source due to friction. This effect, called attenuation, can be described by multiplying the amplitude of the wave with the attenuation profile σ. When two or more waves ψφi are created on the same medium at the same time from several source locations φi , the resultant disturbance Ψ at any point x on the medium is the algebraic sum of the amplitudes of all constituents. Using
46
S. Atasoy et al.
complex represetation, the interference pattern Ψ (x) is given by the amplitude of the sum of individual complex wave functions ψφi (x): n (2) Ψ (x) = ψφi (x) , i=1
where | · | denotes the absolute value of the complex number. Note that the absolute value of the sum of a set of complex numbers is different than the sum of their real parts (See Figure 2). This is known as the superposition principle of the waves. The superposition of waves with fixed frequencies yields to a special phenomenon called interference. In this special case, if the waves reaching a point on the medium are in-phase (aligned in their ascent and descents), they will amplify each other’s amplitudes (constructive interference); conversely if they are out-of-phase, they will diminish each other at that point (destructive interference). This results in a new wave profile called interference pattern. Figure 2a illustrates the interference pattern created by the superposition of 3 waves with the same frequency. The interference pattern is computed by the absolute value of the sum of complex wave functions. The complex addition results in a waveform which is not present in any of the constituting waves nor in the simple sum of the contituent wave amplitudes (Figure 2b) but occurs due to their combination (Figure 2c). Therefore, interference property of waves provides an emergent mechanism for describing the relations between the sources and for propagating local information. 4.2
Interference Description
Using the introduced definitions in Section 4.1, our method formulates the contextual relations of the parts composing the image in an emergent manner. Let Ω ⊂ R2 be the medium on which the waves are created, where x ∈ Ω denotes a point on Ω. In practice the medium is described with a grid discretizing the space. The function I : Ω → R, called stimulus field, assigns a source intensity A0 = I(φ) to the positions φ on the medium Ω. This function contains the local information of the pattern which is propagated over the whole medium. Depending on the application, it can be chosen to be a particular cue defined on the image domain such as gradient magnitude, extracted edges or features. Each value I(φ) of the stimulus field I induces a circular wave ψφ (I(φ), ω; x) on the medium. We use the frequency of the wave to describe the relation between the distribution of sources and the scale at which one wants to describe the pattern. This is discussed in detail in Section 4.3. The profile of the created circular wave is described as: ψφ (I(φ), ω; x) = I(φ) · ei(ω(||x−φ||) · σ ,
(3)
where σ is the attenuation profile. We define the attenuation profile of the wave induced by the source φ with frequency ω as: σ(φ, ω; x) = ω · e−ω·||x−φ|| ,
(4)
Wave Interference for Pattern Description
(a)
47
(b)
Fig. 3. Wave patterns for different source locations with the 3rd frequency (k = 3) shown in a) and 8th frequency (k = 8) shown in b)
where ||x − φ|| denotes the Euclidean distance between a point x and the source location φ. This attenuation profile allows us to conserve the total energy of the waves while changing their frequency. Note that the attenuation profile is a function of the distance to the source (||x − φ||) as well as the frequency ω. The higher the frequency, the sharper is the decay of the attenuation profile. This yields to a localized effect for high frequencies and a more spread effect for low frequencies. Once the location and intensities of the sources on the stimulus field are determined, the interference pattern Ψ (I, ω; x) on the medium for a fixed frequency ω is computed simply by the superposition of all wave patterns: (5) ψφi (I(φi ), ω; x) . Ψ (I, ω; x) = φi ∈Ω 4.3
Multi-frequency Analysis
Computing the interference patterns for several frequencies allows for the analysis of the stimulus field at different scales. As discussed in Section 4.2, high frequency waves are more localized and propagate the content in a smaller neighbourhood, whereas low frequency waves are more spread over a larger area. Figure 3 illustrates the circular wave patterns for several frequencies and source locations. Formally, we define the interference description (ID) Θ(I) of an input I as the set of interference patterns Ψ (I, ωi ) for a range of frequencies {ω1 , ω2 , ..., ωn }: Θ(I) = {Ψ (I, ω1 ), Ψ (I, ω2 ), ..., Ψ (I, ωn )} ,
(6)
i with ωi = 2π·k P , where ki ∈ N is the wave-number, i.e. the number of periods the wave has on the medium and P is the length of the grid.
4.4
Descriptor Comparison
In this section, we present a method to compare two IDs, Θ(I1 ) and Θ(I2 ) in a rotation invariant manner. To this end, we first define a coherency measure c of an interference pattern Ψ (x):
48
S. Atasoy et al.
|Ψ (x)| φi ∈Ω |ψφi (x)|
c(Ψ (x)) =
.
(7)
The coherency measures the power of the interference pattern compared to the sum of the powers of the constituent waves. As φi ∈Ω |ψφi (x)| provides the upper limit to the power of the interference amplitude |Ψ (x)| for each location x on the medium, the coherency measure c normalizes the values of the interference profile to the interval [0, 1] (Figure 4c-d). This enables the comparison of different IDs independent of their absolute values and therefore makes the comparison independent of the number of sources. We subdivide this coherency image c(Ψ (x)) into m + 1 levelsets {γ0 , γ1 , · · · , γj , · · · , γm }
γj = {x|c(Ψ (x)) = j/m}
(8)
and take the sum of the coherency values for each interval between the two T consecutive levelsets. This yields an m-dimensional measure h = [h1, · · · , hm ] , hj = γj−1 ≤x<γj c(Ψ (x)) computed from an interference pattern Ψ (x) (Figure 4e). Performing this levelset histogram calculation for the whole range of frequencies n of an ID (Θ(I)) results in a m × n dimensional measurement h(Θ(I)) = [h(Ψ (I, ω1 )), · · · , h(Ψ (I, ωn ))] (Figure 4f). Quantifying the amount of coherency in each levelset enables a rotation-invariant description of the created interference patterns. The distance between the two IDs Θ(I1 ) and Θ(I2 ) is then simply computed as the cosine measure between the levelset histograms: h(Θ(I1 ))T · h(Θ(I2 )) dist(I1 , I2 ) = acos . (9) ||h(Θ(I1 ))|| · ||h(Θ(I2 ))|| This dissimilarity, measures the distance between the two patterns in a rotation invariant manner (due to the levelset histograms) while taking the contextual relations between the constituent parts (due to the interference pattern) for each pattern.
Fig. 4. Rotation invariant multi-frequency descriptor. a) Input image, b) sources located on the medium Ω as marked in green, c) interference patterns computed for a range of frequencies Θ(I), d) coherence measure computed on the interference patterns c(Θ(I)), e) Levelsets {γ1 , · · · , γm } and f) the levelset histogram, where the columns correspond to descriptors at different frequencies.
Wave Interference for Pattern Description
5
49
Experiments and Results
Firstly, we demonstrate that the ID exhibits the Gestalt properties discussed in Section 2. Then, we show its application for shape matching and retrieval and pattern recognition. For our experiments, we first create a grid (medium Ω : [−π × π] × [−π × π]) and map the locations of the stimulus field to the sources on the medium. Each source creates a circular wave pattern propagating outwards from this source location. Also, we define the stimulus field to be the gradient magnitude of the input image. Note that the use of more sophisticated feature extraction methods such as phase congruency [17] can further improve the performance. The choice of gradient magnitude is only due to its computational simplicity. The strength of the disturbance (A0 ) at a location x on the medium is set to be the gradient magnitude at that location. 5.1
Verification of Gestalt Properties
In this Section we demonstrate how the proposed description exhibits the Gestalt properties emergence, reification and invariance. Emergence: The emergence property states that the whole (structure) is more than the sum of its parts. This is due to the interrelations of the constituents. In ID, the whole (i.e. interference pattern as shown in Figure 2c) is given by the amplitude of the complex interference field (sum of the complex wave functions). This interference pattern is different from the sum of the amplitudes of the consituent waves (Figure 2b). This is due to the constructive and destructive interference effect (as explained in Section 4.1) and can be seen mathematically as: n n n ψφi (x) = Real[ψφi (x)] = ψφi (x) , (10) Ψ (x) = i=1
i=1
i=1
where the left side of the inequality defines the interference pattern and the right side gives the sum of the amplitudes of its constituent waves. The difference between the whole and the sum of its parts leads to the emergence of a new profile which is not present in any of the constituent circular waves nor in their simple sum but occurs due to their interrelations. Reification: Reification is the constructive property seen in human perception [1, 18]. In Figures 5b and f, a triangle and a square are perceived, although they are not explicitly delineated. This is due to the particular spatial arrangement of the packman shapes. ID results in emergence of the same interference patterns in the location where a triangle is perceived in Figures 5a and b and where a square is perceived in Figures 5c and d. Therefore, ID also allows to recognize patterns with missing contours such as the triangle image in Figure 1d.
50
S. Atasoy et al.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Fig. 5. Reification property seen in wave interference. Similar interference patterns emerge for a complete triangle a) and a Kanizsa triangle image b) or a complete square e) and a Kanizsa square image f). c) and d) demonstrate the interference patterns for the 4th frequency (k = 4) for the triangle and Kanizsa triangle, respectively. The source locations are illustrated in black crosses. g) and h) show the interference patterns of images in e) and f) for the 8th frequency (k = 8).
Invariance: The invariance of ID to scale change is achieved due to the mapping of the stimulus field onto the same medium (discretized grid of fixed size). This creates sources along contours of the same size, however, the image with a smaller scale results in less number of sources sampled along the contour. The structure of the created interference patterns are the same as shown in Figures 6a,b. As the proposed comparison method in Section 4.4 is independent of the number of sources, the two IDs result in very similar levelset histograms as shown in Figure 6a,b. We further demonstrate the invariance of the ID for rotation, affine and perspective transformations, as well as deformation. To this end, we simulate a binary triangle image and apply the corresponding transformations. Figure 6c-g displays for each transformation the input images, interference patterns for the same example frequency and the corresponding levelset histograms computed as described in Section 4.4. Note the similarity of the IDs and the levelset histograms despite the large geometric transformations between the images.
Fig. 6. Invariance of ID for scale change (a,b), rotation (c,d), affine transformation (c,e), perspective transformation (c,f) and deformation (c,g). First row shows the input patterns, second row illustrates interference pattern for the 6th frequency k = 6 and last row demonstrates the levelset histograms computed from the IDs h(Θ(I)).
Wave Interference for Pattern Description bone comma glas hcircle heart misk apple bat beetle bell bird bottle brick butterfly camel car carriage cattle cellular phone chicken children chopper classic crown cup deer device device1 device2 device3 device4 device5 device6 device7 device8 device9 dog elephant face fish flatfish fly fork fountain frog guitar hammer hat horse horseshoe jar key lizzard lmfish octopus pencil personal car pocket rat ray sea snake shoe spoon spring stef teddy tree truck turtle watch
51
45
40
35
30
25
20
15
10
5
cobone mm a hcglas ir hecle a m rt ap isk ple be bat etl e be bir ll bo d ttle bu bric tte k ca rfly me ca ca l rr r ce llu ciage lar att ph le o ch ne ch icke ch ildre n op n cla per s crosic w cun d p de eer v de ice device1 device2 device3 device4 device5 device6 device7 device8 vic e ele do9 ph g an fac t e fla fish tfis h fl fou fory nta k in fro ha guitag mm r er h ho ho at rse rs sh e oe jar lizzkey lm ard oc fish to pe p pu rso en s na cil pol car ck et ra se a s ra t na y s ke sphoe spoon rin s g tedtef d tr y truee tu ck wartle tch
0
Fig. 7. The confusion matrix of ID matching as applied on the MPEG7 database
5.2
Application to Shape Matching and Retrieval
In this experiment, we demonstrate the use of IDs for shape matching. We apply the presented description approach to match objects from the contour images of the MPEG7 CE shape database containing 1400 images with 70 different categories. For each image, we first compute the interference patterns for 10 frequencies. Then we compute the similarity of each shape image against all other images in the database by comparing the IDs as described in Section 4.4. Figure 7 demonstrates the performance of ID matching as a confusion matrix computed from the complete MPEG7 database. After matching the shape images, we assign the best matched category to each image for retrieval considering the 20 minimum distances. Figure 8 displays the recall and precision over the 70 categories. Figure 9a demonstrates 3 example shapes; chopper, guitar, and spoon, where chopper leads to high recall and precision in the retrieval because of its characteristic structure. Due to their high similarity, guitar is confused with spoon in the final category retrieval. However, as demonstrated in Figure 9a, the interference patterns of the two shapes still show some differences despite the significant similarity between them. The confusion is caused due to the discretized levelset histogramming as illustrated in Figure 9a last row. In order to evaluate the inter-category distances, we embed the levelset histogram vectors (Section 4.4) linearly in to the 2D space using the classical multi-dimensional scaling (MDS). Figure 9b shows the 2D MDS plot of the representatives of each category. The MDS
52
S. Atasoy et al. 1
0.8
0.6
0.4
0.2
cobon mme hcglaa ir s hecle mart ap isk pl be bae etl t bee ll bobird bu brttle tte ick ca rfly m ce car cael llu ria r lar ca ge ph ttle ch on ch ick e ch ildreen o clappen crossicr w cun p d de deevicer devic e e v de ice1 device2 device3 device4 devic 5 devicee6 device7 vi 8 ele dce9 ph og a facnt e fla fish tfis h fou fofly ntark fr in haguitog mmar ho h hear rse ors t sh e oe ja liz ke r za y l oc mfisrd pe top h rso pe us na nci pol ca l ck r e se rat a s ra t na y k spshoe spoone rin g tesdtef d tre y tru e tu ck wartle tch
0
1
0.8
0.6
0.4
0.2
cobon mme hcglaa ir s hecle mart ap isk pl be bae etl t bee ll bobird bu brttle tte ick ca rfly m ce car cael llu ria r lar ca ge ph ttle ch on ch ick e ch ildreen o clappen crossicr w cun p d de deevicer devic e e v de ice1 device2 device3 d vic 4 deevice5 devicee6 device7 vi 8 ele dce9 ph og a facnt e fla fish tfis h fou fofly ntark fr in haguitog mmar ho h hear rse ors t sh e oe ja liz ke r za y l oc mfisrd pe top h rso pe us na nci pol ca l ck r e se rat a s ra t na y k spshoe spoone rin g tesdtef d tre y tru e tu ck wartle tch
0
Fig. 8. a) Recall and b) precision values for shape category retrieval of IDs when applied to all categories in the MPEG7 shape database CHOPPER
GUITAR
SPOON 15
10
5
0
−5
−10
−15
−20
−15
−10
(a)
−5
0
5
10
15
20
(b)
Fig. 9. a) First row: input images for 3 example shapes. Second row: the corresponding interference patterns for the 5th frequency (k = 5). Third row: Levelset histograms computed with 5 levelset intervals for 10 frequencies. b) 2D MDS plot showing a representative for each category shape. Even in 2D representation similar categories are closely located whereas distinct categories are more separated.
plot demonstrates how similar shape categories locate more closely whereas more distinct structures are more separated. 5.3
Application to Pattern Recognition
In this experiment we demonstrate that ID can be used to recognize patterns created by different texture elements. Therefore, we use images with large inter-class variability in the representation where in each image the pattern is generated with different texture elements. Although the details of the patterns are different, they all share the same global layout. This global structure is enhanced in the low frequency interference patterns. Therefore we create IDs Θ(I) for k = {1, · · · , 4}. For a quantitative evaluation we compare the levelset histograms created from the IDs to the histograms of gradients and of the intensities. Figure 10b illustrates IDs created from the images shown in Figure 10a. Figure 10c demonstrates their matching levelset histograms. IDs lead to similar but distinctive patterns,
Wave Interference for Pattern Description
53
(a)
(c) ID (b)
(d) Intensities
(e) Gradient
Fig. 10. Application of ID for pattern recognition. a) Input images containing the same global layout created with different texture elements. b) IDs at the frequency (k = 3) of the images a-d). c) Levelset histograms as compared to histogram of intensities d) and gradients e). For visualization the histograms are reshaped to 5 × 4 matrices.
whereas the gradients or intensities lead to less similar histograms (Figure 10d,e) that also lack the discriminative power. The noisy background in the 4. image in Figure 10a results in high gradient values in the background. This is also reflected in its effect on the histogram of gradients (4. image in Figure 10e). Naturally, this includes several sources due to the background, as the stimulus field I(φ) is defined to be the gradient magnitude. However, as shown in Figure 10b, ID tolerates a certain degree of background noise.
6
Discussion and Conclusions
In this work, we have introduced a method for pattern description that is based on wave interference. Due to the characteistics of interference phenomenon, our method intrinsically accounts for the contextual relations between the parts of a pattern. This eliminates the need for defining a set of rules or for any prior learning. Furthermore, the particular mathematical formulations of the presented method allow for the computation of higher order relations in an efficient manner, i.e. a simple complex addition. In a series of experiments, we have showed that the proposed description is in agreement with three key Gestalt properties relevant for pattern description. Future work will direct towards extending this approach for describing a whole scene including different objects and patterns. This requires creating an interference description for each object and then describing the scene hierarchically at different scales. In this work, we particularly pointed out the analogies of the wave interference phenomenon and some desired properties of pattern description and demonstrated its efficient application for pattern recognition, shape matching and retrieval.
54
S. Atasoy et al.
Acknowledgements. This research was supported by the Graduate School of Information Science in Health (GSISH) and the TUM Graduate School. The authors would like to thank Wellcome Trust and EPSRC for funding The Centre of Excellence in Medical Engineering and Tobias Wood and Ozan Erdogan for valuable discussions.
References 1. Koffka, K.: Principles of Gestalt psychology. Routledge, New York (1999) 2. Lowe, D.: Object recognition from local scale-invariant features. In: International Conference on Computer Vision, ICCV, vol. 2, pp. 1150–1157 (1999) 3. Levy, B.: Laplace-beltrami eigenfunctions: Towards an algorithm that understand s geometry. In: IEEE International Conference on Shape Modeling and Applications, pp. 13–25 (2006) 4. Peinecke, N., Wolter, F.E., Reuter, M.: Laplace spectra as fingerprints for image recognition. Computer-Aided Design 39, 460–476 (2007) 5. Sclaroff, S., Pentland, A.: Modal matching for correspondence and recognition. IEEE Trans. on Pattern Analysis and Machine Intelligence 17, 545–561 (1995) 6. Shechtman, E., Irani, M.: Matching local self-similarities across images and videos. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 511–518 (2007) 7. Zass, R., Shashua, A.: Probabilistic graph and hypergraph matching. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 1–8 (2008) 8. Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Trans. on Pattern Analysis and Machine Intelligence, 509– 522 (2002) 9. Zhang, D., Lu, G.: A comparative study of Fourier descriptors for shape representation and retrieval. In: Asian Conference on Computer Vision, ACCV (2002) 10. Kuthirummal, S., Jawahar, C., Narayanan, P.: Planar shape recognition across multiple views. In: International Conference on Pattern Recognition, vol. 16, pp. 456–459 (2002) 11. Kazhdan, M., Funkhouser, T., Rusinkiewicz, S.: Rotation invariant spherical harmonic representation of 3d shape descriptors. In: Eurographics/ACM SIGGRAPH Symposium on Geometry Processing, pp. 156–164 (2003) 12. Loncaric, S.: A survey of shape analysis techniques. Pattern Recognition 31, 983– 1001 (1998) 13. Dollar, P., Tu, Z., Belongie, S.: Supervised learning of edges and object boundaries. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, vol. 2 (2006) 14. Bileschi, S., Wolf, L.: Image representations beyond histograms of gradients: The role of Gestalt descriptors. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 1–8 (2007) 15. Williams, L., Jacobs, D.: Stochastic completion fields: A neural model of illusory contour shape and salience. Neural Computation 9, 837–858 (1997) 16. Desolneux, A., Moisan, L., Morel, J.: From gestalt theory to image analysis: a probabilistic approach. Springer, Heidelberg (2007) 17. Kovesi, P.: Image features from phase congruency. Videre: Journal of Computer Vision Research 1, 1–26 (1999) 18. Carman, G., Welch, L.: Three-dimensional illusory contours and surfaces. Nature (1992)
Colour Dynamic Photometric Stereo for Textured Surfaces Zsolt Jank´ o1,2, , Ama¨el Delaunoy1 , and Emmanuel Prados1 1
2
INRIA Rhˆ one-Alpes, Montbonnot, France Computer and Automation Research Institute, Budapest, Hungary
[email protected]
Abstract. In this paper we present a novel method to apply photometric stereo on textured dynamic surfaces. We aim at exploiting the high accuracy of photometric stereo and reconstruct local surface orientation from illumination changes. The main difficulty derives from the fact that photometric stereo requires varying illumination while the object remains still, which makes it quite impractical to use for dynamic surfaces. Using coloured lights gives a clear solution to this problem; however, the system of equations is still ill-posed and it is ambiguous whether the change of an observed surface colour is due to the change of the surface gradient or of the surface reflectance. In order to separate surface orientation from reflectance, our method tracks texture changes over time and exploits surface reflectance’s temporal constancy. This additional constraint allows us to reformulate the problem as an energy functional minimisation, solved by a standard quasi-Newton method. Our method is tested both on real and synthetic data, quantitatively evaluated and compared to a state-of-the-art method.
1
Introduction and Related Works
1.1
Motivation
Reconstructing dynamic surfaces from images has become a popular research topic of computer vision during the last few years. Extending the time-honoured methods of static surface reconstruction to the dynamic case is reasonable. However, active techniques (laser scanner, structured light projection system, etc.) that project time-varying textures on the surface are rather difficult to apply on deformable surfaces. Passive techniques, such as conventional stereo or multicamera systems, are not biased by temporal variation, and they are more frequently applied for dynamic surfaces [1–7]. However, their accuracy is not sufficient to reconstruct small details, such as wrinkles and creases of garments. Since the seminal work of Woodham [8], photometric stereo has become a powerful technique to recover local surface orientation from brightness variations generated by the shading. In contrast to multi-camera systems, photometric
Corresponding author.
R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 55–66, 2011. Springer-Verlag Berlin Heidelberg 2011
56
Z. Jank´ o, A. Delaunoy, and E. Prados
stereo uses image sequence acquired from the same viewpoint, under differing known directional illumination conditions, and it is capable of reconstructing small details, such as surface bumpiness, as well. However, when one wishes to apply photometric stereo on dynamic surfaces, he meets a technical difficulty: how to alter lights while the object remains still. It is clear that for intensively varying surfaces switching the lights is rather inconvenient and impractical. The work of Smith and Smith [9] distinguishes between temporal and spectral multiplexing approaches based on how they isolate illumination conditions from one another. Temporal multiplexing means adaptation of static to dynamic photometric stereo, that is alternating lights. In works [10, 11] light sources are rapidly pulsed in synchronisation with the camera using a timing controller. This solution is technically difficult to attain and requires slow object motion. Additionally, for a human actor vibration of the light can be very annoying. Spectral multiplexing approaches can avoid light pulsation by using light frequency to separate illumination conditions. Here all the light sources are in play simultaneously. This category is also termed as colour photometric stereo, as coloured lights are used to recover surface shape [9, 12–14]. Our method falls within the latter category: we use three light sources with colours red, green and blue, respectively. These three colours appear almost uniformly in the visible light spectrum (400-700nm), hence they are easy to separate from one another. Furthermore, RGB is a standard colour model for sensing, representation and display of images and applied in many conventional cameras. The application of coloured lights to photometric stereo is described in more details below. 1.2
Problem Formulation
A general model of image formation is
λ=700
I(u) = λ=400
T i n(u) E(i, λ)R(i, u, λ)di S(λ)dλ Ω λ=700 T i n(u) E(i, λ)R(i, u, λ)S(λ) dλ di, (1) =
Ω
λ=400
where I(u) is the response of a camera sensor at image point u, n(u) the corresponding surface normal, and Ω ⊂ S2 is the set of all possible directions of incoming lights i (S2 being the unit sphere of R3 ). Denoting the wavelength by λ, E(i, λ) is the light’s spectral power distribution, R(i, u, λ) is the reflectance of the surface point projected to the image point u, and S(λ) is the sensor’s spectral sensitivity. In this work we assume that the scene is illuminated by three light sources (a Red one, λr = 650nm, a Green one, λg = 510nm, and a Blue one, λb = 475nm) and that it is viewed by a single camera with three sensors which mostly respond to the same wavelengths of the lights (λr , λg and λb for the red, green and blue captors, respectively). (See fig. 1.) Sensors’ sensitivity can then be
Colour Dynamic Photometric Stereo for Textured Surfaces
57
Fig. 1. The spectral sensitivities of a typical camera. From [15].
well-approximated by Dirac delta functions as Sk (λ) = sk δ(λ − λk ), k = r, g, b, with suitable constants sk . Since the sensors are sensitive to different, well-separable wavelengths, using three light sources with three different colours (according to the above, red, green and blue, respectively) means that one image is enough to separate the three lights that are necessary for photometric stereo. Based on these assumptions and approximations, the response of the three sensors, i.e. eq. (1) simplifies to Ik (u) = lTk n(u)αk (u),
k = r, g, b.
(2)
Here lk = ik E(ik , λk ) is the unit vector ik towards the light source k multiplied by its intensity1 . αk (u) = R(ik , u, λk )sk is proportional to the albedo, i.e. the fraction of incident energy reflected by the surface. As in previous works that consider the same problem, here we assume the scene to be Lambertian. Also, we will call the triplet α = (αr , αg , αb ) albedo further on. Carefully examining eq. (2) one can see that the reconstruction problem (i.e. recovering normals and albedos from the colours observed) is ill-posed. Beyond the image colour values Ik (u), vectors lk are also given, assuming calibrated light sources. The rest of the unknowns are to be recovered, namely the normal n (2 unknowns) and the albedo α (3 unknowns). On the contrary, we have only 3 equations, for k = r, g, b. (Note that in conventional photometric stereo, when surface is illuminated by three light sources separately at three different time instants, we have three colour images instead of one, and 6 additional equations. Although the normals can then easily be calculated, altering lights is not practical for dynamic surfaces.) In order to solve eq. (2) and to recover surface normals and albedos, one should add further constraints. Such a constraint can be that αr = αg = αb , i.e. the surface is white or gray; however, this assumption is rather restrictive. Another possibility is to capture an additional image under a broad spectrum white light and to initially measure reflectance properties. The light source should be located 1
For simplicity we assume directional light sources at infinite distance.
58
Z. Jank´ o, A. Delaunoy, and E. Prados
close to the camera, and acquisition must be performed at a separate time instant to avoid coupling with the RGB lights. The latter condition makes it impractical to apply for time-varying surfaces. The method of Hernandez et al. [13] works on uniformly coloured surfaces, and a calibration tool is employed to estimate this colour. In contrast to them, we do not want to restrict our method to uniform albedos, either not to apply an initial estimation of reflectance properties. The work of Smith and Smith [9] meets these requirements, as they use narrow band colour photometric stereo in the infra-red region of the spectrum. In the infra-red region the reflected light is less sensitive to surface colour changes, i.e. different colours become metameric to one another. Due to this, surface orientation and colour can be separated. However, their system is rather complicated and requires special and expensive tools. In this work we present a novel method that extends the approach of Hernandez et al. [13] to textured surfaces. The principal idea is to track surface motion and exploit the fact that surface colour, i.e. the albedo is constant through time. Based on this assumption we formulate an energy functional according to eq. (2) that penalises albedos’ temporal change and reinforces spatial and temporal smoothness. Contribution. The principal contribution of this work is a novel photometric stereo method for textured dynamic surfaces. To the best of our knowledge, our method is the first one in the category of dynamic photometric stereo which successfully works on textured surfaces as well, using only a single standard camera with a simple and practical light configuration. The problem is formulated by an energy functional and its optimisation. The method is able to reconstruct local surface orientations and to separate them from reflectance properties.
2
Proposed Solution
In this paper, we focus on a representation of dynamic surfaces as animated normal maps. An animated normal map is a sequence of normal maps where temporal correspondence is given. As discussed above, we consider a setup of one colour camera and three light sources filtered with red, green and blue filters, respectively. We denote the image sequence by It , the image domain by Dt and the image points by ut ∈ Dt , where t ∈ T means a discrete time instant, T = {0, 1, . . . }. Here u0 , u1 , u2 , . . . denote the corresponding image points, that belong to the same surface part, but not the same co-ordinates. That is, ut+1 = ut + d(ut ), where d(ut ) is a 2D displacement vector. The dynamic surface S to be reconstructed can then be represented by the normals, the albedos and the displacement vectors: S = {n(ut ), α(ut ), d(ut ) : ut ∈ Dt , t ∈ T } .
(3)
Although the normal map representation does not reflect the complete structure of the surface, it can easily be integrated to obtain a depth map and a surface
Colour Dynamic Photometric Stereo for Textured Surfaces
59
mesh. Note that recovering depth map from normals gives bad low frequency details; however, if a rough surface mesh is already given, a normal map can be used to represent fine details, small variations of the surface (e.g. creases on garments) as a bump map. 2.1
Energy Functional
The optimality of surface S is quantified by an energy functional composed of two terms. The first term forces the model to suit the photometric constraint while enforcing albedo temporal constancy and unit length normals (data term); the second term guarantees spatial and temporal smoothness (regularisation term): E(S) = ED (S) + ER (S).
(4)
The first part of the data term is local, it does not consider correspondences between consecutive frames. In accordance with eq. (2) it measures the photometric error:
2 1 1
It,k (ut ) − αk (ut ) max(lTk n(ut ), 0) . (5) EDph (S) = |T | |Dt | t∈T
ut ∈Dt k=r,g,b
To guarantee unit length of normals and albedo temporal constancy we define the following terms: 1 1 n(ut )T n(ut ) − 12 , EDn (S) = (6) |T | |Dt | t∈T
EDa (S) =
ut ∈Dt
1 1 2 α(ut+1 ) − α(ut ) . |T | |Dt | t∈T
(7)
ut ∈Dt
The data term is then the weighted sum of the three terms above: ED (S) = μph EDph (S) + μn EDn (S) + μa EDa (S).
(8)
The regularisation term is destined to measure the smoothness of the model. It is composed of a weighted sum of three terms: ER (S) = μs ERs (S) + μr ERr (S) + μt ERt (S).
(9)
The first term quantifies the spatial smoothness, i.e. the deviation of neighbouring normals: 1 1 1 ERs (S) = n(ut ) − n(vt )2 , (10) |T | |Dt | |N | t∈T
ut ∈Dt
vt ∈N (ut )
where N means the set of neighbouring image points. The second term measures the rigidity of the model. We require displacements of nearby points to be as similar as possible: 1 1 1 2 d(ut ) − d(vt ) . (11) ERr (S) = |T | |Dt | |N | t∈T
ut ∈Dt
vt ∈N (ut )
60
Z. Jank´ o, A. Delaunoy, and E. Prados
Third, normals and displacement vectors are reinforced to change smoothly in time (temporal smoothness): ERt (S) =
1 1 2 n(ut+2 ) − 2n(ut+1 ) + n(ut ) + |T | |Dt | t∈T
ut ∈Dt
d(ut+2 ) − 2d(ut+1 ) + d(ut )2 , (12) where ut+1 = ut + d(ut ) and ut+2 = ut + d(ut ) + d(ut + d(ut )). 2.2
Optimisation
In optimisation, quasi-Newton methods are widely used to minimise energy functionals. We have selected one of the most popular members of this class, namely the Broyden-Fletcher-Goldfarb-Shanno (BFGS) method to optimise our energy functional (4). In our experiments we used the limited memory variant (L-BFGS) of the algorithm, included in the ALGLIB library implemented by Sergey Bochkanov [16]. For an iterative optimisation method, such as the L-BFGS, it is essential to choose a good starting point. Our initialisation consists of three phases. First, motion is tracked to estimate the displacement of the points, second, a rough guess is given to the albedos based on the variation of the colour values, and finally the normals are computed using the albedos. Due to the fact that the scene is illuminated by three coloured light sources, image colour is not sufficient to follow surface motion, as surface points change their colour significantly over time. Consider fig. 2a where the same surface particles change their colour from red to green and blue, depending on whether they look towards the red, the green or the blue light source. Although from a global aspect the colours vary, local changes of the texture remain the same, thus they are traceable. According to this, we apply a feature detector method, namely the Sobel edge detector, in order to emphasise local variations in the images. (See fig. 2b.) To track the motion and estimate displacement vectors d(ut ) we applied the high-accuracy optical flow method of Brox et al. [17] and its variation [18], implemented by Chari [19], on the edge maps. Note that the related work of Hernandez et al. [13] also uses optical flow to track the deformation of the surface. However, they use optical flow after normal and depth maps are already computed and their goal is to put them in correspondence. In contrast, we use optical flow before obtaining the normals, in order to constrain the ill-posed problem by assuming temporally constant albedos. Some other methods [11, 20, 21] also use optical flow to improve photometric stereo for dynamic objects, but since they do not use coloured light sources (no spectral multiplexing), we omit discussing them in details. With the estimated displacement vectors in hand, we can give a rough approximation to the albedos. On closer inspection of eq. (2) one can see that Ik (ut ) reaches its maximum at a time instant t if n(ut ) looks towards light source k, and at that time lTk n(ut ) = lk , i.e. αk (ut ) = Ik (ut )/ lk . Consequently, if we assume the surface dynamic enough and the sequence long enough, so that all
Colour Dynamic Photometric Stereo for Textured Surfaces
a)
61
b)
Fig. 2. a) Due to coloured illumination, deforming the surface can change its colour from red to green and blue; b) in contrast, edge detection emphasises local colour changes
normals turn towards each light sources at least once, the following estimation is a good approximation to the albedos: α∗k (u) = max t∈T
Ik (ut ) , k = r, g, b. lk
(13)
In the final step of the initialisation normals are computed solving a system of linear equations based on eq. (2) using the already estimated albedos.
3 3.1
Experiments and Discussion Datasets
Three real test sequences have been created in our studio [22] using three coloured light sources and one camera. The datasets contain sequences of about 150 images acquired from the same viewpoint. The image resolution is 1624 × 1224. Since our method requires calibrated light sources, we applied a simple calibration method. For this we used some additional calibrated cameras and created a short sequence of a white balance card that provides a precise uniform surface that is spectrally neutral under all lighting conditions. (See fig. 3.) Detecting the card’s corners their 3D co-ordinates and hence the fitting plane can be determined using the multiple camera setup. The light sources’ properties (direction and intensity) are computed from the plane’s normal and the sensor response observed.
Fig. 3. Calibration sequence: from the corners of the white balance card the plane’s varying normal can be computed and used to calibrate light sources
62
Z. Jank´ o, A. Delaunoy, and E. Prados
Fig. 4. Results on real datasets: input images (left), albedos (centre), normals (right)
Colour Dynamic Photometric Stereo for Textured Surfaces
63
In order to quantitatively assess the results, we have also created a synthetic dataset (fig. 6, first column). The dataset contain a 10 frames sequence of 800 × 800 images about textured waves. Ground truth normals enable to measure the reconstruction error and to compare our result to another state-of-the-art method. 3.2
Results
Fig. 4 shows the results on real datasets: the original input images, the albedo maps and the colour normal images. Our method is capable of separating surface reflectance (albedo) from surface direction (normal). This is clearly demonstrated by the results: the albedo map is constant over time, the red-green-blue colour changes (visible in the input images) disappear; at the same time, the texture, for instance the stripes of the shirt are not present in the normal image. If we assumed uniform albedo, as in [13], stripes would appear in the normal image, not in the albedo map. To visualise the quality of the results we have integrated the normal maps to obtain 3D surfaces. For this we have used the method of Kovesi [23] and his implementation that is available on his web-site [24]. Fig. 5 shows examples for the surfaces obtained for each real datasets. Our supplemental material contains the complete videos of the reconstructed normals and surfaces.
Fig. 5. Examples for reconstructed surfaces for each real datasets
We have tested our method on synthetic data as well and compared the result to the state-of-the-art method of Hernandez et al. [13]. Since their method assumes uniform albedo, texture changes incorrectly appear in the normal maps. (See figs. 6 and 7.) On the contrary, our method clearly separates albedo and normal maps. To quantitatively assess the results, we have compared the normals obtained to the ground truth data. Considering the angle (in degrees) between the resulted and the ground truth normal, the mean and the confidence interval were calculated: 3.45 ◦ ± 0.0035 with our results and 15.32 ◦ ± 0.0106 for the method of [13]. Fig. 8 shows the histogram of the error image, which illustrates the error distribution. Note that the reconstruction’s accuracy decreases towards the edge of the image, where tracking is less precise (fig. 8, left).
64
Z. Jank´ o, A. Delaunoy, and E. Prados
Fig. 6. Results on a synthetic dataset: input images (left), albedos (centre), normals (right). First two rows: our method. Last two rows: assuming uniform albedo [13].
Fig. 7. Examples for reconstructed surfaces for a synthetic dataset. Left: our method. Right: assuming uniform albedo [13].
Colour Dynamic Photometric Stereo for Textured Surfaces
65
Fig. 8. Error images (left) and their histogram (right). First row: our method. Second row: assuming uniform albedo [13].
4
Conclusion
In this paper we have discussed the problem of applying photometric stereo on dynamic surfaces. We have presented a novel colour photometric stereo method that successfully recovers normals for textured surfaces as well. Our method is an extension of [13], which works only for surfaces with uniform albedo. Assuming the albedos’ temporal constancy and tracking the surface’s local changes initially, we have formulated the problem as an optimisation that forces the model to fit the photometric constraint and encourage spatial and temporal smoothness. Tests on real and synthetic datasets demonstrate the efficiency of our method. Acknowledgments. This work was supported by the French National Agency for Research (ANR) under grant ANR-06-MDCA-007 and partially by the NKTH-OTKA grant CK78409 and by the HUNOROB project (HU0045, 0045/NA/2006-2/P-9), a grant from Iceland, Liechtenstein and Norway through the EEA Financial Mechanism and the Hungarian National Development Agency. – The authors thank Visesh Chari for his optical flow code and Thomas Dupeux for his help at data acquisition.
References 1. Aganj, E., Pons, J.P., S´egonne, F., Keriven, R.: Spatio-temporal shape from silhouette using four-dimensional Delaunay meshing. In: Proc. ICCV 2007 (2007)
66
Z. Jank´ o, A. Delaunoy, and E. Prados
2. Courchay, J., Pons, J.-P., Monasse, P., Keriven, R.: Dense and accurate spatiotemporal multi-view stereovision. In: Zha, H., Taniguchi, R.-i., Maybank, S. (eds.) ACCV 2009. LNCS, vol. 5995, pp. 11–22. Springer, Heidelberg (2010) 3. Furukawa, Y., Ponce, J.: Dense 3D motion capture from synchronized video streams. In: Proc. CVPR 2008 (2008) 4. Pons, J.P., Boissonnat, J.D.: Delaunay deformable models: Topology-adaptive meshes based on the restricted Delaunay triangulation. In: Proc. CVPR 2007 (2007) 5. Starck, J., Hilton, A.: Surface capture for performance based animation. IEEE Computer Graphics and Applications 27, 21–31 (2007) 6. Vedula, S., Baker, S., Kanade, T.: Image-based spatio-temporal modeling and view interpolation of dynamic events. ACM Trans. on Graphics 24, 240–261 (2005) 7. Vlasic, D., Baran, I., Matusik, W., Popovi´c, J.: Articulated mesh animation from multi-view silhouettes. ACM Transactions on Graphics 27 (2008) 8. Woodham, R.J.: Photometric stereo: A reflectance map technique for determining surface orientation from image intensity. In: Image Understanding Systems and Industrial Applications, Proc. SPIE, vol. 155, pp. 136–143 (1978) 9. Smith, M.L., Smith, L.N.: Dynamic photometric stereo–a new technique for moving surface analysis. Image and Vision Computing 23, 841–852 (2005) 10. Timo, P., Detlef, P., Pertti, K. (inventors): Arrangement and method for inspection of surface quality. European Patent Application, EP 1 030 173 A1 (2000) 11. Vlasic, D., Peers, P., Baran, I., Debevec, P., Popovi´c, J., Rusinkiewicz, S., Matusik, W.: Dynamic shape capture using multi-view photometric stereo. In: Proc. SIGGRAPH Asia 2009, pp. 1–11 (2009) 12. Drew, M.S.: Shape from color. Technical report, School of Computing Science, Simon Fraser University, Vancouver, B.C. (1992) 13. Hernandez, C., Vogiatzis, G., Brostow, G., Stenger, B., Cipolla, R.: Non-rigid photometric stereo with colored lights. In: Proc. ICCV 2007, pp. 1–8 (2007) 14. Kontsevich, L., Petrov, A., Vergelskaya, I.: Reconstruction of shape from shading in color images. J. Opt. Soc. Am. A 11, 1047–1052 (1994) 15. Finlayson, G.D., Hordley, S.D., Lu, C., Drew, M.S.: On the removal of shadows from images. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 59–68 (2006) 16. Bochkanov, S.: ALGLIB, http://www.alglib.net 17. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004) 18. Sand, P., Teller, S.: Particle video: Long-range motion estimation using point trajectories. In: Proc. CVPR 2006, pp. 2195–2202 (2006) 19. Chari, V.: High accuracy optical flow, http://www.mathworks.com/matlabcentral/ fileexchange/17500-high-accuracy-optical-flow 20. Ma, W.C., Hawkins, T., Peers, P., Chabert, C.F., Weiss, M., Debevec, P.: Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In: Proc. EGSR 2007 (2007) 21. Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., Debevec, P.: Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Trans. Graph. 24, 756–764 (2005) 22. INRIA: GrImage Platform, http://grimage.inrialpes.fr/index.php 23. Kovesi, P.: Shapelets correlated with surface normals produce surface. In: Proc. ICCV 2005, pp. 994–1001 (2005) 24. Kovesi, P.: MATLAB and octave functions for computer vision and image processing, http://www.csse.uwa.edu.au/~ pk/Research/MatlabFns/
Multi-Target Tracking by Learning Class-Specific and Instance-Specific Cues Min Li, Wei Chen, Kaiqi Huang, and Tieniu Tan National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences {mli,wchen,kqhuang,tnt}@nlpr.ia.ac.cn
Abstract. This paper proposes a novel particle filtering framework for multi-target tracking by using online learned class-specific and instancespecific cues, called Data-Driven Particle Filtering (DDPF). The learned cues include an online learned geometrical model for excluding detection outliers that violate geometrical constraints, global pose estimators shared by all targets for particle refinement, and online Boosting based appearance models which select discriminative features to distinguish different individuals. Targets are clustered into two categories. Separatedtarget is tracked by an ISPF (incremental self-tuning particle filtering) tracker, in which particles are incrementally drawn and tuned to their best states by a learned global pose estimator; target-group is tracked by a joint-state particle filtering method in which occlusion reasoning is conducted. Experimental results on challenging datasets show the effectiveness and efficiency of the proposed method.
1
Introduction
Multi-target tracking (MTT) plays an important role in applications like visual surveillance, intelligent transportation and behavior analysis. It is still a challenging problem in computer vision community. Challenges include but are not limited to target auto-initialization, inter-object occlusions and intensive computational cost caused by joint-state optimization. Extensive work has been done on multi-target tracking for the last decades. Some formulates MTT as a data-association problem between candidate regions and target trajectories. Multiple hypothesis tracker (MHT) [1] and joint probabilistic data association filter(JPDAF) [2] are two classical data-association methods widely used in computer vision context, e.g. the MHT in [3] and JPDAF in [4]. However, MHT/JPDAF has to exhaust all possible associations and thus is computationally intensive. Recently, sampling-based techniques have been proposed for data association. In [5], an MCMC (Markov Chain Monte Carlo) based variant of the auxiliary variable particle filter is proposed, in which the MCMC sampling simulates the probability of a data association. Since dateassociation based methods highly depend on reliable low-level detection results and often do not consider the interactions between targets, they are subject to bad detection results or occlusions. R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 67–81, 2011. Springer-Verlag Berlin Heidelberg 2011
68
M. Li et al.
There is also much work that focuses on multi-target tracking by Bayesian inference in state space. Data association may be involved, but not necessary. Isard et al. [6] propose a Bayesian multi-blob tracker which combines a multi-blob likelihood function with a particle filter. In [7], Adaboost and mixture particle filters [8], which model the filtering distribution as an M-component mixture model, are combined to detect and track multi-targets fully automatically. In [9], joint posterior is approximated by a variational density, based on which a set of autonomous while collaborative trackers are used to overcome the coalescence problem (several trajectories stick to one same target). Qu et al. [10] propose an interactively distributed multi-target tracking algorithm using a magnetic-inertia potential model to solve the multiple object labeling problem in the presence of occlusions. Khan et al. [11] propose a novel MCMC sampling step to obtain an efficient multi-target filter in joint-state space. However, this model can not deal with occlusion. Yang et al. [12] propose a game-theoretic multiple target tracking algorithm, in which the tracking problem is solved by finding the Nash Equilibrium of a game. Zhang et al. [13] propose a species based PSO (particle swarm optimization) algorithm for multiple object tracking, which brings a new view to the tracking problem from a swarm intelligence perspective. Hess et al. [14] present a discriminative training method for particle filters and attempt to directly optimize the filter parameters in response to observed errors. Breitenstein et al. [15] use the continuous confidence of pedestrian detectors and online trained, instance-specific classifiers as a graded observation model to track multi-person in the particle filtering framework. In this paper, a data-driven particle filtering (DDPF) method for multi-target tracking is proposed, and multi-human tracking is used as a study case. Object detection is not a necessary part, but considering target auto-initialization, detection is still introduced into the tracking process. The proposed DDPF framework is not a pure stochastic state inference method, instead, a set of classspecific or instance-specific cues, meaning some kind of models learned online from low-level data, are used to make the state inference process more efficient and reliable. Targets are clustered into tow categories. Separated-target is tracked by an ISPF (incremental self-tuning particle filtering) [16] method, in which particles are incrementally drawn and tuned to their best states by a learned global pose estimator; target-group is tracked by a joint-state space particle filtering method in which occlusion reasoning between group members is conducted. The remainder of the paper is organized as follows. Section 2 gives an overview of the proposed framework. Section 3 introduces our detection and data association method. The Bayesian inference process for multi-target tracking is described in Section 4. Experimental results and analysis are presented in Section 5. Finally, we draw our conclusions in Section 6.
2
Overview of the Proposed Framework
In particle filtering framework, tracking can be casted as an inference task in a Markov model with hidden state variables [17]:
Multi-Target Tracking by Learning Class-Specific and Instance-Specific Cues
69
p(Xt |Ot ) ∝ p(ot |Xt )
p(Xt |Xt−1 )p(Xt−1 |Ot−1 )dXt−1
(1)
where Xt describes the state variable at time t and Ot = {o1 , o2 , ..., ot } is a set of observations. The tracking process is governed by the observation model p(ot |Xt ) and the dynamic model p(Xt |Xt−1 ) between two states. In our framework, separated-target is tracked by a single particle filter, and Xt contains only one single state St (St = {x, y, s}, meaning x, y translation and scale); for a group with n targets, Xt contains the joint states of several targets St = {Sit }ni=1 , as well as the occlusion matrix π t = {πtij }i<j≤n with n(n − 1)/2 independent elements. Occlusion variable πtij has three values: −1 means target i is occluded by target j, 1 means the opposite, and 0 means they have no occlusion relationship.
Fig. 1. The flowchart of the proposed Data-Driven Particle Filtering (DDPF)
Fig. 1 shows the flowchart of the proposed Data-Driven Particle Filtering framework for multi-target tracking. In the target auto-initialization module, a detector is used to detect targets in video frames, and detection outliers that violate projective projection are excluded by a geometrical model learned by Local Weighted Projection Regression (LWPR) [18]. Then remaining detections are associated with existing targets by a data-association method, in which online Boosting classifiers are utilized. New targets are born from detections without any association and they form a Target Pool with old targets. We then classify targets in Target Pool into two categories: separated-target and target-group by a simple clustering method. Separated-target is tracked by an ISPF tracker in single-state space, and target-group is tracked by an MCMC-PF method in joint state space with occlusion reasoning. Global pose estimators and instancespecific Boosting classifiers are utilized in the state optimization process to make it more efficient and reliable. After deleting some vanished targets, the obtained optimal states are then used to update the global pose estimators, online Boosting classifiers and object’s appearance model.
70
3
M. Li et al.
Detection and Association
Since the PSOF (Pyramidal Statistics of Oriented Filtering) [19] method exhibits good performance under different imaging conditions, we use it in our human detection module. Geometrical Model. No perfect detector exists, any detector may produce false positives. To eliminate outliers that obviously violate perspective projection, like object 2 shown in Fig. 2 (a), a geometrical model that describes the relationship between object’s height (denoted by H) and object’s foot point (denoted by (xf , yf ), see Fig. 2 (b)) position is learned by an online regression method called Local Weight Projection Regression (LWPR) [18], which can use piece-wise linear models to approximate non-linear function. Details of LWPR can be seen in [18]. In every frame, training samples are all current detections and they are fed to the LWPR regressor to incrementally learn the geometrical model H(xf , yf ). A detection with height h and foot point (xf , yf ) is regarded as a outlier if: |H(xf , yf ) − h| > Th H(xf , yf )
(2)
where Th is a threshold (usually set to 0.3). An example of the learned geometrical model is shown in Fig. 2 (c), which is nearly a plane. The surface shape of the geometrical model depends on the the number and locations of vanishing points of scene structures, and a plane surface usually means one-point perspective. One advantage of the proposed geometrical model is that it can incrementally learn scene geometrical information without camera calibration.
Fig. 2. An example of online learned geometrical model for excluding detection outliers that violate perspective projection. (a) Detection results. Object 2 (in red) is an outlier. (b) Definitions of foot point and object height. (c) Learned surface of object height vs object’s foot point. Blue points are online collected training samples.
Data Association. After object detection and filtering the outliers, detection results should be associated with trajectories of old targets. First, a similarity matrix between trajectory-detection pairs is computed. The similarity score Simda between a trajectory tr and a detection d is computed as a product of appearance similarity and distance similarity:
Multi-Target Tracking by Learning Class-Specific and Instance-Specific Cues Simda (tr, d) = Simc (tr, d) · pN (|d − tr|)
71 (3)
2 N (|d − tr|; 0, σdet )
where pN (|d − tr|) ∼ is a Gaussian function used to measure the distance similarity between tr and d and Simc (tr, d) is the output (scaled to (0, 1) by a sigmoid function) of an online Boosting classifier used to measure the appearance similarity. Then, a simple matching algorithm is conducted: the pair (tr∗ , d∗ ) with the maximum score is iteratively selected, and the rows and columns belonging to trajectory tr and detection d in the similarity matrix are deleted. Note that only the association with a matching score above a threshold is considered as a valid match.
Fig. 3. Comparison of appearance similarity matrices. (a) trajectories and detections. (b) obtained by feature template matching. (c) obtained by Boosting classifiers. In (b) and (c), yellow block means the maximum of a column and red means the second maximum.
Online Boosting Classifiers. Instance-specific Boosting classifiers are used in data association, as well as the observation model for tracking. Our Boosting classifier is similar to [20] and is trained online on one target against all others. Patches used as positive training examples are sampled from the bounding box of the optimal state given by the tracking process. The negative training set is sampled from all other targets, augmented by background patches. Fig. 3 shows an comparison of appearance similarity matrices obtained by feature template matching (see Section 4.2) and Boosting classifiers. As shown in Fig. 3 (b) and (c), the discriminability of online Boosting is much more powerful than feature template matching, because the average ratio of column maximum to second maximum of online Boosting is 14.5, while that of feature template matching is only 1.3.
4
Tracking by Particle Filtering
To decompose the full joint-state optimization problem in multi-target tracking into a set of “easy-to-handle” low-dimensional optimization problems, we adopt a “divide-and-conquer” strategy. Targets are clustered into two categories: separated-target and target-group. Separated-target is tracked by a single-state particle filter and target-group is tracked by a joint-state particle filter. A set of targets G = {tri }ni=1 with n ≥ 2 is considered as a target-group if: Clustering Rule: ∀ tri G, ∃ trj G(j = i), s.t. |Si Sj |/min(|Si |, |Sj |) > Tc
72
M. Li et al.
Fig. 4. An example of training and using of the full-body pose estimator. (a) the training process. (b) pose-tuning process. S0 (red) is a random particle, S3 (yellow) is the tuned particle with the maximum similarity, others are intermediate states (black) (c) the curve of similarity vs iteration# for (b).
where Si is the optimal state of target tri at frame #t-1 or its prediction state at frame #t using only previous velocity, | · | is the area of a state, and Tc is a threshold (usually set to 0.4). This rule makes target pairs that have or are going to have significant overlapping areas be in the same group. Targets that do not belong to any group are separated targets. 4.1
Global Pose Estimators
Sampling-based stochastic optimization methods usually need to sample a lot of particles, the number of which may grow exponentially with the increase of state-dimension. However, most particles have almost no contribution to state estimation because of their extremely small weights, while they consume most of the computational resources. To tremendously decrease the unnecessary computational cost, ISPF (incremental Self-tuning Particle Filtering)[16] uses an online learned pose estimator to guide random particles to move towards the correct directions, thus every random particle becomes intelligent and dense sampling is unnecessary. Inspired by [16], we also use pose estimators in multi-target tracking. Different from one pose estimator per target in [16], we train two global pose estimators online for all targets. One is ff b trained with full-body human samples, the other is fhs trained with only the head-shoulder parts of human samples. fhs is used when most parts of a target is occluded by other targets except the head-should part. Fig. 4 shows an example of how to train and use a pose estimator. As shown in Fig. 4 (a), we do state perturbation dS around the optimal state S of each target to get training samples, then HOG (Histogram of Oriented Gradients) [21] like features F are extracted and LWPR is used to learn the regression function dS = f (F). By using the regression function, as shown in
Multi-Target Tracking by Learning Class-Specific and Instance-Specific Cues
73
Fig. 4 (b) and (c), given an initial random state around a target, we can guide it move to its best state with the maximum similarity in several iterations (each iteration includes two main steps: ΔSi = f (F(Si )) and Si+1 = Si − ΔSi , see [16] for details). 4.2
Observation Model
To make appearance model robust to partial occlusions, we use a part-based representation for each target, shown in Fig. 5 (a) p1 ∼ p7 . As shown in Fig. 5 (a), each part is represented by a L2-normalized HOG histogram and a L2normalized IRG (Intensity/Red/Green) color histogram: Fj ={Fjhog , Fjirg }. The final representation is the concatenation of all features F = {Fj }m j=1 , where m is the number of all parts. Note that feature F is also the input of online boosting classifiers, and it is not the same F used in pose estimator.
Fig. 5. Part based object representation and calculation of the visibility vector. (a) Target is divided into m parts (denoted by {pj }m j=1 , here m=7), and each part is represented by a HOG histogram and an IRG (Intensity/Red/Green) histogram denoted as Fj ={Fjhog , Fjirg }. (b) An example of calculation of visibility vector. Elliptical regions are the valid areas of targets. Knowing the configuration of states {Si } and their occlusion relations {π ij }, the visibility vector of S1 can be easily obtained: (1,1,1,1,0,1,1).
Part based Feature Template Matching. Only using the online trained Boosting classifiers for appearance model may suffer from occlusions, thus a partbased template matching method is introduced to our appearance model. Given the state configuration X = {S, π } of a target-group (subscript t is dropped), where S={Si }ni=1 is the joint state and π = {π ij }1≤i<j≤n is the occlusion matrix, an m-dimensional visibility vector V =(V1 , ..., Vm ) for each target can be calculated (1: visible, 0: invisible). In our work, a part is invisible if the ratio of its area covered by other targets exceeds some threshold (usually set to 0.5). Fig. 5 (b) shows an example of how to calculate the visibility vector. For separated target, all parts are visible. The template similarity score of a state S (subscript is dropped) to a target tr is defined as: m SimT (tr, S) = 1 −
Fj − Fj · Vj 2· m j=1 Vj
j=1
(4)
74
M. Li et al.
m where F = {Fj }m j=1 is the feature vector specified by S, F = {Fj }j=1 is the reference template of target tr, and V =(V1 , ..., Vm ) is the visibility vector. If m j=1 Vj =0, which means target tr is totally occluded by other targets, SimT (tr, S) is set to the average template matching score of background patches. Multi-Source Similarity Fusion Given a random state S, its appearance similarity Sima to a target tr is defined as a weighted combination of multiple sources: Sima (tr, S) = α1 · pN (|d∗ − S|) + α2 · SimT (tr, S) + α3 · Simc (tr, S)
(5)
where d∗ is the associated detection of target tr and {αi }3i=1 are the weights of different sources with α1 + α2 + α3 =1 (α1 is set to 0 if there is no associated detection). The first term depicts how close S is to the associated detection d∗ , the second term is the similarity obtained by template matching, and the third term is the similarity obtained by the Boosting classifier of target tr. Usually we set the weights α1 , α2 , α3 to 0.2, 0.4, 0.4 respectively. The final observation model of a target group G = {tri }ni=1 with state configuration X (subscript t is dropped) can be defined as: P (o|X) =
n
P (oj |{Si }n i=1 , π ) =
j=1
4.3
n
Sima (trj , Sj )
(6)
j=1
Dynamical Model
In particle filtering framework, besides observation model P (ot |Xt ), another important part is the dynamical model P (Xt |Xt−1 ). As in [9], we also introduce interaction potentials to the dynamical model, which aims to penalize the case that two targets occupy the same position. Suppose G = {tri }ni=1 is a target group with joint state {Si } and occlusion matrix π , the dynamical model is defined as follows: P (Xt |Xt−1 ) =
n 1 πt) U(π ψi (Sit |Si(t−1) ) ψij (Sit , Sjt ) Zc i=1 1≤i<j≤n
(7)
π t ) is a uniform distribution with only where Zc is the normalizing factor, U(π two values: 0 and 1, ψi is a local motion prior modeled by a Gaussian N (Sit − Si(t−1) ; 0 , Σ s ), and ψij is the interaction potential between target-pair defined as: ψij = exp(−
i(t−1) − S j(t−1) 2 βS ) Sit − Sjt 2
(8)
i(t−1) and S j(t−1) are the optimal states of target tri and trj at frame where S #t-1, and β is a constant factor (usually set to 0.1). 4.4
Tracking Separated-Target
Separated target is tracked by the ISPF (Incremental Self-tuning Particle Filtering) [16] tracker. Its observation model and dynamical model can be easily
Multi-Target Tracking by Learning Class-Specific and Instance-Specific Cues
75
derived, because they are special cases of those of target-group. The main steps of ISPF include: 1) particles are incrementally drawn; 2) each random particle is tuned to its neighborhood best state according some similarity feedback; 3) state optimization is terminated if the maximum similarity score of all tuned particle satisfies a TSD (target similarity distribution) modeled as an online Gaussian, or the maximum number of particles Ns is reached. However, there are some differences between the ISPF tracker in this paper and that in [16]. First, we train two global pose estimators for all target, not one for each. Second, the observation models are very different. We introduce online Boosting classifier to appearance model, meanwhile we do not use IPCA (incremental principle component analysis), but a time-varying feature template updated every frame with a constant learning rate to represent target appearance. This is because it is computationally expensive to maintain and update one subspace for each part of each target. Details of how ISPF work can be seen in [16]. 4.5
Tracking Target-Group
With the defined observation model (6) and dynamical model (7), the Bayesian posterior density (1) can be approximated by particle based density: π t )P (ot |Xt ) P (Xt |Ot ) ≈ c · U(π
ψij (Sit , Sjt )
Ng n k=1 i=1
1≤i<j≤n
(k)
ψi (Sit |Si(t−1) )
(9)
where c is the normalizing factor, Ng is the number of particles, and n is the number of targets in a target-group. The next problem is how to efficiently sample particles for this posterior which lies in a high dimensional state space. Markov chain Monte Carlo (MCMC) sampling [11] is an effective solution to this problem. MCMC methods work by defining a Markov Chain over the space of configurations X, such that the stationary distribution of the chain is equal to a target distribution P (X|O). The Metropolis-Hastings (MH) algorithm [22] is one way to simulate from such a chain. It works as follows: Algorithm 1. Metropolis Hastings Algorithm Start with a random particle X(1) , then iterate for k = 1, ..., N 1. Sample a new particle X from the proposal density Q(X ; X(k) ); 2. Calculate the acceptance ratio: a = min{
P (X |O)Q(X(k) ; X ) , 1} P (X(k) |O)Q(X ; X(k) )
(10)
3. Accept X and set X(k+1) =X with probability a, or set X(k+1) =X(k) . The proposal density used here is a product of a uniform distribution for occlusion varibles and the joint local motion prior centered on previous optimal states: (k)
π t ) Q(Xt ; Xt ) = Q(Xt ) ∼ U(π
ˆ i(t−1) ). ψi (Sit |S
(11)
i
Besides using MCMC-sampling to efficiently sample particles for posterior density, we also utilize global pose estimators to slightly refine the random sampled
76
M. Li et al.
particles for each target if possible (at least the head-should part is visible), so as to make the joint-state particle “smart”. The main steps of our data-driven MCMC-based particle filtering are given in Algorithm 2 . Algorithm 2. Data-Driven MCMC-based Particle Filtering ˆ t−1 , propagate it with the 1. Start with previous optimal state configuration X local joint motion prior density, then iterate for k = 1, ..., Ng – (a) Sample a new particle X from the proposal density (11) – (b) Calculate the visibility vector V i for each target tri according to the joint states {Si } and current occlusion matrix π . : for each target tri , tune its pose {S } to {S } by a global – (c) Refine X to X i i pose estimator (the full-body pose estimator takes priority) if at least the head-should part is visible. – (d) Recalculate the visibility vector for each target and compute particle ) according to the observation model (6); weight w = P (o|X – (e) Calculate posterior density (9) and the acceptance ratio (10) with tuned state X and set {X(k+1) , w(k+1) }={X , w } with probability a, or reject – (f) Accept X t t (k+1) (k+1) (k) (k) , wt }={Xt , wt }. it and set {Xt ∗
∗
t = X(k ) with w(k ) = max{w(k) }. 2. state estimation: X t t t (k) (k) 3. Resample Ng particles from {Xt , wt } and set all weights to 1/Ng
5
Experimental Results
Three sequences are used to test the robustness and efficiency of our tracking algorithm. They are the NLPR-Campus (Fig. 6, our own dataset), PETS09 [23] (Fig. 9) and PETS04 [24] (Fig. 11) sequence. Some important parameters are set as follows: the maximum number of particles Ns for separated-target is set to 100, and the number of particle Ng for target-group is set to 200. Gaussian noise covariance matrix Σ S =diag{72, 72 , 0.022}. Without taking the detection time into consideration, the MATLAB implementation of our method can run at about 1 ∼ 0.4 frames/sec on a PC with a P4 2.4 GHz CPU. Robustness and Efficiency Evaluation. Two sequences (the NLPR-Campus and PETS09 sequence) are used to test the robustness of our algorithm. The PSOF detector [19] is used to detect objects in the two sequences with the detection rate of about 0.5 ∼ 0.7, and the online learned geometrical models decrease false alarms by about 10% in each sequence. Targets are only initialized on the border areas of an image except in the first frame, and their trajectories are terminated if they move out of image borders. In the NLPR-Campus sequence, as shown in Fig 6, six objects with three of them having interactions appear in the scene. At about frame #118, obj1 (means the object with ID 1) is occluded by obj2 slightly, but occlusion reasoning is not conducted because their valid overlapping area (the intersection area of two
Multi-Target Tracking by Learning Class-Specific and Instance-Specific Cues
77
Fig. 6. Tracking results of our method in the NLPR-Campus Sequence
Fig. 7. Occlusion reasoning between object 1 and 2 in the NLPR-Campus Sequence. (a) Occlusion relationship. (b) Appearance similarity scores of object 1 and 2.
Fig. 8. Particles used in the NLPR-Campus seqence. (a) and (b): random particles (blue rectangles) and estimated optimal states (yellow rectangles). (c) The curve of average number of particles per object vs frame #.
ellipses) has not exceeded a threshold (0.4). At about frame #131, obj1 and obj2 form a target-group, and thus the MCMC-based particle filtering process with occlusion reasoning is activated. As shown in Fig. 7 (b), after occlusion reasoning, the appearance similarity score of obj1 increases, this is because our part-based template matching score becomes more accurate when occlusion relationship is correctly recovered (shown in Fig. 7 (a)). At about frame #138, obj1 is totally occluded by obj2 and this occlusion last for 50 frames. At about frame #189, obj1 passes by obj2 and obj3 and reappears, our algorithm successfully tracks
78
M. Li et al.
Fig. 9. Tracking results of our method in the PETS09 sequence
Fig. 10. Particles used in the PETS09 seqence. (a) and (b): random particles (blue rectangles) and estimated optimal states (yellow rectangles). (c) The curve of average number of particles per object vs frame #.
him. Although obj1 and obj3 wear similar clothes (white T-Shirt and black trousers), ID-switching does not happen, because the online-Boosting classifiers can choose discriminative features to distinguish them. Our algorithm can not only robustly track every object, but also work efficiently. As shown in Fig. 8 (c), except in the frames in which significant inter-occlusion happens (#131 ∼ #189), our algorithm uses an average number of less than 10 particles per object. The average number of particles per object per frame throughout the sequence is 21.4. In Fig. 8 (a) and (b), some random particles are shown. It can be seen that separated-targets (obj1/obj2/obj3 at #90; obj3/obj4/obj5/obj6 at #164) often consume a very small number of particles (due to the usage of global pose estimators), while large number of particles are used by target-group (obj1-obj2 at #164) in which joint-state optimization is conducted. The PETS09 sequence is more challenging, as shown in Fig. 9. There are not only frequent inter-object occlusions (obj3-obj1 at #25; obj2-obj1 at #104; obj8-obj2 at #104; obj8-obj3 and obj6-obj4 at #163; obj6-obj1 and obj11-obj10 at #347 ), but also frequent background-object occlusions caused by a signboard (obj3 at #104/#144, and obj10 at #270). Besides, obj1, obj2 and obj3 wear
Multi-Target Tracking by Learning Class-Specific and Instance-Specific Cues
79
Fig. 11. Tracking results of our method in the PETS04 sequence
very similar clothes and have frequent interactions. In this complex scene, our algorithm consistently robustly tracks all objects and maintains their IDs correctly. However, the average computational cost is very low. As shown in Fig. 10 (a) and (b), only those objects in a target-group (obj2-obj7 at #131) or occluded by the signboard (obj3 at #131) consume large number of particles, while others consume several to ten particles per object. The curve of average number of particles per object vs frame No. is shown in Fig. 10 (c), and the average number of particles per object per frame is 26.4. The robustness of our algorithm shown on this sequence relies on two main factors. The first factor is the online learned Boosting classifiers. They make the detection association process robust, as well as provide part of the similarity score in the observation model. The second is the MCMC-based joint-state optimization strategy utilized when object-pairs have significant overlapping areas. Because of using part-based appearance model and occlusion reasoning, the MCMC-based particle filtering process is very robust to partial occlusions. The efficiency of our method obviously benefits from the target clustering strategy and the online trained global pose estimators. Accuracy Evaluation. To evaluate the accuracy of our tracking algorithm, as shown in Fig. 11 the P ET S04 sequence is used, in which three objects are labeled and their groundtruth trajectory can be downloaded from [24]. We use the groundtruth object bounding boxes in the first frame to initialize targets, and no detector is used during tracking. The RMSE (root mean square error) between estimated position and groundtruth of our method is shown in Table. 1. It is about 3.9 pixels per object per frame. As comparison, the tracking accuracies of three other state-of-the-art multi-target tracking algorithms: Yang’s work [12], Qu’s work [10] and Zhang’s work [13] are also given in Table 1 (their results are reported in [13]). It can be seen that the tracking accuracy of our algorithm outperforms [12] (RMSE: 11.2 pixels) and [10] (RMSE: 6.3 pixels), and a little inferior to [13] (RMSE: 3.2 pixels). This is because in Zhang’s work [13], object’s appearance is modeled by an online learned subspace which is a little better than our template based representation. Table 1. Tracking accuracy comparison on the PETS04 sequence Algorithm Our method Yang’s work Qu’s work Zhang’s work RMSE per object per frame (pixels) 3.9 11.2 6.3 3.2
80
M. Li et al.
Discussion. From the above experimental results, we can obtain some useful findings: 1) Online learned Boosting classifiers plays an important role to distinguish targets with similar appearance. 2) Without camera calibration, the online learned geometrical model can effectively reduce false detections that violate perspective projection. 3) The online learned global pose estimators tremendously increase the tracking efficiency of our algorithm. 4) The MCMC-based joint-state optimization process can effectively deal with partial occlusions.
6
Conclusions
This paper proposes a Data-Driven Particle Filtering (DDPF) framework for multi-target tracking, in which online learned class-specific and instance-specific cues are used to make the tracking process robust as well as efficient. The learned cues include an online learned geometrical model for excluding detection outliers that violate geometrical constraints, global pose estimators shared by all targets for particle refinement, and online boosting based appearance models which select discriminative features to distinguish different individuals. Targets are clustered into two categories: separated-target and target group. Separatedtarget is tracked by an ISPF (Incremental Self-tuning Particle Filtering) method in low-dimensional state space and target-group is tracked by an MCMC-based particle filtering in joint-state space with occlusion reasoning. Experimental results demonstrate that our algorithm can automatically detect and track objects, effectively deal with partial occlusions, as well as possess high efficiency and stat e-of-the-art tracking accuracy. Acknowledgement. This work is supported by National Natural Science Foundation of China (Grant No. 60736018, 60723005), National Hi-Tech Research and Development Program of China (2009AA01Z318), National Science Founding (60605014, 60875021).
References 1. Reid, D.: An algorithm for tracking multiple targets. IEEE Trans. Automatic Control 24, 84–90 (1979) 2. Bar-Shalom, Y., Fortmann, T., Scheffe, M.: Joint probabilistic data association for multiple targets in clutter. In: Proc. Conf. Information Sciences and Systems (1980) 3. Cox, I., Leonard, J.: Modeling a dynamic environment using a bayesian multiple hypothesis approach. Artificial Intelligence 66, 311–344 (1994) 4. Rasmussen, C., Hager, G.: Probabilistic data association methods for tracking complex visual objects. IEEE Trans. on PAMI 23, 560–576 (2001) 5. Khan, Z., Balch, T., Dellaert, F.: Multitarget tracking with split and merged measurements. In: Proc. of CVPR 2005 (2005) 6. Isard, M., MacCormick, J.: Bramble: A bayesian multipleblob tracker. In: Proc. of ICCV 2001 (2001) 7. Okuma, K., Taleghani, A., de Freitas, N., Little, J.J., Lowe, D.G.: A boosted particle filter: Multitarget detection and tracking. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3021, pp. 28–39. Springer, Heidelberg (2004)
Multi-Target Tracking by Learning Class-Specific and Instance-Specific Cues
81
8. Vermaak, J., Doucet, A., Perez, P.: Maintaining multi-modality through mixture tracking. In: Proc. of ICCV 2003 (2003) 9. Yu, T., Wu, Y.: Decentralized multiple target tracking using netted collaborative autonomous trackers. In: Proc. of CVPR 2005 (2005) 10. Qu, W., Schonfeld, D., Mohamed, M.: Real-time interactively distributed multiobject tracking using a magnetic- inertia potential model. In: Proc. of ICCV 2005 (2005) 11. Khan, Z., Balch, T., Dellaert, F.: Mcmc-based particle filtering for tracking a variable number of interacting targets. PAMI 27, 1805–1819 (2005) 12. Yang, M., Yu, T., Wu, Y.: Game-theoretic multiple target tracking. In: Proc. of ICCV 2007 (2007) 13. Zhang, X., Hu, W., Li, W., Qu, W., Maybank, S.: Multi-object tracking via species based particle swarm optimization. In: ICCV 2009 WS on Visual Surveillance (2009) 14. Hess, R., Fern, A.: Discriminatively trained particle filters for complex multi-object tracking. In: Proc. of CVPR 2009 (2009) 15. Breitenstein, M., Reichlin, F., Leibe, B., Koller-Meier, E., Gool, L.: Robust tracking-by-detection using a detector confidence particle filter. In: ICCV 2009 (2009) 16. Li, M., Chen, W., Huang, K., Tan, T.: Visual tracking via incremental self-tuning particle filtering on the affine group. In: Proc. of CVPR 2010 (2010) 17. Arulampalam, M.S., Maskell, S., Gordon, N., Clapp, T.: A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking. Journal of IEEE Trans. on Signal Processing 50, 174–188 (2002) 18. Vijayakumar, S., Souza, A.D., Schaal, S.: Incremental online learning in high dimensions. Neural Computation 17, 2602–2634 (2005) 19. Li, M., Zhang, Z., Huang, K., Tan, T.: Pyramidal statistics of oriented filtering for robust pedestrian detection. In: Proc. of ICCV 2009 WS on Visual Surveillance (2009) 20. Grabner, H., Bischof, H.: On-line boosting and vision. In: Proc. of CVPR 2006 (2006) 21. Dalal, N., Triggls, B.: Histograms of oriented gradients for human detection. In: CVPR (2005) 22. Hastings, W.: Monte carlo sampling methods using markov chains and their applications. Biometrika 57, 97–109 (1970) 23. PETS09dataset, http://www.cvg.rdg.ac.uk/PETS2009/a.html 24. PETS04dataset, http://homepages.inf.ed.ac.uk/rbf/CAVIARDATA1/
Modeling Complex Scenes for Accurate Moving Objects Segmentation Jianwei Ding, Min Li, Kaiqi Huang, and Tieniu Tan National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, P.R. China {jwding,mli,kqhuang,tnt}@nlpr.ia.ac.cn Abstract. In video surveillance, it is still a difficult task to segment moving object accurately in complex scenes, since most widely used algorithms are background subtraction. We propose an online and unsupervised technique to find optimal segmentation in a Markov Random Field (MRF) framework. To improve the accuracy, color, locality, temporal coherence and spatial consistency are fused together in the framework. The models of color, locality and temporal coherence are learned online from complex scenes. A novel mixture of nonparametric regional model and parametric pixel-wise model is proposed to approximate the background color distribution. The foreground color distribution for every pixel is learned from neighboring pixels of previous frame. The locality distributions of background and foreground are approximated with the nonparametric model. The temporal coherence is modeled with a Markov chain. Experiments on challenging videos demonstrate the effectiveness of our algorithm.
1
Introduction
Moving object segmentation is fundamental and important for intelligent video surveillance. The accuracy of segmentation is vital to object tracking, recognition, activity analysis, etc. Traditional video surveillance systems use stationary cameras to monitor scenes of interest. The usual approach for this kind of system is background subtraction. In many video surveillance applications, the background of scene is not stationary. It may contain waving trees, fountains, ocean ripples, illumination changes, etc. It is a difficult task to design a robust background subtraction algorithm that can extract moving object accurately from complex scenes. Many background subtraction methods have been proposed to address this problem. Stauffer and Grimson [1] used Gaussian mixture model (GMM) to account for the multimodality of color distribution for every pixel. Because of its high efficiency and good accuracy for most conditions, GMM has been used widely. And many researchers have proposed improvements and extensions of the basic GMM method, such as [2–4]. But an implicit assumption for GMMbased method is that the temporal variation of a pixel could be modeled by one or multiple Gaussian distribution. Elgammal et al. proposed Kernel density estimation (KDE) [5] to model color distribution, which does not have any R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 82–94, 2011. Springer-Verlag Berlin Heidelberg 2011
Modeling Complex Scenes for Accurate Moving Objects Segmentation
83
Fig. 1. Moving object segmentation in complex scenes using background subtraction. The first column shows the original images, the second column gives the corresponding segmentation using background subtraction.
underlying distribution. But the bandwidth of kernel is difficult to choose and the computation cost is high. Sheikh and Shah [6] incorporated spatial cue with temporal information into a single model. They got accurate segmentation result in a Maximum a posteriori-Markov Random Field (MAP-MRF) decision framework. Some other sophisticated techniques can be seen in [7–11]. However, there is a drawback of background subtraction. Thresholding is often used to segment foreground from background. Fig. 1 gives an example of moving objects segmentation in complex scenes using background subtraction. How to find an optimal segmentation is an energy minimization problem in MAP-MRF framework. Graph cuts [12] is one of the popular energy minimization algorithm. Boykov et al. [13] proposed an interactive image and video segmentation technique using graph cuts and got good results. This technique was then extended to solve other foreground/background segmentation problems. In [14], Layered Graph cuts (LGC) were proposed to extract foreground from background layers in stereo video sequences. In [15], Liu Feng et al. proposed an method which fused color, locality and motion information. Their method can segment moving objects with limited motion, but works off-line. Sun et al. [16] proposed background cut which combined background subtraction, color and contrast cues. Their method could work in real time. But static background was assumed in the initialization phase. In [17], multiple cues were probabilistically fused in bilayer segmentation of live video. In [18], the complex background was learned by multi-scale discriminative model. But training samples are needed in [17, 18]. We propose an online and unsupervised technique which learns complex scenes to find optimal foreground/background segmentation in a MRF framework. Our method absorbs the advantages of foreground/background modeling and Graph cuts algorithm. But our method works online and does not need prior information or training data. To improve the segmentation accuracy, color, locality, temporal coherence and spatial consistency are fused together in the framework. All these features are easy to get and calculate. We do not choose optical flow as [15, 17] because the reliable computation of optical flow is expensive and it suffers from aperture problem. The models of color, locality and temporal coherence are learned online from complex scenes. Color feature is the dominant one to discriminate foreground from background in our algorithm. To model the color distribution of background image precisely and adaptively, a novel mixture
84
J. Ding et al.
of nonparametric regional model and parametric pixel-wise model is presented to improve the robustness of algorithm under different background change. The foreground color model for every pixel in current frame is learned from the neighboring foreground pixels in previous frame. The locality distributions of background and foreground are also considered to reduce error. The distribution is approximated using a nonparametric model. Besides, an additional spatial uniform distribution is also considered in the model. The label of each pixel at current moment is not independent of that in previous time. This temporal coherence of labels is modeled with a first order Markov chain to increase accuracy. A simple strategy to calculate the transition probabilities is also provided. The remainder of this paper is organized as follows. Section 2 describes the general model for moving object segmentation. Section 3 gives the modeling of complex scenes. Section 4 presents the experiments and results. Section 5 makes conclusions of our work.
2
General Model for Moving Object Segmentation
In this section, we will describe the general model for moving object segmentation in a MAP-MRF framework.
Fig. 2. A simple Flowchart of the proposed algorithm
Moving object segmentation can be considered as a binary labeling problem. Let I be a new input image and P be the set of all pixels in I. The labeling problem is to assign a unique label lp ∈ {f oreground(= 1), background(= 0)} to each pixel p ∈ P. The maximum a posteriori estimation of L = {l1 , l2 · · · , l|P| } for I can be obtained by minimizing the Gibbs energy E(L) [13]: L∗ = arg min E(L)
(1)
The basic energy model E(L) used in [13] only considered color and spatial continuity cues, which are not sufficient [14]. The locality cue of a pixel is an
Modeling Complex Scenes for Accurate Moving Objects Segmentation
85
important spatial constraint that can help to judge the label value. Besides, the label of a pixel in current time is not independent of that in previous time. So the cue of temporal coherence is also needed to increase the accuracy for label decision. We extend the basic energy model and define E(L) as: E(L) = λc Ec (lp ) + λs Es (lp ) + λt Et (lp ) + EN (lp , lq ) (2) p∈P
p∈P
p∈P
p∈P q∈Np
where λc , λs and λt are factors that measure the weights of color term, locality term and temporal coherence term respectively. Np is the 4-connection or 8connection neighborhood of pixel p. Ec (lp ), Es (lp ), Et (lp ) and EN (lp , lq ) are described below. Color term Ec (lp ) is the color term of pixel p, denoting the cost of color vector cp when the label is lp . Ec (lp ) is defined as the negative logarithm of color likelihood: Ec (lp ) = − log P r(cp |lp ) (3) Locality term Es (lp ) is the locality term of pixel p, denoting the cost of locality vector sp when the label is lp . Es (lp ) is defined as the negative logarithm of locality likelihood: (4) Es (lp ) = − log P r(sp |lp ) Temporal coherence term Et (lp ) is the temporal coherence term of pixel p, denoting the cost of label lp in current time T when the label was lpT −1 in previous time T − 1. We use a first-order Markov chain to model the temporal coherence and define Et (lp ) as: Et (lp ) = − log P r(lpT |lpT −1 )
(5)
Spatial consistence term There is spatial consistence for neighboring pixels. They are more likely to have similar colors and same labels. EN (lp , lq ) is used to convey the spatial consistence between neighboring pixels p and q when the labels are lp and lq respectively. Similar to [13, 17], EN (lp , lq ) is expressed as an Ising term and defined as: EN (lp , lq ) = δlp =lq
+ exp(−βd2pq ) 1+
(6)
where δlp =lq is a Kronecker delta function. The constant is a positive parameter. β is also a parameter and set to β = (2 cp − cq )−1 , where · is the expectation over color samples of neighboring pixels in an image. dpq denotes the color difference between pixel p and q. To reduce computation cost, can be set to zero and β is taken as a constant, which is used in [13]. A simpler version of the Ising term can be seen in [6], which ignored the color difference. Among the four energy terms, only the spatial consistence term can be calculated directly from the new input image. But the color likelihood, the locality likelihood and the probability transition matrix should be known first to get the other three energy terms. They can be calculated by constructing color model, locality model and temporal coherence model. The three models are learned from the complex scenes and updated online.
86
J. Ding et al.
At the beginning of the algorithm, an initial background model is constructed using the color information of input images. The background model is updated quickly with high learning parameters until it is stable. After initialization, the flowchart of the algorithm is shown in Fig. 2. At each time T, the first step is modeling the scene and calculate the energy E(L). The second step is to minimize the energy E(L) and output the segmentation. To minimize the energy E(L), we use the implementation of [19]. The third step is to update the scene model with slow learning parameters.
3
Modeling the Complex Scenes
The first and most important step is modeling the scenes. As it may contain dynamic texture or various illumination changes in complex scenes, color, locality, temporal coherence and spatial consistence are fused together to increase the accuracy of segmentation. The spatial consistence model defined by Eq. (6) can be calculated directly on the input image. We only have to construct the color model, the locality model and the temporal coherence model. They will be described below. For simplicity, the update of the scene model is also included in this section. 3.1
Color Model
The color term is the dominant one among the four terms in Eq. (2). Different color features can be used in our algorithms, such as RGB, HSV, Lab. To discriminate foreground from background effectively, we choose RGB color features without color transformation. Let cp = (Rp , Gp , Bp )T be the color vector for each pixel p ∈ P. The modeling of color distribution includes three parts: modeling the background color distribution, modeling the foreground color distribution and updating the color model . Modeling the background color distribution. In [6], the background modeling methods are classified into two categories, which are methods that use pixel-wise model and methods that use regional model. The pixel-wise model is more precise than the regional model to measure the probability of pixel p belonging to background, but it is sensitive to spatial noise, illumination change and small movement of background. To model the color likelihood for each pixel p belonging to background, we define it as a mixture of regional model and pixel-wise model: P r(cp |lp = 0) = αc P rregion (cp |lp = 0) + (1 − αc )P rpixel (cp |lp = 0)
(7)
where αc ∈ [0, 1] is a mixing factor, and P rregion (cp |lp = 0) is the probability of observing cp at the neighbor region of pixel p, and P rpixel (cp |lp = 0) is the probability of observing cp at pixel p. The local spatial and temporal color information is used coherently in Eq. (7). Fig. 3 gives an comparatione of the three models. Our model can approximate the background color distribution precisely and is also robust to various spatial noise.
Modeling Complex Scenes for Accurate Moving Objects Segmentation
87
Fig. 3. Comparation of the three models. The three columns are color distributions of the pixel marked as red in the original image using the pixel-wise model, the regional model and the proposed mixture model, respectively.
The regional background color model can be learned from the local neighboring spatial-temporal volume Vp centered in pixel p with a nonparametric method: 1 P rregion (cp |lp = 0) = KH (cp − cm ) (8) |Vp | cm ∈Vp
where KH (x) = |H|−1/2 K(H−1/2 x) and K(x) is a d-variate kernel function and H is the bandwidth matrix. We select the commonly used Gaussian kernel, so 1 KH (x) = |H|−1/2 (2π)−1/2 exp(− xT H−1 x) (9) 2 To reduce computation cost, the bandwidth matrix H can be assumed to be the form of H = h2 I . The volume Vp is defined as Vp = {ct (u, v)||u − i| ≤ l, |v − j| ≤ l, T − Td < t < T }, where ct (u, v) is the color vector of pixel at the u-th row and v-th column of the image taken at time t, and Td is the period. Pixel p is at the location of i-th row and j-th column of the current input image at time T, and l is a constant value that measures the size of the local spatial region. The computation cost is very high to calculate Eq. (8) directly because of the large sample size. Besides, it is hard to choose proper kernel bandwidth H in high dimensional spaces even it has been simplified. We use the binned kernel density estimator to approximate Eq. (8). The accuracy and effectiveness of binned kernel density estimator has been proved in [20]. Similar strategy has also been used in [6, 7].
88
J. Ding et al.
We use a Gaussian mixture model as the pixel-wise model for every pixel p: K ωk N (cp |μk , Σk ) (10) P rpixel (cp |lp = 0) = k=1
where ωk , μk and Σk are the weight, the mean color vector and the covariance matrix of the k-th component respectively, and K is the number of components. To simplify computation, the covariance matrix is assumed to be of the form: Σk = σk2 I. Modeling the foreground color distribution. The color likelihood P r(cp |lp = 1) for each pixel p belonging to foreground at time T can be learned adaptively based on data from the neighboring foreground pixels in the previous frame at time T-1. Here there is an implicit assumption that a foreground pixel will remain in the neighboring area. Besides, at any time the probability of observing a foreground pixel at pixel p of any color is uniform. Thus, the foreground color likelihood can be expressed as a mixture of a uniform function and the kernel density function: 1 KH (cp − cm ) (11) P r(cp |lp = 1) = βc ηc + (1 − βc ) |Up | cm ∈Up
where βc ∈ [0, 1] is the mixing factor, and ηc is a random variable with uniform probability ηc (cp ) = 1/2563 . The set of neighboring foreground pixels in the previous frame is defined as Up = {ct (u, v)||u − i| ≤ w, |v − j| ≤ w, t = T − 1}, where w is a constant that measures the size of the neighboring region at pixel p, and pixel p is at the i-th row and j-th column of current image. The kernel density function is the same as Eq. (9). Updating the color model. The background color model is updated adaptively and online with the new input image after segmentation. For the nonparametric regional background color model, a sliding window of length lb is maintained. For the parametric pixel-wise model, similar maintenance strategy [1] is used here. If the new observation cp belongs to the k-th distribution, the parameters of the k-th distribution at time T is updated as follows: 2 σk,T
= (1 −
μk,T = (1 − ρ)μk,T −1 + ρcp
(12)
+ ρ(cp − μk,T ) (cp − μk,T ) ωk,T = (1 − ρ)ωk,T −1 + ρMk,T
(13) (14)
2 ρ)σk,T −1
T
where ρ is the learning rate. Mk,T is 1 when cp matches the k-th distribution and 0 otherwise. 3.2
Locality Model
It is not sufficient to use color alone if the color of foreground is similar to that of background. It can reduce error to take account of the location of moving objects. The spatial location of pixel p is denoted as sp = (xp , yp )T . As the shape of objects can be arbitrary, we use a nonparametric model similar to
Modeling Complex Scenes for Accurate Moving Objects Segmentation
89
[15] to approximate the spatial distribution of background and foreground. The difference is that we take account of the spatial uniform distribution for any pixel at any location. The locality distribution of background and foreground are defined as: 1 (sp − sm )T (sp − sm ) P r(sp |lp = 0) = αs ηs + (1 − αs ) max exp(− ) (15) 2πσ 2 m∈Vs 2σ 2 1 (sp − sm )T (sp − sm ) max exp(− ) (16) P r(sp |lp = 1) = αs ηs + (1 − αs ) 2πσ 2 m∈Us 2σ 2 where αs ∈ [0, 1] is the mixing factor, and ηs is a random variable with uniform probability ηs (sp ) = 1/M × N . Here M × N means the image size. Vs and Us are the sets of all background pixels and foreground pixels in the previous frame respectively. The parameter σ is the bandwidth of kernel. An example of locality distributions of background and foreground is shown in Fig. 4. From this figure we can see that the pixels near moving objects have higher probability belonging to foreground, and the pixels far away from moving object have higher probability belonging to background. 3.3
Temporal Coherence Model
For any pixel p ∈ P, if the label is foreground at time T-1, it is more likely for p to remain foreground than background at time T. So the label of pixel p at time T is not independent of labels at previous time. This is called the label temporal
Fig. 4. Locality distributions of background and foreground
Fig. 5. A simple example to show the temporal coherence of labels. The first image depicts the scene, the second image shows the foreground mask at time T-1, and the last image shows the foreground mask at time T. A small portion of pixels whose labels have changed are marked as red in the last image.
90
J. Ding et al.
coherency. Fig. 5 is an example to show the coherency. It can be modeled using a Markov chain. Here, a first order Markov chain is adopted to convey the nature of temporal coherence. There are four (22 ) possible transitions for the first order Markov chain model since there are only two candidate labels of pixel p. But there remain two degrees of freedom because there is relationship between P r(lpT = 1|lpT −1 ) and P r(lpT = 0|lpT −1 ), which is P r(lpT = 1|lpT −1 ) = 1 − P r(lpT = 0|lpT −1 ). We denote them as P rF F and P rBB , where P rF F denotes the probability of remaining foreground, and P rBB denotes the probability of remaining background. A simple strategy is described here to calculate P rF F and P rBB . They are both initialized to be 0.5. At time T-2, the number of foreground pixels and background pixels are denoted as nTF −2 and nTB−2 respectively. When there are detected moving objects, nTF −2 > 0 and nTB−2 > 0 . Then at time T-1 the number of pixels which remain foreground is nTF −1 and the number of pixels which remain background is nTB−1 . We can approximate P rF F and P rBB at time T by: P rFT F = T = P rBB
1 nT −1 αT + (1 − αT ) F 2 nTF −2 1 nT −1 βT + (1 − βT ) B 2 nTB−2
(17) (18)
where αT ∈ [0, 1] and βT ∈ [0, 1] are mixing factors. If nTF −2 = 0 or nTB−2 = 0, P rF F and P rBB remain the previous values.
4
Experiments and Results
We tested our algorithm on a variety of videos, which include fountain, swaying trees, camera motion, etc. The algorithm was implemented using C++, on a computer with 2.83 GHz Intel Core2 CPU and 2GB RAM, and achieved the processing speed of about 6 fps at the resolution of 240 × 360. Since GMM [1] and KDE [5] are two widely used background subtraction algorithms, we compare the performance of our method with them. To evaluate the effectiveness of the three algorithms, no morphological operations like erosion and dilation are used in the experiments. Besides, both qualitative analysis and quantitative analysis are used to demonstrate the effectiveness of our approach. Fig. 6 shows qualitative comparision result on outdoor scenes with fountain. The first column are frames 470, 580, 680 and 850 in the video. There are moving objects in the first, third and fourth images. The second column is the manual segmentation result. The third column shows the result of our method. The fourth and fifth columns are results of GMM and KDE respectively. The difficulty of moving objects extraction in this video is that the background changes continuously and violently. It is difficult to adapt the background model to the dynamic texture for the algorithms of GMM and KDE. But our method works well in the presence of fountain and gives accurate segmentation.
Modeling Complex Scenes for Accurate Moving Objects Segmentation
91
Fig. 6. Segmentation results in the presence of fountain. The first column shows the original images, the second column gives the manual segmentation, the third column presents the result of our method, the last two columns show the results of GMM and KDE respectively.
Fig. 7. Segmentation results in the presence of waving trees. The first column shows the original images, the second column gives the manual segmentation, the third column presents the result of our method, the last two columns show the results of GMM and KDE respectively.
Fig. 7 gives qualitative comparison result on outdoor scenes with waving trees. The video comes from [8]. Our method can detect most of the foreground pixels accurately and has little false detection. Fig. 8 gives segmentation on another two challenging videos by our method. The videos come from [10]. In the two original sequences, there are ocean ripples and waving trees respectively.
92
J. Ding et al.
Fig. 8. Segmentation results in complex scenes. The first column shows the original images which contain ocean ripples, the second column presents the corresponding result of our method. The third column shows the original images which contain waving trees, the fourth column presents the corresponding result of our method.
Fig. 9. Segmentation results in the presence of camera motion. The first column shows the original images, the second column gives the manual segmentation, the third column presents the result of our method, the last two columns show the results of GMM and KDE respectively.
In order to better demonstrate the performance of our algorithm, a quantitative analysis is provided here. The test video and corresponding ground-truth data come from [6]. This video contains average nominal motion of about 14 pixels.
Modeling Complex Scenes for Accurate Moving Objects Segmentation 1
1
0.9
0.8
0.8
0.7
0.7 0.6
0.6 0.5
Recall
Precision
0.9
Ours GMM with rate 0.005 GMM with rate 0.05 GMM with rate 0.1 KDE
0.4 0.3 0.2
0.5 0.4 0.3 0.2
0.1 0 250
93
Ours GMM with rate 0.005 GMM with rate 0.05 GMM with rate 0.1 KDE
0.1
300
350
frame number
400
450
500
0 250
300
350
frame number
400
450
500
Fig. 10. Precision and Recall comparisons of the three algorithms. The GMM is tested with different learning rate:0.005, 0.05, 0.1.
Fig. 9 shows a qualitative illustration of the results as compared to the ground truth, GMM and KDE. From this figure, we can see that our method can segment moving objects accurately and has almost no false detections in the presence of camera motion. Fig. 10 shows the corresponding quantitative comparison in terms of precision and recall. The precision and recall are defined as: N umber of true positives detected T otal number of positives detected N umber of true positives detected Recall = T otal number of true positives
P recision =
From Fig. 10 we can see that the detection accuracy both in terms of precision and recall is consistently higher than GMM and KDE. Several learning rates are selected to test for GMM. Though the precision and recall of our method are both very high, there are still some false positives and false negatives detected by our method. These pixels are mainly at the edge or contour of true objects. That is because spatial continuity information is used in the proposed method.
5
Conclusions
In this paper, we have proposed an online and unsupervised technique which learns multiple cues to find optimal foreground/background segmentation in a MRF framework. Our method adopts the advantages of foreground/background modeling and Graph cuts algorithm. To increase the accuracy of foreground segmentation, color, locality, temporal coherency and spatial continuity are fused together in the framework. The distributions of different features for background and foreground are modeled and learned separately. Our method can segment moving object accurately in complex scenes. That was proved by experiments on several challenging videos. Acknowledgement. This work is supported by National Natural Science Foundation of China (Grant No. 60736018, 60723005), National Hi-Tech Research and Development Program of China (2009AA01Z318), National Science Founding (60605014, 60875021).
94
J. Ding et al.
References 1. Stauffer, C., Grimson, W.: Learning pattern of activity using real-time tracking. IEEE Trans. Pattern Analysis and Machine Intelligence 22, 747–757 (2000) 2. KaewTraKulPong, P., Bowden, R.: An improved adaptive background mixture model for realtime tracking with shadow detection. In: Proc. European Workshop on Advanced Video Based Surveillance Systems (2001) 3. Zivkovic, Z.: Improved adaptive gaussian mixture model for background subtraction. In: Proc. Int. Conf. Pattern Recognition, pp. 28–31 (2004) 4. Lee, D.S.: Effective gaussian mixture learning for video background subtraction. IEEE Trans. Pattern Analysis and Machine Intelligence 27, 827–832 (2005) 5. Elgammal, A., Harwood, D., Davis, L.: Non-parametric model for background subtraction. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1843, pp. 751–767. Springer, Heidelberg (2000) 6. Sheikh, Y., Shah, M.: Bayesian modeling of dynamic scenes for object detection. IEEE Trans. Pattern Analysis and Machine Intelligence 27, 1778–1792 (2005) 7. Ko, T., Soatto, S., Estrin, D.: Background subtraction on distributions. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part III. LNCS, vol. 5304, pp. 276– 289. Springer, Heidelberg (2008) 8. Toyama, K., Krumm, J., Brumitt, B., Meyers, B.: Wallflower: Principles and practice of background maintenance. In: Proc. ICCV, pp. 255–261 (1999) 9. Monnet, A., Mittal, A., Paragios, N., Ramesh, V.: Background modeling and subtraction of dynamic scenes. In: Proc. ICCV, pp. 1305–1312 (2003) 10. Li, L., Huang, W., Gu, I.Y.H., Tian, Q.: statistical modeling of complex backgrounds for foreground object detection. IEEE Transactions on Image Processing 13, 1459–1472 (2004) 11. Pless, R., Larson, J., Siebers, S., Westover, B.: Evaluation of local models of dynamic backgrounds. In: Proc. CVPR, pp. 73–78 (2003) 12. Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., Rother, C.: A comparative study of energy minimization methods for markov random fields. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 16–29. Springer, Heidelberg (2006) 13. Boykov, Y., Jolly, M.: Interactive graph cuts for optimal boundary region segmentation of objects in n-d images. In: Proc. ECCV, pp. 105–112 (2006) 14. Kolmogorov, V., Criminisi, A., Blake, A., Cross, G., Rother, C.: Bi-layer segmentation of binocular stereo video. In: Proc. CVPR, pp. 1186–1193 (2005) 15. Liu, F., Gleicher, M.: Learning color and locality cues for moving object detection and segmentation. In: Proc. CVPR, pp. 320–327 (2009) 16. Sun, J., Zhang, W., Tang, X., Shum, H.-Y.: Background cut. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 628–641. Springer, Heidelberg (2006) 17. Criminisi, A., Cross, G., Blake, A., Kolmogorov, V.: Bilayer segmentation of live video. In: Proc. CVPR, vol. 1, pp. 53–60 (2006) 18. Zha, Y., Bi, D., Yang, Y.: Learning complex background by multi-scale discriminative model. Pattern Recognition Letters 30, 1003–1014 (2009) 19. Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. Pattern Analysis and Machine Intelligence 26, 1124–1137 (2004) 20. Hall, P., Wand, M.: On the accuracy of binned kernel estimators. J. Multivariate Analysis (1995)
Online Learning for PLSA-Based Visual Recognition Jie Xu1,2 , Getian Ye3 , Yang Wang1,2 , Wei Wang1,2 , and Jun Yang1,2 1
National ICT Australia University of New South Wales 3 Canon Information Systems Research Australia {jie.xu,yang.wang,jun.yang}@nicta.com.au,
[email protected],
[email protected] 2
Abstract. Probabilistic Latent Semantic Analysis (PLSA) is one of the latent topic models and it has been successfully applied to visual recognition tasks. However, PLSA models have been learned mainly in batch learning, which can not handle data that arrives sequentially. In this paper, we propose a novel on-line learning algorithm for learning the parameters of PLSA. Our contributions are two-fold: (i) an on-line learning algorithm that learns the parameters of a PLSA model from incoming data; (ii) a codebook adaptation algorithm that can capture the full characteristics of all the features during the learning. Experimental results demonstrate that the proposed algorithm can handle sequentially arriving data that batch PLSA learning cannot cope with, and its performance is comparable with that of the batch PLSA learning on visual recognition.
1
Introduction
Visual recognition involves the task of automatically assigning visual data to the semantic categories. Recently probabilistic latent topic models such as probabilistic Latent Semantic Analysis (PLSA) [1], Latent Dirichlet Allocation (LDA) [2] and their extensions have been applied to the visual recognition tasks. The PLSA has been successfully employed on the tasks of object categorization [3], scene recognition [4], and also action recognition [5], [6]. However, PLSA has the following disadvantages. Firstly, the conventional learning for PLSA is batch learning, and hence it can not handle data that arrives sequentially. Secondly, PLSA relies on a codebook of words for the “bag-of-words” representation. For visual recognition, k-means clustering algorithm is usually employed to produce a codebook of visual words [3],[5],[4]. An optimal codebook can be constructed by taking into account of all the training features. However, it would be infeasible to apply the same algorithm on large datasets with excessive features. In practice, only a subset of features is chosen for the clustering, and it results in a suboptimal codebook. In order to address the aforementioned problems, we propose an on-line learning algorithm for learning the parameters of PLSA model from the incoming data. The contributions of this paper are two-fold: (i) R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 95–108, 2011. Springer-Verlag Berlin Heidelberg 2011
96
J. Xu et al.
an on-line learning algorithm that learns the parameters of PLSA model from incoming data; (ii) a codebook adaptation algorithm that can capture the full characteristics of all the features during the on-line learning. Related Work. Using the proposed on-line algorithm, the trained model can adapt to new data without re-using the previous training data. Incremental EM is employed to speed up the learning of PLSA in [7]. However, the algorithm is not an on-line one since previous training data has to be re-used, together with the new data, for model adaptation. Previous work on on-line learning of PLSA and other latent models can be found in the text mining area [8], [9], [10]. However, all these algorithms are proposed for text mining only, and there is no visual codebook and its adaptation involved. The most related work to our algorithm is the QB-PLSA proposed by Chien et al. in [8]. QB-PLSA employs the QB estimation [11] to enable the on-line learning of PLSA, which represents the priors of PLSA parameters using Dirichlet densities. Our algorithm, on the other hand, employs the on-line EM for stochastic approximation of PLSA, and makes no assumptions of the parameter distributions. To the best of our knowledge, our work proposes the first on-line PLSA algorithm for vision problems. We apply the proposed algorithm to two visual recognition tasks, i.e., scene recognition and action recognition. The task of scene recognition is referred to as the classification of semantic categories of places using visual information collected from scene images. The work in the literature can be classified into one of the two following approaches. The first approach considers global or local feature-based image representation [12],[13],[14], whereas the second one relies on the intermediate concept-based representation [15],[16],[4],[17],[18]. Among the work from the second category, some models learn the concepts in a supervised manner [15],[17], while the others learn without supervision [16],[4],[18]. The PLSA-based scene recognition belongs to the second category, and it has been shown by Bosch et al. [4] to outperform previous models [15],[16]. As for the action recognition, it is an important task in video surveillance. Numerous techniques have been proposed for action recognition in the literature. A popular approach is based on the analysis of motion cues in human actions [19],[20]. Despite its popularity, the robustness of this type of approach depends heavily on the tracking systems, especially under complex situations in videos. To handle complex situations, another type of approach attempts to build dynamic human action models [21],[22],[23]. However, the large number of parameters from the models results in high model complexities. To avoid the model complexities, local feature-based video representation is employed. Salient features are extracted from the videos, and videos are then converted into intermediate representations based on the features, before being fed into different classifiers [24],[25],[26],[27]. The PLSA has been employed for action recognition based on the “bag-of-words” representation using spatial-temporal interest points [24], and it has been shown to perform well on this task [5]. The experimental results of our proposed algorithm show that the proposed algorithm can handle large datasets that batch PLSA learning cannot cope with, and its performance is comparable with that of the batch PLSA learning on visual recognition.
Online Learning for PLSA-Based Visual Recognition
97
The rest of the paper is organized as follows: Section 2 reviews the batch learning algorithm for PLSA-based visual recognition. Our proposed algorithm is described in Section 3, followed by the experimental results in Section 4. Our final conclusions are summarized in Section 5.
2
PLSA-Based Visual Recognition
The PLSA was originally proposed for text modeling, in which words are considered as the elementary building units for documents. Rather than modeling documents as sets of words, PLSA models each document as a mixture of latent topics, and each topic is characterized by a distribution over words. In the context of visual recognition, the visual features are considered as words. Denote zk ∈ Z = {z1 , z2 , ..., zK } as the latent topic variable that associates the words and documents, and X as the training co-occurrence matrix consists of co-occurrence counting of (di , wj ), collected from N visual documents D = {d1 , d2 , ..., dN }, based on a codebook of V visual words W = {w1 , w2 , ..., wV }. The joint probability of the visual document di , the hidden topic zk and the visual word wj is assumed to have the following form [1]: p(di , zk , wj ) = p(wj |zk )p(zk |di )p(di ),
(1)
where p(wj |zk ) represents the probability of the word wj occurring in the hidden topic zk , p(zk |di ) is the probability of the topic zk occurring in the document di , and p(di ) can be considered as the prior probability of di . The joint probability of observation pair (di , wj ) can be generated by marginalizing over all the topic variables zk : p(zk |di )p(wj |zk ). (2) p(di , wj ) = p(di ) k
Let n(di , wj ) be the number of occurrences of the word wj in the document di , and all of them constitute the co-occurrence matrix X. The prior probabil ity p(di ) can be computed as p(di ) ∝ j n(di , wj ). The parameters of PLSA then Kdistributions over K latent variables, and such that V contain multinomial p(w |z ) = 1 and j k j=1 k=1 p(zk |di ) = 1. We represent these KV + KN probabilities using θ = {{p(wj |zk )}j,k , {p(zk |di )}k,i }. The log likelihood of X given θ can be written as L(X|θ) =
N V
n(di , wj ) log p(di , wj ).
(3)
i=1 j=1
A maximum likelihood estimation of θ is obtained by maximizing the log likelihood using an Expectation Maximization (EM) algorithm [28]. Starting from an initial estimate of θ, the expectation (E) step of the algorithm computes the following expectation function ˆ = Ez [log P (X, Z|θ)|X, ˆ Q(θ|θ) θ] ∝
N V i=1 j=1
n(di , wj )
K k=1
p(zk |di , wj ) log[p(zk |di )p(wi |zk )].
98
J. Xu et al.
The maximization (M) step then calculates the new estimate θˆ by maximizing ˆ computed at E-step. The EM algorithm alterthe expectation function Q(θ|θ) nates between both steps until the log likelihood converges. It is shown in [1] that, the E step is equivalent to computing the posterior probability p(zk |di , wj ), given the estimated p(wj |zk ), p(zk |di ): p(wj |zk )p(zk |di ) p(zk |di , wj ) = K , l=1 p(wj |zl )p(zl |di )
(4)
while the M step corresponds to the following updates, given the computed p(zk |di , wj ) from the E-step: new
p(wj |zk )
N
i=1 n(di , wj )p(zk |di , wj ) , N m=1 i=1 n(di , wm )p(zk |di , wm )
= V
(5)
V
new
p(zk |di )
j=1 n(di , wj )p(zk |di , wj ) = K V . l=1 j=1 n(di , wj )p(zl |di , wj )
(6)
In (6), the denominator can also be written as n(di ), i.e., the total number of words occurring in di . During the inference stage, given a testing visual document dtest , the topicbased intermediate representation p(zk |dtest ) are computed by using the “foldin” heuristic described in [1]. The heuristic employs the EM algorithm in the same way as in the learning, while fixing the values of coefficients p(wj |zk ) obtained from the training stage.
3
Proposed Algorithm
We are to present our algorithm for PLSA under the setting of on-line learning, where training data is assumed to arrive at successive time slices. De(t) (t) (t) note D(t) = {d1 , d2 , ..., dNt } as the data stream arriving at time t. and (t)
(t)
(t)
F (t) = {f1 , f2 , ..., fNf } as the feature descriptors from D(t) . Let W (t) = t
{w1 , w2 , ..., wVt } be the codebook of visual words at different time slice t, X (t) be the co-occurrence matrix at time t. The goal of our algorithm is to learn the parameters of PLSA model from data streams. 3.1
On-line Codebook Adaptation for PLSA
To train a PLSA model for visual recognition, a codebook of visual words is required. The k-means clustering algorithm is usually employed to produce the codebook using all the features in the training set. Niebles et al. show that a bigger sized codebook often results in a higher recognition accuracy [5]. The size of the codebook is usually proportional to the number of features. However in practice, due to memory limitations, only a limited number of features are considered for the codebook construction when dealing with a large dataset.
Online Learning for PLSA-Based Visual Recognition
99
Algorithm 1. On-line Codebook Adaptation 1: INPUTS: (t) (t) (t) 2: F (t) = {f1 , f2 , ..., fNf } - feature vectors in the data stream at time slice (t) t
3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14:
W (t−1) = {w1 , w2 , ..., wV t−1 } - the codebook from time slice (t − 1) Σ (t−1) = {(μ1 , σ1 ), (μ2 , σ2 )..., (μVt−1 , σV t−1 )} - the distance statistics for W (t−1) ¯ = {(¯ Σ μ, σ ¯ 2 )} - the mean distance statistics computed from the initial set β - the learning factor OUTPUTS: W (t) = {w1 , w2 , ..., wVt } - updated codebook for time slice (t) Σ (t) = {(μ1 , σ1 ), (μ2 , σ2 )..., (μVt , σVt )} - the distance statistics for W (t) for each time slice t ≥ 1 do Vt ← Vt−1 (t) for each feature fn do choose a codeword wk , where k = arg mini ( (t)
(t)
fn −wi 2 2 ) σi2
15: d ← fn − wk 22 16: if d ≤ μk + 2.5σk then 17: μk ← μk + β(d − μk ) 18: σk2 ← σk2 + β(d − μk )(d − μk ) (t) 19: wk ← wk + β(fn − wk ) 20: else (t) ¯ , σV2 t ← σ ¯2 21: Vt ← Vt + 1, wVt ← fn , μVt ← μ 22: end if 23: end for 24: end for
The produced codebook is not necessarily optimal as it only considers a subset of features. In order to construct a codebook that captures the full characteristics of all features in the data, we present here an codebook adaptation algorithm for the codebook construction. (t) Given a feature vector fn , and the corresponding wk , denote δ as the dif(t) (t) ference vector between fn and wk , i.e., δ = fn − wk . We simply assume that each dimension of δ follows an i.i.d. zero mean Gaussian distribution, and hence (t) the squared Euclidean distance between fn and wk can be approximately modeled by a Gaussian distribution [29]. The statistics μk and σk are the mean and standard deviation for the squared distances between wk and all the features assigned to wk . The codebook adaptation algorithm is presented in Algorithm 1. As shown in Algorithm 1, the inputs include the current feature set F (t) , the previous codebook W (t−1) , and also the distance statistics Σ (t−1) for W (t−1) . The Σ (t−1) consists of the corresponding means and variances for the squared distances between the features and corresponding codewords. During the adaptation process, (t) each feature chooses one codeword to update. For each feature vector fn , we (t) 2 f −w select the codeword wk that has the minimum of n σ2 k 2 . Let d be the squared k
100
J. Xu et al. (t)
distance between fn and wk . If the difference between d and μk is within 2.5 (t) σk , both wk and (μk , σk ) will be updated using fn . The updating factor β is (t) set to be a flat rate (e.g., β = 0.05). If no match is found between fn and all (t) the codewords, a new codeword is then initiated using fn . The related distance 2 statistics are initiated using μ ¯, and σ ¯ , which are the corresponding mean values computed from the initial training set. 3.2
On-Line EM for PLSA Learning
The learning of PLSA model in Section 2 employs the batch EM for parameter estimation. However, batch EM is not applicable to data that arrives sequentially. To enable the learning of PLSA under this situation, we propose an on-line learning algorithm for PLSA. The idea of the proposed algorithm is to update the model parameter θ using D(t) received at each time slice. For a better understanding of the algorithm, we begin the on-line learning using the notations for batch learning in Section 2. To enable the on-line learning for PLSA, we firstly define two statistics n(d, wj )zk and n(d, w)zk in the follows: n(d, wj )zk =
N
n(di , wj )p(zk |di , wj ), j ∈ {1, 2, ..., V }, k ∈ {1, 2, ..., K}. (7)
i=1
n(d, w)zk =
V N
n(di , wj )p(zk |di , wj ), k ∈ {1, 2, ..., K}.
(8)
j=1 i=1
Based on the above definitions, (5) in M-step can be re-formulated as: N n(di , wj )p(zk |di , wj ) n(d, wj )zk p(wj |zk ) = V i=1 . = N n(d, w)zk m=1 i=1 n(di , wm )p(zk |di , wm )
(9)
The main idea of our proposed on-line EM is to replace the values of n(d, wj )zk and n(d, w)zk with weighted mean values. We define the weighted mean for n(d, w)zk at time t as follows: n(d, w)(t) zk = η(t)
t
((
τ =1
t
s=τ +1
λ(s))
Nτ V
(τ )
(τ )
n(di , wj )p(τ ) (zk |di , wj ))
(10)
i=1 j=1
(t) where η(t) = ( tτ =1 ts=τ +1 λ(s))−1 , and p(t) (zk |di , wj ) represents the posterior probability computed using visual document stream D(t) and W . Here, the parameter λ(s) (0 ≤ λ(s) ≤ 1, s = 1, 2, 3, ...) is a time-dependent decaying factor that determines the contribution of the data stream at each time slice. Equation (10) can be simplified as (please refer to the appendix for details of derivation): Nt V (t) (t) (t−1) = (1 − η(t))n(d, w) + η(t)[ n(di , wj )p(t) (zk |di , wj )]. n(d, w)(t) zk zk i=1 j=1
(11)
Online Learning for PLSA-Based Visual Recognition
101
Algorithm 2. On-line EM Algorithm for PLSA Learning 1: INITIAL STAGE (0) (0) 2: estimate n(d, w)zk , n(d, wj )zk from the initial training set. 3: construct the initial codebook W (0) using F (0) from the initial training set 4: ON-LINE LEARNING STAGE 5: for each image stream at time slice t do 6: Codebook Adaptation: 7: compute W (t) using F (t) and W (t−1) based on Algorithm 1 (t) 8: compute X (t) = {n(di , wj )} with W (t) and F (t) 9: On-line EM Parameter Estimation: (t) 10: initialize p(t) (wj |zk ), p(t) (zk |di ) based on W (t) 11: for each EM iteration n do 12: E-step: (t)
(t)
13: 14:
compute p(t) (zk |di , wj ) using p(t) (wj |zk ) and p(t) (zk |di ) with (4) M-step:
15: 16:
computen(d,w)zk using X (t) , n(d,w)zk , and p(t)(zk |di , wj ) with (11) (t) (t−1) (t) compute n(d, wj )zk using X (t) , n(d, wj )zk , and p(t) (zk |di , wj ) with (12) (t) (t) compute p(t) (wj |zk ) using n(d, w)zk and n(d, wj )zk with (13) (t) (t) compute p(t) (zk |di ) using X (t) and p(t) (zk |di , wj ) with (6) Check convergence:
17: 18: 19: 20: 21: 22: 23: 24: 25:
(t)
(t−1)
(t)
(t)
compute L(n) (X (t) |θ) using p(t) (wj |zk ) and p(t) (zk |di ) with (3) (n) (n−1) |< if new likelihood L(n) satisfies convergence condition | L L−L (n−1) then terminate EM iteration end if end for end for (t)
Similarly, n(d, wj )zk can also be derived as: Nt (t) (t) (t−1) n(d, wj )(t) + η(t)[ n(di , wj )p(t) (zk |di , wj )]. zk = (1 − η(t))n(d, wj )zk i=1
(12) Using (11), and (12), the on-line estimations for p(wj |zk ) at time t can be written as follows: (t)
p(t) (wj |zk ) =
n(d, wj )zk (t)
n(d, w)zk
(13)
. (t)
(t)
We can see from (11) and (12) that the values of n(d, w)zk and n(d, wj )zk (t−1) (t−1) are obtained by combining the previous values n(d, w)zk and n(d, wj )zk (t) with the current data D using different weights respectively. The weights are controlled by the normalization coefficient η(t) (1/t ≤ η(t) ≤ 1) which is used as a learning rate. Sato et al. pointed out in [30] that the on-line EM is a stochastic
102
J. Xu et al.
approximation. To guarantee the estimate converges to a local maximum of the likelihood function of the data, η(t) must satisfy the following conditions: lim η(t) = 0,
t→∞
∞ t=1
η(t) = ∞,
∞
η 2 (t) < ∞.
(14)
t=1
We employ the (1/t)-schedule as it is designed to match these conditions. Under this schedule, η(t) is set to be 1/t, which means each stream is weighted equally, i.e., λ(s) = 1. To combine this on-line algorithm with the codebook adaptation in Section 3.1, η(t) is set to 1 for any newly created codeword wj , since there is no past experience for the new codeword to learn from. An overview of the on-line learning algorithm with codebook adaptation is pre(0) (0) sented in Algorithm 2. The initial parameter estimates n(d, wj )zk , n(d, w)zk , are estimated from an initial training set D(0) using standard EM of PLSA. During the on-line learning stage, our algorithm performs the codebook adaptation using F (t) from data stream D(t) at each time slice t, and then conducts the on-line learning based on the updated codebook W (t) . As a result, the proposed algorithm can automatically and adaptively learn from data streams over the entire learning life.
4
Experiments
We evaluate the performance of the proposed algorithm for both scene recognition and human action recognition. All the experiments are conducted on a PC with an Intel 2.09 GHz Core Duo CPU and 1GB of RAM. We use the Matlab implementation of PLSA code provided by Fergus et al. [31]. We compare the on-line learning and batch learning in the experiments. The Picasa Scene Data. To demonstrate the advantages of the proposed algorithm, we conduct our first experiment based on the web images. We issue queries to the Google Picasa Web Album using the provided API [32], and receive the images sequentially. All the collected images are resized to 320-by-256 pixels. Sample images can be found in the supplementary material. We repeat the experiments for 5 runs, and the average recognition performance is used for comparison. At each run, we firstly collect 238 images as the initial training set, and also 238 images as the testing set. We then collect different number streams of images for the on-line learning. We use 2, 3, and 4 streams. We collected 4743 images from 7 different scene categories in total at each run. For feature extraction, we use the SIFT descriptor [33] as the local feature descriptor. Feature points are densely sampled on regular grids from each image, and 80 SIFT descriptors are extracted per image. We adopt the scene recognition approach using PLSA and k-NN classifier proposed in [4] in the experiments. For the PLSA parameters, the codebook size is set to 1300, and the number of topics is fixed to 25. The parameter k for the k-NN classification is set to 17.
Online Learning for PLSA-Based Visual Recognition
65.5
63.5 63 62.5 62 0
1 Streams
(a) 2 Image Streams
2
65
68
With codebook adaptation Without codebook adaptation
Recognition rate (%)
64
With codebook adaptation Without codebook adaptation
Recognition rate (%)
Recognition rate (%)
64.5
64.5 64 63.5 63 0
1
Streams
2
(b) 3 Image Streams
3
103
With codebook adaptation Without codebook adaptation
67 66 65 64 63 0
1
2 Streams
3
4
(c) 4 Image Streams
Fig. 1. The performance of on-line learning algorithm based on the Google Picasa Web Album
For the batch learning, we intend to learn a batch model using all the images from both the initial set and the on-line training set, which is 4505 in total. Let |D|, |W |, and |Z| be the number of images, the codebook size, and the number of topics. The memory requirement for our Matlab based implementation is 64 ∗ |D||W ||Z| bytes (In this experiment, |D|, |W |, and |Z| are 4505, 1300, and 25 respectively). This memory requirement exceeds the Matlab memory capacity on our computer, and therefore the conventional batch PLSA learning cannot be performed on our system. Compared with the batch learning algorithm, the proposed algorithm has the advantages of lower memory consumption since only a subset of the training data is processed at each time. At each run of the on-line learning, an initial model is learned using the batch learning algorithm on the initial training set. An initial codebook is also obtained by clustering all the features from the initial set. We then update the initial model using image streams. After each update, the topic vectors for the images from both the initial set and the testing set are re-computed using the updated model. The category of each testing image is determined by the k-NN search in the initial set. To see the effect of codebook adaptation, the experiments are conducted using the on-line learning algorithms with and without codebook adaptation respectively. Figure 1 reveals the overall trend of recognition rates throughout the on-line learning. It can be seen from Figure 1 that, codebook adaptation helps to improve the recognition performance under different stream size settings. Table 1 summarizes the experimental results, and the highest accuracy of 67.20% is obtained by the algorithm with codebook adaptation. Table 1. Summary of performance on the Picasa scene dataset. Results are reported as the average accuracies of 5 runs. #streams 2 3 4 Without codebook adaptation 64.37% 64.00% 64.02% With codebook adaptation 65.54% 65.42% 67.20% Batch Learning N/A
104
J. Xu et al.
Mean accuracies 70
68.5
Recognition rate (%)
Recognition rate %
69 68 67 66 65 64
68
Proposed QB−PLSA
67.5 67 66.5 66
63 62 1
3
5
7
9
11
#Neighbours
(a)
13
15
17
19
65.5 0
1
Streams
2
3
(b)
Fig. 2. The recognition performance on the OT scene dataset. (a) The average recognition rates for different k’s of the k-NN algorithm on the testing set, based on the batch-trained PLSA model. (b) The performance of both the proposed algorithm and QB-PLSA.
The OT Scene Dataset. In our second experiment, we compare the proposed algorithm with the conventional batch PLSA, and also the QB-PLSA proposed in [10], using the OT scene dataset [34]. The OT dataset contains 2688 images from 8 different scene categories, and they are 360 coasts, 328 forest, 260 highway, 308 inside of cities, 374 mountain, 410 open country, 292 streets, and 356 tall buildings. The average size of each image is 256-by-256 pixels. For feature extraction, the SIFT descriptor [33] is employed as the local feature descriptor. We again use the scene recognition approach using PLSA and kNN classifier in this experiment, by following the settings detailed in [35]. The number of topics for PLSA is set to 25, and the codebook size is set to 650. We optimize the k for the k-NN classifier in the experiments. The experiments are repeated for 5 runs, and the average recognition performance is used for comparison. At each run, the dataset is randomly divided into 538 initial training images, 1612 on-line training images, and also 538 testing images. No labeling information is used in the random division. The on-line training set is further divided into 3 streams to test the on-line learning. At each run of the batch learning, a PLSA model is learned using all the images from both the initial set and the on-line training set, and tested using the testing set. The average performance is used for comparison. Figure 2a depicts the average recognition rate for different k’s of the k-NN algorithm on the testing set. It can be seen from the figure that, the best recognition rate of 69.52% for the batch-learned PLSA is obtained when k = 19. This figure is consistent with the testing accuracy reported in [35], based on the similar settings. As a result, k = 19 for the k-NN algorithm will serve in the following sections as a baseline for the evaluation of the proposed algorithm. At each run of the on-line learning, we compare the proposed algorithm with the QB-PLSA. QB-PLSA is proposed for text mining, and there is no codebook adaptation involved. To enable QB-PLSA for scene recognition, the initial codebook generated from the initial training set is used as the visual codebook
Online Learning for PLSA-Based Visual Recognition
105
for QB-PLSA. Compared with the QB-PLSA, the proposed algorithm employs the codebook adaptation to capture the feature characteristics in the new data; it is more flexible as there is no assumption made on the distributions of the PLSA parameters. The average recognition performances for both on-line algorithms are demonstrated in Figure 2b. It can be observed from Figure 2b that, the proposed algorithm achieves better recognition performance than QB-PLSA throughout the entire learning. Table 2 summarizes the recognition accuracies of all learning algorithms, and it shows that the performance of the proposed algorithm is comparable with that of the batch-learned PLSA model. Table 2. Summary of performance on the OT scene dataset. Results are reported as the average accuracies of 5 runs. Algorithms Accuracies QB-PLSA 65.95% Proposed 68.00% Batch Learning 69.53%
The KTH Action Dataset In our last experiment, we use the KTH human action dataset [36] for evaluation. The KTH dataset is a single-view video dataset in human actions, which consists of 598 action videos. Six categories of human actions can be found in this dataset, and they are boxing, jogging, running, walking, hand waving and hand clapping. Each of the action is performed several times by 25 subjects in different environments with scale changes. There are only a single action in each video. For feature extraction, we choose the separable linear filter in [24] for spacetime interest point extraction in videos. Each extracted space-time patch is described as a concatenated vector of its brightness gradient. Then all the descriptors are projected to 100 dimensions using PCA. The codebook size is set to 1000 for the KTH dataset. The experiments are conducted based on the leave-one-out testing paradigm (LOO). Since the KTH dataset contains a large number of features, only a subset of features are chosen to construct the batch codebook [5], and we construct With Codebook Adaptation Without Codebook Adaptation
80.5 80 79.5 79 78.5 78 77.5 77 0
1
Streams
(a) 2 Video Streams
2
79.5
79.5
79
79
78.5 With Codebook Adaptation Without Codebook Adaptation
78 77.5
Recognition rate (%)
Recognition rate (%)
81
Recognition rate (%)
81.5
78.5 78 77.5
77
77
76.5 0
76.5 0
1
Streams
2
(b) 3 Video Streams
3
With Codebook Adaptation Without Codebook Adaptation
1
2
Streams
3
4
(c) 4 Video Streams
Fig. 3. The performance of on-line learning algorithm on the KTH action dataset
106
J. Xu et al.
Table 3. Summary of performance on the KTH action dataset. Results are reported as the averages of 25 runs. #streams 2 3 4 Without codebook adaptation 77.70% 77.36% 77.20% With codebook adaptation 81.00% 79.06% 79.33% Batch Learning 80.17%
the codebook by using videos of only three subjects. After the codebook construction, a PLSA model is learned from the videos of 24 subjects using batch learning algorithm, and is tested against the videos of remaining subject. We repeat the LOO batch learning for 25 runs. At each LOO run of the on-line learning, all the videos of one subject are used as the initial training set. The initial training set is used to construct the initial codebook, and also to train the initial model, using batch PLSA learning algorithm. The remaining training videos form the on-line learning set for adapting the model. The on-line training set is randomly split into different number of video streams without using the labeling information, and we use 2, 3, and 4 video streams in the experiments. After learning from each video stream, the adapted model is tested against the testing videos. We evaluate both on-line learning algorithms with and without codebook adaptation. A confusion matrix is computed for each testing, and the recognition accuracies for the trained model are reported as the mean values of diagonal elements from the average confusion table of 25 runs. The performances of the proposed algorithm under different streams settings are depicted in Figure 3. Figure 3 also shows the codebook adaptation helps to improve the recognition performance for the on-line learning. Table 3 summarizes the performance of batch learning and on-line learning. It can be observed from the table that our batch-trained PLSA achieves the average accuracy of 80.17%, which is consistent with that reported in [5]. The table reveals that the performance of the proposed algorithm is comparable with that of the conventional batch algorithm on this dataset. Finally we compare the proposed algorithm with the QB-PLSA, with 7 video streams. This time we use the adapted codebook for QB-PLSA at each
Recognition rate (%)
80 79.5
Proposed QB PLSA
79 78.5 78 77.5 77 76.5 0
1
2
3
4
5
6
7
Streams
Fig. 4. The performance of both the proposed algorithm and QB-PLSA
Online Learning for PLSA-Based Visual Recognition
107
stream.The accuracies of both algorithms on each of the video streams are depicted in Figure 4. The figure shows that the proposed algorithm also outperforms the QB-PLSA on the KTH dataset.
5
Conclusions
In this paper, we have proposed an on-line learning algorithm for PLSA-based visual recognition. The proposed algorithm can automatically and adaptively learn the parameters of a PLSA model during the on-line learning. We evaluate the proposed algorithm using datasets for scene recognition and human action recognition. Experimental results demonstrate that the proposed can handle data that batch PLSA learning cannot deal with, and its performance is comparable with that of the batch PLSA learning on visual recognition. Acknowledgement. This work was done when G. Ye was affiliated with National ICT Australia. National ICT Australia (NICTA) is funded by the Australian Government as represented by the Department of Broadband, Communications and the Digital Economy and the Australian Research Council through the ICT Center of Excellence program. Wei Wang is supported by ARC Discovery Grant DP0987273.
References 1. Hofmann, T.: Unsupervised Learning by Probabilistic Latent Semantic Analysis. Machine Learning 42, 177–196 (2001) 2. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet Allocation. The Journal of Machine Learning Research 3, 993–1022 (2003) 3. Sivic, J., Russell, B., Efros, A., Zisserman, A., Freeman, W.: Discovering objects and their location in images. In: ICCV (2005) 4. Bosch, A., Zisserman, A., Munoz, X.: Scene classification using a hybrid generative/discriminative approach. TPAMI 30, 712–727 (2008) 5. Carlos Niebles, J., Wang, H., Li, F.F.: Unsupervsied Learning of Human Action Categories Using Spatial-Temporal Words. IJCV 42, 993–1022 (2008) 6. Savarese, S., DelPozo, A., Niebles, J., Fei-Fei, L.: Spatial-temporal correlations for unsupervised action classification. In: WMCV (2008) 7. Xu, J., Ye, G., Wang, Y., Herman, G., Zhang, B., Yang, J.: Incremental EM for Probabilistic Latent Semantic Analysis on Human Action Recognition. In: AVSS (2009) 8. Chou, T., Chen, M.: Using incremental PLSI for threshold-resilient online event analysis. IEEE Transactions on Knowledge and Data Engineering 20, 289 (2008) 9. AlSumait, L., Barbar´ a, D., Domeniconi, C.: Online LDA: Adaptive Topic Model for Mining Text Streams with Application on Topic Detection and Tracking. In: ICDM (2008) 10. Chien, J., Wu, M.: Adaptive Bayesian latent semantic analysis. IEEE Transactions on Audio, Speech, and Language Processing 16, 198–207 (2008) 11. Hamilton, J.: A quasi-Bayesian approach to estimating parameters for mixtures of normal distributions. Journal of Business & Economic Statistics 9, 27–39 (1991)
108
J. Xu et al.
12. Vailaya, A., Figueiredo, M., Jain, A., Zhang, H., Technol, A., Alto, P.: Image classification for content-based indexing. TIP 10, 117–130 (2001) 13. Wu, J., Rehg, J.: Where am I: Place instance and category recognition using spatial PACT. In: CVPR (2008) 14. Gupta, P., Arrabolu, S., Brown, M., Savarese, S.: Video Scene Categorization by 3D Hierarchical Histogram Matching. In: ICCV (2009) 15. Vogel, J., Schiele, B.: Natural scene retrieval based on a semantic modeling step. LNCS, pp. 207–215. Springer, Heidelberg (2004) 16. Fei-Fei, L., Perona, P.: A Bayesian hierarchical model for learning natural scene categories. In: CVPR (2005) 17. Quattoni, A., Torralb, A.: Recognizing Indoor Scenes. In: CVPR (2009) 18. Li, L., Socher, R., Fei-Fei, L.: Towards Total Scene Understanding Classification, Annotation and Segmentation in an Automatic Framework. In: CVPR (2009) 19. Yilmaz, A., Shah, M.: Recognizing Human Actions in Videos Acquired by Uncalibrated Moving Cameras. In: ICCV (2005) 20. Little, J., Boyd, J.: Recognizing people by their gait: the shape of motion. Journal of Computer Vision Research 1, 1–32 (1998) 21. Ikizler, N., Forsyth, D.: Searching Video for Complex Activities with Finite State Models. In: ICCV, vol. 3 (2007) 22. Pruteanu-Malinici, I., Carin, L.: Infinite hidden markov models for unusual-event detection in video. TIP 17, 811–822 (2008) 23. Wang, Y., Mori, G.: Max-margin hidden conditional random fields for human action recognition. In: CVPR (2009) 24. Dollar, P., Vincent, R., Garrison, C.: Behavior recognition via sparse spatiotemporal features. In: VS-PETS (2005) 25. Ke, Y., Sukthankar, R., Hebert, M.: Efficient visual event detection using volumetric features. In: ICCV, vol. 1 (2005) 26. Yeffet, L., Wolf, L.: Local Trinary Patterns for Human Action Recognition. In: ICCV (2009) 27. Gilbert, A., Illingworth, J., Bowden, R.: Fast Realistic Multi-Action Recognition using Mined Dense Spatio-temporal. In: ICCV (2009) 28. Dempster, A., Laird, N., Rubin, D., et al.: Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society 39, 1–38 (1977) 29. Papoulis, A., Pillai, S.U.: Probability, Random Variables, and Stochastic Processes. McGraw-Hill, New York (2002) 30. Sato, M.: Convergence of on-line EM algorithm. ICONIP 1, 476–481 (2000) 31. Fergus, R., Fei-Fei, L., Torralba, A.: ICCV 2005 short course on Object Recognition (2005), http://people.csail.mit.edu/fergus/iccv2005/bagwords.html 32. Google: Developer’s Guide to Picasa Web Albums Data API (2009), http://code.google.com/apis/picasaweb/overview.html 33. Lowe, D.: Distinctive image features from scale-invariant keypoints. IJCV 60, 91– 110 (2004) 34. Oliva, A., Torralba, A.: Modeling the shape of the scene: A holistic representation of the spatial envelope. IJCV 42, 145–175 (2001) 35. Horster, E., Lienhart, R., Slaney, M.: Continuous visual vocabulary modelsfor plsabased scene recognition. In: CIVR (2008) 36. Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions: A local svm approach. In: CVPR (2004)
Emphasizing 3D Structure Visually Using Coded Projection from Multiple Projectors Ryo Nakamura, Fumihiko Sakaue, and Jun Sato Department of Computer Science and Engineering Nagoya Institute of Technology Nagoya, 466-8555, Japan
Abstract. In this paper, we propose a method for emphasizing 3D structure of the scene visually by blending patterned lights projected from multiple projectors. The proposed method enables us to emphasize specific 3D structure of the scene without capturing image of the scene and without estimating 3D structure of the scene. As a result, the 3D structure can be emphasized visually without any delay of computation. In this paper, we propose a method for generating the patterned light of projectors which enables us to emphasize arbitrary 3D structure of the scene efficiently.
1
Introduction
In computer vision, much attention has been paid for obtaining 3D structure information from camera images [1–3]. This is because the 3D structure information is useful for controlling various robots and machines [4] and for assisting human activities by augmenting 3D scenes [5–8]. In general, cameras, computers and displays are required for augmenting 3D scenes. This is because the cameras and computers are required for recovering 3D structure information, and computers and displays are required for presenting virtual information to human users. We can also use the combination of projectors and cameras for recovering 3D structures and presenting virtual information [9, 10]. However, all these existing systems require computational costs for recovering 3D information and for generating virtual information for human users. These computational costs are often very large and hamper the realization of these systems. In this paper, we consider the system configuration of augmented reality in very different way from those in the existing methods. We do not use image sensors to obtain information of the 3D scene, we do not use computers to recover 3D information, and we do not use displays to present 3D information to human users. We show that even if these components are not used, the augmentation of the 3D scene can be achieved, and useful 3D information can be presented to human users. In this paper, we show that by emitting patterned light from multiple projectors, we can emphasize specific 3D areas or specific shapes in the 3D space by coloring these areas and structures with specific colors. The shape of emphasized R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 109–122, 2011. c Springer-Verlag Berlin Heidelberg 2011
110
R. Nakamura, F. Sakaue, and J. Sato
(a)
(b)
(c)
Fig. 1. Emphasizing 3D structure visually by using multiple projectors. The specific areas or specific structures in the 3D space are emphasized by coloring the 3D scene directly using multiple projectors. In (a) convex parts of the surface are emphasized by coloring with red, while in (b) an object in a specific area is emphasized by coloring with red. In (c) the irregularity of products in a factory is emphasized.
areas can be chosen freely. For example, we can design patterned lights so that the convex part of the surface is emphasized visually as shown in Fig. 1 (a), which may be useful for vehicle drivers to see the irregularity of road surfaces. It is also possible to emphasize objects in a specific area in the 3D space visually as shown in Fig. 1 (b), which can be applied for extracting the irregularity of products in factories as shown in Fig. 1 (c). In our method, we only use multiple projectors. Instead of using computers, we use blending of lights emitted from multiple projectors for augmenting the 3D structures. Instead of using displays, we use object surfaces directly to show the augmented results to human users. As a result, we do not require cameras, computers and displays to realize augmentation of the 3D scene, and we only need multiple projectors. Since we do not use computers, the augmentation of the 3D scene can be achieved without any delay of computation. That is, all are achieved in the light speed.
2
Emphasizing 3D Structure Using Coded Projection
In this section, we explain the overview of 3D structure emphasis, which is based on coded lights emitted from multiple projectors. Let us consider a point xi on the image of projector i, where a light is emitted with the radiance of Ei . Suppose the ray of the light intersect with a plane π at a point X in the 3D space. Then, the irradiance Ii at point X is described as follows: Ii =
−Ei (L N) ((Si − X) Li )2
(1)
where, Si denotes the center of projection of projector i, and Li denotes the orientation of the projector, that is the direction of the principal axis of the projector. N represents the surface normal of π. Unlike the standard point light source, projector light is calibrated so that the front-parallel surface to the
Emphasizing 3D Structure Visually Using Coded Projection
111
Fig. 2. The billboard space and its projection to projector images
projector has the same irradiance everywhere on the surface. Thus, the irradiance of the surface changes according to the distance of the surface along the principal axis of the projector as shown in (1). Now, if we have K projectors and if these projectors illuminate the same point X in the 3D space, the irradiance I at point X can be described as follows: I=
K
Ii
i=1
=
K
Wi Ei
(2)
i=1
where, Wi represents the contribution of the point xi of ith projector to the irradiance of point X as follows: −(L N) Wi = (3) ((Si − X) Li )2 As shown in (3), the weight Wi of each projector light to the irradiance of a 3D point is different from each other. What we want to achieve in this research is to emphasize a certain area in the 3D space by coloring objects in the area emitting light according to the weight of each projector light. However, since a single ray of light may illuminates any point on the line of ray, it is not an easy task to generate proper pattern of multiple projector lights. In the following sections, we propose a method for generating projection patterns linearly to achieve arbitrary 3D space emphasis.
3
Billboard Model for Space Representation
We first consider the quantization of the 3D space. The voxel space is often used for representing the 3D space. However, since we consider the projection of light,
112
R. Nakamura, F. Sakaue, and J. Sato
we represent the 3D space by using a set of small billboards as shown in Fig. 2. We call it a billboard space. We derive the image pattern of projector lights, so that we can emphasize specific areas in the 3D space by projecting light from multiple projectors to the billboard space. Thus, the objective emphasis pattern is given in the billboard space. Suppose we have K projectors, each of which projects light from image point xi (i = 1, · · · , K) to an identical 3D point X as follows: xi = Pi X
(i = 1, · · · , K)
(4)
where, Pi denotes the projection matrix of ith projector. Note, in this paper, points are represented by using the homogeneous coordinates in equations. We assume these projectors are fully calibrated, and thus the projection matrices Pi (i = 1, · · · , K) are known. Then, by projecting a billboard by using (4), the projection of the billboard can be obtained in the projector image. Since each billboard has a certain width and a height, the projected billboard may extend over several pixels in the projector image, and the boundary of the billboard may not coincide with the boundary of image pixels in general as shown in Fig. 2. Thus, we define the contribution Wij of jth image pixel to ith billboard as the occupied area of the image pixel in the projected billboard as follows: Wij =
S(bi ∧ pj ) S(bi )
(5)
where, bi denotes the projection of ith billboard in the image, and pj denotes the jth image pixel. S(·) denotes the area. Then, we define a vector Wi which represents the contribution of each image pixel to the ith billboard as follows: (6) Wi = Wi1 . . . Wij . . . WiN where, N denotes the number of image pixels in a projector image. Now, we consider a vector Y, which represents the projection pattern of N pixels in the projector image: Y = Y1 . . . Yi . . . YN (7) where, Yi represents the image intensity of ith pixel in the projector image. Then, the intensity Ii of the ith billboard can be described as follows: Ii = Wi Y
(8)
By using (8) we next consider a method for deriving the projection patterns Yi (i = 1, · · · , K) of K projectors, which generate objective 3D emphasis in the 3D billboard space.
4
Generating Projection Patterns for 3D Emphasis
As we have seen in section 3, the projection pattern Y generates the intensity Ii of the ith billboard as shown in (8). If the projectors are calibrated, each
Emphasizing 3D Structure Visually Using Coded Projection
113
billboard can be projected to a projector image, and Wi (i = 1, · · · , M ) can be computed from (5). Let us consider a vector I which consists of the objective billboard intensities as follows: I = [I1 , · · · , Ii , · · · , IM ]
(9)
Then, the relationship between the projection pattern Y and the objective billboard pattern I can be described as follows: ⎡ ⎤ W1 ⎢ W2 ⎥ ⎢ ⎥ (10) ⎢ .. ⎥ Y = I ⎣ . ⎦ WM
We next consider multiple projectors. Suppose we have K projectors, and these projectors illuminate the same billboard space. Then, since the compound of multiple lights is described by the addition of radiances, the relationship between the projection patterns Yi (i = 1, · · · , K) of K projectors and the objective billboard pattern I can be described as follows: ⎡ 1 ⎤⎡ ⎤ Y1 W1 W12 . . . W1K ⎢ W21 W22 . . . W2K ⎥ ⎢ Y2 ⎥ ⎢ ⎥⎢ ⎥ (11) ⎢ .. .. .. .. ⎥ ⎢ .. ⎥ = I ⎣ ⎣ . ⎦ . ⎦ . . . 1 2 K YK WM WM . . . WM where, Wij denotes the weight vector Wi of jth projector. If we rewrite (11) as LY = I, then a least square solution of (11) can be computed as follows: Y = L+ I
(12)
where, L+ denotes the pseudo inverse of L, and is described as L+ = (L L)−1 L . Then, by projecting Yi (i = 1, · · · , K) from K projectors, the billboard space can be emphasized as it is designed by I. If we want to illuminate the billboard space with color, we design the objective billboard pattern I in each color channel, and do the above procedure in each color channel. Then, we can color the billboard space with arbitrary color patterns.
5
Representation of Negative Intensity
If we compute the projection patterns Yi (i = 1, · · · , K) by using the above method, they include negative intensities in general. However, negative intensities cannot be projected from real projectors. Thus, in this research, we represent negative intensities virtually by considering a virtual zero intensity at the middle of the intensity range.
114
R. Nakamura, F. Sakaue, and J. Sato
projector intensity
1 255
virtual zero level
0 positive intensity negative intensity
Fig. 3. Representation of virtual negative intensity
Suppose the 3D space is illuminated by a projector light with certain constant intensity I0 . Then, if we project a light with higher intensity than I0 , we can represent the positive intensity as shown in Fig. 3. On the other hand, if we project a light with lower intensity than I0 , we can represent the negative intensity virtually. Thus, by adding a certain amount of intensity to the objective billboard pattern I and computing the projection pattern Yi from (12), we can generate projector patterns properly, which represent negative intensities as well as positive intensities. Then, we can illuminate the 3D space as we wish. Up to now, we proposed a method for emphasizing the 3D scene by projecting coded lights from multiple projectors. We next show the efficiency of the proposed method in real projector experiments.
6 6.1
Experiments Real Projector Experiments
We first show the results from an experiment with real projectors. In this experiment, we used three projectors as shown in Fig. 4 (a). The resolution of these projectors is 640 × 480. We first calibrated these three projectors by using a calibration board and a camera, and computed the projection matrices of these projectors. Although we need a camera for the initial calibration of multiple projectors, we do not need it after the calibration.
projector 3 projector 2
projector 1
(a) Setup of three projectors
(b) 3D object
Fig. 4. The setup of three projectors and a 3D object used in our experiment
Emphasizing 3D Structure Visually Using Coded Projection
(a) objective 3D pattern
115
(b) synthesized 3D pattern
Fig. 5. The objective color pattern of the billboard space, i.e. objective emphasis pattern, and the color pattern of the billboard space, which was synthesized from the projection patterns shown in Fig. 6
(a) projector 1
(b) projector 2
(c) projector 3
Fig. 6. The projection patterns computed from the proposed method. (a), (b) and (c) show projection patterns for projector 1, 2 and 3 respectively, which were computed from the objective coloring pattern shown in Fig. 5 (a).
㽳 㽲 㽴 㽵㽶㽷 㽸㽺 㽹 position of the 3D object
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
Fig. 7. The results of 3D space emphasis. The 3D object was moved in the 3D space, and its colorization was observed at 9 different positions. The image number and the position number in the left image are corresponding to each other.
116
R. Nakamura, F. Sakaue, and J. Sato
(a) objective 3D pattern
(b) synthesized 3D pattern
Fig. 8. The objective color pattern of the billboard space and the color pattern of the billboard space, which was synthesized from the projection patterns shown in Fig. 9
Then, projection patterns of these projectors were computed from the projection matrices and an objective coloring pattern of the billboard space. In this experiment, the objective coloring pattern of the billboard space was defined as shown in Fig. 5 (a), i.e. the cylindrical space is colored with red in the 3D space. The computed projection patterns of three projectors are shown in Fig. 6. As shown in this figure, negative as well as positive intensities are required for generating objective 3D emphasis patterns, and it is realized by using the virtual negative intensity representation. For reference, we generated the coloring pattern of the 3D billboard space by using synthetic projector images. The coloring pattern of the billboard space synthesized from Fig. 6 is shown in Fig. 5 (b). As shown in this figure, the cylindrical space is colored with red properly. However, some errors exist in coloring. These errors may decrease if we use more projectors. We next projected the extracted projection patters by using three real projectors, and put a 3D object shown in Fig. 4 (b) into the space and moved it in the space. The results of coloring of this object are shown in Fig. 7. In this figure, we show the results of coloring at 9 different positions in the 3D space. These 9 positions are shown in the left image of Fig. 7. As shown in Fig. 7, the 3D object is colored with red when it is at the center, and is not colored at the other positions. Thus, we find that the 3D object can be colored and emphasized as we designed in the billboard space by using the proposed method. We next designed the objective coloring pattern, so that 3D objects are not colored at the center of the 3D space and are colored with red at the other positions as shown in Fig. 8. The projection patterns of three projectors for this objective coloring pattern were computed from our method as shown in Fig. 9, and the 3D object was colored by the projection patterns as shown in Fig. 10. From Fig. 10, we find that the proposed method can color the 3D space, so that only a specific area is not colored. From these results, we find that the proposed method can emphasize arbitrary shapes and structures in the 3D space.
Emphasizing 3D Structure Visually Using Coded Projection
(a) projector 1
(b) projector 2
117
(c) projector 3
Fig. 9. The projection patterns computed from the proposed method. (a), (b) and (c) show projection patterns for projector 1, 2 and 3 respectively, which were computed from the objective coloring pattern shown in Fig. 8 (a).
㽳 㽲 㽴 㽵㽶㽷 㽸㽺 㽹 position of the 3D object
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
Fig. 10. The results of 3D space emphasis. The 3D object was moved in the 3D space, and its colorization was observed at 9 different positions. The image number and the position number in the left image are corresponding to each other.
6.2
Accuracy Evaluation
We next evaluate the accuracy of coloring of the proposed method. The accuracy of coloring was evaluated by the RMS error E between the objective color pattern and the color pattern generated from the proposed method as follows: M 1 2 (Ii − Ii ) (13) E= M i=1 where, M denotes the number of billboards, and Ii and Ii denote the objective intensity and the intensity generated from the proposed method at ith billboard in the 3D space respectively. Representation of Negative Intensity. We first evaluate the efficiency of the representation of virtual negative intensity described in section 5.
118
R. Nakamura, F. Sakaue, and J. Sato
synthesized 3D pattern objective 3D pattern (a) (0.5 − 1) case
objective 3D pattern synthesized 3D pattern (b) (0 − 1) case Fig. 11. The objective 3D pattern and the synthesized 3D pattern from our method. (a) shows the results when the objective intensity range varies from 0.5 to 1. (b) shows the results when the objective intensity range varies from 0 to 1.
In this experiment, we assume that the intensity of each projector varies from 0 to 1, and thus the intensity of compound light varies from 0 to 3. We evaluated the RMS error between the objective color patterns and generated color patterns from our method in the billboard space. In this experiment, we chose two different intensity ranges of objective color patterns. One is from 0 to 1, and the other is from 0.5 to 1. Fig. 11 (a) shows the objective pattern and the generated pattern in the case of (0.5 − 1), and Fig. 11 (b) shows those in the case of (0− 1) respectively. Fig. 12 (a) and (b) show projection patterns of three projectors in these two cases. The horizontal axis shows the horizontal position in the projection pattern and the vertical axis shows the intensity of projection pattern. As shown in Fig. 12 (b), the projection pattern of (0 − 1) case includes negative intensities. These negative intensities are replaced by 0 before projecting from projectors, and thus generated pattern in the billboard space includes large errors. On the other hand, the projection pattern of (0.5 − 1) case is within the range from 0 to 1, and can be projected from projectors as it is. As a result, the RMS error of (0.5 − 1) case is 25.6, while that of (0 − 1) case is 63.6. Thus, we find that the virtual representation of negative intensity is very efficient.
Emphasizing 3D Structure Visually Using Coded Projection
projector 1
projector 2 (a) (0.5 − 1) case
projector 3
projector 1
projector 2 (b) (0 − 1) case
projector 3
119
Fig. 12. The projection pattern of three projectors. The horizontal axis in each graph shows the horizontal position in the projection pattern, and the vertical axis shows the intensity of projection pattern. (a) shows the results when the objective intensity varies from 0.5 to 1, while (b) shows the results when the objective intensity varies from 0 to 1.
6.3
Constellation of Multiple Projectors
We next evaluate the relationship between the constellation of multiple projectors and the RMS error of color patterns generated from our method in the billboard space. The three projectors are positioned at a fixed distance from the center of the billboard space, and the relative direction θ between each pair of projectors viewed from the center of the billboard space is changed from 5◦ to 45◦ . Thus, the distance between each pair of projectors is large when the relative direction θ is large. The color patterns generated from our method in the case of θ = 5◦ , 20◦ , 30◦ and 45◦ are shown in Fig. 13 (a), (b), (c) and (d) respectively, and the RMS errors of these generated patterns are shown in Fig. 14. The horizontal axis of this graph shows the relative direction between each pair of projectors and the vertical axis shows the RMS error of color patterns generated from our method. As shown in this graph, the coloring in the 3D space is more accurate when the relative direction of each pair of projectors is large. This property is analog to the accuracy of stereo cameras, where the 3D measurement is more accurate when the relative distance between two cameras is larger. 6.4
Number of Projectors
We next evaluate the relationship between the number of projectors and the accuracy of 3D coloring.
120
R. Nakamura, F. Sakaue, and J. Sato
(a) 5◦
(b) 20◦
(c) 30◦
(d) 45◦
RMS error in intensity
Fig. 13. The 3D color patterns generated from multiple projectors, when the relative direction between each pair of projectors viewed from the center of the billboard space is 5◦ , 20◦ , 30◦ and 45◦ 㪋㪇 㪊㪏 㪊㪍 㪊㪋 㪊㪉 㪊㪇 㪉㪏 㪉㪍 㪉㪋 㪉㪉 㪉㪇
㪌㫦 㪈㪇㫦 㪈㪌㫦 㪉㪇㫦 㪉㪌㫦 㪊㪇㫦 㪊㪌㫦 㪋㪇㫦 㪋㪌㫦
relative direction (degrees)
Fig. 14. The relationship between the relative direction between each pair of projectors and the RMS error of 3D color patterns generated from our method
In this experiment we changed the number of projectors used for coloring the 3D space from 2 to 7, and evaluated the RMS error of generated color patterns in the 3D space. For avoiding the effect of the variation in projector positions, we fixed the position of the left most and the right most projectors, and inserted projectors between these two with a constant interval. The 3D color patterns generated from our method in the case of 2, 3, 4, 5, 6 and 7 projectors are shown in Fig. 15, and the RMS errors of these generated patterns are shown in Fig. 16. The horizontal axis of this graph shows the number of projectors, and the vertical axis shows the RMS error of 3D color patterns generated from our method. As shown in this graph, the coloring in the 3D space is more accurate when we use more projectors in general. However, the accuracy degrades when we use 6 or more than 6 projectors. This is because the simple linear computation without any restriction provides us large intensity variations in projector images when we use many projectors, and the variation of intensity
Emphasizing 3D Structure Visually Using Coded Projection
(a) 2 projectors
(d) 5 projectors
(b) 3 projectors
(c) 4 projectors
(e) 6 projectors
(f) 7 projectors
121
RMS error in intensity
Fig. 15. The 3D color patterns generated from the proposed method, when we use 2, 3, 4, 5, 6 and 7 projectors 㪊㪉 㪊㪇 㪉㪏 㪉㪍 㪉㪋 㪉㪉 㪉㪇
㪉
㪊
㪋
㪌
㪍
㪎
number of projectors
Fig. 16. The relationship between the number of projectors and the RMS error of 3D color patterns generated from our method
exceeds the range of projector intensity. This problem may be avoided by using conditional least square methods adding conditions in the range of each projector intensity.
7
Conclusion
In this paper, we proposed a method for emphasizing 3D structure of the scene visually by using patterned lights projected from multiple projectors. For generating projector images of arbitrary 3D space emphasis, we proposed a method for designing the objective 3D color patterns in the billboard space, and computing the projector images by counting the contribution of each projector to the 3D space coloring.
122
R. Nakamura, F. Sakaue, and J. Sato
The proposed method enables us to emphasize specific 3D structure of the scene without capturing image of the scene and without estimating 3D structure of the scene. As a result, the 3D structure can be emphasized visually without any delay of computation.
References 1. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003) 2. Faugeras, O., Luong, Q.-T., Papadopoulo, T.: The Geometry of Multiple Images. MIT Press, Cambridge (2001) 3. Agarwal, S., Snavery, N., Simon, I., Seitz, S.M., Szeliski, R.: Building Rome in a Day. In: Proc. International Conference on Computer Vision (2009) 4. Malis, E.: Visual servoing invariant to change in camera intrinsic parameters. In: Proc. International Conference on Computer Vision, vol. 1, pp. 704–709 (2001) 5. Kanade, T., Rander, P., Narayanan, P.J.: Virtualized Reality: Constructing Virtual Worlds from Real Scenes. IEEE Multimedia 4(1), 34–47 (1997) 6. Oka, K., Sato, I., Nakanishi, Y., Sato, Y., Koike, H.: Interation for entertainment contents based on direct manipulation with bare hands. In: Proc. IWEC, pp. 397– 404 (2002) 7. Milgram, P., Kishino, F.: A taxonomy of mixed reality visual display. IEICE Transactions on Information and System E77-D(12), 1321–1329 (1994) 8. Sukuthankar, R., Stockton, R., Mullin, M.: Smarter presentations: Exploiting homography in camera-projector systems. In: Proc. ICCV 2001, vol. 1, pp. 247–253 (2001) 9. Hall-Holt, O., Rusinkiewicz, S.: Stripe Boundary Codes for Real-Time StructuresLight Range Scanning of Moving Objects. In: Proc. International Conference on Computer Vision, Proc. ICCV, pp. 359–366 (2001) 10. Zhang, L., Curless, B., Seitz, S.: Rapid Shape Acquisition using Color Structured Light and Multi-Pass Dynamic Programming. IEEE 3D Data Processing Visualization and Transmission (2002)
Object Class Segmentation Using Reliable Regions Vida Vakili and Olga Veksler Computer Science Department University of Western Ontario London, Canada
Abstract. Image segmentation is increasingly used for object recognition. The advantages of segments are numerous: a natural spatial support to compute features, reduction in the number of hypothesis to test, region shape itself can be a useful feature, etc. Since segmentation is brittle, a popular remedy is to integrate results over multiple segmentations of the scene. In previous work, usually all the regions in multiple segmentations are used. However, a typical segmentation algorithm often produces generic regions lacking discriminating features. In this work we explore the idea of finding and using only the regions that are reliable for detection. The main step is to cluster feature vectors extracted from regions and deem as unreliable any clusters that belong to different classes but have a significant overlap. We use a simple nearest neighbor classifier for object class segmentation and show that discarding unreliable regions results in a significant improvement.
1
Introduction
Over the past couple of decades, most object recognition approaches were based on the sliding window scanning [1–5], localizing the bounding box of an object. While very successful for some applications, such as face and pedestrian detection [2], there are several well known issues with the sliding window approach. First, it is mostly appropriate for object classes that are well-approximated by a rectangle, such as cars, but not for object classes with thin parts, such as giraffes. Another problem is that there is no precise pixel-level segmentation of the detected object. Also, sliding window approach is expensive computationally, since a large number of sub-windows at different scales needs to be processed in a single image. In recent years, there has been a lot of interest in using results of a generic image segmentation algorithm to help obtain pixel-precise object segmentation [6–15]. Segmentation is used to subdivide an image into a set of regions. These regions do not have predefined shapes like rectangular patches and their boundaries are more likely to align with the boundaries of objects in the scene. Therefore the shape and boundary properties of regions can be used for feature extraction. Furthermore, it has been shown in [16] that regions from unsupervised bottomup segmentation can provide a better spatial support for object detection than R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 123–136, 2011. c Springer-Verlag Berlin Heidelberg 2011
124
V. Vakili and O. Veksler
rectangular windows. Another advantage of using image regions is scalability and potential savings in computational efficiency. Image regions provide a usually much smaller set of hypothesis to examine compared to the sliding window approach, and at “natural” scales that are obtained through segmentation. Ideally, a good segmentation algorithm would cleanly separate the objects or at least the object parts. However, the state-of-the art in segmentation is still very far from this goal. With hierarchical segmentation there is a higher chance that a complete object or an object part is contained at some hierarchy level. Therefore some methods use a single hierarchical segmentation algorithm [14, 17] for object recognition. However, even hierarchical algorithms suffer from instability, the results depends on the choice of the feature space and parameter settings. Another strategy is to use smaller image regions, or ”superpixels”, as in [15], and regularize the classifier by aggregating histograms in the neighborhood of each superpixel. However, if superpixels are too small, they may not have enough spatial support for feature extraction. If superpixels are bigger, then there is a larger chance for an error due to inaccuracies in superpixel boundaries, since superpixels are used as indivisible units. A more commonly used strategy is to integrate the results over multiple different segmentations of the same scene [6, 13, 18, 19]. Usually several unsupervised bottom-up segmentation algorithms, each with different parameters are used. With more segments, there is a higher chance that some segments contain whole objects or easily recognizable object parts. In support of using multiple segmentations, in [16] they show that usually regions generated by a single segmentation of a scene do not contain entire objects. Previous work that relies on multiple segmentations either makes an assumption that at least one segmentation separates the object from the background [6, 19], or uses all segments as reliable for object class recognition [13]. Clearly many regions from multiple segmentations are rather generic blobs that do not contain features useful for discrimination. This is because most bottomup segmentation algorithms are implicitly or explicitly designed to partition an image into a set of uniform patches in some feature space. The main idea of our approach is to find reliable regions and discard the ones that are not reliable for object recognition. This should result in more stable decisions at the time of object recognition. In the training stage, first multiple image segmentations are obtained by varying the parameters in FH [20] and Mean Shift [21] segmentation algorithms. Each segment is described by a large set of features based on color, texture, etc. Then discriminative features are selected using the Relief-F algorithm [22]. For robustness, we do not want to decide on reliability individually for each segment. It is more reliable to make this decision for a whole cluster containing similar segments. A single object class is usually segmented into different possibly overlapping parts, each part may bear little similarity to the other parts. Therefore we cluster each object class (separately from the other classes) using the k-means algorithm, with the aim that each cluster contains similar segments of the object class. Clusters from different objects could have significant intersection, which
Object Class Segmentation Using Reliable Regions
125
makes them unreliable for recognition. The overlapping clusters contain segments that are generic blobs not corresponding to an easily recognizable object part. These clusters are discarded. In the test stage, first multiple image segmentations are generated from the test image and features are extracted for each segment. Reliable segments are found based on their distance to the reliable clusters obtained at the training stage. Since our goal is to learn to determine unreliable segments, and not necessarily to come up with the state of the art object class segmentation system, we use a simple nearest neighbor classifier as opposed to more sophisticated techniques. Initially, each reliable segment is classified independently, and the results may not be spatially coherent. To encourage coherency and integrate the results of multiple segmentations, we use graph-cut optimization [23] to generate the final result. This leads to a significant improvement in accuracy, compared to single segmentation results. Experimental results on MSRC database show that discarding unreliable regions results in a significant accuracy improvement. Even though the goal was not to come up with the state-of-the art classifier, surprisingly, a simple nearest neighbor classifier with reliable regions is not too far from the state-of-the art methods.
2
Related Work
Our work is most related to the methods that use multiple segmentation for various object recognition tasks. An early work that uses segmentation for recognition is [24]. Their goal is associating labels to image regions, and use multiple segmentation algorithms to evaluate which one is better for their task. The method introduced in [6] seeks to discover visual object categories and their segmentation in a collection of unlabeled images starting with regions generated with several unsupervised image segmentation approaches. However, they make an assumption that is often violated in practice. They assume that at least one of the regions contains the full object. Multiple segmentations were also used by [18] to label each pixel with its rough geometric surface class. Multiple segmentations contribute to a wider set of hypothesis to explore as compared to a single segmentation. Their work popularized the use of multiple segmentations for various computer vision tasks. In [19], they use multiple stable segmentations to localized objects in a weakly supervised framework. A segmentation is considered stable if small perturbations of the image do not cause significant change in segmentation results. The idea is that stable segments might be more adequate for object recognition. However, stable regions could still correspond to generic blobs, not necessarily be useful for object recognition. In [13], they classify the regions generated from multiple segmentations in supervised manner and then integrate all segments to get the final result. However in their approach they have an assumption that all the regions generated by multiple segmentations are beneficial for object recognition, which is likely not be true in practice.
126
V. Vakili and O. Veksler
The importance of incorporating multiple segmentations into object recognition, as opposed to a single segmentation, is investigated in [16]. They show two important results: good spatial support is important for recognition and multiple segmentations provide better spatial support for objects than bounding boxes and a single segmentation.
3
Proposed Approach
In this section, we describe our approach to the object class segmentation with reliable regions. In the training stage, reliable regions for each object class are determined and retained, and unreliable regions are discarded. In the test stage, unreliable regions are discarded as well. Reliable regions are classified with the nearest neighbor classifier using the pool of reliable regions obtained at the time of training. Finally region classifications are combined and regularized using graph cuts to get the final object map. 3.1
Obtaining Multiple Segmentations
To obtain multiple segmentations, we use two popular and efficient segmentation algorithms, FH [20] and the Mean-Shift [21]. Multiple segmentations are generated by varying the parameters of each algorithm. We obtain 18 distinct segmentations for the FH algorithm, and 16 for the mean-shift algorithm. Figure 1 shows some examples of segmentation images that are generated for one of the input images. This figure illustrates the possible variation in segmentation regions and different characteristics of the segmentation algorithms. 3.2
Features from Regions
We explored features that are based on size, location, color, texture, probability of the boundary, and edge histograms. Location, shape, color and texture features are the same as in [18]. Probability of the boundary is based on the boundary detector introduced in [25], which is better at detecting object boundaries than the conventional edge detection methods because it is less distracted by texture gradients. We take the mean value of probability of the boundary within a two pixel wide radius of the region boundary. Distribution of intensity gradients can be useful for shape and appearance description. Therefore two kinds of edge descriptors are used to capture edge cues: Histogram of Oriented Gradient (HOG) [3] and Edge Orientation Histogram (EOH) [26]. In our work HOG and EOH are computed for 8 bins for all the pixels in each region and also for pixels around the boundary of regions within two pixels radius. They are normalized by the size of the region or the size of the boundary of the region, correspondingly. The initial set of features has size 219. In order for clustering to have a reliable performance, it is necessary to reduce dimensionality [27]. We use the Relief-F algorithm [22] for feature selection in this work. We retain the top 15 features ranked
Object Class Segmentation Using Reliable Regions
127
Fig. 1. Some examples of segmentation results from FH and Mean Shift algorithms with different input parameters
by Relief-F. These features are based on color, location, edge histogram and texture filters. It is interesting to note that boundary features were not selected. 3.3
Clustering
For robustness, we want to make a decision on the reliability of a whole group of similar segments, rather than decide for each segment separately. Therefore we need to perform segment clustering within each object class. A single object class is usually segmented into several possibly overlapping parts, sometimes bearing little similarity to each other. As illustrated in Figure 2 a cow generally gets segmented into the head, body, and legs. Therefore segments resulting from multiple segmentations of multiple instances of a single object class typically consist of several distinct clusters. To facilitate discovery of reliable segments, first the clustering structure resulting from multiple segments of a single object class must be captured. We use k-means [28] to cluster each object class. The number of clusters for each object is chosen between 4 to 8 so that the resulting number of clusters maximizes the silhouette coefficient. Figure 3 shows some representative samples for four out of six clusters from the cow class and three out of five clusters from the car class. Each cluster corresponds to a separate column. For cow class, the clusters tend to correspond to bodies, heads, vertical segments, horizontal segments, respectively. For the car class, the clusters seem to correspond to the horizontal structures, body, and the “lights and wheels” parts.
128
V. Vakili and O. Veksler
Fig. 2. First column shows the original images and the second column their typical segmentations
Fig. 3. Left: Four clusters from the cow class, right: 3 clusters from the car class. Each column corresponds to a separate cluster. Six representative members are shown for each cluster.
Object Class Segmentation Using Reliable Regions
3.4
129
Discarding Unreliable Segments
So far we used all regions generated from multiple segmentations in the clustering, however clusters from different objects could have lots of intersection with each other which makes them unreliable for object detection. To get rid of unreliable clusters, those clusters that have a large amount of intersection with clusters of different object classes should be discarded. In our work, first the average intersection of each of the clusters with all other clusters of different classes is calculated and then for each object class two or more clusters with the least amount of intersection are retained. These clusters are reliable and will lead to more certain and stable decision at the time of object recognition. There are many different ways to measure cluster overlap. Since the number of training samples is relatively small and the dimensionality of our data is relatively high, we avoid measuring cluster overlap in the original feature space to avoid the problems associated with high dimensionality. Instead we project the two clusters for which we want to measure the overlap into the best one dimensional subspace so that the samples from each cluster stay as separated as possible. This step is performed with Fisher linear discriminant [29]. According to Fisher linear discriminant, two clusters should be projected on the line in the direction V which maximizes: J(V ) =
(μ1 − μ2 )2 s21 + s22
(1)
μ is the mean and s2 is the scatter of projection. The best V for projection is found with eigenvector analysis. After two clusters are projected on the best line that separates them, the value of equation (2) can be used to quantify the overlap in the projected space: J(cli , ckj ) =
(μ1 − μ2 )2 σ12 + σ22 ,
(2)
where cli denotes cluster i from object class l; σ12 and σ22 are the variances of projections of two clusters. The larger is the value in equation (2), the better is the separation between the two clusters. To measure how reliable one cluster is, the average of equation (2) is calculated over all clusters of other classes, see equation (3): l k j,k=l J(ci , cj ) l , (3) J(ci ) = ι where ι is the number of clusters such that their object label is not equal to l. For each object class we choose two or more clusters with the best value of J(cli ). In Figure 3, for the cow class, the first and second clusters (corresponding to the first and second column) are reliable, and the last two are unreliable. For the car class, the first cluster is unreliable and the last two are reliable. It can be seen from the figure that the reliable clusters contain segments that are more
130
V. Vakili and O. Veksler
easily recognizable as parts of the object class. Unreliable clusters tend to have segments that are more generic and shared between classes, such as blobs of certain eccentricity and orientation.
4
Object Class Segmentation
So far the process of creating a pool of reliable clusters from training images was discussed. Given a novel image to segment into its constituent object classes, we first generate multiple segmentations using FH and mean-shift algorithms. For each segment we extract fifteen features selected as discriminative at the training stage. We discard unreliable segments using the following procedure. We compute the distance from the current segment to the cluster centers obtained at the training stage. If the current segment is closest to an unreliable cluster center, it is itself deemed as unreliable and discarded. Otherwise it is classified using the k-nearest neighbor classifier. We set k = 8 for the experiments. Notice that since we discard unreliable segments for a test image, there is a chance that a pixel does not get classified because it is contained only in unreliable segments. We found experimentally that the percentage of such pixels is small. In most cases, if a pixel is in unreliable generic blob in one segmentation, it often does belong to a reliable region for another segmentation, usually at a different scale. The small percentage of pixels that do not get classified at this stage do get classified at the next stage, which we now describe. Since all the segments are classified independently, to improve coherence and integrate the results of multiple segmentations, we use graph cut optimization framework [23]. We seek a final labeling f that minimizes the following energy: Dp (fp ) + wpq · δ(fp = fq ) (4) E(f ) = p∈P
(p,q)∈N
The first term in equation (4) is called a data term and it is the penalty for pixel p to be assigned label fp . Dp measures how well label fp suits pixel p. This constraint ensures that the current labeling f is consistent with the observed data. We set the data term as follows. The more often pixel p was classified as object class l, the smaller is Dp (l), making object class l more likely for p. Specifically, Dp (fp ) is defined as the negative number of segments that contain pixel p and get classified as fp . The second term in equation (4) is known as the smoothness term. The smoothness term measures the extent to which f is not smooth. We use the Potts Model. Here δ(fp = fq ) is 0 if fp = fq and 1 otherwise. A typical choice (Ip −Iq )2
for wpq is λe− 2σ2 , where Ip and Iq are the intensities of pixels p and q. Parameter σ is related to the level of variation between neighboring pixels. The parameter λ is used to control the relative importance of the data term versus the smoothness term. We set λ to 10 in this work. This energy function can be approximately optimized with the alpha-expansion algorithm of [23]. The code for alpha-expansion is taken from Boykov et al. [23, 30, 31].
Object Class Segmentation Using Reliable Regions
131
Notice that if a pixel belongs only to unreliable segments, i.e. it did not get classified with any object class for any segmentation, then its data terms are equal to zero for all classes. Such pixel gets classified based only on propagation from it surrounding context due to the smoothness term.
5
Experimental Results
We tested our approach on the MSRC 21-class data set [32]. In this data set pixels close to object boundaries and in non-object regions are labeled void in the ground truth images. These pixels are ignored in evaluation. Since ground truth may contain different labels for any given segment, for training, we use only the segments that have more than 70% label consistency, that is at least 70% of pixels should belong to a single class. Table 1 gives the quantitative results using graph cuts for integrating multiple segmentations. The second row shows the results when only reliable regions are used for object detection and the first row is the results of using all regions generated by all of the segmentations. As it can be seen, the results are better by 11% in pixel-averaged and class-averaged accuracy when just the reliable regions are used. For all object classes except the sheep, sky and boat object classes, using just the reliable regions outperform the approach where all regions are used. This shows that lots of regions do not have informative features to distinguish an object or an object part. Using all of these regions instead of just the reliable ones can mislead object recognition.
boat
body
cat
dog
road
book
chair
sign
bird
bike
flower
car
face
water
sky
aeroplane
cow
sheep
tree
grass
building
Pixel Average Graph cut (all
Class Average
Table 1. Quantitative performance using multiple segmentations. In the first row, all regions from multiple segmentations are used for recognition while in the second row just reliable regions are used.
57 44 47 81 60 23 49 96 31 46 60 41 39 65 35 22 43 24 63 30 11 33 31
regions) Graph cut
68 55 60 90 70 50 43 87 46 87 64 54 46 71 43 26 67 25 75 39 50 37 24
(reliable regions)
We also measure the improvement graph cut integration of multiple segmentations gives over single segmentation results. We compute accuracy improvement of graph cut results over each of the 34 single segmentation result. On average, the class-averaged accuracy increases by 7.8% with graph cuts, with standard deviation of 0.03%. The smallest improvement is 4.0% and the best improvement is 14.5%. Figure 4 shows some successful qualitative results. Some partial successful examples or failures are shown in Figure 5. The first column contains the input
132
V. Vakili and O. Veksler
Fig. 4. Successful qualitative results generated using graph cuts for integrating multiple image segmentations. a) Original images. b) Ground truth images. c) Results generated using all the regions. d) Results generated using reliable regions. The black pixels in all result images are not labeled in ground truth images and therefore are not considered in our evaluation. This picture is best viewed in color.
Object Class Segmentation Using Reliable Regions
133
images, the second column shows the ground truth segmentation. The third and fourth columns present maps of the most likely class at each pixel using all regions and reliable regions, respectively. The black pixels in the result images are considered as void in ground truth images and are not considered in our evaluation. These are the pixels for which no accurate ground truth could be constructed by the authors of the dataset. As can be seen from Figure 4 and Figure 5 using all regions leads to less accurate results as opposed to using only the reliable regions.
Fig. 5. Partially successful or failure results generated using graph cuts for integrating multiple image segmentations. a) Original images. b) Ground truth. c) Results generated using all the regions. d) Results generated using reliable regions. The black pixels in all result images are not labeled in ground truth images and therefore are not considered in our evaluation. This picture is best viewed in color.
5.1
Comparison to Previous Work
The purpose of this work was not to come up with the state-of-the art algorithm but to show that discarding unreliable regions from multiple segmentations gives better results for object recognition. This was shown using a simple nearest neighbor framework. Even though the goal was not a state of the art algorithm, we compare the performance of the proposed system to the state-of the art recent work [7, 8, 13]. Notice that they [7, 8, 13] use much more sophisticated machine learning algorithms and features. Nevertheless our overall performance is not that much worse, which is surprising, given the simplicity of our classifier. Furthermore, we are even able to achieve better accuracy for some classes. Our approach is superior on “water”, “sign” , “bird” “chair”, “dog” and “boat” object classes.
134
V. Vakili and O. Veksler
boat
cat
dog
road
book
chair
sign
bird
bike
car
face
sky
7
Verbeek and
cow
Shotton et al. [7] 72 58 62 98 86 58 50 83 60 53 74 63 75 63 35 19 92 15 86 54 19 62
tree
body
flower
water
aeroplane
sheep
grass
building
Pixel Average
Class Average
Table 2. The comparison of our approach with that of Shotton et al. [7], Verbeek and Triggs [8] and Pantofaru et al. [13]
74 64 52 87 68 73 84 94 88 73 70 68 74 89 33 19 78 34 89 46 49 54 31
Triggs [8] Pantofaru et
74 60 68 92 81 58 65 95 84 81 75 65 68 53 35 23 85 16 83 48 29 48 15
al. [13] reliable regions
68 55 60 90 70 50 43 87 46 87 64 54 46 71 43 26 67 25 75 39 50 37 24
The running time of our approach could be improved significantly, it is currently in minutes. The most time consuming part is obtaining multiple segmentations and extracting features.
6
Conclusion
We develop a novel object recognition algorithm based on selecting reliable regions from multiple segmentations. We produce pixel-wise segmentation of the objects in the image. To increase the robustness of our system to poor segmentations and also in order integrate the results of multiple segmentations, information from multiple image segmentations is integrated using graph cut optimization framework. The main contribution of our work is a clustering approach to find the reliable regions for object recognition. Experiments show that when using only the reliable regions, the results improve by approximately 11% both in pixel-averaged and class-averaged accuracy.
References 1. Rowley, H., Baluja, S., Kanade, T.: Human face detection in visual scenes. In: Advances in Neural Information Processing Systems, pp. 875–881 (1996) 2. Viola, P., Jones, M.: Robust real-time face detection. International Journal of Computer Vision 57, 137–154 (2004) 3. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Conference on Computer Vision and Pattern Recognition, p. 886 (2005) 4. Ferrari, V., Fevrier, L., Jurie, F., Schmid, C.: Groups of adjacent contour segments for object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 36–51 (2008) 5. Lampert, C., Blaschko, M., Hofmann, T., Zurich, S.: Beyond sliding windows: Object localization by efficient subwindow search. In: Conference on Computer Vision and Pattern Recognition (2008)
Object Class Segmentation Using Reliable Regions
135
6. Russell, B., Efros, A., Sivic, J., Freeman, W., Zisserman, A.: Using multiple segmentations to discover objects and their extent in image collections. In: Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 1605–1614 (2006) 7. Shotton, J., Winn, J., Rother, C., Criminisi, A.: Textonboost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation. In: European Conference on Computer Vision, pp. 1–15 (2006) 8. Verbeek, J., Triggs, B., Inria, M.: Region classification with markov field aspect models. In: Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007) 9. Verbeek, J., Triggs, B.: Scene segmentation with conditional random fields learned from partially labeled images. In: Proc. NIPS (2008) 10. He, X., Zemel, R.: Learning hybrid models for image annotation with partially labeled data. In: Advances in Neural Information Processing Systems, NIPS (2008) 11. Schroff, F., Criminisi, A., Zisserman, A.: Object class segmentation using random forests (2008) 12. Gould, S., Rodgers, J., Cohen, D., Elidan, G., Koller, D.: Multi-class segmentation with relative location prior. International Journal of Computer Vision 80, 300–316 (2008) 13. Pantofaru, C., Schmid, C., Hebert, M.: Object recognition by integrating multiple image segmentations. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part III. LNCS, vol. 5304, pp. 481–494. Springer, Heidelberg (2008) 14. Gu, C., Lim, J., Arbelaez, P., Malik, J.: Recognition Using Regions. In: Proceedings of the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Proc. CVPR, pp. 1030–1037 (2009) 15. Fulkerson, B., Vedaldi, A., Soatto, S.: Class segmentation and object localization with superpixel neighborhoods. In: IEEE International Conference on Computer Vision, pp. 670–677 (2009) 16. Malisiewicz, T., Efros, A.: Improving spatial support for objects via multiple segmentations. In: British Machine Vision Conference (BMVC), vol. 2, BMVC (2007) 17. Todorovic, S., Ahuja, N.: Learning subcategory relevances for category recognition. In: Proceedings of the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008) 18. Hoiem, D., Efros, A., Hebert, M.: Recovering surface layout from an image. International Journal of Computer Vision 75, 151–172 (2007) 19. Galleguillos, C., Babenko, B., Rabinovich, A., Belongie, S.: Weakly supervised object localization with stable segmentations. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 193–207. Springer, Heidelberg (2008) 20. Felzenszwalb, P., Huttenlocher, D.: Efficient graph-based image segmentation. International Journal of Computer Vision 59, 167–181 (2004) 21. Comaniciu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 603– 619 (2002) 22. Kira, K., Rendell, L.: A practical approach to feature selection. In: Proceedings of the Ninth International Workshop on Machine Learning Table of Contents, pp. 249–256. Morgan Kaufmann Publishers Inc., San Francisco (1992) 23. Boykov, Y., Veksler, O., Zabih, R.: Efficient approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 1222–1239 (2001)
136
V. Vakili and O. Veksler
24. Barnard, K., Duygulu, P., Guru, R., Gabbur, P., Forsyth, D.: The effects of segmentation and feature choice in a translation model of object recognition. In: Conference on Computer Vision and Pattern Recognition, pp. 675–682 (2003) 25. Martin, D., Fowlkes, C., Malik, J.: Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Transactions on Pattern Analysis and Machine Intelligence, 530–549 (2004) 26. Levi, K., Weiss, Y.: Learning object detection from a small number of examples: The importance of good features. In: Conference on Computer Vision and Pattern Recognition, vol. 2 (2004) 27. Berkhin, P.: A survey of clustering data mining techniques. Grouping Multidimensional Data, 25–71 (2006) 28. Hartigan, J.: Clustering algorithms. Wiley, New York (1975) 29. Fisher, R.: The use of multiple measures in taxonomic problems. Ann. Eugenics 7, 179–188 (1936) 30. Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1124–1137 (2004) 31. Kolmogorov, V., Zabih, R.: What energy functions can be minimized via graph cuts? IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 147– 159 (2004) 32. Shotton, J., Winn, J., Rother, C., Criminisi, A.: The MSRC 21-class object recognition database (2006)
Specular Surface Recovery from Reflections of a Planar Pattern Undergoing an Unknown Pure Translation Miaomiao Liu1 , Kwan-Yee K. Wong1 , Zhenwen Dai2 , and Zhihu Chen1 1 2
Dept. of Computer Science, The University of Hong Kong, Hong Kong Frankfurt Institute for Advanced Studies, Goethe-University Frankfurt
Abstract. This paper addresses the problem of specular surface recovery, and proposes a novel solution based on observing the reflections of a translating planar pattern. Previous works have demonstrated that a specular surface can be recovered from the reflections of two calibrated planar patterns. In this paper, however, only one reference planar pattern is assumed to have been calibrated against a fixed camera observing the specular surface. Instead of introducing and calibrating a second pattern, the reference pattern is allowed to undergo an unknown pure translation, and a closed form solution is derived for recovering such a motion. Unlike previous methods which estimate the shape by directly triangulating the visual rays and reflection rays, a novel method based on computing the projections of the visual rays on the translating pattern is introduced. This produces a depth range for each pixel which also provides a measure of the accuracy of the estimation. The proposed approach enables a simple auto-calibration of the translating pattern, and data redundancy resulting from the translating pattern can improve both the robustness and accuracy of the shape estimation. Experimental results on both synthetic and real data are presented to demonstrate the effectiveness of the proposed approach.
1
Introduction
Shape recovery has always been a hot topic in computer vision over the past few decades. Traditional shape recovery methods can be classified into different categories according to the information they employed, such as structure from motion [1], shape from silhouettes [2], and shape from shadings [3]. Most of the existing methods, however, can only handle diffuse objects due to the fact that the information they exploited is derived under the assumption of a diffuse surface. For instance, structure-from-motion methods cannot be applied to a specular (mirror) object as the features on the object surface are the reflections of its surrounding environment and are therefore not viewpoint independent. Similarly, shape-from-silhouettes methods have difficulties in handling specular objects as it is not a trivial task to extract the silhouette of a mirror object. The underlying principle of shape-from-shadings methods states that the shadings of the object follow the Lambert’s cosine law, and therefore it is obvious that R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 137–147, 2011. c Springer-Verlag Berlin Heidelberg 2011
138
M. Liu et al.
specular surfaces cannot be recovered using shape-from-shadings methods. In addition, the shape of a specular object also cannot be obtained using a laser scanner since the surface may often reflect the laser ray away from the sensor of the scanner. In order to allow shape recovery of specular objects, special algorithms have to be developed under the assumption of a specular surface whose appearance follows the law of reflection. In [4], Zisserman et al. pointed out that a moving observer can tell the physical properties, such as convexity or concavity, of a specular surface. Fleming et al. [5] also showed that human beings have the ability to distinguish the shape of a specular surface even when no knowledge about the environment is available. In [6], Oren and Nayar explored the relation between image trajectories of virtue features induced by specular reflection and 3D geometry of the specular object they travelled on. Roth and Black [7] proposed to use both optical flow and specular flow to recover an object which is a mixture of specular and diffuse materials. In [8], Adato et al. analyzed valued information of specular flow induced by the movement of a camera, and derived a non-linear partial differential equation (PDE) to describe the relation between the 3D shape and motion of the specular object. In [9], Canas et al. linearized this non-linear PDE in order to make it easier for practical use. The aforementioned works are regarded as shape from specular flow (SFSF) methods. SFSF methods generally require a dense observation of specular flow, but dense specular flows often merge together in an undetermined way. Besides, SFSF methods recover the specular surface by solving PDE, and the solution is greatly affected by the initial boundary condition. Furthermore, the motion of the specular object must be small and continuous in order to derive the PDE. All these limitations make SFSF methods not very useful in practice. Great efforts have also been devoted to specular shape recovery techniques under known surroundings. In [10, 11], Savarese and Perona demonstrated that local surface geometry properties of a mirror object can be determined by analyzing the local differential properties of the reflection of two calibrated lines. In [12], Bonfort and Sturm introduced a voxel carving method to recover a specular surface using a normal consistency criterion. Seitz et al. [13] reduced the specular shape recovery problem to a light path reconstruction problem. They showed that surface points can be obtained if the positions of two reference points are known in space for one single view, or the position of one reference point is known and is visible in two different views. In [14], a phase shift method was used to determine the correspondence between a reference point in space and its reflection on the image plane. Nehab et al. [15] reduced the shape recovery problem to an image matching problem by minimizing a cost function based on normal consistency. Note that calibration plays an important role in specular shape recovery methods under known surroundings. Bonfort et al. [16] used a mirror to calibrate a reference plane after changing its position every time. In [17], Rozenfeld et al. suggested an approach to recover the surface by using an empirically calculated 1D homography. Their proposed method only needs sparse plane-image correspondences, but depends on optimization greatly.
Specular Surface Recovery from a Translating Planar Pattern
139
In this paper, we adopt the approach of specular shape recovery under known surroundings. Unlike previous methods which assume a fully calibrated environment, we propose a novel solution based on observing the reflections of a translating planar pattern. We assume a reference planar pattern is initially calibrated against a fixed camera observing a specular surface. This pattern is then allowed to undergo an unknown pure translation. The motion of the pattern as well as the shape of the specular object can then be estimated from the reflections of the translating pattern. The main contributions of this paper are 1) a closed form solution for recovering the motion of the translating pattern, and 2) a novel method for shape recovery based on computing the projections of the visual rays on the translating pattern. The proposed auto-calibration method allows an easy calibration of the translating pattern, and data redundancy resulting from the translating pattern can improve both the robustness and accuracy of the shape estimation. The proposed shape recovery method produces a depth range for each pixel in which the true surface point exists. This in turn provides a measure of the accuracy and confidence of the estimation. The rest of the paper is organized as follows. Section 2 describes the physical configuration of our proposed translating pattern approach. Based on geometric constraints, a closed form solution is derived for recovering the unknown motion of the translating pattern in Section 3. A degenerate case is also studied and analyzed. Section 4 introduces a projection based method for specular shape recovery using the estimated motion. Experimental results on both synthetic and real data are presented in Section 5, followed by conclusions in Section 6.
2
Translating Planar Pattern
Consider a pinhole camera, with camera center O, looking at a specular object S which reflects a nearby planar pattern Π to the image I (see Fig.1). Consider now a point P on S. Let A be a point on Π which is reflected by S at P to a point x on I. Suppose now the planar pattern undergoes a pure translation given by an unknown translation vector T, and let Ω denotes the planar pattern at the new location after translation. Let B be a point on Ω such that it is collinear with A and P . According to the law of reflection, B will also be reflected by S at P to the same point x on I. As shown in [13, 16], the 3D position of P can be obtained by intersecting the visual ray (defined by O and x) with the reflection ray (defined by A and B). In order to construct the reflection ray, both the 3D positions of A and B are needed, and their accuracies would directly affect the accuracy of the estimated 3D position of P . The relative positions of A and B on the planar pattern can be resolved by encoding the pattern using gray code and observing the corresponding intensities at x. Besides, in order to obtain the 3D positions of A and B, the relative positions and orientations of Π and Ω with respect to O must also be known. In this paper, Π is assumed to have been calibrated against O initially. The problem of calibrating Ω is thus reduced the problem of estimating the unknown translation vector T. In the next section, a closed form solution for T will be derived based on some geometric constraints.
140
M. Liu et al.
S
Ω
∏ P
A
B
T x
I
O Fig. 1. Configuration of a translating planar pattern. A pinhole camera with camera center O is viewing a specular object S which reflects a nearby planar pattern Π to the image I. Ω is a planar pattern obtained by translating Π by an unknown translation vector T. A and B are points on Π and Ω respectively which are reflected by S at P to the same image point x on I.
3
Recovering the Unknown Translation
Referring to the configuration described in the previous section, Ω and Π are related by a pure translation given by the translation vector T. Consider a point X on Π. Let X be a point on Ω such that it has the same relative position on the planar pattern as X (i.e., both X and X are encoded by the same gray code). X and X are referred to as a pair of corresponding points, and are related by X = X + T, where X and X denote the position vectors for X and X respectively. Now referring back to Fig. 1, and recalling that A and B are points on Π and Ω, respectively, which are reflected by S at P to the same image point x on I. Let Q be a point on Π which is encoded by the same gray code as B (i.e., B and Q form a pair of corresponding points). Hence, we have B = Q + T,
(1)
and the vector AB in the direction of the reflection ray can be written as AB = AQ + T.
(2)
Consider the plane Γ formed by the visual ray and the reflection ray at P , and let N be its unit normal vector. It is obvious that AB and N are mutually orthogonal and therefore AB · N = 0. (3)
Specular Surface Recovery from a Translating Planar Pattern
141
Substituting (2) into (3) gives (AQ + T) · N = 0.
(4)
Note that Q can be inferred from the gray code of B, and the vector AQ can be constructed from the 3D positions of A and Q on the calibrated planar pattern Π. The unit normal vector N can be computed as the vector cross product of Ox and OA. Hence, the only unknown in (4) is the translation vector T, which has three degrees of freedom. Since a pair of reflections observed at one pixel provides one linear constraint on T, T can therefore be estimated using a linear least squares method from the reflections observed at three or more pixels: ⎛
⎞−1 ⎛ NT A1 Q1 · N1 1 ⎜ NT ⎟ ⎜ ⎜ 2 ⎟ ⎜ A2 Q2 · N2 T = −⎜ . ⎟ ⎜ .. ⎝ .. ⎠ ⎝ . NT n
⎞ ⎟ ⎟ ⎟ ⎠
(5)
An Qn · Nn
If n > 3, the matrix composed of normal vectors will be a rectangle matrix, and the symbol of inverse in (5) denotes the Moore-Penrose pseudo-inverse. The pseudo-inverse gives the solution in a least-squared sense. However, when the specular object is a plane or a surface of revolution, there exist some degenerate cases which are described below in Proposition 1 and Proposition 2. Proposition 1. If the specular object is a plane, the translation vector T cannot be determined by using the proposed theory. Proof. Let S be a specular plane. Without loss of generality, let P be an arbitrary point on S and NP be the surface normal at P . Consider now the visual ray OP . By the law of reflection, the visual ray OP , the surface normal NP and the reflection ray must be all lie on the same plane. Let ΓP denotes such a reflection plane. Note all points on S have the same surface normal NP . It follows that all the reflection planes will intersect along a line passing through the camera center O and with an orientation same as NP , and hence their normal vectors are all orthogonal to NP . This makes the matrix composed of the normal vectors in (5) to have a rank of at most 2, and therefore T cannot be recovered using (5). Proposition 2. If the specular object is a surface of revolution and its revolution axis passes through the camera center, the translation vector T cannot be determined by using the proposed theory. Proof. Let S be a specular surface of revolution with its revolution axis l passing through the camera center O. Without loss of generality, let P be an arbitrary point on S and NP be the surface normal at P . Consider now the visual ray OP . By the law of reflection, the visual ray OP , the surface normal NP and the reflection ray must all lie on the same plane. Let ΓP denotes such a reflection plane. By the properties of a surface of revolution, NP lies on the plane defined
142
M. Liu et al.
by P and l. Since l passes through O, it is easy to see that l also lies on ΓP . It follows that all the reflection planes will intersect at l and hence their normal vectors are all orthogonal to l. This makes the matrix composed of normal vectors in (5) to have a rank of at most 2, and therefore T cannot be recovered using (5). Since a sphere is a special kind of surface of revolution with any line passing through its center being a revolution axis, Corollary 1 follows immediately. Corollary 1. A single sphere cannot be recovered by using the proposed theory.
4
Specular Surface Recovery
With the estimated translation vector T, the position of the reference planar pattern after translation can be determined. If accurate correspondences between points on the reference planar pattern and pixels on the image can be obtained, the surface of the specular object can be recovered by triangulating the corresponding visual ray and reflection ray at each pixel (see Section 2). However, most of the encoding strategies cannot achieve a real point-to-point correspondences. Usually, one pixel corresponds to one encoded area (e.g., a square on the reference planar pattern). In [16], a dense matching of the pixels on the image and positions on the reference plane was implemented by minimizing an energy function which penalized the mis-encoding problem, and bilinear interpolation was used to achieve a sub-pixel accuracy in the matching. Bilinear interpolation, however, is indeed not a good approximation as it is well-known that ratio of lengths is not preserved under perspective projection. Besides, the reflection of the pattern by a non-planar specular surface would also introduce distortion in the image of the pattern, and such a distortion depends on both the shape and distances of the object relative to the camera and the planar pattern. In order to obtain good shape recovery results, the resolution of the encoding pattern and the positions of the plane need to be chosen carefully. In this paper, the specular surface is assumed to be smooth and locally planar. After associating a gray code (and hence an encoded area on the reference plane) to each pixel, the algebraic centers ci for pixels with the same gray code is computed and associated with the center Ci of the corresponding encoded area on the reference plane. This is the approximation to the point-to-point correspondences in the underlying encoding strategy. This approximation has been verified with experiments on synthetic data, and experimental results show that the error distances introduced by approximating the ground truth reflection points on the plane with the centers of the encoded areas are negligible. Nevertheless, this approximation can only be applied to obtain a matching between the image pixels and points on one reference plane, as the algebraic centers ci computed for one reference plane position will not in general coincide with those computed for a different reference plane position. Instead of trying to locate a point match on the translated reference plane Ω for each ci , the visual ray for ci is projected onto the translated reference plane Ω using its corresponding point Ci on the initial reference plane Π as the projection center (see Fig. 2). The projection of
Specular Surface Recovery from a Translating Planar Pattern
143
Ω Π S
Ci
P
B
Ci
O
Fig. 2. Shape recovery of specular object. The visual ray at ci is projected onto the translated reference plane Ω using its corresponding point Ci on the initial reference plane Π as the projection center. The projection of the visual ray on Ω will result in a line which will intersect the corresponding encoded area as observed at ci . This intersection line segment can be back projected onto the visual ray and this gives a depth range in which the surface point should exist.
the visual ray on Ω will result in a line which will intersect the corresponding encoded area (which is a square) as observed at ci . This intersection line segment can be back projected onto the visual ray and this gives a depth range in which the surface point should exist. If Ω is far from Π, the depth range will be small and the midpoint can be taken as the estimated surface depth. Alternatively, the depth range can be made smaller by increasing the resolution of the encoding pattern (i.e., decreasing the size of the encoded area). This, however, is not a practical strategy since the encoding pattern will be blurred in the imaging process and may result in an aliasing effect.
5
Experimental Analysis
The proposed approach was tested using both synthetic and real image data. The synthetic images were generated using the pov-ray software. For the real experiments, images were taken by using a simple setting composed of a monitor, a camera and a drawer. The monitor was used for displaying gray coded patterns. The camera used was a Cannon 450D with a focal length of 50mm. The drawer was used for carrying the monitor and making it move in pure translation motions. 5.1
Synthetic Experiments
In the synthetic experiments, the scene was composed of two spheres or two planes with different positions and orientations. Note that the cases with multiple
144
M. Liu et al.
Fig. 3. Shape recovery result for a sphere. Left column: Ground truth observed in two different views. Right column: Recovered shape in two different views. 1.4 1.2 1 0.8 0.6 0.4 0.2 0
0
50
100
150
Fig. 4. Relationship between depth recovery error and translation scale. The scene is composed of two spheres which have the same radius of 19mm, and the translation scale ranges from 1mm to 150mm. 12880 pixels are used for shape recovery. Y-axis shows the average depth recovery error for all pixels, and X-axis represents the translation scale.
spheres or multiple planes are not degenerate, and the shape of the objects can be recovered using the proposed method. In order to analyze the effect of the translation scale on the result, the reference plane was translated with multiple scales. The reconstruction for a part of a sphere is shown in Fig. 3, which also gives a comparison with the ground truth data. Fig. 4 shows the relation between the translation scale of the reference plane and the shape recovery error. It can
Specular Surface Recovery from a Translating Planar Pattern
145
Fig. 5. Shape recovery result for a plane. Left column: Ground truth observed in two different views. Right column: Recovered shape in two different views.
Fig. 6. Setting for the real experiment. Left Column: Setting used in the real experiment. Right Column: sample pattern images used for encoding. The proposed method is used to recover the area reflecting the reference plane on the specular object.
be seen that, within the visible reflection range for the specular object, larger translation scale would result in smaller reconstruction error. After experimenting with a range of translation scales, 80mm is chosen as the translation scale for the second position of the reference plane. The reconstruction error is smaller than 0.5%. The reconstruction for the plane is shown in Fig.5, which again also gives a comparison with the ground truth data. 70mm is chosen as the translation scale and the average reconstruction error is less than 0.45%.
146
M. Liu et al.
0 −0.1 −0.2 −0.3
1
−0.4 2.5
0.9 0.8
2
0.7
1.5
0.6
1 0.5
0.5
Fig. 7. Shape recovery result for a spoon. The recovered shape is the area with reflections on the spoon.
5.2
Real Experiments
The real experiment was conducted to recover the shape of a small spoon. The experiment setting was quite simple. The monitor was carried by a drawer and translated on the table (see Fig. 6). The monitor displayed gray coded pattern images and a set of images were taken when the monitor translated in different scales. A traditional structured light method [18] was used to get the correspondences between a coded area on the reference plane and a pixel on the image. The first reference planar pattern is initially calibrated. The reference plane can be translated along a certain direction in different scale and the reconstruction results can be obtained by choosing the best motion scale or combine the multiple motion information together to get a better result. Although the spoon is small and it only reflects a small area on the monitor, the shape can be well recovered (see Fig. 7).
6
Conclusions
In this paper, we have proposed a novel solution for specular surface recovery based on observing the reflections of a translating planar pattern. Unlike previous methods which assume a fully calibrated environment, only one reference planar pattern is assumed to have been calibrated against a fixed camera observing the specular surface, and this pattern is allowed to undergo an unknown pure translation. The motion of the pattern as well as the shape of the specular object can then be estimated from the reflections of the translating pattern. The main contributions of this paper are 1) A closed form solution for recovering the motion of the translating pattern. This allows an easy calibration of the translating pattern, and data redundancy resulting from the translating pattern (i.e., reflections of multiple planar patterns at each pixel location) can improve both the robustness and accuracy of the shape estimation. 2) A novel method for shape recovery based on computing the projections of the visual rays on the translating pattern. This produces a depth range for each
Specular Surface Recovery from a Translating Planar Pattern
147
pixel in which the true surface point exists. This in turn provides a measure of the accuracy and confidence of the estimation. Experimental results on both synthetic and real data are presented, which demonstrate the effectiveness of our proposed approach. As for the future work, we are now trying to solve the more challenging problems in which 1) the initial reference planar pattern is also uncalibrated and 2) the motion of the reference planar pattern is not limited to pure translation.
References 1. Tomasi, C.: Shape and motion from image streams under orthography: a factorization method. International Journal of Computer Vision 9, 137–154 (1992) 2. Wong, K.Y.K., Cipolla, R.: Structure and motion from silhouettes. In: ICCV, pp. 217–222 (2001) 3. Woodham, R.: Photometric method for determining surface orientation from multiple images. Optical Engineering 19, 139–144 (1980) 4. Zisserman, A., Giblin, P., Blake, A.: The information available to a moving observer from specularities. Image and Vision Computing 7, 38–42 (1989) 5. Fleming, R.W., Torralba, A., Adelson, E.H.: Specular reflections and the perception of shape. Journal of Vision 4, 798–820 (2004) 6. Oren, M., Nayar, S.K.: A theory of specular surface geometry. International Journal of Computer Vision 24, 105–124 (1997) 7. Roth, S., Black, M.J.: Specular flow and the recovery of surface structure. In: CVPR, pp. 1869–1876 (2006) 8. Adato, Y., Vasilyev, Y., Ben-Shahar, O., Zickler, T.: Toward a theory of shape from specular flow. In: ICCV, pp. 1–8 (2007) 9. Canas, G.D., Vasilyev, Y., Adato, Y., Zickler, T., Gortler, S., Ben-Shahar, O.: A linear formulation of shape from specular flow. In: ICCV, pp. 1–8 (2009) 10. Savarese, S., Perona, P.: Local analysis for 3d reconstruction of specular surfaces. In: CVPR, pp. 738–745 (2001) 11. Savarese, S., Perona, P.: Local analysis for 3d reconstruction of specular surfaces - part ii. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 759–774. Springer, Heidelberg (2002) 12. Bonfort, T., Sturm, P.: Voxel carving for specular surfaces. In: ICCV, pp. 691–696 (2003) 13. Seitz, S., Matsushita, Y., Kutulakos, K.: A theory of inverse light transport. In: ICCV, pp. 1440–1447 (2005) 14. Yamazaki, M., Iwata, S., Xu, G.: Dense 3D reconstruction of specular and transparent objects using stereo cameras and phase-shift method. In: Yagi, Y., Kang, S.B., Kweon, I.S., Zha, H. (eds.) ACCV 2007, Part II. LNCS, vol. 4844, pp. 570–579. Springer, Heidelberg (2007) 15. Nehab, D., Weyrich, T., Rusinkiewicz, S.: Dense 3d reconstruction from specularity consistency. In: CVPR, pp. 1–8 (2008) 16. Bonfort, T., Sturm, P., Gargallo, P.: General specular surface triangulation. In: Narayanan, P.J., Nayar, S.K., Shum, H.-Y. (eds.) ACCV 2006. LNCS, vol. 3852, pp. 872–881. Springer, Heidelberg (2006) 17. Rozenfeld, S., Shimshoni, I., Lindenbaum, M.: Dense mirroring surface recovery from 1d homographies and sparse correspondences. In: CVPR, pp. 1–8 (2007) 18. Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. In: CVPR, pp. 195–202 (2003)
Medical Image Segmentation Based on Novel Local Order Energy LingFeng Wang1 , Zeyun Yu2 , and ChunHong Pan1 1
2
NLPR, Institute of Automation, Chinese Academy of Sciences {lfWang,chpan}@nlpr.ia.ac.cn Department of Computer Science University of Wisconsin-Milwaukee
[email protected]
Abstract. Image segmentation plays an important role in many medical imaging systems, yet in complex circumstances it is still a challenging problem. Among many difficulties, problem caused by the image intensity inhomogeneity is the key aspect. In this work, we develop a novel localhomogeneous region-based level set segmentation method to tackle this problem. First, we propose a novel local order energy, which interprets the local intensity constraint. And then, we integrate this energy into the objective energy function. After that, we minimize the energy function via a level set evolution process. Extensive experiments are performed to evaluate the proposed approach, showing significant improvements in both accuracy and efficiency, as compared to the state-of-the-art.
1
Introduction
Level set model has been extensively applied to medical image segmentation, because it holds several desirable advantages over traditional image segmentation methods. First, it can achieve sub-pixel segmentation accuracy of object boundaries. Second, it can change the topology of the contour automatically. Existing level set methods can be mainly classified into two groups: i.e. the edge-based [1–3] methods and the region-based [4–14] methods. Edge-based methods utilize the local edge information to attract the active contour toward the object boundaries. These methods have two inevitable limitations [4], such as depending on the initial contour as well as sensitive to the noise. Region-based methods, on the other hand, are introduced to overcome the two limitations. These methods identify each interest region by a certain region descriptor, such as intensity, to guide the evolution of the active contour. The principle of regionbased methods tend to rely on the assumption of homogeneity of the region descriptor. In this work, we improve the traditional region-based method to tackle the segmentation difficulties, such as the inhomogeneity of the image intensity. There are mainly two types of region-based segmentation methods: i.e. the global-homogeneous [4–6] and local-homogeneous [7–14]. One of the most important global-homogeneous methods is proposed by Chan et al. [4]. This method R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 148–159, 2011. c Springer-Verlag Berlin Heidelberg 2011
Medical Image Segmentation Based on Novel Local Order Energy
149
first assumes that image intensities are statistically homogeneous in each segmented region. Then, it utilizes the global fitting energy to represent this homogeneity. Improved performance of the global-homogeneous methods has been reported in [4–6]. However, the details of the object always cannot be segmented out by these methods. As shown in Fig.1, parts of brain tissues are not segmented out. The local-homogeneous methods are proposed to solve intensity inhomogeneity. These methods assume that the intensities in a relatively small local region are separable. Li et al. [7, 10] introduced a local fitting energy due to a kernel function, which is used to extract the local image information. Motivated by the local fitting energy proposed in [10], the local Gaussian distribution fitting energy is presented in [14]. An et al. [8] offered a variational region-based algorithm, which combines the Γ -Convergence approximation with a new multi-scale piecewise smooth model. Unfortunately, the local-homogenous methods have two inevitable drawbacks. First, these methods are sensitive to the initial contour. Second, the segmentation results have obvious errors. As shown in Fig.1, although the two initial contours are very similar, the segmentation results of [10] are quite different. Furthermore, both the two results have obvious segmentation errors. We use the blue ellipses to indicate these errors. Motivated by the previous works [4, 7, 8, 10, 15], the local intensity constraint is adopted to realize our local-homogeneous region-based segmentation method. The purposes of using the local intensity constraint are from two aspects. First, the local characteristic can soften the global constraint of globalhomogeneous methods. Second, using additional constraint can strengthen the instability of local-homogenous methods. In this work, we first develop a local order energy as local intensity constraint. The proposed local order energy assumes that intensities in a relatively small local region has order. For example, the intensities in the target are larger than those in the non-target. Generally, the local order assumption is easy to be satisfied in medical image. Then, we integrate the local order energy into the objective energy function, and minimize the energy function via a level set evolution process. Experimental results show the proposed method has many advantages and improvements. First, compared to the global-homogeneous methods, the detail information is well segmented. Second, the two main drawbacks of the traditional local-homogeneous methods are well solved. Third, the proposed method is also insensitive to noise. The remainder of the presented paper is organized as follows: Section 2 describes the proposed segmentation method in detail; Section 3 presents some experiments and results; the conclusive remark is given in Section 4.
2
The Proposed Method
In this section, we first briefly introduce two traditional segmentation methods, i.e. the global-homogenous methods and the local-homogenous methods. Then, we describe the proposed local order energy, which interprets the local intensity constraint, in detail. Finally, we use level set evolution process to minimize the energy function, and present our algorithm in Alg.1.
150
L. Wang, Z. Yu, and C. Pan
ÿ
ÿ
Fig. 1. A comparison of the proposed method to the traditional methods, i.e. the Chan et al. [4] (a global-homogeneous method) and the Li et al. [10] (a local-homogenous method). Note that, two different yet very similar initial contours are used to test the initialization sensitive of three algorithms.
2.1
Global-Homogenous and Local-Homogenous Methods
Let Ω ⊂ R2 be the image domain, and I : Ω → R be the given gray-level image. Global-homogenous methods formulated the image segmentation problem as to find a contour C. The contour segments the image into two non-overlapping regions, i.e. the region of inside contour as Cin (the target region) and the region of outside contour as Cout (the non-target region). By specifying two constants cin and cout , the fitting energy (global fitting energy) is defined as follows, I(x) − cin 2 dx E global (C, cin , cout ) = λin Cin I(x) − cout 2 dx + μ|C| (1) + λout Cout
where |C| is the length of the contour C, and λin , λout and μ are positive constants. As expressed in Eqn.1, the specified two constants cin and cout globally govern all the pixel intensities in Cin and Cout , respectively. Thus, Chan’s method is denoted as global-homogeneous method. The two constants cin and cout can further be interpreted as the two cluster centers, i.e. the target cluster cin and the non-target cluster cout . Motivated by Chan’s work, Li et al. [10] proposed a local fitting energy due to a kernel function, given by local f it E (2) C, cin (x), cout (x) = E C, cin (x), cout (x) dx + μ|C| yielding,
2 E f it C, cin (x), cout (x) = λin Kσ x − y I(y) − cin (x) dy Cin 2 Kσ x − y I(y) − cout (x) dy + λout Cout
(3)
Medical Image Segmentation Based on Novel Local Order Energy
151
where K(.) is the Gaussian kernel function Kσ x =
2 2 1 e−x /2σ pa(2π)n/2 σ n rameterized with the scale parameter σ. More comprehensive interpretations of the local fitting energy are introduced in [10] in detail. The fitting values cin (x) and cout (x) are locally govern the pixel intensities centered at position x, whose size is controlled by σ. Thus, Eqn.2 is named as the local fitting energy, and Li’s method is denoted as local-homogeneous method. Similarly, cin (x) and cout (x) can be interpreted as two local cluster centers at position x. The main difference between the above two methods is how to choose the cluster centers. The global-homogenous method simply utilizes two constants cin and cout as the cluster centers, and formulates them as the global fitting energy. In contrary, the local-homogenous method adopts the local cluster centers, and formulates them as the local fitting energy.
2.2
Local Order Energy
Generally, medical image has inhomogeneity problem. Thus, local fitting energy interprets the segmentation problem better than global fitting energy. However, local fitting energy also has some problems. The most serious one is that local fitting energy function has many local optimal solutions. Thus, it is hard to reach the global optimal solution through the optimization method, such as the level set evolution process. Meanwhile, this problem brings two inevitable drawbacks. First, the solution is sensitive to the initialization. Second, the segmentation result has some obvious errors. Actually, the local optimal solutions of global fitting energy is less than those of local fitting energy. That is, the global-homogenous method has more stable solution than the local-homogenous method. The main reason is that global fitting energy utilizes a strong intensity constraint. However, the local fitting energy weakens this intensity constraint by using the local characteristic. Thus, it is reasonable to add other local intensity constraints. Here, we adopt the local order energy as the local intensity constraint. The proposes of using this energy are from two sides. First, using local constraints can solve the inhomogeneity problem better than using global constraints. Thus, the detail information can be well segmented out. Second, adding other constraints can obviously decrease local optimal solutions number. Thus, the segmentation result can be stable and with small errors. In the next, we first give a straightforward example to interpret the local optimal solution problem. Then, we present the local order energy, which is one of the important local intensity constraint. Fig.2 presents a straightforward example to interpret the local optimal solution problem. We use a simple synthetic image with half bright pixels and half dark pixels as the test image. As shown in Fig.2, the curves Cright (in red) and Cwrong (in blue) represent the right and wrong segmentation results, respectively. When specifying the local cluster centers cin (x) and cout (x), both two local fitting energies E local (Cright , cin (x), cout (x)) and E local (Cwrong , cin (x), cout (x)) calculated from Cright and Cwrong can reach to the local minimums. That is, both E local (Cright , cin (x), cout (x)) and E local (Cwrong , cin (x), cout (x)) are the local optimal solutions. As shown in the second row of Fig.2, we assign three boundary
152
L. Wang, Z. Yu, and C. Pan
pixels (in green dashed curve) with cin (x) = 1 and other pixels with cin (x) = 1 and cout (x) = 0. Here, the local fitting energy E local (Cright , cin (x), cout (x)) reaches to the local minimum. As shown in the third row of Fig.2, we assign three boundary pixels with cout (x) = 1 and other pixels with cin (x) = 1 and cout (x) = 0. Here, the local fitting energy E local (Cwrong , cin (x), cout (x)) reaches to the local minimum. Note that, the curve Cwrong is the wrong segmentation result. We can also enumerate other wrong results, which are similar with Cwrong .
First Row
1
1
1
1
1
1
1
1
1
1
1
1
1wrong 0 1 0right 1
0
0
0
0
0
0
0
0
0
0
0
0
0
Second Row
right
cin (x)
cout ( x)
1 1 1 1 1 x x x x x 1 1 1 1 1 x x x x x 1 1 1 1 1 x x x x x
x x x x x 0 0 0 0 0 x x x x x 0 0 0 0 0 x x x x x 0 0 0 0 0
Third Row
wrong
cin (x)
cout ( x)
1 1 1 1 x x x x x x 1 1 1 1 x x x x x x 1 1 1 1 x x x x x x
x x x x 1 0 0 0 0 0 x x x x 1 0 0 0 0 0 x x x x 1 0 0 0 0 0
Fig. 2. Overview of the comparison of two segmentation results. 1 and 0 are the intensity values of the image. The first row gives two segmentation curves Cright (in red) and Cwrong (in blue). The second row presents the cin (x) and cout (x) of each pixel when curve is Cright , while the third row presents them when curve is Cwrong . Note that, the symbol x means the random value.
There are two types of values. The first one is the value of non-target cluster centers cout (x) inside the curve C. The second one is the value of target cluster centers cin (x) outside the curve C. From the Eqn.2 and Eqn.3, we find that these two types of values are none sense when calculating the local fitting energy. That is, we can set them as the random values. As shown in Fig.2, we use symbol x to represent the random value. However, these values have useful meanings in
Medical Image Segmentation Based on Novel Local Order Energy
153
practice. Such as, cin (x) means the local target cluster center, and the cout (x) means the local non-target cluster center. Moreover, these values are recursive used when perform level set evolution process. The evolution process will be described in Subsection 2.3. Or you can find the similar process in [10] in detail. In this work, these values are used to formulate the local order energy, which is a local intensity priori constraint. Actually, in a small local region of a medical image, the intensity values always have order. Such as the target intensity values are larger than the non-target ones. Thus, the target cluster center value cin (x) is larger than the non-target cluster center value cout (x) at position x. The local order information is represented by the local order energy, given by, E order C, cin , cout = sgn cout (x) − cin (x) dx (4) Ω
where sgn(.) is the sign function, sgn(x) =
1 x>0 0 Else
(5)
As shown in Eqn.4, the local order energy uses all cluster centers, which include the above two types of values. Furthermore, the local order energy, is one of the important local intensity constraint, because it restricts the local target intensity should larger than the non-target intensity statistically. Combining the local fitting energy defined in Eqn.2, we can get the new local energy E E = E local + E order
(6)
where E local and E order are defined by Eqn.2 and Eqn.4, respectively. Since the local fitting energy and the local order energy are both defined from the contour C, the final energy E is represented by the contour C too. 2.3
Level Set Formulation and Energy Minimization
Level set formulation is performed to solve the above energy E. In level set methods, a contour C is represented by the zero level set of a Lipschitz function φ : Ω → R, which is called a level set function. Accordingly, the energy of our method can be defined as 2 F φ, cin , cout = λin Kσ x − y I(y) − cin (x) H φ(y) dydx 2 + λout Kσ x − y I(y) − cout (x) 1 − H φ(y) dydx sgn cout (x) − cin (x) dx + μL(φ) + νP(φ) (7) +γ Ω
where λin , λout , γ, μ, and ν are the weighting constants, H(.) is the Heaviside function, L(φ) = δ(φ(x))|∇φ(x)|dx is the length term, δ(.) is the derivative
154
L. Wang, Z. Yu, and C. Pan
function of Heaviside function, and P(φ) = 12 (|∇φ(x)| − 1)2 dx. The term P(φ) is the regularization term [3], which serves to maintain the regularity of the level set function. In implement, Heaviside
function is approximated by a smooth function H (x) = 12 1 + π2 arctan( x ) , and the corresponding derivative function is defined by δ (x) = H (x) = π(2+x2 ) . When implementation, the coefficient is set as = 1.0. Standard gradient descent method is performed to minimize the energy function F (φ, cin , cout ) based on two sub-steps. For the first step, we fix the level set functional φ, and calculate two cluster centers cin (x) and cout (x). For the second step, we fix the two cluster centers cin (x) and cout (x), and calculate the level set functional φ. Variational method is first applied to calculate cin and cout for a fixed level set functional φ, given by, cin = max cin , cout cin , cout cin = min (8) yielding,
Kσ (x) ⊗ H φ(x) I x cin =
Kσ (x) ⊗ H φ(x)
cout
Kσ (x) ⊗ 1 − H φ(x) I x = Kσ (x) ⊗ 1 − H φ(x)
(9) where ⊗ is the convolution operation. Then, keeping cin and cout fixed, the energy functional is minimized with respect to the level set functional φ by using the standard gradient descent, given by, ∂φ = − δ (φ) λin Tin − λout Tout ∂t ∇φ ∇φ + μδ div + ν ∇2 φ − div (10) |∇φ| |∇φ| where Tin and Tout are defined as follows, 2 Tin (x) = Kσ (y − x)I(y) − cin (x) dy 2 Tin (x) = Kσ (y − x)I(y) − cin (x) dy
(11)
The more information of the implementation of the proposed local prior intensity level set method is illustrated in Alg.1 in detail.
3
Experiment Results
A number of real medical images are used to evaluate the performance of the proposed scheme by comparing the two traditional methods, i.e. Chan et al. [4] and Li et al. [10], which are the deputations of global-homogeneous and localhomogeneous methods, respectively. When implementation, the coefficients are
Medical Image Segmentation Based on Novel Local Order Energy
155
Algorithm 1. Local Intensity Priori Level Set Segmentation Data: initial level set φinitial , input image I, max iteration maxiter, and weighting constants Result: level set φ and segmentation result. 1 1 Initial the output level set function φ = φinitial ; 2 for i ← 1 to maxiter do 3 Calculate cin and cout based on φi by Eqn.8 and Eqn.9; 4 Calculate level set φi+1 based on cin and cout by Eqn.10 and Eqn.11; 5 if φi equals to φi+1 then 6 Break the iteration; 7 end 8 end 9 Saving level set φ, and segmenting the input image based on φ;
Initializations
Chanÿ s Method
Liÿ s Method
Proposed Method
Fig. 3. The comparison of our method to Chan [4] and Li [10] on CT image
empirically set as λin = 1, λout = 1, ν = 1, μ = 0.004 · h · w, where h and w are the height and width of the test image. All the medical images are downloaded from the web-site 1 and 2 . The visual comparison is first shown in Fig.1, and the more comparisons are illustrated in Fig.3, Fig.4, and Fig.5, respectively. We use three different image acquisition techniques, i.e. CT, MR, and Ultrasound, to evaluate our approach. As illustrated in these figures, although the images are rather noisy and part of the boundaries are weak, the detail information are still segmented very well comparing to the Chan’s [4] and Li’s [10] methods. Furthermore, the proposed approach is not sensitive to the initialization compared to the Li’s [10] method. 1 2
http://www.bic.mni.mcgill.ca/brainweb/ http://www.ece.ncsu.edu/imaging/Archives/ImageDataBase/Medical/index.html
156
L. Wang, Z. Yu, and C. Pan
Initializations
Chanÿ s Method
Liÿ s Method
Proposed Method
Fig. 4. The comparison of our method to Chan [4] and Li [10] on Ultrasound image
Initializations
Chanÿ s Method
Liÿ s Method
Proposed Method
Fig. 5. The comparison of our method to Chan [4] and Li [10] on MR image
The numeric comparisons are presented in Fig.6 and Fig.7. The segmentation results are compared to the ground truth by using the following two error measures: i.e. false negative (FN), which interprets number of target pixels that are missed, and false positive (FP), which interprets number of non-target pixels are marked as target. Denoting Rour target are the result of target region of by gt li our method. Similarly, Rchar , R , target Rtarget are the Chan’s, Li’s, and ground in truth. Thus, the FN and FP are defined as follows FP =
R∗target − R∗target ∩ Rgt target Rgt target
FN =
Rt argetgt − R∗target ∩ Rgt target Rgt target (12)
Medical Image Segmentation Based on Novel Local Order Energy
Scene 1
Scene 2
Scene 3
Scene 4
Chan FN Mean
FP Cov
Mean
Scene 5
Li Cov
Time(s)
FN Mean
Mean
Scene 6
Our
FP Cov
157
Time(s)
Cov
FN Mean
FP Cov
Mean
Cov
Time(s)
Scene 1
0.131 0.001 0.018 0.000
3.293
0.142 0.125 0.163 0.151
12.49
0.026 0.000 0.019 0.000
12.91
Scene 2
0.144 0.001 0.021 0.001
3.395
0.152 0.153 0.102 0.173
13.16
0.017 0.000 0.015 0.000
13.46
Scene 3
0.123 0.000 0.012 0.000
3.351
0.178 0.203 0.128 0.145
13.04
0.020 0.000 0.017 0.000
13.12
Scene 4
0.126 0.000 0.014 0.001
3.362
0.101 0.146 0.137 0.126
13.07
0.018 0.000 0.016 0.000
13.22
Scene 5
0.132 0.001 0.016 0.000
3.325
0.122 0.116 0.124 0.136
12.86
0.011 0.000 0.014 0.000
13.10
Scene 6
0.126 0.000 0.014 0.001
3.338
0.186 0.107 0.149 0.127
12.81
0.019 0.000 0.021 0.000
13.09
Fig. 6. An overview of the comparisons to Chan’s [4] and Li’s [10] by using two error measures, i.e. the (FN) and (FP)
Chan Li Ours
0.2 0.15 0.1 0.05 0
Chan Li Ours
0.25
False Positive
False Negative
0.25
0.2 0.15 0.1 0.05
0
5
10
15
20
Power of Noise(%)
25
30
35
0
0
5
10
15
20
25
30
35
Power of Noise(%)
Fig. 7. An comparisons to Chan’s [4] and Li’s [10] when adding noise to the images. The horizontal axis shows the power of noise, while the vertical axis shows two error measures, i.e. the false negative (FN) and false positive (FP).
where {∗|∗ ∈ chan, li, our}. In Fig.6, six different scenes are choosed, and for each scene, 100 experiments are performed with the different initial contour. As shown in the figure, both the means and variances of the above two error measures are smaller than the traditional methods. That is to say, the proposed method is of better accuracy (smaller error mean) and stability (smaller error variance) than the traditional ones. Furthermore, the computation cost of our approach is similar with the Li’s approach. Fig.7 shows the comparisons when adding noise to the images. As illustrated in the figure, the error of our method increase slower than the traditional methods when the power of noise enlarging. That is, the proposed method is less sensitive to the noise. We also use Chan’s, Li’s , and our methods to medical image reconstruction. The visual result of our method is shown in Fig.8. As shown in this figure, the brain tissues are preserved well. The main reason is our segmentation method gives excellent the segmentation result. The numerical comparison of reconstruction
158
L. Wang, Z. Yu, and C. Pan
Scene
3D Reconstruction Results
Frame 1
Frame 1 To 20
Frame 21 To 40
Frame 61 To 80
Frame 81 To 100
Frame 41 To 60
Frame 30
Frame 100
Fig. 8. Medical image reconstruction result by our method Table 1. The comparison of reconstruction errors Algorithm Chan’s [4] Li’s [10] Ours FN 0.161 0.143 0.019 FP 0.016 0.126 0.017
error is shown in Tab.1. The error is defined as the mean of all slice’s FN and FP, given by N N = 1 = 1 FP FN FNi FPi (13) N i=1 N i=1 where N is the slice number, the FNi , FPi are the ith slice’s FN and FP (defined in Eqn.12). As shown in this table, the our two errors are obviously smaller than the Chan’s and Li’s.
4
Conclusions and Future Works
In this work, we propose a novel region-based level set segmentation method. The main contribution of this work is proposing the local order energy, which is used to represent the local intensity priori constraint. The proposed algorithm has been shown on a number of experiments to address the medical image segmentation problem both efficiently and effectively. Meanwhile, compared to the traditional methods, the proposed method has many advantages and improvements: 1. the detail information is well segmented; 2. the main drawbacks of the traditional local-homogeneous methods are well solved; 3. the proposed method is insensitive to noise. Future works will be adding the other local intensity constraints to improve the segmentation algorithm, or adding the shape information
Medical Image Segmentation Based on Novel Local Order Energy
159
to enhance the segmentation results. Moreover, the proposed algorithm can not only be used in medical image segmentation, but also be applied on other types of images. Acknowledgement. This work was supported by the National Natural Science Foundation of China (Grant No. 60873161 and Grant No. 60975037).
References 1. Caselles, V., Kimmel, R., Sapiro, G.: Geodesic active contours. IJCV 22, 61–79 (1997) 2. Yezzi, A., Kichenassamy, S., Kumar, A., Olver, P., Tannenbaum, A.: A geometric snake model for segmentation of medical imagery. IEEE Trans. on MI 16, 199–209 (1997) 3. Li, C., Xu, C., Gui, C., Fox, M.D.: Level set evolution without re-initialization: A new variational formulation. In: CVPR, pp. 430–436 (2005) 4. Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. on IP 10, 266–277 (2001) 5. Tsai, A., Anthony Yezzi, J., Willsky, A.S.: Curve evolution implementation of the mumfordcshah functional for image segmentation, denoising, interpolation, and magnification. IEEE Trans. on IP 10, 1169–1186 (2001) 6. Vese, L.A., Chan, T.F.: A multiphase level set framework for image segmentation using the mumford and shah model. IJCV 50, 271–293 (2002) 7. Li, C., Kao, C.Y., Gore, J.C., Ding, Z.: Implicit active contours driven by local binary fitting energy. In: CVPR, pp. 1–7 (2007) 8. An, J., Rousson, M., Xu, C.: γ-convergence approximation to piecewise smooth medical image segmentation. In: Ayache, N., Ourselin, S., Maeder, A. (eds.) MICCAI 2007, Part II. LNCS, vol. 4792, pp. 495–502. Springer, Heidelberg (2007) 9. Piovano, J., Rousson, M., Papadopoulo, T.: Efficient segmentation of piecewise smooth images. In: SSVMCV, pp. 709–720 (2007) 10. Li, C., Kao, C.Y., Gore, J.C., Ding, Z.: Minimization of region-scalable fitting energy for image segmentation. IEEE Trans. on IP 17, 1940–1949 (2008) 11. Li, C., Huang, R., Ding, Z., Gatenby, C., Metaxas, D., Gore, J.: A variational level set approach to segmentation and bias correction of images with intensity inhomogeneity. In: Metaxas, D., Axel, L., Fichtinger, G., Sz´ekely, G. (eds.) MICCAI 2008, Part II. LNCS, vol. 5242, pp. 1083–1091. Springer, Heidelberg (2008) 12. Li, C., Gatenby, C., Wang, L., Gore, J.C.: A robust parametric method for bias field estimation and segmentation of mr images. In: CVPR (2009) 13. Li, C., Li, F., Kao, C.Y., Xu, C.: Image segmentation with simultaneous illumination and reflectance estimation: An energy minimization approach. In: ICCV (2009) 14. Wang, L., Macione, J., Sun, Q., Xia, D., Li, C.: Level set segmentation based on local gaussian distribution fitting. In: Zha, H., Taniguchi, R.-i., Maybank, S. (eds.) ACCV 2009. LNCS, vol. 5994, pp. 293–302. Springer, Heidelberg (2010) 15. Xiang, S., Nie, F., Zhang, C., Zhang, C.: Interactive natural image segmentation via spline regression. IEEE Trans. on IP 18, 1623–1632 (2009)
Geometries on Spaces of Treelike Shapes Aasa Feragen1 , Francois Lauze1 , Pechin Lo1 , Marleen de Bruijne1,2 , and Mads Nielsen1 1 2
eScience Center, Dept. of Computer Science, University of Copenhagen, Denmark Biomedical Imaging Group Rotterdam, Depts of Radiology & Medical Informatics, Erasmus MC – University Medical Center Rotterdam, The Netherlands {aasa,francois,pechin,marleen,madsn}@diku.dk
Abstract. In order to develop statistical methods for shapes with a tree-structure, we construct a shape space framework for treelike shapes and study metrics on the shape space. The shape space has singularities, which correspond to topological transitions in the represented trees. We study two closely related metrics, TED and QED. The QED is a quotient euclidean distance arising from the new shape space formulation, while TED is essentially the classical tree edit distance. Using Gromov’s metric geometry we gain new insight into the geometries defined by TED and QED. In particular, we show that the new metric QED has nice geometric properties which facilitate statistical analysis, such as existence and local uniqueness of geodesics and averages. TED, on the other hand, has algorithmic advantages, while it does not share the geometric strongpoints of QED. We provide a theoretical framework as well as computational results such as matching of airway trees from pulmonary CT scans and geodesics between synthetic data trees illustrating the dynamic and geometric properties of the QED metric.
1
Introduction
Trees are fundamental structures in nature, where they describe delivery systems for water, blood and other fluids, or different types of skeletal structures. Trees and graphs also track evolution, for instance in genetics [3]. In medical image analysis, treelike shapes appear when studying structures such as vessels [4, 5] or airways [6–8], see Fig. 1. In more general imaging problems, they appear in the form of shock graphs [9] describing general shapes.
Fig. 1. Treelike structures found in airways and vessels in lungs [1], in breast cancer vascularization, and in retinal blood vessels [2] R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 160–173, 2011. Springer-Verlag Berlin Heidelberg 2011
Geometries on Spaces of Treelike Shapes
161
An efficient model for tree-shape would have applications such as shape recognition and classification. In medical imaging, this could give new insight into anatomical structure and biomarkers for diseases that affect shape – e.g. COPD affects the edge shapes and topological structures of airway trees [10]. We want to do statistical analysis on sets of treelike shapes, and in order to meaningfully define averages and modes of variation in analogy with the classical PCA analysis, we need at least local existence and uniqueness of geodesics and averages. Immediate questions are then: How do we model treelike structures and their variation? Can we encode global, tree-topological structure as well as local edgewise geometry in the geometry of a single shape space? Do geodesics exist, and are they unique? Do the dynamics of geodesics reflect natural shape variation? Here we provide the theoretical foundations needed to answer these questions, accompanied by a first implementation. We define shape spaces of ordered and unordered treelike shapes, where transitions in internal tree-topological structure are found at shape space singularities in the form of self-intersections. This is a general idea, which can be adapted to other situations where shapes with varying topology are studied – e.g. graphs or 3D shapes described by medial structures. The paper is organized as follows: In Section 1.1 we give a brief overview of related work. In Section 2 we define the treespace. Using Gromov’s approach to metric geometry [11] we gain insight into the geometric properties of two different metrics; one which is essentially tree edit distance (TED) and one which is a quotient euclidean distance (QED). We pay particular respect to the properties of geodesics and averages, which are essential for statistical shape analysis. In Section 4 we discuss how to overcome the computational complexity of both metrics. In particular, the computational cost can be drastically reduced by interpreting order as an edge semi-labeling, which can often be obtained through geometric or anatomic restrictions. In Section 5 we discuss a simple QED implementation, and in Section 6 we illustrate the properties of QED through computed geodesics and matchings for synthetic planar trees and 3D pulmonary airway trees. The paper contains mathematical results, but proofs are omitted due to length constraints. 1.1
Related Work
Metrics on sets of trees have been studied by different research communities in the past 20 years. The best-known approach in the computer vision community is perhaps TED, as used for instance by Sebastian et al [9] for comparing shapes via their shock graph representations. TED performs well for tasks such as matching and registration [9]. Our goal is, however, to adapt the classical shape statistics to treelike shapes. The TED metric will nearly always have infinitely many geodesics between two given trees, and thus it is no longer sufficient, since it becomes hard to meaningfully define and compute an average shape or find modes of variation. Another approach to defining a metric on treespace is that of Wang and Marron et al [12]. They define a metric on attributed trees as well as a “median-mean”
162
A. Feragen et al.
tree, which is not unique, and a version of PCA which finds modes of variation in terms of so called treelines, encoding the maximum amount of structural and attributal variation. Wang and Marron analyze datasets consisting of brain blood vessels, which are trees with few, long branches. The metric is not suitable for studying large trees with a lot of topological variation and noise such as airways, as it punishes structural changes much harder than shape variation with constant topological structure. A rather different approach is that of Jain and Obermayer [13], who define metrics on graphs. Here, graphs are modeled as incidence matrices, and the space of graphs, which is a quotient by the group of vertex relabelings, is given a metric inherited from Euclidean space. Means are computed using Lipschitz analysis, giving fast computations; however the model is rigid when it comes to modeling topological changes, which is essential for our purposes. We have previously [14] studied geodesics between small planar embedded trees in the same type of singular shape space as studied here. These geodesics are fundamental in the sense that they represent the possible structural changes found locally even in larger trees.
2
Geometry of Treespace
Before defining a treespace and giving it a geometric structure, let us discuss which properties are desirable. In order to compute average trees and analyze variation in datasets, we require existence and uniqueness properties for geodesics. When geodesics exist, we want the topological structure of the intermediate trees to reflect the resemblance in structure of the trees being compared – in particular, a geodesic passing through the trivial one-vertex tree should indicate that the trees being compared are maximally different. Perhaps more important, we would like to compare trees where edge matching is inconsistent with tree topology as in Fig. 2a; specifically, we would like to find geodesic deformations in which the tree topology changes when we have such edge matchings, for instance as in Fig. 2b.
(a) Edge matching
(b) Geodesic candidate
Fig. 2. A good metric must handle edge matchings which are inconsistent with tree topology
2.1
Representation of Trees
We represent any treelike (pre-)shape as a pair (T , x) consisting of a rooted, planar binary tree T = (V, E, r) with edge attributes describing edge shape. Here, T describes the tree topology, and the the attributes describe the edge geometry. The attributes are represented by a map x : E → A or, equivalently, by a point
Geometries on Spaces of Treelike Shapes
163
x ∈ AE , where A is the attribute space. Tree-shapes which are not binary are represented by the maximal binary tree in a very natural way by allowing constant edges, represented by the zero scalar or vector attribute. Endowing an internal edge with a zero attribute corresponds to collapsing that edge, see Fig. 4a, and hence an arbitrary attributed tree can be represented as an attributed binary tree. The intuitive idea is illustrated in Fig. 3.
Fig. 3. Treelike shapes are encoded by an ordered binary tree and a set of attributes describing edge shape
The attributes could be edge length, landmark points or edge parametrizations, for instance. In this work, we use trees whose attributes are open curves translated to start at the origin, described by a fixed number of landmark points. Thus, thorughout the paper, the attribute space is (Rd )n where d = 2 or 3 and n is the number of landmark points per edge. Collapsed edges are represented by a sequence of origin points. We need to represent trees of different sizes and structures in a unified way in order to compare them. As a first step, we describe all shapes using the same binary tree T to encode the topology of all the tree-shapes in the treespace – by choosing a sufficiently large binary tree we can represent all the trees in our dataset by filling out with collapsed edges. We call T the maximal binary tree. When given a left-right order on every pair of children in a binary tree, there is an implicitly defined embedding of the tree in the plane. Hence we shall use the terms “planar tree” and “ordered tree” interchangingly, depending on whether we are interested in the planar properties or the order of the tree. We initially define our metric on the set of ordered binary trees; later we use it to compute distances between unordered trees by considering all possible orders. In Section 4 we discuss how to handle the computational challenges of the unordered case. depth n with edges E, any atGiven an ordered maximal binary tree Tn of tributed tree T is represented by a point in X = e∈E (Rd )n . We call X the tree preshape space, since it parametrizes all possible shapes. 2.2
The Space of Trees as a Quotient – A Singular Treespace
We fix the maximal depth n binary tree Tn which encodes the connectivity of the trees. In the preshape space defined above, there will be several points corresponding to the same tree, as illustrated in Fig. 4a. We go from preshapes to shapes by identifying those preshapes which define the same shape. Consider two ordered tree-shapes where internal edges are collapsed. The orders of the original trees induce orders on the collapsed trees. We consider two ordered tree-shapes to be the same when their collapsed ordered topological structures are identical, and the edge attributes on corresponding non-collapsed edges are identical as well, as in Fig. 4a. Thus, tree identifications come with
164
A. Feragen et al.
(a)
(b)
Fig. 4. (a) Higher-degree vertices are represented by collapsing internal edges. The dotted line indicates zero attribute, which gives a collapsed edge. We identify those tree preshapes whose collapsed structures are identical – here x1 and x2 represent the same tree T . (b) The TED moves: Remove an edge a, add an edge b, deform an edge c.
an inherent bijection of subsets of E: If we identify x, y ∈ X = (RN )E , denote E1 = {e ∈ E|xe = 0}, E2 = {e ∈ E|ye = 0}; the identification comes with an order preserving bijection ϕ : E1 → E2 identifying those edges that correspond to the same edge in the collapsed tree-shape. Note that ϕ will also correspond to similar equivalences of pairs of trees with the same topology, but other attributes. The bijection ϕ next induces a bijection Φ : V1 → V2 given by Φ : (xe ) → / E1 } and V2 = {x ∈ X|xe = 0 if e ∈ / E2 } (xϕ(e) ). Here, V1 = {x ∈ X|xe = 0 if e ∈ are subspaces of X where, except for at the axes, the topological tree structure is constant, and for x ∈ V1 , Φ(x) ∈ V2 describes the same shape as x. We define a map Φ for each pair of identified tree-structures, and form an equivalence on X by setting x ∼ Φ(x) for all x and Φ. For each x ∈ X we denote ¯ = (X/ ∼) = by x¯ the equivalence class {x ∈ X|x ∼ x}. The quotient space X {¯ x|x ∈ X} of equivalence classes x ¯ is the space of treelike shapes. The geometric interpretation of the identification made in the quotient is that we are folding and gluing our space along the identified subspaces; i.e. when x1 ∼ x2 we glue the two points x1 and x2 together. See the toy quotient space in Fig. 5 for an intuitive lower-dimensional illustration. 2.3
Metrics on Treespace
Given a metric d on Euclidean space X = e∈E RN we define the standard quo¯ = X/ ∼ by setting tient pseudometric [15] d¯ on the quotient space X k ¯ d(¯ x, y¯) = inf d(xi , yi )|x1 ∈ x ¯, yi ∼ xi+1 , yk ∈ y¯ . (1) i=1
This amounts to finding the optimal path from x¯ to y¯ passing through the identified subspaces, as shown in Fig. 5. We define two metrics on X, which come from two different ways of combining the individual edge distances: The metrics d1 and d2 on X = e∈E RN are in 2 duced by the norms x−y1 = e∈E xe −ye and x−y2 = e∈E xe − ye . From now on, d and d¯ will denote either the metrics d1 and d¯1 , or d2 and d¯2 . We have the following: ¯ which is a conTheorem 1. The distance function d¯ restricts to a metric on X, tractible, complete, proper geodesic space.
Geometries on Spaces of Treelike Shapes
165
This means, in particular, that given any two trees, we can always find a geodesic between them in both metrics d¯1 and d¯2 . Note that d¯1 is essentially the classical tree edit distance (TED) metric, where the distance between two trees is the minimal sum of costs of edge edits needed to transform one tree into another, see Fig. 4b. The d¯2 is a descent of the Euclidean metric, and geodesics in this metric are concatenations of straight lines in flat regions. We call it the QED metric, for quotient euclidean distance. In Section 3 we describe the properties of these metrics with example geodesic deformations between simple trees. Using methods from metric geometry [11] we obtain results on curvature and average trees. We consider two different types of average tree; namely the centroid, as defined and computed in [3], and the circumcenter, defined as follows: ¯ with diameter D a circumcenter of S is defined as a point Given a set S ⊂ X ¯ ¯ x, D/2) = {¯ ¯ d¯2 (¯ x ¯ ∈ X such that S ⊂ B(¯ z ∈ X| x, z¯) ≤ D/2}. ¯ has ¯ ¯∈X Theorem 2. i) Endow X with the QED metric d¯2 . A generic point x ¯ a neighborhood U ⊂ X in which the curvature is non-positive. At non-generic ¯ d¯2 ) is unbounded. points, the curvature of (X, ¯ there exists ¯ ¯ ∈ X, ii) Endow X with the QED metric d¯2 . Given a generic point x a radius rx¯ such that sets contained in the ball B(¯ x, rx¯ ) have unique centroids and circumcenters. ¯ d¯1 ) is everywhere ¯ with the TED metric d¯1 . The curvature of (X, iii) Endow X ¯ d¯1 ) does not have locally unique geodesics anywhere. unbounded, and (X, The practical meaning of Theorem 2 is that i) we can use techniques from metric geometry to look for QED averages, ii) for datasets whose scattering is not too large, there exist unique centroids and circumcenters for the QED metric, and iii) we cannot use the same techniques in order to prove existence or uniqueness of centerpoints for the TED metric; in fact, any geometric method which requires bounded curvature is going to fail for the TED metric. This result motivates our study of the QED metric. 2.4
From Planar Trees to Spatial Trees
The world is not flat, and for most applications it is necessary to study embedded trees in R3 . As far as attributes in terms of edge embeddings and landmarks are concerned, this is not very different from the planar embedded trees – the main difference from the R2 case comes from the fact that trees in R3 have no canonical edge order. The left-right order on children of planar trees gives an implicit preference for edge matchings, and hence reduces the number of possible matches. When we no longer have this preference, we generally need to consider all orderings of the same tree and choose the one which minimizes the distance. ¯ = X/G, ¯ We define the space of spatial treelike shapes as the quotient X where G is the group of reorderings of the standard tree; G is a finite group. The metric ¯ We can prove: ¯ induces a quotient pseudometric d¯ on X. d¯ on X Theorem 3. For d¯ induced by either d¯1 or d¯2 , the function d¯ is a metric and the ¯ is a contractible, complete, proper geodesic space. ¯ d) space (X,
166
A. Feragen et al.
(a)
(b)
Fig. 5. (a) A 2-dimensional toy version of the folding of Euclidean space along two linear subspaces, along with geodesic paths from x ¯ to y¯ and from x ¯ to y¯ . (b) Local¯ ¯ ¯ to-global properties of TED: d1 (T1 , T2 ) = d1 (T1,1 , T2,1 ) + d1 (T1,2 , T2,2 ).
While considering all different possible orderings of the tree makes perfect sense from the geometric point of view, in reality this becomes an impossible task as the size of the trees grow beyond a few generations. In real applications we can, however, efficiently reduce complexity by taking tree- and treespace geometry into account. This is discussed in Section 4.
3
Comparison of the QED and TED Metrics
In this section we list the main differences between the TED and QED metrics and compare their performance on the small trees studied in [14]. ¯ are ¯ and X Geometry. As shown in Theorem 1 and Theorem 3 above, both X complete geodesic spaces. However, by Theorem 2, the QED metric gives locally non-positive curvature at generic points, while the TED metric gives unbounded ¯ This means that we cannot imitate the classical stacurvature everywhere on X. tistical procedures on shape spaces using the TED metric. Note also that the QED metric is the quotient metric induced from the Euclidean metric on the preshape space X, making it the obvious choice for a metric seen from the shape space point of view. Computation. The TED metric has nice local-to-global properties, as illustrated in Fig. 5b. If the trees T1 and T2 are decomposed into subtrees T1,1 , T1,2 and T2,1 , T2,2 as in Fig. 5b such that the geodesic from T1 to T2 restricts to geodesics between T1,1 and T2,1 as well as T2,1 and T2,2 , then d(T1 , T2 ) = d(T1,1 , T2,1 ) + d(T2,1 , T2,2 ). This property is used in many TED algorithms, and the same property does not hold for the QED metric. Performance. We have previously [14] studied fundamental geodesic deformations for the QED metric; that is, deformations between depth 3 trees which encode the local topological transformations found in generic tree-geodesics. These deformations are important for determining whether an internal tree-topological transition is needed to transform one (sub)tree into another – i.e. whether the correct edge registration is one which is inconsistent with the tree topology.
Geometries on Spaces of Treelike Shapes
(a)
167
(b)
Fig. 6. (a) The fundamental geodesic deformations: geodesic 1 goes through an internal structural change, while geodesic 2 does not, as the disappearing branches approach the zero attribute. (b) Two options for structural change.
To compare the TED and QED metric on small, simple trees, consider the two tree-paths in Fig. 6b, where the edges are endowed with non-negative scalar attributes a, b, c, d, e describing edge length. Path 1 indicates a matching of the identically attributed edges c and d, while Path 2 does not make the match. Now, √ the cost of Path 1 is 2e in both metrics, while the cost of Path 2 is 2 c2 + d2 in the QED metric and 2(c + d) in the TED metric. In particular, TED will chose to identify the c and d edges whenever e2 ≤ c2 + 2cd + d2 , while QED makes the match whenever e2 ≤ (c2 + d2 ). That is, TED will be more prone to internal structural changes than QED. This is also seen empirically in the comparison of TED and QED matching in Fig. 7. Note that although the TED is more prone to matching trees with different tree-topological structures, the matching is similar.
(a) Matching in the QED metric.
(b) Matching in the TED metric.
Fig. 7. Given a set of five data trees, we match each to the four others in both metrics
4
Computation and Complexity
Complexity is a problem with computing both TED and QED distances, in particular for 3D trees, which do not have a canonical planar order. Here we discuss how to use geometry and anatomy to find approximations of the metric whose complexity is significantly reduced.
168
A. Feragen et al.
Ordered trees: Reducing complexity using geometry. The definition of d¯ in (1) opens for considering infinitely many possible paths. However, we can significantly limit the search for a geodesic by taking the geometry of treespace into account. For instance, it is enough to consider paths going through each structural transition at most once: Lemma 1. For the metrics d¯1 and d¯2 on X, the shortest path between two points ¯ passes through each identified subspace at most once. in X Another way of limiting the complexity of real computations is to find a generic shape structure. We show that locally in the geodesic, the only internal topological transitions found will be those described by the fundamental geodesic deformations in Section 3. Theorem 4. i) Treelike shapes that are truly binary (i.e. their internal edges are not collapsed) are generic in the space of all treelike shapes. ii) The tree-shape deformations whose local structure is described by the fundamental geodesic deformations in Fig. 6a are generic in the set of all tree deformations. Moreover, the set of pairs of trees whose geodesics are of this form, is generic in the space of all pairs of trees. Genericity basically means that when a random treelike shape is selected, the probability of that shape being truly binary is 1. This does not mean, however, that non-binary trees do not need to be considered. While non-binary trees do not appear as randomly selected trees, they do appear in paths between randomly selected pairs of trees, as in Fig. 6a above. This also means that we can, in fact, work on a less complicated semi-preshape-space where we, instead of identifying all possible representations of the same shape, restrict ourselves to making the identifications illustrated by Fig. 6a. This is similar to the generic form of shock graphs and generic transitions between shock graphs found by Giblin and Kimia [16]. The notions of genericity in the two settings are, however, different since in [16], genericity is defined with respect to the Whitney topology on the space of the corresponding shape boundary parametrizations. Although we do run into non-binary treelike shapes in real-life applications, for instance when studying airway trees, this can be interpreted as an artifact of resolution rather than as true higher-degree vertices. Indeed, the airway extraction algorithms record trifurcations when the relative distances are below certain treshold values. Finally, it is often safe to assume that only very few structural changes will actually happen. For the airway trees studied in Section 6.2, we find empirically that it is enough to allow for one structural change in each depth 3 subtree. Using these arguments, it becomes feasible to computationally handle semilarge structures. In many medical imaging applications, such as airways or blood vessels, we are not interested in studying tree-deformations between very large networks or trees, since beyond a certain point, the tree-structure stops following a predetermined pattern and becomes a stochastic variable.
Geometries on Spaces of Treelike Shapes
169
Unordered trees: Reducing complexity using geometry- or anatomybased semi-labeling schemes. It is well known that the general problem of computing TED-distances between unordered trees is NP-complete [17], and the QED metric is probably generally not less expensive to compute, as indicated also by Theorem 3. Here we discuss how to use geometry and anatomy to find approximations of the metric whose complexity is significantly reduced. In particular, trees appearing in applications are usually not completely unordered, but are often semi-labeled. Semi-labelings can come from geometric or anatomical properties as in the pulmonary airway trees studied in Example 1 below, or may be obtained by a coarser registration method. (Semi-)labelings can also come from a TED distance computation or approximation, which is a reasonable way to detect approximate structural changes since the TED and QED give similar matchings, as seen in Fig. 7. Example 1 (Semi-labeling of the upper airway tree). In Fig. 9a, we see a “standard” airway tree with branch labels (or at least the first generations of it). Most airway trees, as the one shown to the left in Fig. 1, have similar, but not necessarily identical, topological structure to this one, and several branches, especially of low generation, have names and can be identified by experts. The top three generations of the airway tree serve very clear purposes in terms of anatomy. The root edge is the trachea; the second generation edges are the left and right bronchi; and the third generation edges lead to the top and towards the middle and lower lung lobes on the left, and to the top and bottom lobes on the right. Thus we find a canonical semi-labeling of the airway tree, which we use to simplify the problem of computing airway distances in Section 6.2.
5
QED Implementation
We explain a simple implementation of the QED metric. As shown in the next sections, our results are promising – the geodesics for synthetic data show the expected correspondences between edges, and in our experiments on 6 airway trees extracted from CT scans of 3 patients made at two different times, we recognize the correct patient in 5 out of 6 cases. Alignment of trees. We translate all edges to start at 0. We do not factor out scale in the treespace, because in general, we consider scale an important feature of shape. Edge scale in particular is a critical property, as the dynamics of appearing and disappearing edges is directly tied to edge size. Edge shape comparison. We represent each edge in a tree by a fixed number of landmark points – in our case 6 – evenly distributed along the edge, the first one at the origin (and hence neglected). The distance between two edge attributes v1 , v2 ∈ (Rd )5 is defined as the Euclidean distance between them. Although simple, this distance measure does take the scale of edges into account to some degree since they are aligned.
170
A. Feragen et al.
Fig. 8. Different options for structural changes in the left hand side of the depth 3 tree. Topological illustration on the left, corresponding tree-shape illustration on the right.
Implementation. In ordered depth 3 trees we make an implementation using Algorithm 1, where the number of structural changes taking place in the whole tree is limited to either 1 or 2, and in the final case we also rule out the case where the two changes happen along the same “stem”. This leaves us with the options for structural changes illustrated in Fig. 8, applied to the left and right half tree. The complexity of Algorithm 1 is O(nk · C(n, k)), where n is the number of internal vertices, k is the number of structural changes and C(n, k) is the maximal complexity of the optimization in line 8. Algorithm 1. Computing geodesics between ordered, rooted trees with up to k structural changes 1: x, y planar rooted depth n binary trees 2: S = {S} set of ordered identified pairs S = {S1 , S2 } of subspaces of X corresponding to internal topological changes, corresponding to a subspace S ¯ s.t. if {S1 , S2 } ∈ S, then also {S2 , S1 } ∈ S. of X, ˜ ≤ k do 3: for S˜ = {S 1 , . . . , S s } ⊂ S with |S| i i 4: for p ∈ S with representatives pi1 ∈i S1 and pi2 ∈ S2i do 5: p = (p1 , p2 , . . . , ps ) 2 i 6: f (p) = min{d2 (x, p1 ) + s−1 j=1 ds (pi , pi+1 ) + d2 (ps , y)} 7: end for 8: dS˜ = min f (p)|p = (p1 , . . . , ps ), pi ∈ S i , S˜ = {S 1 , . . . , S s } 9: 10: 11: 12: 13: 14:
6
pS˜ = {pi1 , pi2 }si=1 = argminf (p) end for ˜ ≤ k} d = min{dS |S˜ ⊂ S, |S| p = {p1 , p2 }si=1 = {pS˜ |dS˜ = d} geodesic = g = {x → p11 ∼ p12 → p21 ∼ p22 → . . . → ps1 ∼ ps2 → y} return d, g
Experimental Results
The QED metric is new, whereas the matching properties of the TED metric are well known [9]. In this section we present experimental results on real and synthetic data which illustrate the geometric properties of the QED metric. The experiments on airway trees in Section 6.2 show, in particular, that it is feasible to compute the metric distances on real 3D data trees.
Geometries on Spaces of Treelike Shapes
6.1
171
Synthetic Planar Trees of Depth 3
We have uploaded movies illustrating geodesics between planar depth 3 trees, as well as a matching table for a set of planar depth 3 trees, to the webpage http://image.diku.dk/aasa/ACCVsupplementary/geodesicdata.html. Note the geometrically inutitive behaviour of the geodesic deformations. 6.2
Results in 3D: Pulmonary Airway Trees
We also compute geodesics between subtrees of pulmonary airway trees. The airway trees were first segmented from low dose screening computed tomography (CT) scans of the chest using a voxel classification based airway tree segmentation algorithm by Lo et al. [18]. The centerlines were then extracted from the segmented airway trees using a modified fast marching algorithm based on [19], which was originally used in [18] for measuring the length of individual branches. Since the method also gives connectivity of parent and children branches, a tree structure is obtained directly. Leaves with a volume less than 11 mm3 were assumed to be noise and pruned away, and the centerlines were sampled with 5 landmark points on each edge. Six airways extracted from CT scans taken from three different patients at two different times were analyzed, restricting to the first six generations of the airway tree. The first three generations were identified and labeled as in Ex. 1, leaving us with four depth 3 subtrees representing the 4th to 6th generations of a set of 3D pulmonary airway trees. Algorithm 1 was run with both one and two structural changes, and no subtree geodesics (out of 24) had more than one structural change. Thus, we find it perfectly acceptable to restrict our search to paths with few structural changes.
(a)
(b)
Fig. 9. (a) Standard airway tree with branch labels. Drawing from [6] based on [20]. (b) Table of sum of QED distances for the five airway subtrees for 6 airway trees retrieved from CT scans of three patients taken at two different times, denoted P(a, i) where a denotes patient and i denotes time.
In the depth 3 subtrees we expect the deformation from one lung to the other to include topological changes, and we compare them using the unordered QED metric implementation on the subtrees which are cut off after the first 3 generations. As a result we obtain a deformation from the first depth 6 airway tree to
172
A. Feragen et al.
the second, whose restriction to the subtrees is a geodesic. In this experiment we study four airways originating from CT scans of two different patients acquired at two different times, and we obtain the geodesic distances and corresponding matching shown in Fig. 9b. The five subtree distances (top three generations and four subtrees) were added up to give the distance measure in Fig. 9b, and we see that for 5 out of 6 images, the metric recognizes the second image from the same patient.
7
Conclusions and Future Work
Starting from a purely geometric point of view, we define a shape space for treelike shapes. The intuitive geometric framework allows us to simultaneously take both global tree-topological structure and local edgewise geometry into account. We define two metrics on this shape space, QED and TED, which give the shape space a geodesic structure (Theorems 1 and 3). QED is the geometrically natural metric, which turns out to have excellent geometric properties which are essential for statistical shape analysis. In particular, the QED metric has local uniqueness of geodesics and local existence and uniqueness for two versions of average shape, namely the circumcentre and the centroid (Theorem 2). TED does not share these properties, but has better computational properties. Both metrics are generally NP hard to compute for 3D trees. We explain how semi-labeling schemes and geometry can be used to overcome the complexity problems, and illustrate this by computing QED distances between trees extracted from pulmonary airway trees as well as synthetic planar data trees. Our future research will be centered around two points: Development of nonlinear statistical methods for the singular tree-shape spaces, and finding fast algorithms for an approximate QED metric. The latter will allow us to actually compute averages and modes of variation for large, real 3D data trees. Acknowledgements. This work is partly funded by the Lundbeck foundation, the Danish Council for Strategic Research (NABIIT projects 09−065145 and 09− 061346) and the Netherlands Organisation for Scientific Research (NWO). We thank Dr. J.H. Pedersen from the Danish Lung Cancer Screening Trial (DLCST) for providing the CT scans. We thank Marco Loog, Søren Hauberg and Melanie Ganz for helpful discussions during the preparation of the article.
References 1. Pedersen, J., Ashraf, H., Dirksen, A., Bach, K., Hansen, H., Toennesen, P., Thorsen, H., Brodersen, J., Skov, B., Døssing, M., Mortensen, J., Richter, K., Clementsen, P., Seersholm, N.: The Danish randomized lung cancer CT screening trial - overall design and results of the prevalence round. J. Thorac. Oncol. 4, 608–614 (2009)
Geometries on Spaces of Treelike Shapes
173
2. Staal, J., Abramoff, M., Niemeijer, M., Viergever, M., van Ginneken, B.: Ridge based vessel segmentation in color images of the retina. TMI 23, 501–509 (2004) 3. Billera, L.J., Holmes, S.P., Vogtmann, K.: Geometry of the space of phylogenetic trees. Adv. in Appl. Math. 27, 733–767 (2001) 4. Charnoz, A., Agnus, V., Malandain, G., Soler, L., Tajine, M.: Tree matching applied to vascular system. In: Brun, L., Vento, M. (eds.) GbRPR 2005. LNCS, vol. 3434, pp. 183–192. Springer, Heidelberg (2005) 5. Chalopin, C., Finet, G., Magnin, I.E.: Modeling the 3D coronary tree for labeling purposes. MIA 5, 301–315 (2001) 6. Tschirren, J., McLennan, G., Pal´ agyi, K., Hoffman, E.A., Sonka, M.: Matching and anatomical labeling of human airway tree. TMI 24, 1540–1547 (2005) 7. Kaftan, J., Kiraly, A., Naidich, D., Novak, C.: A novel multi-purpose tree and path matching algorithm with application to airway trees. In: SPIE, vol. 6143 I (2006) 8. Metzen, J.H., Kr¨ oger, T., Schenk, A., Zidowitz, S., Peitgen, H.-O., Jiang, X.: Matching of tree structures for registration of medical images. In: Escolano, F., Vento, M. (eds.) GbRPR. LNCS, vol. 4538, pp. 13–24. Springer, Heidelberg (2007) 9. Sebastian, T.B., Klein, P., Kimia, B.: Recognition of shapes by editing their shock graphs. TPAMI 26, 550–571 (2004) 10. Nakano, Y., Muro, S., Sakai, H., Hirai, T., Chin, K., Tsukino, M., Nishimura, K., Itoh, H., Par´e, P.D., Hogg, J.C., Mishima, M.: Computed tomographic measurements of airway dimensions and emphysema in smokers. Correlation with lung function. Am. J. Resp. Crit. Care Med. 162, 1102–1108 (2000) 11. Gromov, M.: Hyperbolic groups. In: Essays in group theory. Math. Sci. Res. Inst. Publ., vol. 8, pp. 75–263 (1987) 12. Wang, H., Marron, J.S.: Object oriented data analysis: sets of trees. Ann. Statist. 35, 1849–1873 (2007) 13. Jain, B.J., Obermayer, K.: Structure spaces. J. Mach. Learn. Res. 10, 2667–2714 (2009) 14. Feragen, A., Lauze, F., Nielsen, M.: Fundamental geodesic deformations in spaces of treelike shapes. In: Proc ICPR (2010) 15. Bridson, M.R., Haefliger, A.: Metric spaces of non-positive curvature. Springer, Heidelberg (1999) 16. Giblin, P.J., Kimia, B.B.: On the local form and transitions of symmetry sets, medial axes, and shocks. IJCV 54, 143–156 (2003) 17. Zhang, K., Statman, R., Shasha, D.: On the editing distance between unordered labeled trees. Inf. Process. Lett. 42, 133–139 (1992) 18. Lo, P., Sporring, J., Ashraf, H., Pedersen, J.J., de Bruijne, M.: Vessel-guided airway tree segmentation: A voxel classification approach. Medical Image Analysis 14, 527–538 (2010) 19. Schlath¨ olter, T., Lorenz, C., Carlsen, I.C., Renisch, S., Deschamps, T.: Simultaneous segmentation and tree reconstruction of the airways for virtual bronchoscopy. In: SPIE, vol. 4684, pp. 103–113 (2002) 20. Boyden, E.A.: Segmental anatomy of the lungs. McGraw-Hill, New York (1955)
Human Pose Estimation Using Exemplars and Part Based Refinement Yanchao Su1 , Haizhou Ai1 , Takayoshi Yamashita2 , and Shihong Lao2 1
Computer Science and Technology Department, Tsinghua, Beijing 100084, China 2 Core Technology Center, Omron Corporation, Kyoto 619-0283, Japan
Abstract. In this paper, we proposed a fast and accurate human pose estimation framework that combines top-down and bottom-up methods. The framework consists of an initialization stage and an iterative searching stage. In the initialization stage, example based method is used to find several initial poses which are used as searching seeds of the next stage. In the iterative searching stage, a larger number of body parts candidates are generated by adding random disturbance to searching seeds. Belief Propagation (BP) algorithm is applied to these candidates to find the best n poses using the information of global graph model and part image likelihood. Then these poses are further used as searching seeds for the next iteration. To model image likelihoods of parts we designed rotation invariant EdgeField features based on which we learnt boosted classifiers to calculate the image likelihoods. Experiment result shows that our framework is both fast and accurate.
1
Introduction
In recent year human pose estimation from a single image has become an interesting and essential problem in computer vision domain. Promising applications such as human computer interfaces, human activity analysis and visual surveillance always rely on robust and accurate human pose estimation results. But human pose estimation still remains a challenging problem due to the dramatic change of shape and appearance of articulated pose. There are mainly two categories of human pose estimation methods: Topdown methods and bottom-up methods. Top-down methods, including regression based methods and example based method, concern of the transformation between human pose and image appearance. Regression based methods [1, 2] learns directly the mapping from image features to human pose. Example based methods [3–7] find a finite number of pose examplars that sparsely cover the pose space and store image features corresponding to each examplar, the resultant pose is obtained through interpolation of the n closest examplars. Bottom-up methods, which are mainly part based methods [8–12], divide human body into several parts and use graph models to characterize the whole body. First several candidates of each part are found using learned part appearance model and then the global graph models are used to assemble each part into a whole body. R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 174–185, 2011. c Springer-Verlag Berlin Heidelberg 2011
Human Pose Estimation Using Exemplars and Part Based Refinement
175
Among all these works, top-down methods are usually quick in speed but their performances rely on the training data. Bottom up methods are more accurate but the exhaustive search of body parts is too time consuming for practical applications. Inspired by the works of Active Shape Model (ASM) [13], we find that iterative local search could give accurate and robust result of shape alignment with acceptable computation cost given a proper initialization. We design a novel iterative inferring schema for bottom-up part based pose estimation. And to model the image likelihood of each part, we designed rotation invariant EdgeField features based on which we learn boosted classifiers for each body part. To provide initial poses to the part based method, we utilize a sparse example based method that gives several best matches with little time cost based on the cropped human body image patch given by a human detector. The combination of top-down and bottom-up methods enables us to achieve accurate and robust pose estimation, and the iterative inferring schema boosts the speed of pose inference with no degeneracy on accuracy. The rest of this paper is organized as follows: section 2 presents an overview of our pose estimation framework, section 3 describes the top-down example based stage, and section 4 is the bottom-up iterative searching stage. Experiments and conclusions are then given in section 5 and section 6.
2
Overview of Our Framework
A brief flowchart of our framework is given in Fig. 1.
6HDUFKLQJ &URSSHG 6HDUFKLQJ 6HDUFKLQJ 6HHGV ,PDJH &DQGLGDWHV 6HHGV ([DPSOHEDVHG *HQHUDWH %3LQIHUHQFH &RYHUDJH" LQLWLDOL]DWLRQ FDQGLGDWHV
5HVXOW
Fig. 1. Examples in dataset 2
The pose estimation framework uses the cropped image patch given as input and consists of two main stages: the initialization stage and the refining stage. In the initialization stage, a sparse top-down example based method is used to find the first n best matches of the examplars, the poses of these examplars are used as the searching seeds in the searching stage. In the refining stage, we utilize a set of boosted body part classifier to measure the image likelihood and a global tree model to restrict the body configuration. Inspired by the works of Active Shape Model, we design an iterative searching
176
Y. Su et al.
algorithm to further refine the pose estimation result. Unlike ASM which finds the best shape in each iteration, we find the first n best poses using BP inference on the tree model and use them as searching seeds for the next iteration. The refined pose estimation result is given when the iteration converges.
3
Example Based Initialization
The objective of example based initialization stage is to provide several searching seeds for the part based refinement stage, so this stage has to be fast and robust. In some previous example-based methods[3, 4, 7], a dense cover of pose space which uses all the training samples as examplars is adopted to achieve better performance, but in our case, a sparse examplar set is enough to get initial poses for the refining stage. We use the Gaussian Process Latent Variable Model (GPLVM) [14] to obtain a low dimensional manifold and use the active set of GPLVM as examplars. An illustration of the GPLVM model and the examplars are given in Fig. 2.
Fig. 2. Illustration of GPLVM and examplars
The adopted image feature is an important aspect of example-based methods. In many previous works, silhouettes of human body are usually used as the templates of examplars. But in practice, it is impossible to get accurate silhouettes from a single image with clutter background. In [3] the author adopts histograms of oriented gradients (HOG) [15] as the image descriptor to characterize the appearance of human body. However, there are two main limitations of using HOG descriptor in a tight bounding box: firstly, the bounding box of human varies dramatically when the pose changes, secondly, a single HOG descriptor isnt expressive enough considering the furious change of the appearance of human body in different pose. So we first extend the tight bounding rectangle to a square, and then divide the bounding square equally into 9 sub-regions,
Human Pose Estimation Using Exemplars and Part Based Refinement
177
HOG descriptors in the sub-regions are concatenated into the final descriptor. And for each examplar, we assigned a weight to each sub-region which represents the percentage of foreground area in the sub-region. Given an image and the bounding box, we first cut out a square image patch based on the bounding box, and then HOGs of all the sub-regions are computed. The distance between the image and an examplar is given by the weighted sum of the Euclid distances of HOGs in each sub-region. We find the first n examplars with shortest distance as the initial poses.
4 4.1
Part Based Refinement Part Based Model
It is natural that human body can be decomposed into several connected parts. Similar with previous works, we model the whole body as a Markov network as shown in Fig. 3(left). The whole body is represented as a tree structure in which each tree node corresponds to a body part and is related to its adjacent parts and its image observation. Human pose can be represented as the configuration of the tree model: L = {l1 , l2 , ..., ln }, and the state of each part is determined by the parameter of its enclosing rectangle: li = {x, y, θ, w, h} (Fig. 3(right)). Each part is associated with an image likelihood, and each edge of the tree between connected part i and part j has an associated, that encodes the restriction of configuration of part i and potential function part j.
xi
pj
pi
ߠ
ߠ
( x, y) h
xj
w
VWDWH REVHUYDWLRQ
Fig. 3. Left: Markov network of human body. Middle:the constraint between two connected parts. Right: the state of a body part.
As shown in Fig. 3(middle), the potential function has two components corresponding to the angle and the position: p θ (li , lj ) + ψij (li , lj ) ψij (li , lj ) = ψij
(1)
The constraint of angle is defined by the Mises distribution: θ (li , lj ) = ek cos(θ−μ) ψij
(2)
178
Y. Su et al.
And the potential function on position is a piece-wise function: p (li , lj ) = { ψij
2
αe−β||pi −pj || , ||pi − pj ||2 < t 0 , ||pi − pj ||2 ≥ t
(3)
The parameter α, β can be learnt from the ground truth data. The piece-wise definition of the potential function not only guarantees the connection between adjacent parts, but also can speed up the inference procedure. The body pose can be obtained by optimizing: p(L|Y ) ∝ p(Y |L)p(Y ) = ψi (li ) ψij (li , lj ) (4) i
4.2
i∈Γ (p)
Image Likelihood
As in previous works, body silhouettes are the most direct cue to measure the image likelihood. However, fast and accurate silhouettes extraction is another open problem. As proven in previous works, boosting classifiers with gradient based features such as HOG feature [15] is an effective way of measuring image likelihood in pedestrian detection and human pose estimation. However, in human pose estimation, the orientation of body parts change dramatically when the pose changes, so when measuring image likelihood of body parts we need to rotate the images to compute the HOG feature values. This time consuming image rotation can be eliminated when using a kind of rotation invariant features. Based on this point, we designed an EdgeField feature. Given an image, we can take each pixel as a charged particle and then the image as a 2D field. Following the definition of electric field force, we can define the force between two pixels p and q: kdp · dq r (5) ||r||3 Where dp , dq are the image gradient on pixel p, q. and r is the vector pointing from p to q. the amplitude of the force is in proportion to the gradient amplitude of p and q, and the similarity between the gradient of p and q, and is in inverse proportion to the square of the distant between p and q. Given a template pixel p and an image I, we can compute the force between the template pixel and the image by summing up the force between the template pixel and each image pixel: f (p, q) =
f (p, I) =
kdp · dq q∈I
||r||3
r
(6)
By applying orthogonal decomposition on f(p,I), we can get: f (p, I) =
f (p, I) · ex f (p, I) · ey
=k
dp · q∈Γ (p) dp · q∈Γ (p)
rq ·ex ||rq ||3 rq ·ey ||rq ||3
=
dp · Fx (p) dp · Fy (p)
= dp F (p) (7)
Human Pose Estimation Using Exemplars and Part Based Refinement
179
Where ex and ey are the unit vector of x and y directions. And F (p) is a 2 by 2 constant matrix in each position p in image I. we call F (p) the EdgeField. Our EdgeField feature is define as a short template segment of curve composed of several points with oriented gradient. The feature value is defined as the projection of the force between the template segment and the image to a normal vector l: f (S, I) = F (p, I) · l = dTp F (p)l (8) p∈S
p∈S
Obviously, the calculation of the EdgeField feature is rotation invariant. To compute the feature value of an oriented EdgeField feature, we only need to rotate the template. The EdgeField can be precomputed from the image by convolution. An example of EdgeField and its contour line are shown in Fig. 4 left and middle. Fig. 4 right shows an example of EdgeField feature.
dpi pi f l
f(S,I)
S I
Fig. 4. Left: EdgeField; Middle: Contour line of EdgeField; Right: EdgeField feature
Based on the EdgeField feature, we can learn a boosted classifier for each body part. We use the improved Adaboost algorithm [14] that gives real-valued confidence to its predictions which can be used as the image likelihood of each part. The weak classifier based on feature S is defined as follow: h(Ii ) = lutj , vj < f (S, Ii ) ≤ vj+1
(9)
Where the range of feature value is divided into several sub ranges{(vj , vj+1 )}, and the weak classifier output a certain constant confidence luti in each sub range. A crucial problem in learning boosted classifier with EdgeField feature is that the number of different EdgeField features is enormous and exhaustive search in conventional boosting learning becomes impractical. So we design a heuristic searching procedure as follow:
180
Y. Su et al.
Algorithm 1. Weak classifier selection for the ith part Input: samples {Iij }, labels of the sample{yij } and the corresponding weight{Dij } 1. Initialize Initialize the open feature list: OL = {S1 , S2 , . . . , Sn } and the close feature list: CL = φ 2. Grow and search: for i = 1 to T lost from OL: Select the feature S ∗ with the lowest classification S ∗ = argmins∈OL (Z(S)) where Z(S) = i Di exp(−yi h(Ii )) OL = OL − {S ∗ }, CL = CL ∪ {S ∗ } Generate new features based on S∗ under the constraint in Fig. 5, then learn parameters and calculate the classification lost and put them in OL. end for 3. Output: Select the feature with the lowest classification loss for OL and CL
This algorithm initializes with EdgeField features with only one template pixel, and generates new features based on current best feature under the growing constraint which keep the template segment of the feature simple and smooth.
Fig. 5. Growing constraint of features: Left: the second pixel (gray square) can only be placed at 5 certain directions from the first pixel (black square); Middle: The following pixels can only be placed in the proceeding direction of current segment and in the adjacent directions. Right: the segment cannot cross the horizontal line crossing the first pixel.
When the position of each pixel p in the feature is given, we need to learn the parameters of the feature: the projection vector l and the gradient of each pixel dp. The objective is to best separate the positive and negative samples. Similar with common object detection task, the feature values of positive samples are surrounded by those of negative samples, so we take the Maximal-RejectionClassifier criterion [18] which minimizes the positive scatter while maximizing the negative scatter: ({dpi , l}) = argminRpos /Rneg
(10)
Where Rpos and Rneg are the covariance of the feature values of positive sample and negative samples respectively.
Human Pose Estimation Using Exemplars and Part Based Refinement
181
Since it is hard to learning l and dp simultaneously, we first discretize l into 12 directions, and learn {dp } for each l. Given l, the feature value can be computed as follow:
dTpi F (pi )l = dTp1 , dTp2 , . . . , dTpn (F (p1 )l, F (p2 )l, . . . , F (pn )l) (11) f (S, I) = pi ∈S
Then the learning of {dp } becomes a typical MRC problem where dTp1 , . . . , dTpn is taken as the projection vector. Following [16], we adopt weighted MRC to take the sample weights into account. Given the boosted classifier of each part, the image likelihood can be calculated as follow: hk , i(Ii )))−1 (12) ψi (li ) = (1 + exp(−Hi (Ii )))−1 = (1 + exp(− k
4.3
Iterative Pose Inference
Belief propagation (BP)[17] is a popular and effective inference algorithm of tree structured model inference. BP calculates the desired marginal distributions p(li—Y) by local message passing between connected nodes. At iteration n, each node t ∈ V calculates a message mnts (lt ) that will be passed to a node s ∈ Γ (t): n ψst (ls , lt )ψt (lt ) × mn−1 (13) mts (ls ) = α ut (ls )dlt lt
u∈Γ (s)
Where α = 1/ mnts (ls )is the normalization coefficient. The approximation of the marginal distribution p(ls |Y ) can be calculated in each iteration: mnut (ls ) (14) pˆ(ls |Y ) = αψt (lt ) u∈Γ (s)
In tree structured models, pˆ(ls |Y ) will converge to p(ls |Y ) when the messages are passed to every nodes in the graph. In our case the image likelihood is discontinuous so that the integral in equation (13) becomes product over all feasible states lt. Although the present of the BP inference reduces the complexity for pose inference, the feasible state space of each part remains unconquerable by exhaustive search. Inspired by the work of Active Shape Model (ASM), we designed an iterative inference procedure which limits each part in the neighborhood of several initial states and inference locally in each iteration. The whole iterative inference algorithm is shown in algorithm 2. In each iteration, the searching seeds are the initial guesses of the states of each part, and we sample uniformly from the neighborhood of each searching seeds to generate searching candidates. And then BP inference is applied over the searching candidates to calculate the marginal distribution over the candidates. And the searching seeds for next iteration are selected from the candidates using MAP according to the marginal distribution.
182
Y. Su et al.
Unlike ASM which uses the reconstructed shape as the initial shape for the local search stage, we keep the first n best candidates as the searching seeds for the next iteration and searching candidates are generated based on the searching seeds. Preserving multiple searching seeds reduce the risk of improper initial pose.
Algorithm 2. Iterative pose inference Input: searching seeds for each part Si = {li1 , li2 , . . . , lin } Generating candidates: Ci = {li |li ∈ N eighborhood(lki ), lki ∈ Si } For each part i: Calculate the image likelihood ψi (li )using boosted classifiers BP inference: Loop until convergence: Calculate messages: For each part s and the messages
t, t ∈ Γ (s) caculate n mts = α lt ∈Ct ψst (ls , lt )ψt (lt ) × u∈Γ (s) mn−1 ut (ls) n Calculate belief: pˆ(ls |Y ) = αψt (lt ) u∈Γ (s) mut (ls ) Find searching seeds for each part i: Si =candidates with the first n largest belief values Go to generating candidates if Si do not converge The estimate pose L = {Si }
5 5.1
Experiments Experiment Data
Our experiments are taken on following dataset: 1. Synthesized dataset: A dataset of 1000 images synthesized using Poser 8. Each image contains a human and the pose is labeled manually. And the backgrounds of these images are the images selected randomly selected from other datasets. 2. The Buffy dataset from [11]: This dataset contains 748 video frames over 5 episodes of the fifth season of Buffy: the vampire slayer. The upper body pose (including head, torso, upper arms and lower arms) are labeled manually in each image. This is a challenging dataset due to the changing of poses, different clothes and strongly varying illumination. As in [11, 12], we use 276 images in 3 episodes as the testing set, and the rest 472 as training set. 3. The playground dataset: This dataset contains 350 images of people play football and basketball in the playground. The full body pose is labeled manually. 150 images from this data set are used as the training set, and the rest 200 is the testing set. 4. The iterative image parsing dataset from[11]: this dataset contains 305 images of people engaged in various activities such as standing, performing exercises, dancing, etc. the full body pose in each image is annotated manually. This dataset is more challenging because there is no constraint on human poses, clothes, illumination and backgrounds. 100 images in this dataset are used as the training set and the rest 205 is the testing set.
Human Pose Estimation Using Exemplars and Part Based Refinement
0.6 0.55
Our descriptor HOG descriptor
0.85
Mean Error
Mean Error
0.7 0.65
1
0.9
Our descriptor HOG descriptor
Mean Error
0.8 0.75
0.8 0.75 0.7
0
5 10 15 number of selected examplars
20
0.65
183
0
5 10 15 number of selected examplars
20
Our descriptor HOG descriptor
0.9 0.8 0.7
0
5 10 15 number of selected examplars
20
Fig. 6. Mean minimum errors of selected examplars in dataset 2,3,4 (from left to right)
5.2
Experiments of Examplar Based Initialization
To learn the examplars for full body pose estimation, we used the whole dataset 1 to learn examplars. The coordinates of 14 joint point of each image are connected as a pose vector; GPLVM is used to learn a low dimensional manifold of the pose space. The active set of GPLVM is taken as the examplars and the corresponding descriptors are stored. The upper body examplars are learnt similarly by using the bounding box of upper body. Given an image and a bounding box of human, we generate 100 candidate bounding box by adding random disturbance to the bounding box. And for each examplar, we find the best candidate bounding box with the shortest distance of the descriptor, and the first 20 examplars with the shortest distance are used as the initial pose in the next stage. We measures the accuracy of this stage on the datasets in terms of the mean of minimum distance of corresponding joint points divided by the part length between ground truth and the top n selected examplars, as shown in Fig. 6. 5.3
Experiments of Pose Estimation
For the part classifier, in the training sets, we cut out image patches according to the bounding box of each part and normalize them into the same orientation and scale and use them as positive samples. Negative samples are obtained by adding large random disturbance to the ground truth states of each part. In the pose estimation procedure, the top 20 matched examplars are used as the searching seeds. 100 searching candidates of each part are generated based on each searching seeds. During each iteration, the candidates with top 20 posterior marginal possibilities are preserved as the searching seeds for the next iteration. To compare with previous methods, we used the code of [12]. The images are cut out based on the bounding box (with some loose), and then are processed by the code. The correctness of each part in the testing sets of dataset 2,3,4 are listed in table 1. A part is correct if the distance between estimated joint points and ground truth is below 50% of the part length. From the result we can see that our method can achieve better correctness with much less time cost. Fig. 7 give some examples in dataset 2,3,4.
184
Y. Su et al. Table 1. Correctness of parts in dataset 2,3,4
part
dataset 2 Method Our in [12] method
dataset 3 Method Our in [12] method
Torso 91% 93% 81% Upper Arm 82% 83% 83% 85% 45% 51% Lower Arm 57% 59% 61% 64% 32% 31% Upper Leg 54% 61% Lower Leg 43% 45% Head 96% 95% 81% Total 78% 80% 52% Time cost 65.1s 5.3s 81.2s
84% 57% 59% 41% 37% 67% 63% 51% 50% 89% 59% 9.1s
dataset 3 Method Our in [12] method 84% 56% 59% 35% 38% 68% 59% 66% 51% 78% 59% 79.1s
87% 64% 62% 41% 43% 67% 68% 69% 60% 80% 64% 8.9s
Fig. 7. Examples in dataset 2,3,4
6
Conclusion
In this paper, we propose a fast and robust human estimation framework that combines the top-down example based method and bottom-up part based method. Example based method is first used to get initial guesses of the body pose and then a novel iterative BP inference schema is utilized to refined the estimated pose using a tree-structured kinematic model and discriminative part classifiers based on a rotation invariant EdgeField feature. The combination for top-down and bottom-up methods and the iterative inference schema enable us to achieve accurate pose estimation in acceptable time cost. Acknowledgement. This work is supported by National Science Foundation of China under grant No.61075026, and it is also supported by a grant from Omron Corporation.
Human Pose Estimation Using Exemplars and Part Based Refinement
185
References 1. Agarwal, A., Triggs, B.: Recovering 3d human pose from monocular images. PAMI 28(1), 44–58 (2006) 2. Bissacco, A., Yang, M., Soatto, S.: Fast human pose estimation using appearance and motion via multi-dimensional boosting regression. In: CVPR, vol. 13, pp. 234– 778 (2007) 3. Poppe, R.: Evaluating example-based pose estimation: Experiments on the humaneva sets. In: CVPR 2nd Workshop on EHuM2, vol. 14, pp. 234–778 (2007) 4. Poppe, R., Poel, M.: Comparison of silhouette shape descriptors for example-based human pose recovery. AFG (2006) 5. Mori, G., Malik, J.: Recovering 3d human body configurations using shape contexts. IEEE PAMI, 1052–1062 (2006) 6. Rogez, G., Rihan, J., Ramalingam, S., Orrite, C., Torr, P.H.S.: Randomized Trees for Human Pose Detection. In: CVPR (2008) 7. Shakhnarovich, G., Viola, P., Darrell, R.: Fast pose estimation with parametersensitive hashing. In: ICCV (2003) 8. Jiang, H., Martin, D.R.: Global pose estimation using non-tree models. In: CVPR (2008) 9. Sigal, L., Black, M.J.: Measure locally, reason globally: Occlusion-sensitive articulated pose estimation. In: CVPR (2006) 10. Sigal, L., Bhatia, S., Roth, S., et al.: Tracking Loose-limbed People. In: CVPR (2004) 11. Ramanan, D.: Learning to parse images of articulated bodies. In: NIPS (2006) 12. Andriluka, M., Roth, S., Schiele, B.: Pictorial Structures Revisited: People Detection and Articulated Pose Estimation. In: CVPR (2009) 13. Hill, A., Cootes, T.F., Taylor, C.J.: Active shape models and the shape approximation problem. In: 6th British Machine Vison Conference, pp. 157–166 (1995) 14. Lawrence, N.D.: Gaussian process latent variable models for visualization of high dimensional data. In: NIPS, vol. 16, pp. 329–336 (2004) 15. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR (2005) 16. Schapire, R.E., Singer, Y.: Improved boosting algorithms using confidence-rated predictions. Machine Learning 37(3), 297–336 (1999) 17. Yedidia, J.S., Freeman, W.T., Weiss, Y.: Understanding Belief Propagation and its Generalizations. IJCAI (2001) 18. Xu, X., Huang, T.S.: Face Recognition with MRC-Boosting. In: ICCV (2005)
Full-Resolution Depth Map Estimation from an Aliased Plenoptic Light Field Tom E. Bishop and Paolo Favaro Department of Engineering and Physical Sciences Heriot-Watt University, Edinburgh, UK {t.e.bishop,p.favaro}@hw.ac.uk
Abstract. In this paper we show how to obtain full-resolution depth maps from a single image obtained from a plenoptic camera. Previous work showed that the estimation of a low-resolution depth map with a plenoptic camera differs substantially from that of a camera array and, in particular, requires appropriate depth-varying antialiasing filtering. In this paper we show a quite striking result: One can instead recover a depth map at the same full-resolution of the input data. We propose a novel algorithm which exploits a photoconsistency constraint specific to light fields captured with plenoptic cameras. Key to our approach is handling missing data in the photoconsistency constraint and the introduction of novel boundary conditions that impose texture consistency in the reconstructed full-resolution images. These ideas are combined with an efficient regularization scheme to give depth maps at a higher resolution than in any previous method. We provide results on both synthetic and real data.
1
Introduction
Recent work [1, 2] demonstrates that the depth of field of Plenoptic cameras can be extended beyond that of conventional cameras. One of the fundamental steps in achieving such result is to recover an accurate depth map of the scene. [3] introduces a method to estimate a low-resolution depth map by using a sophisticated antialiasing filtering scheme in the multiview stereo framework. In this paper, we show that the depth reconstruction can actually be obtained at the full-resolution of the sensor, i.e., we show that one can obtain a depth value at each pixel in the captured light field image. We will show that the fullresolution depth map contains details not achievable with simple interpolation of a low-resolution depth map. A fundamental ingredient in our method is the formulation of a novel photoconsistency constraint, specifically designed for a Plenoptic camera (we will also use the term light field (LF) camera as in [4]) imaging a Lambertian scene (see Fig. 1). We show that a point on a Lambertian object is typically imaged under several microlenses on a regular lattice of correspondences. The geometry of such lattice can be easily obtained by using a Gaussian optics model of the Plenoptic camera. We will see that this leads to the reconstruction of full-resolution views R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 186–200, 2011. c Springer-Verlag Berlin Heidelberg 2011
Full-Res. Depth Map Estimation from an Aliased Plenoptic Light Field
187
consisting of mosaiced tiles from the sampled LF. Notice that in the data that we have used in the experiments, the aperture of the main lens is a disc. This causes the square images under each microlens to be split into a disc containing valid measurements and 4 corners with missing data (see one microlens image in the right image in Fig. 1). We show that images with missing data can also be reconstructed quite well via interpolation and inpainting (see e.g. [5]). Such interpolation however is not necessary if one uses a square main lens aperture and sets the F-numbers of the microlenses and the main lens appropriately. More importantly, we will show that the reconstructed samples form tiled mosaics, and one can enforce consistency at tile edges by introducing additional boundary conditions constraints. Such constraints are fundamental in obtaining the full-resolution depth estimates. Surprisingly, no similar gain in depth resolution is possible with a conventional camera array. In a camera array the views are not aliased (as the CCD sensors have contiguous pixels), and applying any existing method on such data (including the one proposed here) only results in upsampling of the low-resolution depth map (which is just the resolution of one view, and not the total number of captured pixels). This highlights another novel advantage of LF cameras. 1.1
Prior Work and Contributions
The space of light rays in a scene that intersect a plane from different directions may be interpreted as a 4D light field (LF) [6], a concept that has been useful in refocusing, novel view synthesis, dealing with occlusions and non-Lambertian objects among other applications. Various novel camera designs that sample the LF have been proposed, including multiplexed coded aperture [7], Heterodyne mask-based cameras [8], and external arrangements of lenses and prisms [9]. However these systems have disadvantages including the inability to capture dynamic scenes [7], loss of light [8], or poor optical performance [9]. Using an array of cameras [10] overcomes these problems, at the expense of a bulky setup. Light Field, or Plenoptic, cameras [4, 11, 12], provide similar advantages, in more compact portable form, and without problems of synchronisation and calibration. LF cameras have been used for digital refocusing [13] capturing extended depth-of-field images, although only at a (low) resolution equal to the number of microlenses in the device. This has been addressed by super-resolution methods [1, 2, 14], making use of priors that exploit redundancies in the LF, along with models of the sampling process to formulate high resolution refocused images. The same sampling properties that enable substantial image resolution improvements also lead to problems in depth estimation from LF cameras. In [3], a method for estimating a depth map from a LF camera was presented. Differences in sampling patterns of light fields captured by camera arrays and LF cameras were analysed, revealing how spatial aliasing of the samples in the latter case cause problems with the use of traditional multi-view depth estimation methods. Simply put, the LF views (taking one sample per microlens) are highly undersampled for many depths, and thus in areas of high frequency texture, an error term based on matching views will fail completely. The solution proposed in [3]
188
T.E. Bishop and P. Favaro
Fig. 1. Example of LF photoconsistency. Left: A LF image (courtesy of [15]) with an enlarged region (marked in red) shown to the right. Right: The letter E on the book cover is imaged (rotated by 180◦ ) under nine microlenses. Due to Lambertianity each repetition has the same intensity. Notice that as depth changes, the photoconsistency constraint varies by increasing or decreasing the number of repetitions.
was to filter out the aliased signal component, and perform matching using the remaining valid low-pass part. The optimal filter is actually depth dependent, so an iterative scheme was used to update the filtering and the estimated depth map. Whilst this scheme yields an improvement in areas where strong aliasing corrupts the matching term, it actually throws away useful detail that could be used to improve matches if interpreted correctly. Concurrent with this work, [16] proposed estimation of depth maps from LF data using cross-correlation, however this also only gives low resolution results. In this paper we propose to perform matching directly on the sampled lightfield data rather than restricting the interpretation to matching views in the traditional multi-view stereo sense. In this way, the concept of view aliasing is not directly relevant and we bypass the need to antialias the data. A key benefit of the new formulation is that we compute depth maps at a higher resolution than the sampled views. In fact the amount of improvement attainable depends on the depth in the scene, similar to superresolution restoration of the image texture as in [2]. Others have considered super-resolution of surface texture along with 3D reconstruction [17] and depth map super-resolution [18], however in the context of multi-view stereo or image sequences, where the sampling issues of a LF camera do not apply. Specifically, our contributions are: 1. A novel method to estimate a depth value at each pixel of a single (LF) image; 2. Analysis of the subimage correspondences, and reformulation in terms of virtual full-resolution views, i.e., mosaicing, leading to a novel photoconsistency term for LF cameras; 3. A novel penalty term that enforces gradient consistency across tile boundaries of mosaiced full-resolution views.
Full-Res. Depth Map Estimation from an Aliased Plenoptic Light Field
2
189
Light Field Representation and Methodology Overview
We consider a plenoptic camera, obtained by adding a microlens array in front of a conventional camera’s sensor [1, 2, 4]. The geometry of this camera is almost identical to that of a camera array where each camera’s aperture corresponds to a microlens. However, the LF camera has an additional main lens in front of the array, and this dramatically affects the sampling of the LF. We will see in sec. 3 that the precise study of how the spatial and angular coordinates sample the light field is critical to the formulation of matching terms for depth estimation. Let us consider a version of the imaging system where the microlens apertures are small and behave as pinholes.1 In the LF camera, each microlens forms its own subimage Sc (θ) on the sensor, where c ∈ R2 identifies the center of a microlens and θ ∈ R2 identifies the local (angular) coordinates under such a microlens. In a real system the microlens centers are located at discrete positions ck , but to begin the analysis we generalize the definition to virtual microlenses that are free to be centered in the continuous coordinates c. The main lens maps an object in space to a conjugate object inside the camera. Then, pixels in each subimage see light from the conjugate object from different vantage points, and have a one-to-one map with points on the main lens (see Fig. 2). The collection of pixels, one per microlens, that map to the same point on the main lens (i.e., with the same local angle θ) form an image we call a view, denoted by Vθ (c) ≡ Sc (θ). The views and the subimages are two equivalent representations of the same quantity, the sampled light field. We assume the scene consists of Lambertian objects, so that we can represent the continuous light field with a function r(u) that is independent of viewing angle. In our model, r(u) is called the radiance image or scene texture, and relates to the LF via a reprojection of the coordinates u ∈ R2 at the microlens plane, through the main lens center into space. That is, r(u) is the full-resolution all-focused image that would be captured by a standard pinhole camera with the sensor placed at the microlens plane. This provides a common reference frame, independent of depth. We also define the depth map z(u) on this plane. 2.1
Virtual View Reconstructions
Due to undersampling, the plenoptic views are not suitable for interpolation. Instead we may interpolate within the angular coordinates of the subimages. In our proposed depth estimation method, we make use of the fact that in the Lambertian light field there are samples, indexed by n, in different subimages that can be matched to the same point in space. In fact, we will explain how several estimates rˆn (u) of the radiance can be obtained, one per set of samples with common n. We term these full resolution virtual views, because they make use of more samples than the regular views Vθ (c), and are formed from interpolation 1
Depending on the LF camera settings, this holds true for a large range of depth values. Moreover, we argue in sec. 4 that corresponding defocused points are still matched correctly and so the analysis holds in general for larger apertures.
190
T.E. Bishop and P. Favaro
r’
O
microlenses
r
z’
c1
θ
- v’θq v
c2
u
θq μ
d
θq θ1 θ2 θq
v’
cK
Main Lens
v
conjugate image v’ v
θ1
v’
- v’ vθ
λ(u)(c-u) u
z’ (u-c) u’ v’ O
Main Lens
z‘
θ0
c v’-z‘
θ0
v
θ Sensor
Fig. 2. Coordinates and sampling in a plenoptic camera. Left: In our model we begin with an image r at v , and scale it down to the conjugate image r at z , where O is the main lens optical center. Center: This image at r is then imaged behind each microlens; for instance, the blue rays show the samples obtained in the subimage under the central microlens. The central view is formed from the sensor pixels hit by the solid bold rays. Aliasing of the views is present: the spatial frequency in r is higher than the microlens pitch d, and clearly the adjacent view contains samples unrelated to the central view via interpolation. Right: Relation between points in r and corresponding points under two microlenses. A point u in r has a conjugate point (the triangle) at u , which is imaged by a microlens centered at an arbitrary center c, giving a projected point at angle θ. If we then consider a second microlens positioned at u, the image of the same point is located at the central view, θ = 0, on the line through O.
in the non-aliased angular coordinates. Similarly to [1], these virtual views may be reconstructed in a process equivalent to building a mosaic from parts of the subimages (see Fig. 3). Portions of each subimage are scaled by a magnification factor λ that is a function of the depth map, and these tiles are placed on a new grid to give a mosaiced virtual view for each n. As λ changes, the size of the portions used and the corresponding number of valid n are also affected. We will describe in detail how these quantities are related and how sampling and aliasing occurs in the LF camera in sec. 3, but firstly we describe the overall form of our depth estimation method, and how it uses the virtual views. 2.2
Depth Estimation via Energy Minimization
We use an energy minimization formulation to estimate the optimal depth map z = {z(u)} in the full-resolution coordinates u. The optimal z is found as the minimum of the energy E defined as: E1 (u, z(u)) + γ2 E2 (u, z(u)) + γ3 E3 (z(u)) (1) E(z) = u
where γ2 , γ3 > 0. This energy consists of three terms. Firstly, E1 is a data term, penalizing depth hypotheses where the corresponding points in the light field do not have matching intensities. For the plenoptic camera, the number of virtual views rˆn (u) depends on the depth hypothesis. We describe this term in sec. 4.
Full-Res. Depth Map Estimation from an Aliased Plenoptic Light Field
191
Fig. 3. Mosaiced full-resolution views generated from subimages. Each microlens subimage in a portion of the light field image shown on the left is partitioned into a grid of tiles. Rearranging the tiles located at the same offset from the microlens center into a single image produces mosaiced full-resolution views as shown in the middle. As we used a Plenoptic camera with a circular main lens aperture, some pixels are not usable. In the top view, the tiles are taken from the edge of each microlens and so there is missing data which is inpainted and shown on the image to the right. We use these full-resolution views to perform accurate data matching for depth reconstruction.
E2 is a new matching term we introduce in sec. 4.1, enforcing the restored views to have proper boundary conditions. It effectively penalizes mismatches at the mosaiced tile boundaries, by imposing smoothness of the reconstructed texture (that also depends on the depth map z). The third term, E3 , is a regularization term that enforces piecewise smoothness of z. Because the terms E1 and E2 only depend on z in a pointwise manner, they may be precomputed for a set of depth hypotheses. This enables an efficient numerical algorithm to minimize eq. (1), which we describe in sec. 4.3. Before detailing the implementation of these energy terms in sec. 4, we show in sec. 3 how such views are obtained from the sampled light field.
3
Obtaining Correspondences in Subimages and Views
We summarise here the image formation process in the plenoptic camera, and the relations between the radiance, subimages and views. We consider first a continuous model and then discretise, explaining why aliasing occurs. In our model, we work entirely inside the camera, beginning with the full-resolution radiance image r(u), and describe the mapping to the views Vθ (c) or microlens subimages Sc (θ). This mapping is done in two steps: from r(u) to the conjugate object (or image) at z , which we denote r , and then from r to the sensor. This process is shown in Fig. 2. There is a set of coordinates {θ, c} that corresponds to the same u, and we will now examine this relation. We see in Fig. 2 that the projection of r(u) to r (u ) through the main lens u , where v is the main lens to microlens array center O results in u = z v(u) distance. This is then imaged onto the sensor at θ(u, c) through the microlens
192
T.E. Bishop and P. Favaro
at c with a scaling of − v −zv (u) . Notice that θ(u, c) is defined as the projection of u through c onto the sensor relative to the projection of O through c onto the sensor; hence, we have z (u) v (c − u) v − z (u) v = λ(u)(c − u),
θ(u, c) =
(2)
where we defined the magnification factor λ(u) = v −zv (u) z v(u) , representing the signed scaling between the regular camera image r(u) that would form at the microlens array, and the actual subimage that forms under a microlens. We can then use (2) to obtain the following relations: r(u) = Sc (θ(u, c)) = Sc λ(u)(c − u) , v D (3) ∀c s.t. |θ(u, c)| < v 2 = Vλ(u)(c−u) (c), θ v D r(u) = Vθ (c(θ, u)) = Vθ u + (4) ∀ |θ| < . λ(u) v 2
The constraints on c or θ mean only microlenses whose projection − vv θ is within the main lens aperture, of radius D 2 , will actually image the point u. Equations (3) and (4) show respectively how to find values corresponding to r(u) in the light field for a particular choice of either c or θ, by selecting the other variable appropriately. This does not pose a problem if both c and θ are continuous, however in the LF camera this is not the case and we must consider the implications of interpolating sampled subimages at θ = λ(u)(c − u) or θ . sampled views at c = u + λ(u) 3.1
Brief Discussion on Sampling and Aliasing in Plenoptic Cameras
In a LF camera with microlens spacing d, only a discrete set of samples in each . view is available, corresponding to the microlens centers at positions c = ck = dk, where k indexes a microlens. Also, the pixels (of spacing μ) in each subimage . sample the possible views at angles θ = θq = μq, where q is the pixel index; these coordinates are local to each microlens, with θ0 the projection of the main lens center through each microlens onto the sensor. Therefore, we define the discrete . observed view Vˆq (k) = Vθq (ck ) at angle θq as the image given by the samples . for each k. We can also denote the sampled subimages as Sˆk (q) = Vθq (ck ). For a camera array, the spatial samples are not aliased, and a virtual view Vθ (c) may be reconstructed accurately, which allows for the sub-pixel matching used in regular multi-view stereo. However, in the LF camera, there is clearly a large jump between neighboring samples from the same view in the conjugate image (see Fig. 2). Even when the microlenses have apertures sized equal to their spacing, as described in [3], the resulting microlens blur size changes with
Full-Res. Depth Map Estimation from an Aliased Plenoptic Light Field
193
depth. Equivalently there is a depth range where the integration region size in the conjugate image is less than the sample spacing. This means that the sampled views Vˆq (k) may be severely aliased, depending on depth. As a result, simple interpolation of Vˆq (k) in the spatial coordinates k is not enough to reconstruct an arbitrary virtual view Vθ (c) at c = ck . 3.2
Matching Sampled Views and Mosaicing
In an LF camera we can reconstruct Vθ (c) and avoid aliasing in the sampled views by interpolating instead along the angular coordinates of the subimages, and using the spatial samples only at known k. Using eq. (3), we can find the estimates rˆ(u) of the texture by interpolating as follows: r(u) = Vλ(u)(ck −u) (ck ) ⇒ rˆ(u) = Vˆ λ(u) (k), μ
(ck −u)
∀k s.t. |λ(u)(ck − u)| <
v D v 2
(5)
. Let us now expand ck as u + Δu + nd, where Δu = ck0 − u is the vector from . u to the closest microlens center ck0 = u − mod (u, d). We can now enumerate the different possible reconstructed images rˆn (u) for all valid choices of n ∈ Z2 that lead to the condition on ck being satisfied. Then, we find rˆn (u) = Vˆ λ(u) (Δu+nd) k0 + n , μ
∀n s.t. |λ(u)(Δu + nd)| <
v D v 2
(6)
The reconstructed virtual views rˆn (u) for different n should match at position u, if we have made the right hypothesis for the depth z(u) and hence λ(u) (notice that given λ(u) one can immediately obtain z(u) and vice versa). This interpretation is similar to that of the method used in [1], except that rather than averaging together the full-resolution mosaics rˆn (u), we keep them separated.
4
Energy Terms for Full-Resolution Depth Maps
We now describe the energy terms in sec. 2.2. Firstly, we define the matching term E1 that penalizes differences between the different virtual views2 rˆn (u) as E1 u, z(u) =
1 W (u)
rˆn (u) − rˆn (u).
∀n=n n,n ∈Z2 n s.t. |λ(u)(Δu+nd)|< vv
(7)
D 2 D 2
n s.t. |λ(u)(Δu+n d)|< vv
W (u) is a normalization term, counting the number of valid pairs in the sum. The number of possible n, and hence number of virtual views, changes based on the magnification factor λ, according to the depth hypothesis. Therefore 2
Interestingly, unlike matching low resolution views in multi-view stereo, the virtual views are already aligned to each other in regions with the correct depth assumption.
194
T.E. Bishop and P. Favaro x border gradients:
pixels used for x-gradients:
xx
xy 2 high-res views (wrong depth assumption)
partial borders interpolated
borders propagated via linear interpolation
cell boundary
Fig. 4. Left: The energy term E2 penalizes unnatural discontinuities in the reconstructed full-resolution views, at the mosaic cell boundaries. Interpolation is used to complete partial views, and interior pixels from the borders. Right: The x-border gradients at the red pixels use the supports shown, using the difference between the pairs of gradients given between the yellow pairs.
W (u) varies with both position and depth hypothesis. Also, when using a Plenoptic camera with a circular main lens aperture, we need to deal with missing data. In this case, parts of some of the views will be missing where tiles were taken from the borders of a microlens (see Fig. 3). For this reason, we consider using inpainting (we use a PDE implementation in MATLAB3 ) to slightly reduce the amount of missing data, i.e., to extend the field of view of each microlens based on extrapolating the data contained in neighboring lenses. Furthermore, rather than comparing all views with each other as in eq. (7), we have found that using a penalty in E1 that compares each mosaic rˆn (u) (at valid locations) with the median obtained by inpainting each rˆn (u) can be more robust, and also considerably faster for large W . We have made the assumption that microlens blur is insignificant at working depths. If blur is present, it will be the same amount for each point being matched, and will not cause any difference for reasonably planar surfaces. In the case of a depth transition, highly textured surfaces will appear slightly larger than they really are, due to matching occurring at all points within the true point blur disc. Ideally, we could also deconvolve each mosaic independently before matching, however, this is beyond the scope of this paper. 4.1
Mosaiced Boundary Condition Consistency
Given the full-resolution mosaiced views, rˆn (u), we may also establish a selfconsistency criterion in E2 . This term says that each reconstructed rˆn (u) should have consistent boundary conditions across blocks within each view. That is, the mosaicing process should not introduce unnatural jumps if the depth is estimated 3
http://www.mathworks.com/matlabcentral/fileexchange/4551
Full-Res. Depth Map Estimation from an Aliased Plenoptic Light Field
195
correctly. Therefore we expect the x- and y-derivatives to be mostly smooth across boundaries (assuming the probability of an image edge coinciding with a mosaic tile along all its edges to be low), and set E2 z(u) = |∇xx rˆn (u)| + |∇xy rˆn (u)| + |∇yx rˆn (u)| + |∇yy rˆn (u)|, n∈Z2
n∈Z2
|θck (n) (u)|< vv D2
|θck (n) (u)|< vv D2
u∈Rboundary-x
u∈Rboundary-y
(8) where ∇ab denotes the 2nd order derivative with respect to the coordinates a and b, Rboundary-x includes the pixels on the left and right tile boundaries, and Rboundary-y the top and bottom boundaries (only non-inpainted regions). The restriction of this constraint to the boundary of the tiles is due to eq. (6). We observe that for a fixed n, eq. (6) tells us that, as we vary u, several consecutive samples are taken from the same microlens subimage at k0 , but there is a discontinuity present in the reconstruction as u moves past a microlens boundary and k0 then jumps to a different microlens k. Changing n will choose a different offset portion of each subimage to reconstruct (effectively a full-resolution virtual view, centered at an angle qn = dλ(u) μ n ). Within each tile the data is continuous and just comes from a scaled version of the same subimage and so we should not expect to penalize differently for a different scaling/depth. / Rboundary via bilinear interpolation We can instead define E2 (z(u)) for u ∈ of the values at the boundary. While this term does not help super-resolving the depth map, it helps enforcing its overall consistency. 4.2
Regularization of the Depth Map
Due to the ill-posedness of the depth estimation problem, we introduce regularization. Our focus here is not on the choice of prior, so for simplicity we use a standard total variation term (∇ denotes the 2D gradient with respect to u): E3 (z(u)) = ∇z(u)1 . 4.3
(9)
Numerical Implementation
At this point we have all the necessary ingredients to work on the energy introduced in eq. (1). In the following we use the notation introduced earlier on where the depth map z is discretized as a vector z = {z(u)}. Then, we pose the depth estimation problem as the following minimization zˆ = arg min E(z) = arg min Edata (z) + γ3 E3 (z) z
z
(10)
where we combine E1 and E2 in the data term Edata = E1 + γ2 E2 ; in our experiments the tradeoff between E3 and Edata is fixed by γ2 = 1 and γ3 = 10−3 . We then minimize this energy by using an iterative solution. We first notice that the term Edata can be written as a sum of terms that depend on a single entry of z at a time. Hence, we find a first approximate solution by
196
T.E. Bishop and P. Favaro
ignoring the regularization term and performing a fast brute force search for each entry of z independently. This procedure yields the initial solution z0 . Then, we approximate Edata via a Taylor expansion up to the second order, i.e., 1 Edata (zt+1 ) Edata (zt )+∇Edata (zt )(zt+1 −zt )+ (zt+1 −zt )T HEdata (zt )(zt+1 −zt ) 2 (11)
where ∇Edata is the gradient of Edata , HEdata is the Hessian of Edata , and zt and zt+1 are the solutions at iteration t and t+1 respectively. To ensure that our local approximation is convex we let HEdata be diagonal and each element be positive by taking its absolute value (component-wise) |HEdata (zt )|. In the case of the regularization energy we use a Taylor expansion of its gradient up to the linear term. When we compute the Euler-Lagrange equations of the approximate energy E with respect to zt+1 this linearization results in ∇Edata (zt ) + |HEdata (zt )|(zt+1 − zt ) − γ3 ∇ ·
∇(zt+1 − zt ) =0 |∇zt |
(12)
which is a linear system in the unknown zt+1 , and can be solved efficiently via the Conjugate Gradient method.
5
Experiments
We begin by testing the performance of the method on artificially generated light fields, before considering data from a real LF camera. Synthetic Data. We simulated data from a LF camera, with main lens of focal length F = 80mm focused at 0.635m, with parameters d = 0.135mm, μ = 0.09mm, v = 0.5mm, Q = 15 and microlens focal length f = 0.5mm. To compare with the antialiasing method of [3], we estimate the view disparity . μ , in pixels per view, for a set of textured planes at known depths (the s(u) = dλ(u) disparity between the views is zero for the main lens plane in focus, and increases with depth). We used the Bark texture from the Brodatz texture dataset, and in each case just used the energy without regularization. We plot the true value of s for each plane against the mean value from the depth maps found by both the method of [3] and the proposed method, along with error bars in Fig. 5. Clearly in the case of no noise the proposed algorithm performs a lot more reliably with a smaller variance of the error, although with increasing noise the benefit over the method of [3] reduces. Both methods perform poorly around the main lens plane in focus: in the case of the method of [3], the disparity becomes smaller than the interpolation accuracy, whereas the proposed method begins to fail as the magnification factor tends towards 1, and there are not enough virtual views to form the median. This is not surprising as it relates to the undersampling of the space by the LF camera around these depths. Real Data. We obtained datasets from a portable LF camera with the same system parameters as above. The scene range was 0.7–0.9m, corresponding to disparities of 0.16–0.52 pixels per view. One of the low resolution views is shown,
0.6
0.6
0.5
0.5
0.4
0.4
estimated disparity
estimated disparity
Full-Res. Depth Map Estimation from an Aliased Plenoptic Light Field
0.3 0.2 0.1
0.3 0.2 0.1
Antialiasing method (Bishop et al.) Proposed method true s
0 ï0.1
197
0.1
0.2
0.3
0.4
true disparity, s, in pixels per view
0.5
0.6
Antialiasing method (Bishop et al.) Proposed method true s
0 ï0.1
0.1
0.2
0.3
0.4
0.5
0.6
true disparity, s, in pixels per view
Fig. 5. Results on synthetic data. We plot the estimated disparity from a textured plane versus the true disparity s (which depends on the depth of the plane) for our method and the method from [3]. Error bars shown at 3 standard deviations. Left: data with no noise. Right: Data with noise standard deviation 2.5% of signal range.
Fig. 6. Results on real data. Top row, left to right: One low resolution view from the light field; One full-resolution virtual view, with depth hypothesis in the middle of the scene range (note blocking artefacts); Low resolution depth map using multiview stereo method [3]; Bottom row: Unregularized full-resolution depth map (note that the textureless areas shown in red do not yield a depth estimate); After regularization; High resolution all-in-focus image obtained using the estimated depth map.
198
T.E. Bishop and P. Favaro
Fig. 7. Results on real data. From left to right: One low resolution view from the light field; Regularized depth estimate; High resolution image estimate from the light field, using estimated depth map. Source LF data in bottom row courtesty of [16].
along with a full-resolution virtual view used in the algorithm, and the estimated depthmap before and after regularization in Fig. 6 and Fig. 7. We also applied the method to data from [16] in the bottom row of Fig. 7. In each case, before applying regularization, we automatically select regions where the confidence is too low due to lack of texture (the energy has very small values for all depths at these pixels, but is estimated very reliably elsewhere). For this size image (2100× 1500 pixel virtual views in the case of Fig. 6), our MATLAB implementation takes about half an hour to estimate E1 and E2 , using up to n = 37 virtual views at the maximum depths, although solving eq. (12) only takes a couple of minutes. Notice the main burden is not in the regularisation scheme but simple interpolation and difference operations between a large number of views (e.g. 40*37=1480 views if using n = 37 and 40 depth levels), which could be done at least an order of magnitude faster using, for example, GPU acceleration.
6
Conclusions
We have demonstrated how depth can be estimated at each pixel of a single (LF) image. This is an unprecedented result that highlights a novel quality of Plenoptic cameras. We have analysed the sampling tradeoffs in LF cameras and shown
Full-Res. Depth Map Estimation from an Aliased Plenoptic Light Field
199
how to obtain valid photoconsistency constraints, which are unlike those for a regular multi-view matching. We also presented a novel boundary consistency constraint in reconstructed full-resolution views to penalize incorrect depths, and proposed a novel algorithm to infer full-resolution depth maps using these terms, with efficient regularization. Acknowledgement. This work has been supported in part by EPSRC grant EP/F023073/1(P), grant EP/I501126/1, and SELEX/HWU/2010/SOW3.
References 1. Lumsdaine, A., Georgiev, T.: The focused plenoptic cameras. In: ICCP 2009 (IEEE International Conference on Computational Photography) (2009) 2. Bishop, T.E., Zanetti, S., Favaro, P.: Light field superresolution. In: ICCP 2009 (IEEE Int. Conference on Computational Photography) (2009) 3. Bishop, T.E., Favaro, P.: Plenoptic depth estimation from multiple aliased views. In: ICCV 2009 Workshops: 3DIM (The 2009 IEEE International Workshop on 3-D Digital Imaging and Modeling), Kyoto, Japan (2009) 4. Ng, R., Levoy, M., Br´edif, M., Duval, G., Horowitz, M., Hanrahan, P.: Light field photography with a hand-held plenoptic camera. Technical Report CSTR 2005-02, Stanford University (2005) 5. Weickert, J., Welk, M.: Tensor field interpolation with pdes. In: Weickert, J., Hagen, H. (eds.) Visualization and Processing of Tensor Fields, pp. 315–325. Springer, Berlin (2006) 6. Levoy, M., Hanrahan, P.: Light field rendering. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1996, pp. 31–42. ACM Press, New York (1996) 7. Liang, C.K., Lin, T.H., Wong, B.Y., Liu, C., Chen, H.H.: Programmable aperture photography: Multiplexed light field acquisition. ACM Transactions on Graphics (Proc. SIGGRAPH 2008) 27 (2008) 8. Veeraraghavan, A., Raskar, R., Agrawal, A.K., Mohan, A., Tumblin, J.: Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Graph (Proc. SIGGRAPH 2007) 26, 69 (2007) 9. Georgeiv, T., Zheng, K.C., Curless, B., Salesin, D., Nayar, S., Intwala, C.: Spatioangular resolution tradeoff in integral photography. In: Akenine-Mller, T., Heidrich, W. (eds.) Eurographics Symposium on Rendering (2006), Adobe Systems, University of Washington, Columbia University (2006) 10. Vaish, V., Levoy, M., Szeliski, R., Zitnick, C., Kang, S.B.: Reconstructing occluded surfaces using synthetic apertures: Stereo, focus and robust measures. In: Proc. CVPR 2006, vol. 2, pp. 2331–2338 (2006) 11. Adelson, E.H., Wang, J.Y.: Single lens stereo with a plenoptic cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 99–106 (1992) 12. Fife, K., El Gamal, A., Wong, H.S.: A 3d multi-aperture image sensor architecture. In: Custom Integrated Circuits Conference, CICC 2006, pp. 281–284. IEEE, Los Alamitos (2006) 13. Ng, R.: Fourier slice photography. ACM Trans. Graph. 24, 735–744 (2005) 14. Levin, A., Freeman, W.T., Durand, F.: Understanding camera trade-offs through a bayesian analysis of light field projections. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part IV. LNCS, vol. 5305, pp. 88–101. Springer, Heidelberg (2008)
200
T.E. Bishop and P. Favaro
15. Georgiev, T., Lumsdaine, A.: Depth of field in plenoptic cameras. In: Alliez, P., Magnor, M. (eds.) Eurographics 2009 (2009) 16. Georgiev, T., Lumsdaine, A.: Reducing plenoptic camera artifacts. Computer Graphics Forum 29, 1955–1968 (2010) 17. Goldluecke, B., Cremers, D.: A superresolution framework for high-accuracy multiview reconstruction. In: IEEE International Conference on Computer Vision (ICCV), Kyoto, Japan, pp. 342–351 (2009) 18. Li, F., Yu, J., Chai, J.: A hybrid camera for motion deblurring and depth map super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008, pp. 1–8 (2008)
Indoor Scene Classification Using Combined 3D and Gist Features Agnes Swadzba and Sven Wachsmuth Applied Informatics, Bielefeld University, Germany {aswadzba,swachsmu}@techfak.uni-bielefeld.de
Abstract. Scene categorization is an important mechanism for providing high-level context which can guide methods for a more detailed analysis of scenes. State-of-the-art techniques like Torralba’s Gist features show a good performance on categorizing outdoor scenes but have problems in categorizing indoor scenes. In contrast to object based approaches, we propose a 3D feature vector capturing general properties of the spatial layout of indoor scenes like shape and size of extracted planar patches and their orientation to each other. This idea is supported by psychological experiments which give evidence for the special role of 3D geometry in categorizing indoor scenes. In order to study the influence of the 3D geometry we introduce in this paper a novel 3D indoor database and a method for defining 3D features on planar surfaces extracted in 3D data. Additionally, we propose a voting technique to fuse 3D features and 2D Gist features and show in our experiments a significant contribution of the 3D features to the indoor scene categorization task.
1
Introduction
An important ability for an agent (both human and robot) is to build an instantaneous concept of the surrounding environment. Such holistic concepts activate top-down knowledge that guides the visual analysis in further scene interpretation tasks. For this purpose, Oliva et al. [1] introduced the so-called Gist features that represent the spatial envelope of a scene given as a 2D image. Based on this general idea a variety of scene recognition approaches have been proposed that work well for outdoor scenes – like “buildings”, “street”, “mountains”, etc. – but frequently break down for indoor scenes, e.g. “kitchen”, “living room”, etc.
Fig. 1. Example room from 3D database, corresponding 3D data for defining 3D features, and computed 2D Gist image R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 201–215, 2011. Springer-Verlag Berlin Heidelberg 2011
202
A. Swadzba and S. Wachsmuth
Recently, there are some approaches, e.g., [2], that try to combine global Gist information with local object information to achieve a better performance on different indoor categories. Although, they achieve significant improvement, the training of such methods relies on a previous hand labeling of relevant regions of interest. Psychological evidence even suggests that objects do not contribute to the low-level visual process responsible for determining the room category. In early experiments, Brewer and Treyen provided hints that with the perception of room schemata objects (e.g., books) were memorized to be in the experimental room (e.g., office) though they were not really present. Henderson et al. [3, 4] even found through their fMRI studies brain areas that are sensitive to 3D geometric structures enabling a distinction of indoor scenes from other complex stimuli but do not show any activation on close-up views of scene-relevant objects, e.g., a kitchen oven. This emphasizes the contribution of the scene geometry to the ability of categorizing indoor scenes. Currently, most approaches to scene recognition are based on color and texture statistics of 2D images neglecting the 3D geometric structure of scenes. Therefore, our goal is to capture the 3D spatial structure of scene categories (see Fig. 1 for impression of used data) in a holistically defined feature vector. More precisely, as man-made environments mainly consist of planar surfaces we define appropriate statistics on these spatial structures capturing their shape and size and their orientations to each other. To analyze their performance on the indoor scene categorization problem a new dataset is introduced consisting of six different room categories and 28 individual rooms that have been recorded with a Time-of-Flight (ToF) sensor in order to provide 3D data. The classification results are contrasted to results achieved with 2D Gist features. Further, possible combination schemes are discussed. In the following, the paper discusses related work, describes the 3D indoor database and the computation of our 3D scene descriptor and combination schemes with 2D features. Finally, evaluation results are discussed.
2
Related Work
As navigation and localization was one of the first tasks exhaustively investigated in robotics research some interesting approaches to solve the problem of recognizing known rooms and categorizing unknown rooms can be found. Known rooms can be recognized by invariant features in 360◦ laser scans [5], by comparing current 2D features to saved views [6], and by recognizing specific objects and their configurations [7]. Burgard’s group has examined a huge bunch of different approaches to the more general place categorization problem ranging from the detection of concepts like “room”, “hall”, and “corridor” by simple geometric features defined on a 2D laser scan [8] to subconcepts like “kitchen” [9] using ontologies encoding the relationship between objects and the subconcepts [10]. Alternatively, occurrence statistics of objects in place categories [11, 12] and interdependencies among objects and locations [13] have been tried out. Parallel to the described approaches, Torralba [14] has developed a lowlevel global image representation by wavelet image decomposition that provides
Indoor Scene Classification Using Combined 3D and Gist Features
203
relevant information for place categorization. These so-called Gist features perform well on a lot of outdoor categories like “building”, “street”, “forest”, etc. but have problems to categorize indoor scenes reliably. Therefore, they have extended their approach by local discriminative information [2]. They learn prototypes from hand-labeled images, which define the mapping between images and scene labels. Similar to those prototypes, Li and Perona represent an image by a collection of local regions denoted as codewords which they obtain by unsupervised learning [15]. Similar approaches based on computing visual or semantic typicality vocabularies and measuring per image the frequency of these words or concepts are introduced by Posner [16], Bosch [17], and Vogel [18]. A local features based technique used in this paper for comparison is proposed by Lazebnik et. al [19]. They partition an image into increasingly fine sub-regions, the so-called spatial pyramid, and represent these regions through SIFT descriptors. In [6] the application of such local features for place categorization on a mobile robot in a real environment is tested with moderate success. So far, to the best of our knowledge, 3D information only has been used for detecting objects [20–22] or classifying each 3D point into given classes which could be “vegetation”, “facade”, ... [23], “chair”, “table”, ... [24], or “plane”, “sphere”, “cylinder”, ... [25]. The introduced features are of local nature since each 3D point is represented by a feature vector computed from the point’s neighborhood. They are not applicable as we aim for classifying a set of points as a whole.
3
The 3D Indoor Database
In order to systematically study room categorization based on the 3D geometry of rooms a sufficiently robust method for extracting 3D data is necessary. Even though recent papers show an impressive performance in extracting the 3D room frame [26, 27] or depth-ordered planes from a single 2D image [28] they are at the moment not precise enough in computing all spatial structures given by, e.g., the furniture, on the desired level of detail. Therefore, existing 2D indoor databases cannot be utilized. Recording rooms in 3D could be done in principle by stereo camera systems, laser range finders, or Time-of-Flight (ToF) cameras. As room structures are often homogeneous in color and less textured stereo cameras will not provide enough data. In order to acquire instantaneously dense 3D point clouds from a number of rooms laser range finders are unhandy and not fast enough. Thus, we use ToF cameras to acquire appropriate data but our approach is not limited to this kind of data. The Swissranger SR3100 from Mesa Imaging1 is a near-infrared sensor delivering in real-time a dense depth map of 176 × 144 pixels resolution [29]. The depth is estimated per pixel from measuring the time difference between the signal sent and received. Besides the 3D point cloud (→ Fig. 3b), the camera delivers a 2D amplitude image (→ Fig. 3a) which looks similar to a normal gray-scale image but encodes for each pixel the amount of reflected infrared light. The noise ratio of the sensor depends on the reflection 1
http://www.mesa-imaging.ch
204
A. Swadzba and S. Wachsmuth
bath.1
bath.2
bath.3
bed.1
bed.2
bed.3
bed.4
eat.4
eat.5
eat.6
kit.1
eat.1
eat.2
kit.2
kit.3
kit.4
kit.5
kit.6
kit.7
liv.1
liv.2
liv.3
liv.4
liv.5
off.1
off.2
off.3
eat.3
Fig. 2. Photos of 28 different rooms scanned in 3D by a Swissranger SR3100. The images were taken for visualization at the position of the 3D camera. The database contains 3 bathrooms, 4 bedrooms, 6 eating places, 7 kitchens, 5 livingrooms, and 3 offices.
properties of the materials contained in the scene. Points measured from surfaces reflecting infrared light in a diffuse manner have a deviation of 1 to 3 cm. For acquiring many different arranged rooms per category, we recorded data in a regular IKEA home-center2 . Its exhibition is ideal for our purpose because it is assembled by 3D boxes representing single rooms. Each box is furnished differently. We placed the 3D camera at an arbitrary position of the open side and scanned the room for 20 to 30 seconds while the camera is continuously moved by round about 40◦ left/right and 10◦ up/down. This simulates a robot moving its head for perceiving the entire scene. We acquired data for 3 bathrooms, 4 bedrooms, 6 eating places, 7 kitchens, 5 livingrooms, and 3 offices3 . Fig. 2 shows photos of the scanned rooms taken at the position of the 3D camera.
4
Learning a Holistic Scene Representation
This section presents the computation of our novel 3D scene descriptor and the 2D Gist descriptor from a Swissranger frame F . Further, details are given on learning of room models from these feature types and the combination of classification results from both features and several consecutive frames to achieve a robust scene categorization on data of a so far unseen room. Scene Features From 3D Data. As man-made environments mostly consist of flat surfaces a simplified representation of the scene is generated by extracting bounded planar surfaces from the 3D point cloud of a frame. Noisy data is 2 3
http://www.ikea.com/us/en/ http://aiweb.techfak.uni-bielefeld.de/user/aswadzba
Indoor Scene Classification Using Combined 3D and Gist Features
(a) liv.5: frame 162
(b)
205
(c)
Fig. 3. (a): Swissranger amplitude image; (b): corresponding 3D point cloud with extracted planar surfaces (highlighted by color), an angle between two planes; a box enclosing a plane with its edges parallel to the two largest variance directions, and a plane transformed so that the vectors indicating the two largest variance directions in the data are parallel to the coordinate axis; (c): the responses of the Gabor filters applied to the amplitude image are averaged to one image visualizing the information encoded by the Gist feature vector.
smoothed by applying median filters to the depth map of each frame. Artifacts at jumping edges (which is a common problem in actively sensing sensors) are removed by declaring points as invalid if they lie on edges in the depth image. A normal vector ni is computed for each valid 3D point pi via Principal Component Analysis (PCA) of points in the neighborhood N5×5 in the Swissranger image plane. The decomposition of the 3D point cloud into connected planar regions is done via region growing as proposed in [30] based on the conormality measurement defined in [31]. Iteratively, points are selected randomly as seed of a region and extended with points of its N3×3 neighborhood if the points are valid and conormal. Two points p1 and p2 are conormal, when their normals n1 and n2 hold:
α = (n1 , n2 )
arccos(n1 · n2 ) π − arccos(n1 · n2 )
: arccos(n1 · n2 ) <= : else
π 2
with α < θα
(1)
pi Here, we have chosen θα = 18 . The resulting regions are refined by some runs of the RANSAC algorithm [32]. Fig. 3b displays for a frame F the resulting set of planes {Pj |nj · p − dj = 0}j=1...m where nj is computed through PCA of points {pi } assigned to Pj and dj by solving the plane equation for the centroid of these plane points. In the following, the computation of a 3D feature vector x3D based on the extracted planar patches in F is described. This vector encodes the 3D geometry of the perceived scene independent from colors and textures, the view point, and the camera orientation. For m planes {Pj }j=1,...,m in frame F shape and size characteristics and for all plane pairs {(Pk , Pl )}k,l∈(1...m),k=l angle and size ratio characteristics are computed. For each plane characteristic, cs , cA , c÷ , and c , the resulting values are binned into a separated histogram which is normalized to length 1 through dividing the histogram by the number of planes or plane pairs, respectively. Then, the resulting four vectors are concatenated to one feature vector encoding general spatial properties of the scene like the orientation of the patches to each other, their shapes, and their size characteristics:
206
A. Swadzba and S. Wachsmuth
x3D = x , xs , xA , x÷ , with min(a, b) 1 cs = → xs = · Hs ({csj }), j = 1 . . . m, max(a, b) m 1 cA = a · b → x A = · HA ({cA j = 1 . . . m, j }), m A min(cA 2 k , cl ) c÷ · H÷ ({c÷ k, l = 1 . . . m, → x÷ = kl = kl }), A m(m − 1) max(ck , cA ) l 2 c · H ({c → x = k, l = 1 . . . m. kl = (nk , nl ) kl }), m(m − 1)
(2) (3) (4) (5) (6)
Shape characteristic cs . Each planar patch P is spanned by the points assigned to this patch during plane extraction. PCA of these points provides three vectors a, b, n with a indicating the direction of the largest variance in the data and b the direction orthogonal to a with the second largest variance. n gives the normal vector of the planar patch. Fig. 3b shows a planar patch P transformed so that a and b are parallel to the coordinate axes. A minimum bounding box is computed with its orientation parallel to a and b including all plane points. The height of the box is assumed to be a and the width b. As can be seen in Eq. 3 the feature vector xs is a normalized histogram Hs over {csj } with h = 5 bins in the interval ]0, 1[ and a bin width of Δh = 0.2. It encodes the distribution of shapes which range from elongated (cs near 0) to squarish (cs near 1). Size characteristic cA . Considering the minimum bounding box of P the covered area can be easily computed. The normalized histogram HA (→ Eq. 4) consists of h = 6 bins with the boundaries [0; (25cm)2 [, [(25cm)2 ; (50cm)2 [, [(50cm)2 ; (100cm)2 [, [(100cm)2 ; (200cm)2 [, [(200cm)2 ; (300cm)2 [, and [(300cm)2 ; ∞[. Size ratio characteristic c÷ . For a plane pair (Pk , Pl ) the size ratio denotes with a value near 0 that two planes significantly differ in their size whereas the planes cover nearly the same area if the value is near 1. The histogram H÷ is designed equally to histogram Hs (h = 5, Δh = 0.2). Assuming m(m−1) pairs of planes 2 the feature vector is computed using Eq. 5. Angle characteristic c . Using Eq. 1, the acute angle between two planes (Pk , Pl ) can be computed using the planes’ π normals. Eq. 6 gives the corresponding histogram H (h = 9, Δh = 18 , [0, π2 ]). Scene Features From 2D Data. Due to the fact that the ToF camera delivers additionally to the 3D point cloud an amplitude image which can be treated like a normal grayscale image we are able to compute Torralba’s Gist features4 . As the feature vector x3D captures information about the arrangement of planar patches in the scene, xGist encodes additional global scene information. These Gist features are texture features computed from a wavelet image decomposition [14]. Each image location is represented by the output of filters tuned to different orientations and scales. Here, we used 8 orientations and 4 scales applied to a 144 × 144 clipping of the 176 × 144 amplitude image bilinear resized 4
http://people.csail.mit.edu/torralba/code/spatialenvelope
Indoor Scene Classification Using Combined 3D and Gist Features
207
to 256 × 256. The resulting representation is downsampled to 4 × 4 pixels. Thus, the dimensionality of xGist is 8 × 4 × 16 = 512. Fig. 3c shows the average of filter t responses on the amplitude image. Since the depth values of the Swissranger are organized in a regular 2D grid this so-called depth image can also be used to compute in the same way a so-called Depth-Gist feature vector, xDGist . It is an alternative approach to compute features from 3D data. Training Room Models and Combining Single Classifications. In the learning stage of our system, we decided to learn a discriminant model for each class which can be used to compute the distance of a feature vector to each class boundary. As we use Support Vector Machines (SVM) the class boundaries are defined by the learned support vectors. The learning of a class model is realized by a one-against-all learning technique where feature vectors which are in-class form the positive samples. The same amount of negative examples is sampled uniformly from all other classes. Formally written, for our set of six classes Ω ={bath., bed., eat., kit., liv., off.} we learn a set of discriminant functions G(x) which map the n-dimensional feature vector x on a |Ω|-dimensional distance vector d with di denoting the distance to the class boundary of ωi [33]: (7) G(x) : x → d = d1 , . . . , d6 = di = gi (x) | gi : n → i=1...6 , E d : d → e = e1 , . . . , e6 = 0, ej = 1, 0 with dj = max{di }i=1,...,6(8) . The classification is then realized by our decision function E(.) which maps the distance vector d to a binary vector e where the j-th component is equal 1 if dj = max{. . . , di , . . .} and all other components are set to 0. In our scenario a sequence of frames will be recorded from an unknown room while tilting and panning the camera. A classification result for a single frame might not be very reliable as the view field of the camera is limited so that only a small part of the room can be seen per frame. By considering the classification results of a set of consecutive frames {Ft }t∈Δt , the decision is becoming more reliable. In the following, we are going to present two different fusion schemes, V1×E and V2×E . Voting Scheme V2×E . It relies on the idea to perform a classification decision on the frame level, to sum decisions within the window Δt weighted by their reliability, and to make a second decision on the resulting sum: ¨ Δt = ¨ Δt , d ¨Δt = E d e (9) A dt · E dt , dt = G xt A d = l dj1 − l dj2 , dj1 = max{di },
dj2
t∈Δt
1 . 1 + e−x = max{di \dj1 } l(x) =
(10) (11)
Eq. 11 determines the reliability of a decision by the difference between the value of the best and the value of the second best class. The bigger the difference the more reliable is the decision. For a plausible comparison of different classification results the distances to the class boundaries have to be normalized. For this
208
A. Swadzba and S. Wachsmuth
purpose we have chosen the strictly monotic logistic function l(.) which maps the distances on values in ]0, 1[ resulting in the weighting function A(.) (→ Eq. 10). Voting Scheme V1×E . Making a winner-takes-all class decision based on one frame might be vulnerable to noise. Therefore, an alternative approach is to skip the decision on the frame level. Instead, it passes the support for all classes to the final classification. The benefit of this scheme arises in cases where some frames give more or less equal support for two or more classes. Intuitively, making a hard decision towards one of these classes involves a high risk for misclassification. The idea is to lower this risk by keeping the support for all classes and counting on frames in Δt with significant support for one class providing a reliable class decision. Formulating this mathematically, normalized distance vectors dt are collect over time resulting in an accumulated distance vector d˙ Δt on which the decision function E(.) is applied: e˙ Δt = E d˙ Δt , d˙ Δt = L dt , L d = . . . , l(di ), . . . . (12) t∈Δt
Combining Different Feature Types. The voting and weighting technique based on different sums allows also for a straight forward combination of different feature types with differing dimensions. In our case, a set of discriminant functions G3D (.) is learned based on 3D features and another set GGist (.) based on Gist features . For a frame window {Ft }t∈Δt and a set of feature vectors Gist 3D {x3D t , xt }t∈Δt the corresponding Gist distance vectors are denoted by dt = Gist Gist and d x . The combined decision is formulated as G3D x3D = G t t t follows using the described voting schemes: 3D 3D 3D ¨ = ¨ Gist , d ¨ +d ¨Comb E dt , (13) =E d A d3D e Δt Δt Δt Δt t ¨ Gist d Δt
=
t∈Δt
3D d˙ Δt =
t∈Δt
t∈Δt
3D Gist Comb Gist E dt e˙ Δt = E d˙ Δt + d˙ Δt , A dGist t
(14)
Gist , d˙ Δt = . L d3D L dGist t t
(15)
t∈Δt
Rejection. Depending on the selected window size, the speed of the camera drive, and the current frame rate of the camera the actually acquired frames might only show uninformative scene views or may be disturbed by persons moving in front of the camera. The resulting classifications are unreliable and can be excluded by using the modified decision function E (.) (→ Eq. 16). Classifications are rejected (0) if distances to the class boundary of the best class and the second best class do not differ significantly. During evaluation θrej = 0.05 is used leading to rejection of not more than 20% of the test frames. We apply E (.) only in the final classification stage to vector dΔt . d −d 1 : j1dj j2 > θrej , 1 E d = 0, ej1 , 0 , where ej1 = (16) 0 : else
1
1
0.9
0.9 classification rate
classification rate
Indoor Scene Classification Using Combined 3D and Gist Features
0.8 0.7 0.6 0.5
0.8 0.7 0.6 0.5 0.4
0.4 50
100
150 window size
200
250
50
300
1 0.9 classification rate
1 0.9 0.8 0.7 0.6 0.5 0.4
100
150 window size
200
250
300
(b) voting scheme: V1×E ; rej.: no
(a) voting scheme: V1×E ; rej.: θrej = 0.05
classification rate
209
0.8 0.7 0.6 0.5 0.4
50
100
150 window size
200
250
300
(c) voting scheme: V2×E ; rej.: θrej = 0.05
50
100
150 window size
200
250
300
(d) voting scheme: V2×E ; rej.: no
Fig. 4. This figure shows the development of the classification results when enlarging the window Δt from 1 to 300 frames (→ x-axis). Two voting schemes are evaluated V1×E and V2×E introduced in Eq. 15 and 14. The tested feature vectors and combinations are: green → (x3D , xGist ), orange → (xDGist , xGist ), blue→ xGist , cyan → xDGist , ¯ denotes the mean standard deviation of the classifired → x3D , and black → xSP . σ cation curve. r¯ denotes the mean rejection (rej.) rate. (best viewed in color)
5
Evaluation of Room Type Categorization
This section describes the evaluation of the performance in determining the room type for a set of frames of an unknown room, using our proposed combination of spatial 3D and Gist features (x3D , xGist ), Eq. 14 and Eq. 15. Its performance is compared to classification results using pure x3D or pure xGist . Additionally, the Depth-Gist feature vector xDGist and its combination with the standard Gist feature vector, (xDGist , xGist ), is tested as an alternative combination of features from 2D and 3D data. We also apply a scene classification scheme to our dataset that is based on spatial pyramid matching proposed by Lazebnik et al. [19]. The required dictionary is built on the amplitude images of our training sets with a dictionary size of 100 and 8500-dimensional feature vectors5 , xSP . The evaluation utilizes the 3D indoor database of rooms presented in Sec. 3. All 2D descriptors are computed on the amplitude image to achieve a fair comparison for the 3D descriptor as the same amount of scene details is considered. Further, to avoid a bias arising from chosen test sequences all presented classification 5
http://www.cs.unc.edu/~ lazebnik/
210
A. Swadzba and S. Wachsmuth
results are averaged over 10 training runs. Within each run a room per class is chosen randomly extracting all frames of the room as test sequence while the frames of the remaining rooms form the training set. For a better comparison of the different features SVM models with an RBF kernel are trained where the kernel’s parameters are optimized through a 10-crossvalidation scheme choosing a model with the smallest possible set of support vectors (Occam’s razor law). Here, we utilized the SVMlight implementation of the support vector machine algorithm [34, 35]. Additionally, the models learnt on the IKEA database are tested for their applicability on Swissranger frames acquired in two real flats. An important parameter of our system is the number of consecutive frames in Δt used for determining the current class label through voting. As our sequences are recorded with a camera standing at one or two positions and being moved continuously simulating a robot’s head looking around, the possible values could range from one frame to all frames of the sequence. Performing room category decision on one frame is expected to be very fast but vulnerable to noise. The more frames are considered the more stable the classification result should become. There is a trade-off between getting a room type label quickly and reliably. The curves in Fig. 4 show the development of the classification rates if the window size Δt is enlarged from 1 to 300 frames. As assumed the performance of all features increases if the window gets bigger. Taking a deeper look on all subfigures of Fig. 4 it can be seen that our proposed combination (x3D , xGist ) of spatial 3D and Gist features clearly outperforms the other features under all voting and rejection conditions. E.g., the classification rates in Fig. 4a (mean: 0.84, max: 0.94) are higher than the corresponding values of the second best curve (which are 0.71 and 0.79 using combined Depth-Gist and Gist features). The error is reduced by 45 % and 71 %, respectively. This reduction is especially remarkable, because it points out that features needed to be carefully defined on 3D data in order to capture informative spatial structures in a sufficiently generalized way. Here, we are able to show that this can be done with lower dimensional features. Figure 4 shows also the influence of different voting and rejection schemes on the categorization performance. Skipping the winner-takes-all decision on the frame level, Eq. 15, influences the pure Gist based approaches only marginally while classification rates using x3D are significantly increased from (mean: 0.55, max: 0.60) to (mean: 0.65, max: 0.80). As shown in Fig. 4a the spatial 3D features perform comparable to Gist features, even though the dimensionality of 25 is 20 times smaller than the 512 dimensions of the Gist vectors. Lazebnik’s spatial pyramid features, xSP , show the worst performance even though 8500dimensional feature vectors are defined. The high mean standard deviation value of 0.19 arises from the fact that they work well for one or two classes while they do not work at all for other classes (→ Fig. 5). Rejection of frames when no clear decision can be taken is a reasonable step as some frames will not show meaningful views of a scene. As expected the classification results are improved (compare 4a with 4b and 4c with 4d). The influence of rejection is higher on voting scheme V1×E than on V2×E . Taking decisions on
Indoor Scene Classification Using Combined 3D and Gist Features
211
Fig. 5. This figure shows the confusion matrices of all examined features. Here, the window size for voting over consecutive frames is set to Δt = 60. The average classification rates for individual classes are listed along the diagonal, all other entries show misclassifications.
the frame level leads to clearer decisions on the window level with the drawback of being more vulnerable to noise. In Fig. 5 confusion matrices of the curves in Fig. 4a at window size Δt = 60 are given. Taking into account the frame rate of the camera there is an initial delay of 2 to 6 seconds before the system would deliver class labels or reach its full categorization performance. The confusion matrices of x3D and xGist give an impression of their complementary nature. Therefore, fusion of both features leads to improved indoor scene classification capabilities. To test the generalizability of the room models trained on the entire IKEA database we additionally acquired sequences from rooms (bedroom, eating place, kitchen, and living room) of two real flats, F1 and F2. Per room 300 frames were acquired with the Swissranger positioned at the door opening. The room models are acquired using 3D and Gist features combined with voting scheme V1×E over a voting window Δt = 60. Fig. 6 shows pictures of the rooms and the distribution of the resulting class labels using the learnt room models. The continuous red line denotes the ground truth labeling and the black circles mark the
212
A. Swadzba and S. Wachsmuth
Fig. 6. This figure shows the classification results of test sequences (bed.F1, eat.F1, kit.F1, liv.F1, bed.F2, eat.F2, kit.F2, and liv.F2) acquired in two real flat, F1 and F2. The red line refers to the ground truth and the black circles mark the classification results. The shown results are achieved by using 3D features and Gist features in combination, the voting scheme V1×E and a voting window of size Δt = 60. (best viewed in color)
classification results. In general, a quite good overall categorization performance of 0.65 and 0.84 is achieved. Especially, the sequences, kit.F1, kit.F2, liv.F1, liv.F2, and eat.F2 are correct recognized nearly over the whole sequence. Only some frames in the middle of kit.F2 are rejected or wrong classified due to the fact that the camera was directed towards the kitchen window which let to a lot of noisy data because of the disturbing effect of the sun light on the ToF measurement principle. This effect is also responsible for the misclassification of the complete eat.F1-sequence. Suppressing this effect is a subject to a technical solution since there already exists a suppression of background illumination for ToF-sensors [36]. Also, an atypical missing or arrangement of furniture, like only a bed in a room or the bed placed in the corner, leads to misclassifications and rejections as happened for bed.F1 and bed.F2. This problem can be tackled by introducing such rooms into the training of the room models. Finally, we can state that our light-weighted 25-dimensional feature vector, x3D , encodes enough meaningful information about the structural properties of rooms leading to a remarkable good performance in determining the room type for a given sequence of frames. The small dimensionality of our 3D features recommends this features for the use on robots with limited computing resources and restricted capabilities to acquire thousands of training images. Complementary information from Gist features xGist computed on the corresponding amplitude images further improves the categorization.
Indoor Scene Classification Using Combined 3D and Gist Features
6
213
Conclusion
In this paper we have shown that feature vectors capturing the 3D geometry of a scene can significantly contribute to solve the problem of indoor scene categorization. We introduce a 3D indoor database and propose a method for defining features on extracted planar surfaces that encode the 3D geometry of the scene. Additionally, we propose voting techniques which allows to fuse single classification results from several consecutive frames and different feature types like, e.g., 3D features and 2D Gist features. In our experiments, we have been able to show that Gist features and 3D features complement one another when combined through voting. The 3D cue leads earlier to better labeling results. As the fusion of classification results from consecutive frames provide a more stable scene categorization result we plan to investigate the categorization performance on registered 3D point clouds. Such a registered cloud holds a bigger part of the scene which should lead to a more stable classification ability compared to the current situation where the scene classification has to be done on a partial scene view. Also we would like to introduce class hierarchies where on the top level the room types like kitchen are located and on the lower levels subparts of the scenes like “a wall with bookshelves” or “sideboard-like-furniture for placing things on it”. Acknowledgment. This work is funded by the Collaborative Research Center CRC 673 “Alignment in Communication”.
References 1. Oliva, A., Torralba, A.: Modeling the shape of the scene: A holistic representation of the spatial envelope. IJCV 42, 145–175 (2001) 2. Quattoni, A., Torralba, A.: Recognizing indoor scenes. In: CVPR (2009) 3. Henderson, J.M., Larson, C.L., Zhu, D.C.: Full scenes produce more activation than close-up scenes and scene-diagnostic objects in parahippocampal and retrosplenial cortex: An fMRI study. Brain and Cognition 66, 40–49 (2008) 4. Henderson, J.M., Larson, C.L., Zhu, D.C.: Cortical activation to indoor versus outdoor scenes: An fMRI study. Experimental Brain Research 179, 75–84 (2007) 5. Topp, E., Christensen, H.: Dectecting structural ambiguities and transistions during a guided tour. In: ICRA, pp. 2564–2570 (2008) 6. Ullah, M.M., Pronobis, A., Caputo, B., Luo, J., Jensfelt, P., Christensen, H.I.: Towards Robust Place Recognition for Robot Localization. In: ICRA (2008) 7. Ranganathan, A., Dellaert, F.: Semantic modeling of places using objects. In: Robotics: Science and Systems (2007) ´ 8. Mozos, O.M., Stachniss, C., Burgard, W.: Supervised learning of places from range data using adaboost. In: ICRA, pp. 1730–1735 (2005) 9. Zender, H., Mozos, O., Jensfelt, P., Kruijff, G.J.M., Burgard, W.: Conceptual spatial representations for indoor mobile robots. RAS 56, 493–502 (2008) 10. Galindo, C., Saffiotti, A., Coradeschi, S., Buschka, P., Fern´ andez-Madrigal, J.A., Gonz´ alez, J.: Multi-hierarchical semantic maps for mobile robotics. In: IROS, pp. 3492–3497 (2005)
214
A. Swadzba and S. Wachsmuth
11. Vasudevan, S., G¨ achter, S., Nguyen, V., Siegwart, R.: Cognitive maps for mobile robots - an object based approach. RAS 55, 359–371 (2007) 12. Kim, S., Kweon, I.S.: Scene interpretation: Unified modeling of visual context by particle-based belief propagation in hierarchical graphical model. In: Narayanan, P.J., Nayar, S.K., Shum, H.-Y. (eds.) ACCV 2006. LNCS, vol. 3852, pp. 963–972. Springer, Heidelberg (2006) 13. Pirri, F.: Indoor environment classification and perceptual matching. In: Intl. Conf. on Knowledge Representation, pp. 30–41 (2004) 14. Torralba, A., Murphy, K.P., Freeman, W.T., Rubin, M.A.: Context-based vision system for place and object recognition. In: ICCV (2003) 15. Fei-Fei, L., Perona, P.: A bayesian hierarchical model for learning natural scene categories. In: CVPR (2005) 16. Posner, I., Schr¨ oter, D., Newman, P.M.: Using scene similarity for place labelling. In: Intl. Symp. on Experimental Robotics, pp. 85–98 (2006) 17. Bosch, A., Zisserman, A., Mu˜ noz, X.: Scene classification using a hybrid generative/discriminative approach. PAMI 30, 712–727 (2008) 18. Vogel, J., Schiele, B.: A semantic typicality measure for natural scene categorization. In: Rasmussen, C.E., B¨ ulthoff, H.H., Sch¨ olkopf, B., Giese, M.A. (eds.) DAGM 2004. LNCS, vol. 3175, pp. 195–203. Springer, Heidelberg (2004) 19. Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: CVPR (2006) 20. Huber, D.F., Kapuria, A., Donamukkala, R., Hebert, M.: Parts-based 3D object classification. In: CVPR, pp. 82–89 (2004) 21. Cs´ ak´ any, P., Wallace, A.M.: Representation and classification of 3D objects. Systems, Man, and Cybernetics 33, 638–647 (2003) 22. Johnson, A.E., Herbert, M.: Using spin images for efficient object recognition in cluttered 3d scenes. PAMI 21, 433–449 (1999) 23. Munoz, D., Bagnell, J.A., Vandapel, N., Hebert, M.: Contextual classification with functional max-margin markov networks. In: CVPR (2009) ´ 24. Triebel, R., Schmidt, R., Mozos, O.M., Burgard, W.: Instance-based amn classification for improved object recognition in 2D and 3D laser range data. In: IJCAI, pp. 2225–2230 (2007) 25. Rusu, R.B., Marton, Z.C., Blodow, N., Beetz, M.: Learning informative point classes for the acquisition of object model maps. In: Intl. Conf. on Control, Automation, Robotics and Vision, Hanoi, Vietnam (2008) 26. Lee, D., Hebert, M., Kanade, T.: Geometric reasoning for single image structure recovery. In: CVPR (2009) 27. Hedau, V., Hoiem, D., Forsyth, D.: Recovering the spatial layout of cluttered rooms. In: ICCV (2009) 28. Yu, S.X., Zhang, H., Malik, J.: Inferring spatial layout from a single image via depth-ordered grouping. In: CVPR Workshops (2008) 29. Weingarten, J., Gruener, G., Siegwart, R.: A state-of-the-art 3D sensor for robot navigation. In: IROS, vol. 3, pp. 2155–2160 (2004) 30. Hois, J., W¨ unstel, M., Bateman, J., R¨ ofer, T.: Dialog-based 3D-image recognition using a domain ontology. In: Barkowsky, T., Knauff, M., Ligozat, G., Montello, D.R. (eds.) Spatial Cognition 2007. LNCS (LNAI), vol. 4387, pp. 107–126. Springer, Heidelberg (2007) 31. Stamos, I., Allen, P.K.: Geometry and texture recovery of scenes of large scale. CVIU 88, 94–118 (2002)
Indoor Scene Classification Using Combined 3D and Gist Features
215
32. Fischler, M.A., Bolles, R.C.: Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24, 381–395 (1981) 33. Kuncheva, L.I.: Combining Pattern Classifiers. John Wiley & Sons, Chichester (2004) 34. Vapnik, V.N.: The Nature of Statistical Learning Theory (1995) 35. Joachims, T.: Learning to Classify Text Using Support Vector Machines. PhD thesis, Cornell University (2002) 36. M¨ oller, T., Kraft, H., Frey, J., Albrecht, M., Lange, R.: Robust 3D measurement with PMD sensors. In: 1st Range Imaging Research Day at ETH (2005)
Closed-Form Solutions to Minimal Absolute Pose Problems with Known Vertical Direction Zuzana Kukelova1 , Martin Bujnak2 , and Tomas Pajdla1 1
Center for Machine Perception, Czech Technical University in Prague 2 Bzovicka 24, 85107, Bratislava, Slovakia
Abstract. In this paper we provide new simple closed-form solutions to two minimal absolute pose problems for the case of known vertical direction. In the first problem we estimate absolute pose of a calibrated camera from two 2D-3D correspondences and a given vertical direction. In the second problem we assume camera with unknown focal length and radial distortion and estimate its pose together with the focal length and the radial distortion from three 2D-3D correspondences and a given vertical direction. The vertical direction can be obtained either by direct physical measurement by, e.g., gyroscopes and inertial measurement units or from vanishing points constructed in images. Both our problems result in solving one polynomial equation of degree two in one variable and one, respectively two, systems of linear equations and can be efficiently solved in a closed-form. By evaluating our algorithms on synthetic and real data we demonstrate that both our solutions are fast, efficient and numerically stabled.
1
Introduction
Cheap consumer cameras become precise enough to be used in many computer vision applications, e.g. structure from motion [2, 15, 22, 23], or recognition [13, 14]. The good precision of these cameras allows to simplify camera models and therefore also algorithms used in these applications. For instance, for the pinhole camera model it is common to set the camera skew to zero, the pixel aspect ratio to one and the principal point to the center of the image. Moreover, since most of the digital cameras put the information about the focal length into the image header (EXIF), it is frequently assumed that this is a good approximation of the whole internal calibration of the camera and therefore the camera is considered calibrated. These assumptions allow to use simpler algorithms, e.g. the 5-point relative pose algorithm for calibrated cameras [19, 24], and as it was shown in [12, 15, 16, 22, 23] they work well even when these assumptions are not completely satisfied. More and more cameras become also equipped with inertial measurement units (IMUs) like gyroscopes and accelerometers, compasses, or GPS devices and add information from these devices into the image header too. This could be even more observable in modern cellular phones and other smart devices. Unfortunately, since GPS position precision is in order of meters, it is not sufficient to provide external camera calibration, i.e. the absolute position of the R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 216–229, 2011. Springer-Verlag Berlin Heidelberg 2011
Closed-Form Solutions to Minimal Absolute Pose Problems
217
camera. Neither compass accuracy (which could provide cameras yaw) is good enough to provide camera orientation since its sensor is highly influenced by magnetic field disturbances. On the other hand currently available IMUs (even the low cost ones) provide very accurate roll and pitch angle, i.e. the vertical direction. The angular accuracy of roll and pith angle in low cost IMUs like [26] is about 0.5◦ , and in high accuracy IMUs is less than 0.02◦ . While most today cameras use information from gyroscopes and IMUs only to distinguish whether the image orientation is landscape or portrait, we show that knowing the camera “up vector” can radically simplify the camera absolute pose problem. Determining the absolute pose of a camera given a set of n correspondences between 3D points and their 2D projections is known as the Perspective-n-Point (PnP) problem. This problem is one of the oldest problems in computer vision with a broad range of applications in structure from motion [2, 15] or recognition [13, 14]. PnP problems for fully calibrated cameras for three and more than three points have been extensively studied in the literature. One of the oldest papers on this topic dates already to 1841 [7]. Recently a large number of solutions to the calibrated PnP problems have been published [5, 18, 20, 21] including several solutions to the PnP problems for partially calibrated cameras [1, 3, 9, 25]. These solutions assume unknown focal length [1, 3, 25], unknown focal length and radial distortion [9] or unknown focal length and principal point [25]. In this paper we provide new simple closed-form solutions to two minimal absolute pose problems for cameras with known rotation around two angles, i.e. known vertical direction. In the first problem we estimate the absolute pose of a calibrated camera from two 2D-3D correspondences and a given vertical direction. In the second problem we assume camera with unknown focal length and radial distortion and estimate its pose together with the focal length and the radial distortion from three 2D-3D correspondence and a given vertical direction. In both cases this is the minimal number of correspondences needed to solve these problems. Both these problems result in solving one polynomial equation of degree two in one variable and one respectively two systems of linear equations and can be efficiently solved in a closed-form way. By evaluating our algorithms on synthetic and real data we show that both our solutions are fast, efficient and numerically stable. Moreover, both presented algorithms are very useful in real applications like structure from motion, surveillance camera calibration or applications of registering photographs taken by a camera mounted on a car moving on a plane. Our work builds on recent results from [10] where the efficient algorithm for relative pose estimation of a camera with known vertical direction from three point correspondences was presented. It was demonstrated that in the presence of good vertical direction information this algorithm is more accurate than the classical 5-point relative pose algorithm [19, 24] for calibrated cameras. Next we provide the formulation of our two problems and show how they can be solved
218
2
Z. Kukelova, M. Bujnak, and T. Pajdla
Problem Formulation
In this section we formulate two problems of determining absolute pose of a camera given 2D-3D correspondences and the vertical direction, respectively rotation angles around two axes. Problem 1. Given the rotation of the calibrated camera around two axes and the images of two known 3D reference points, estimate the absolute position of the camera and the rotation of the camera around the third axis (y-axis). Problem 2. Given the rotation of the camera with unknown focal length around two axes and the distorted images of three known 3D reference points, estimate the absolute position of the camera, the rotation of the camera around the third axis (y-axis), the unknown focal length and the parameter of radial distortion. In both problems we use the standard pinhole camera model [8]. In this model the image projection u of a 3D reference point X can be written as λ u = P X,
(1)
where P is a 3 × 4 projection matrix, λ is an unknown scalar value and points u = [u, v, 1] and X = [X, Y, Z, 1] are represented by their homogeneous coordinates. The projection matrix P can be written as P = K [R | t],
(2)
where R is a 3×3 rotation matrix, t = [tx , ty , tz ] contains the information about camera position and K is the 3 × 3 calibration matrix of the camera. In our problems we assume that we know the vertical direction, i.e. the co ordinates of the world vector [0, 1, 0] in the camera coordinate system. This vector can be often obtained from the vanishing point or directly from the IMU by a very simple calibration of the IMU w.r.t. the world coordinate system. It is important that all what has to be known to calibrate the IMU w.r.t. the world coordinate system, is the direction of the “up-vector”, i.e. the direction which is perpendicular to the “ground plane”, i.e. the plane z = 0 in the world coordinate system. This direction is easy to calibrate by reseting the IMU when lied down on the “ground plane”. Note that this “ground plane” need not be horizontal. Moreover, very often IMU’s are precalibrated to use the gravity vector as the up-vector. Then, the IMU is calibrated when the plane z = 0 of the world coordinate system corresponds to the ground plane, i.e. if the coordinates z = 0 are assigned to the points on the ground plane. From the “up-vector” returned by the IMU rotation we can compute the rotation of the camera around two axes, in this case the x-axis (φx ) and the z-axis (φz ). Note that IMU sometimes returns directly these two angles. We will denote this rotation as ⎤ ⎡ ⎤⎡ cos φz − sin φz 0 1 0 0 Rv = ⎣ sin φz cos φz 0 ⎦ ⎣ 0 cos φx − sin φx ⎦ . (3) 0 sin φx cos φx 0 0 1
Closed-Form Solutions to Minimal Absolute Pose Problems
219
Therefore, the only unknown parameter in the camera rotation matrix R is the rotation angle φy around the y-axis, i.e. the vertical axis and we can write R = R (φy ) = Rv Ry (φy ) ,
(4)
where Rv is the known rotation matrix (3) around the vertical axis and Ry (φy ) is the unknown rotation matrix around the y-axis of the form ⎤ ⎡ cos φy 0 − sin φy ⎦. 0 (5) Ry = ⎣ 0 1 sin φy 0 cos φy With this parametrization the projection equation (1) has the form λu = K [R (φy ) | t]X = K [Rv Ry (φy ) | t]X,
(6)
For our problems we “simplify” the projection equation (6) by eliminating the scalar value λ and trigonometric functions sin and cos. φ To eliminate sin and cos we use the substitution q = tan 2y for which it holds that cos φy =
1−q2 1+q2
and sin φy =
2q 1+q2 .
Therefore we can write ⎡ ⎤ 1 − q2 0 −2q 1 + q 2 Ry (q) = ⎣ 0 1 + q 2 0 ⎦ . 2q 0 1 − q2
(7)
The scalar value λ from the projection equation (6) can be eliminated by multiplying equation (6) with the skew symmetric matrix [u]× . Since [u]× u = 0 we obtain the matrix equation [u]× K [Rv Ry (q) | t]X = 0.
(8)
This matrix equation results in three polynomial equations from which only two are linearly independent. This is caused by the fact that the skew symmetric matrix [u]× has rank two. Polynomial equations (8) are the basic equations which we use for solving our two problems. Next we describe how these equations look for our two problems and how they can be solved.
3
Absolute Pose of a Calibrated Camera with Known Up Direction
In the case of Problem 1 for calibrated camera the calibration matrix K is known and therefore the projection equation (8) [u]× K [Rv Ry (q) | t]X = 0 has the form ⎡ ⎤ ⎤ X ⎡ ⎤⎡ 0 −1 v r11 (q) r12 (q) r13 (q) tx ⎢ ⎥ ⎣ 1 0 −u ⎦ ⎣ r21 (q) r22 (q) r23 (q) ty ⎦ ⎢ Y ⎥ = 0, (9) ⎣Z⎦ r31 (q) r32 (q) r33 (q) tz −v u 0 1
220
Z. Kukelova, M. Bujnak, and T. Pajdla
where u is a calibrated image point and rij (q) are the elements of the rotation matrix R = Rv Ry (q). Note that these elements are quadratic polynomials in q. φ In this case we have four unknowns tx , ty , tz , q = tan 2y and since each 2D3D point correspondence gives us two constraints, two independent polynomial equations of the form (9), the minimal number of point correspondences needed to solve this problem is two. Matrix equation (9) gives us two linearly independent equations for each point and each from this equations contains monomials q 2 , q, tx , ty , tz , 1. Since we have two points, we have four of these equations and therefore we can use three from them to eliminate tx , ty and tz from the fourth one. In this way we obtain one polynomial in one variable q of degree two. Such equation can be easily φ solved in a closed-form and generally results in two solutions for q = tan 2y . Backsubstituting these two solutions to the remaining three equations give us three linear equations in three variables tx , ty and tz . By solving these linear equations we obtain solutions to our problem.
4
Absolute Pose of a Camera with Unknown Focal Length and Radial Distortion and Known Camera Up Direction
In Problem 2 we assume that the skew in the calibration matrix K is zero, the aspect ratio is one and the principal point is in the center of the image. Modern cameras are close to these assumptions and as it will be shown in experiments these assumptions give good results even when they are not exactly satisfied. The only parameter from K which cannot be safely set to a known value is the focal length f . Here we assume that the focal length f is unknown and therefore the calibration matrix K has the form diag [f, f, 1]. Since the projection matrix is given only up to a scale we can equivalently write K = [1, 1, w] for w = 1/f . For this problem we further assume that the image points are affected by some amount of radial distortion. Here we model this radial distortion by the one-parameter division model proposed in [6]. This model is given by formula (10) pu ∼ pd / 1 + krd2 ,
where k is the distortion parameter, pu = [uu , vu , 1] , resp. pd = [ud , vd , 1] , are the corresponding undistorted, resp. distorted, image points, and rd is the radius of pd w.r.t. the distortion center. We assume that the distortion center is in the center of the image and therefore rd2 = u2d + vd2 . Since the projection equation (8) contains undistorted image points u but we measure distorted ones we need to put the relation (10) into equation (8). ˆ i = [ˆ ui , vˆi , 1] be the ith measured distorted image point, projection of the Let u
3D point Xi = [Xi , Yi , Zi , 1], and let rˆi2 = uˆ2i + vˆi2 . Then ui = uˆi , vˆi , 1 + kˆ ri2 and the projection equation (8) for the ith point has the form
Closed-Form Solutions to Minimal Absolute Pose Problems
⎡
0 −1 − ⎣ 1 + kˆ ri2 0 −ˆ vi u ˆi
kˆ ri2
221
⎡ ⎤ ⎤ Xi vˆi 1 0 0 r11 (q) r12 (q) r13 (q) tx ⎢ ⎥ Yi ⎥ −ˆ ui ⎦⎣ 0 1 0 ⎦⎣ r21 (q) r22 (q) r23 (q) ty ⎦⎢ ⎣ Zi ⎦ = 0, (11) 0 0 0 w r31 (q) r32 (q) r33 (q) tz 1 ⎤⎡
⎤⎡
where again rij (q) are the elements of the rotation matrix R = Rv Ry (q). φ In this case we have six unknowns tx , ty , tz , q = tan 2y , k, w = 1/f and since each 2D-3D point correspondence gives us two constraints the minimal number of point correspondences needed to solve this problem is three. For three point correspondences the matrix equation (11) gives us nine polynomial equations from which only six are linearly independent. Let denote these nine polynomial equations g1i = 0, g2i = 0 and g3i = 0 for i = 1, 2, 3, where g1i = 0 is the equation corresponding to the first row in the matrix equation (11), g2i = 0 to the second and g3i = 0 to the third row. Now consider the third polynomial equation from the matrix equation (11), i.e. the equation g3i = 0. This equation has the form g3i = c1 q 2 + c2 q + c3 tx + c4 ty + c5 = 0,
(12)
where cj , j = 1, . . . , 5 are known coefficients. The polynomial equation (12) doesn’t contain variables tz , k and w. Since we have three 2D-3D correspondences we have three equations g3i = 0, i = 1, 2, 3. Therefore, we can use two from these equations to eliminate tx and ty from the third one, e.g. from the g31 = 0. In this way we obtain one polynomial equation in one variable q of degree two. Such equation can be easily solved in a closed-form and generally results in two φ solutions for q = tan 2y . Backsubstituting these two solutions to the remaining two equations corresponding to the third row of (11), i.e. g32 = 0 and g33 = 0, gives us two linear equations in two variables tx and ty which can be again easily solved. After obtaining solutions to q, tx and ty we can substitute them to the equations corresponding to the first row of (11). These equations have after the substitution the form g1i = c6 tz w + c7 w + c8 k + c9 = 0
(13)
for i = i, . . . , 3 and known coefficients cj , j = 6, . . . , 9. Equations 13 can be again solved in a linear way. This is because we can consider the monomial tz w as a new variable and solve a linear system of three equations in three variables. Solving this linear system together with solutions to q, tx and ty gives us in general two solutions to our absolute pose problem for camera with unknown focal length and radial distortion given camera rotation around two axis.
5
Experiments
We have tested both our algorithms on synthetic data (with various levels of noise, outliers, focal lengths and radial distortions and different angular deviation
222
Z. Kukelova, M. Bujnak, and T. Pajdla
of the vertical direction) and on real datasets and compared them with the P3P algorithm for calibrated camera presented in [5], the P4P algorithm for camera with unknown focal length from [3] and with the P4P algorithm for camera with unknown focal length and radial distortion [9]. In all experiments we denote our new algorithms as up2p and up3p+f+k, and compared algorithms as p3p [5], p4p+f [3] and p4p+f+k [9]. 5.1
Synthetic Data Set
We initially studied our algorithms on synthetically generated ground-truth 3D scenes. These scenes were generated randomly with the Gaussian distributions in a 3D cube. Each 3D point was projected by a camera, where the camera orientation and position were selected randomly but looking on the scene. Then the radial distortion using the division model [6] was added to all image points to generate noiseless distorted points. Finally, Gaussian noise with standard deviation σ was added to the distorted image points assuming a 1000 × 1000 pixel image. Noise free data set. In the first synthetic experiment, we have studied the numerical stability of both proposed solvers on exact measurements and compared them with the algorithms [3, 5, 9]. In this experiment 1000 random scenes and camera poses were generated. The radial distortion parameter was set to kgt = −0.2 and the focal length to fgt = 1.5. In the case of our calibrated up2p algorithm, the p3p algorithm from [5] and the p4p+f algorithm [3] these values were used to precalibrate cameras. The rotation error was measured as the rotation angle in the angle-axis rep−1 and the translation error as the angle resentation of the relative rotation RRgt between ground-truth and estimated translation vector. In this case the rotation and translation errors for both our algorithms were under the machine precision and therefore we were not able to properly display them here. Figure 1 (Left) shows the log10 relative error of the focal length f obtained by selecting the real root closest to the ground truth value fgt = 1.5. The log10 absolute error of the radial distortion parameter is in Figure 1 (Right). 100 80
p4pf up3p p4pfk
200 150
up3p p4pfk
60 100
40
50
20 0
−16 −14 −12 −10 −8 −6 Log10 relative focal length error
0
−15 −10 −5 Log10 absolute radial distrion error
Fig. 1. Log10 relative error of the focal length f (Left) and Log10 absolute error of the radial distortion parameter k (Right) obtained by selecting the real root closest to the ground truth value fgt = 1.5 and kgt = −0.2
Closed-Form Solutions to Minimal Absolute Pose Problems
223
8 6 4 2 0 0
0.1
0.2
0.5
1
2
up direction angular deviation in degrees
Translation error in degrees
Rotation error in degrees
up2p 10
up2p 10 8 6 4 2 0 0
0.1
0.2
0.5
1
2
up direction angular deviation in degrees
Fig. 2. The influence of the accuracy of the vertical direction on the estimation of rotation (Left) and translation (Right) for our calibrated up2p algorithm
It can be seen that the new proposed up3p+f+k solver for camera with unknown focal length and radial distortion and given vertical direction (Magenta) provides more accurate estimates of both k and f then the p4p+f+k solver presented in [9] (Green) and the p4p+f solver from [3] (Blue). Vertical direction angular deviation. In the real applications the vertical direction obtained either by direct physical measurement by, e.g., gyroscopes and IMUs or from vanishing points constructed in images is not accurate. The angular accuracy of roll and pith angle in low cost IMUs like [26] is about 0.5◦ , and in high accuracy IMUs is less than 0.02◦. Therefore, in the next experiment we have tested the influence of the accuracy of the vertical direction on the estimation of rotation (Figure 2 (Left) and 3 (Top left)), translation (Figure 2 (Right) and 3 (Top right)), focal length (Figure 3 (Bottom left)) and radial distortion (Figure 3 (Bottom right)). In this case the ground truth focal length was fgt = 1.5 and the radial distortion kgt = −0.2. For each vertical direction angular deviation 1000 estimates of both our algorithms, the calibrated up2p (Figure 2) and the up3p+f+k with unknown focal length and radial distortion (Figure 3), were made. The vertical direction angular deviation varied from 0◦ to 2◦ . All results in Figures 2 and 3 are represented by the MATLAB function boxplot which shows values 25% to 75% quantile as a blue box with red horizontal line at median. The red crosses show data beyond 1.5 times the interquartile range. The rotation error is again measured as the rotation angle in the angle-axis −1 and the translation error as the representation of the relative rotation RRgt angle between ground-truth and estimated translation vector. Focal length and radial distortion figures show directly estimated values. It can be seen that even for relatively high error in the vertical direction the median values of estimated focal lengths and radial distortion parameters are very close to the ground truth values (Green horizontal line). Note that the angular rotation and translation errors cannot be smaller than the angular error of the vertical direction.
224
Z. Kukelova, M. Bujnak, and T. Pajdla
6 4 2 0
Focal length estimates
0 0.1 0.2 0.5 1 2 up direction angular deviation in degrees up3p+f+k 3
2
1
0
0 0.1 0.2 0.5 1 2 up direction angular deviation in degrees
Translation error in degrees
8
up3p+f+k 10 8 6 4 2 0 0 0.1 0.2 0.5 1 2 up direction angular deviation in degrees up3p+f+k Radial distortion estimates
Rotation error in degrees
up3p+f+k 10
0.2 0 −0.2 −0.4 −0.6 0 0.1 0.2 0.5 1 2 up direction angular deviation in degrees
Fig. 3. The influence of the accuracy of the vertical direction on the estimation of rotation (Top left), translation (Top right), focal length (Bottom right), radial distortion (Bottom right) for our up3p+f+k algorithm
Data affected by noise. Boxplots in Figure 4 show behavior of both our solvers, the calibrated up2p solver (Cyan) and up3p+f+k solver (Magenta), together with the p3p algorithm [5] (Red), the p4p+f algorithm from [3] (Blue) and the p4p algorithm [9] (Green) in the presence of noise added to image points. In this experiment for each noise level, from 0.0 to 2 pixels, 1000 estimates for random scenes and camera positions and fgt = 1.5, kgt = −0.2, were made.670 Here it can be seen that since our algorithms use less points than [3, 5, 9] they are little bit more sensitive to a noise added to the image points than the algorithms [3, 5, 9]. The difference is only in the rotation error (Figure 4 (Top left)) where both our new solvers outperform existing algorithms [3, 5, 9]. This might be due to the fact, that the rotation axis is fixed in both our algorithms. However, our new algorithms still provide good estimates also for translation (Figure 4 (Top right)), focal length (Bottom left) and radial distortion (Bottom right) in the presence of noise. Ransac experiment. The final synthetic experiment shows behaviour of our algorithms within the RANSAC paradigm [5]. Figure 5 shows the mean value of the number of inliers over 1000 runs of RANSAC as a function of the number of samples of the RANSAC. We again compare our calibrated up2p algorithm (Cyan) with the p3p algorithm for calibrated camera presented in [5] (Red), and our up3p+f+k algorithm for camera with unknown focal length and radial distortion (Magenta) with the p4p+f algorithm from [3] (Blue) and the p4p+f+k algorithm [9] (Green). Figure 5 (Left) shows results for 50% outliers, pixel noise
Closed-Form Solutions to Minimal Absolute Pose Problems
1.5 Translation error in degrees
Rotation error in degrees
1.5
225
1
0.5
0
1
0.5
0 0
0.1 0.5 1 noise in pixels
2
0
0.1 0.5 1 noise in pixels
2
0.1 0.5 1 noise in pixels
2
1.6 Radial distortion estimates
Focal length estimates
−0.14 1.55
1.5
1.45
1.4
−0.16 −0.18 −0.2 −0.22 −0.24 −0.26
0
0.1 0.5 1 noise in pixels
2
0
Fig. 4. Error of rotation (Top left), translation (Top right), focal length estimates (Bottom left) and radial distortion estimates (Bottom right) in the presence of noise for our calibrated up2p algorithm (Cyan), the calibrated p3p algorithm from [5] (Red), our p3p+f+k algorithm (Magenta), the p4p+f algorithm presented in [3] (Blue) and the p4p+f+k algorithm [9] (Green)
0.5px, fgt = 1.5, kgt = −0.2 and the vertical direction angular deviation 0.02◦ and Figure 5 (Right) results for the same configuration but the vertical direction angular deviation 1◦ . These vertical direction angle deviations reflect accessible precisions using low cost and high cost IMUs. Vertical direction precision about 1◦ can be received by taking pictures from hand using standard smartphone. As it can be seen both our new algorithms little bit outperform the remaining algorithms [3, 5, 9]. Computation times. Since both our problems result in solving one polynomial equation of degree two in one variable and one, respectively two, systems of linear equations and can be solved in a closed-form, they are extremely fast. The Matlab implementation of both our solvers runs about 0.1ms. For comparison, our Matlab implementation of the p3p algorithm [5] runs about 0.6ms, the Matlab implementation of the p4p+f algorithm [3] downloaded from [17] about 2ms and the original implementation of the p4p+f+k algorithm [9] even about 70ms. 5.2
Real Data
Synthetically generated vertical direction. In the real experiment we created 3D reconstruction from a set of images captured by an off-the-shelf camera. We used reconstruction pipeline similar to one described in [23]. We take this
Z. Kukelova, M. Bujnak, and T. Pajdla 0.5px noise level, 0.02 degree up direction deviation 300
0.5px noise level, 1.0 degree up direction deviation 300
250
250
200 150 p3p up2p p4pf up3p p4pfk
100 50 0 0
50
100 Ransac cycles
150
200
Mean support
Mean support
226
200 150 p3p up2p p4pf up3p p4pfk
100 50 0 0
50
100 Ransac cycles
150
200
Fig. 5. The mean value of the number of inliers over 1000 runs of RANSAC as a function of the number of samples of the RANSAC
5
0.6 0.4 0.2 0
0.4 Focal length relative error
Translation error in degrees
Rotation error in degrees
1 0.8
4 3 2 1
p3p
up3p+f+k p4p+f
p4p+f+k
0 −0.2 −0.4
0 up2p
0.2
up2p
p3p
up3p+f+k p4p+f
p4p+f+k
up3p+f+k
p4p+f
p4p+f+k
Fig. 6. Results of real experiment for vertical direction randomly rotated with standard deviation 0.5◦
reconstruction as a ground truth reference for further comparison even if it might not be perfect. To simulate a measurement from gyroscope we have extracted “vertical direction” from reconstructed cameras and randomly rotated them by a certain angle. This way we have created 1000 random vertical direction with normal distribution for maximal deviation of 0.02, 0.05 and 0.5 and 1 degrees to simulate industry quality, standard and low cost gyroscope measurement. Note that from the 3D reconstruction we have a set of 2D-3D tentative correspondences as well as correct correspondences. To make registration more complicated we added more outliers to the set of tentative correspondences for those cameras where fraction of correct features was greater than 50%. We randomly connected 2D measurements and 3D points to make 50% of tentative correspondences wrong (outliers). Next, for each random vertical direction we executed locally optimized RANSAC [4] with both our new algorithms. Further we also evaluated 1000 runs of locally optimized ransacs with the calibrated p3p algorithm [5], the p4p+f algorithm presented in [3] and the p4p+f+k solver from [9] and compared all results with ground truth rotation, translation and the focal length. Note that our calibrated up2p algorithm and the p3p algorithm for calibrated camera from [5] require internal camera calibration. Hence we used calibration obtained from the 3D reconstruction. Results for vertical direction randomly rotated with standard deviation 0.5◦ are in Figure 6. It can be seen that our new algorithms (up2p and up3p+f+k) provide comparable estimates of rotation (Left), translation (Center) and focal
Closed-Form Solutions to Minimal Absolute Pose Problems
227
200 mean number of inliers
mean number of inliers
300 150
100 up2p p3p up3p+f+k p4p+f p4p+f+k
50
0 0
200
400 600 ransac cycles
800
250 200 150 up2p p3p up3p+f+k p4p+f p4p+f+k
100 50 0 0
1000
200
400 600 ransac cycles
800
1000
Fig. 7. Real experiment: Example of input images (Left). Results from RANSAC on the image with small radial distortion (Center) and on the image with significant radial distortion (Right).
1 1
2
2
2
3
3 4
4
1 4
3
Fig. 8. Results of real experiment for real IMU sensor from HTC HD2
length (Right) as algorithms p3p [5], p4p+f [3] and p4p+f+k [9] even for this higher vertical direction deviation. Moreover, our algorithms use less points, give only up to two solutions which have to be verified inside ransac loop and are considerably faster than the algorithms [3, 5, 9]. All these aspects are very important in RANSAC and real applications. The mean value of the number of inliers over 1000 runs of RANSAC as a function of the number of samples of the RANSAC for two cameras and vertical direction deviation 0.5◦ can be seen in Figure 7. It can be seen that both our algorithms outperform existing algorithms [3, 5, 9]. Vertical direction obtained from real IMU. In this experiment we used HTC HD2 phone device to obtain real images with IMU data. We took by hand several images of a book placed on a A4 paper. The image size was 800×600 pixels. We assigned 3D coordinates to the paper corners and manually clicked corresponding 2D points in all input images. First, RANSAC with the new solvers was used. Then, a non-linear refinement [8] optimizing focal length, radial distortion, rotation and translation was performed. For our calibrated up2p algorithm we used radial distortion and focal length calculated using non-linearly improved result obtained from up3p+f+k.
228
Z. Kukelova, M. Bujnak, and T. Pajdla
Figure 8 shows 3D model of the paper and book back projected to images using calculated camera poses. Green wire-frame model of the book shows estimated camera from minimal sample using our algorithms (up2p (Left), up3p+f+k (Center+Right)), red color represent model after non-linear refinement. Magenta lines represent gravity vector obtained from phone device. As it can be seen we have obtained quite precise results (reprojection error about 1.5px) even for standard images taken by hand and without calibration of relative position of IMU sensor and camera inside the phone.
6
Conclusion
In this paper we have presented new closed-form solutions to two minimal absolute pose problems for the case of known vertical direction, i.e. known rotation of the camera around two axes. In the first problem we estimate absolute pose of a calibrated camera and in the second of a camera with unknown focal length and radial distortion. We show that information about the vertical direction obtained by direct physical measurement by, e.g. gyroscopes and inertial measurement units (IMUs) or from vanishing points constructed in images, can radically simplify the camera absolute pose problem. In the case of two presented problems the information about the vertical direction leads to very fast, efficient and numerically stable closed-form solutions. For comparison the only existing algorithm for estimating absolute pose of a camera with unknown focal length and radial distortion [9] without the vertical direction leads to LU decomposition of 1134 × 720 matrix and further eigenvalue computations and is very impractical in real applications. Experiments show that both our algorithms are very useful in real application like structure from motion. Moreover, since still more and more cameras and smart devices are equipped with IMUs and can save information about the vertical direction, i.e. roll and pitch angles, to the image header, we see the great future potential of presented solvers. Acknowledgement. This work has been supported by EC project FP7-SPACE218814 PRoVisG, by Czech Government research program MSM6840770038 and by Grant Agency of the CTU Prague project SGS10/072/OHK4/1T/13.
References 1. Abidi, M.a., Chandra, T.: A new efficient and direct solution for pose estimation using quadrangular targets: Algorithm and evaluation. IEEE PAMI 17(5), 534–538 (1995) 2. 2D3. Boujou, www.2d3.com 3. Bujnak, M., Kukelova, Z., Pajdla, T.: A general solution to the P4P problem for camera with unknown focal length. In: CVPR 2008 (2008) 4. Chum, O., Matas, J., Kittler, J.: Locally optimized RANSAC. In: Michaelis, B., Krell, G. (eds.) DAGM 2003. LNCS, vol. 2781, pp. 236–243. Springer, Heidelberg (2003)
Closed-Form Solutions to Minimal Absolute Pose Problems
229
5. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24(6), 381–395 (1981) 6. Fitzgibbon, A.: Simultaneous linear estimation of multiple view geometry and lens distortion. In: CVPR 2001, pp. 125–132 (2001) 7. Grunert, J.A.: Das pothenot’sche Problem, in erweiterter Gestalt; nebst Bemerkungen u ¨ber seine Anwendung in der Geod¨ asie. Archiv der Mathematik und Physik 1, 238–248 (1841) 8. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003) 9. Josephson, K., Byr¨ od, M., ˚ Astr¨ om, K.: Pose Estimation with Radial Distortion and Unknown Focal Length]. In: Proc. Conference on Computer Vision and Pattern Recognition (CVPR 2009), Florida, USA (2009) 10. Kalantari, M., Hashemi, A., Jung, F., Gu´edon, J.-P.: A New Solution to the Relative Orientation Problem using only 3 Points and the Vertical Direction, Computing Research Repository (CoRR), abs/0905.3964 (2009) 11. Kelly, J., Sukhatme, G.S.: Fast Relative Pose Calibration for Visual and Inertial Sensors. Springer Tracts in Advanced Robotics, vol. 54 (2009) 12. Li, X., Wu, C., Zach, C., Lazebnik, S., Frahm, J.-M.: Modeling and Recognition of Landmark Image Collections Using Iconic Scene Graphs. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 427–440. Springer, Heidelberg (2008) 13. Leibe, B., Cornelis, N., Cornelis, K., Gool, L.J.V.: Dynamic 3d scene analysis from a moving vehicle. In: CVPR (2007) 14. Leibe, B., Schindler, K., Gool, L.J.V.: Coupled detection and trajectory estimation for multi-object. In: ICCV (2007) 15. Martinec, D., Pajdla, T.: Robust rotation and translation estimation in multiview reconstruction. In: CVPR (2007) 16. Microsoft Photosynth, http://photosynth.net/ 17. Minimal Problems in Computer Vision, http://cmp.felk.cvut.cz/minimal/ 18. Moreno-Noguer, F., Lepetit, V., Fua, P.: Accurate non-iterative o(n) solution to the pnp problems. In: ICCV (2007) 19. Nister, D.: An efficient solution to the five-point relative pose. IEEE PAMI 26(6), 756–770 (2004) 20. Quan, L., Lan, Z.-D.: Linear n-point camera pose determination. IEEE PAMI 21(8), 774–780 (1999) 21. Reid, G., Tang, J., Zhi, L.: A complete symbolic-numeric linear method for camera pose determination. In: ISSAC 2003, pp. 215–223. ACM, New York (2003) 22. Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: Exploring photo collections in 3D. ACM Transactions on Graphics (SIGGRAPH Proceedings) 25(3) (2006) 23. Snavely, N., Seitz, S.M., Szeliski, R.: Modeling the world from Internet photo collections. International Journal of Computer Vision 80(2), 189–210 (2008) 24. Stewenius, H., Engels, C., Nister, D.: Recent developments on direct relative orientation. ISPRS J. of Photogrammetry and Remote Sensing 60, 284–294 (2006) 25. Triggs, B.: Camera pose and calibration from 4 or 5 known 3d points. In: Proc. 7th Int. Conf. on Computer Vision, Kerkyra, Greece, pp. 278–284. IEEE Computer Society Press, Los Alamitos (1999) 26. Xsens, www.xsens.com
Level Set with Embedded Conditional Random Fields and Shape Priors for Segmentation of Overlapping Objects Xuqing Wu and Shishir K. Shah Quantitative Imaging Laboratory Department of Computer Science University of Houston Houston, TX 77204, USA
Abstract. Traditional methods for segmenting touching or overlapping objects may lead to the loss of accurate shape information which is a key descriptor in many image analysis applications. While experimental results have shown the effectiveness of using statistical shape priors to overcome such difficulties in a level set based variational framework, problems in estimation of parameters that balance evolution forces from image information and shape priors remain unsolved. In this paper, we extend the work of embedded Conditional Random Fields (CRF) by incorporating shape priors so that accurate estimation of those parameters can be obtained by the supervised training of the discrete CRF. In addition, non-parametric kernel density estimation with adaptive window size is applied as a statistical measure that locally approximates the variation of intensities to address intensity inhomogeneities. The model is tested for the problem of segmenting overlapping nuclei in cytological images.
1
Introduction
Segmentation of multiple objects in images wherein the objects may overlap or occlude other objects/regions of interest remains a challenging problem. This is certainly the case for analysis of cytological images, wherein the classification of the imaged sample is based on the shape, distribution and arrangement of cells/nuclei, and other contextual information [1]. To automate this classification process, the computer needs to learn cellular characteristics by extracting corresponding image features from the region of interest (ROI). The accuracy of feature extraction depends on the performance of segmentation results. However, reliable segmentation of cytological images remains challenging because of intensity inhomogeneity and the existence of touching and overlapping cells/nuclei. Li et al. [2] gave a brief review on some methodologies towards the development of automated methods for nuclei or cell segmentation. Among all proposed methods, thresholding, watershed, and deformable models or active contours remain the most popular approaches. In general, watershed segmentation is claimed to be more effective in terms of differentiating connected nuclei or cells. However, the watershed model for segmentation of multiple objects is not suitable R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 230–241, 2011. Springer-Verlag Berlin Heidelberg 2011
Level Set with Embedded CRF and Shape Priors
(a)
(b)
231
(c)
Fig. 1. Overlapping ellipse example: there are two objects in this image, A and B
for separating overlapping objects because any segmented pixel/set is generally associated to a single label and a pixel cannot be granted membership to more than one label. For example, fig. 1(a) shows two objects A and B with an overlapping area C. Fig. 1(b) shows the expected segmentation result under the watershed scheme and fig. 1(c) presents the ground truth with two elliptical objects indicated by red and blue contours. Accurate description of the shape is lost in the segmented image shown in fig. 1(b). Coupled curve evolution [3] and multiphase level set [4] provide a powerful framework for segmenting multiple connected objects. In addition, shape priors can also be easily integrated into the level set formulation resulting in a more constrained segmentation [5]. Nonetheless, these models are subject to the same labeling constraints so as to avoid assigning different labels to overlapping regions. A practical solution to overcome this limitation is to evolve each curve independently [6] and let the shape prior control the boundary in the overlapping region. One of the major drawbacks of the aforementioned continuous level set method is the ad-hoc style of the parameter estimation [7]. In order to leverage the power of structured discriminative models like Conditional Random Fields (CRF), Cobzas and Schmidt [7] proposed to embed CRF into the level set formulation to improve edge regularization, incorporate correlations in labels of neighboring sites and evaluate weighting parameters for each energy term by pre-trained discrete CRF models. A key problem that remains unsolved is the explicit integration of shape information in a discrete CRF model so that parameters that balance the influence from image information and shape priors can be evaluated systematically through embedding. Our work addresses this very problem and is distinguished by three key contributions. First, we propose a novel spatial adaptive kernel density estimator to associate local observations with expected labels. Second, we apply the top-down CRF model [8] as a constraint and embed it into the level set formulation as the shape measurement. Third, we solve the parameter estimation problem for the curve evolution with shape priors by embedding potential functions and parameters learned from the discrete CRF model. The rest of the paper is organized as follows. Section 2 briefly introduces the CRF embedding method proposed by Cobzas and Schmidt [7]. The proposed adaptive kernel density estimator and embedded shape prior are presented in section 3. Estimation of parameters and the experimental evaluation of the developed model for the problem of segmenting overlapping nuclei in cytological images are presented in section 4. The paper is concluded in section 5.
232
2
X. Wu and S.K. Shah
Level Set Methods with Embedded CRF
Image segmentation is a fundamental problem in image analysis and can be viewed as classification or labeling of image pixels/regions, wherein y represents a set of labels to be assigned to each pixel represented by its value x. Since most natural images are not random collection of pixels [9], it is important to find an accurate model to describe information in the image to achieve good segmentation accuracy. Both CRF and level set are popular mathematical models used for solving segmentation problems. A binary CRF with weighted Ising potentials for the task of segmentation is formulated as follows [7]: ⎛ ⎞ 1 yi wT fi (x) + (1 − |yi − yj |) vk fijk (x)⎠ , p(y|x, w, v) = exp − ⎝ Z i∈N
i,j∈E
k
(1) where Z is a normalizing constant known as the partition function, N is the number of nodes, fi (x) denotes features for node i, and fijk represents the k th feature for the edge between nodes i and j. Node labels are yi . The association term yi wT fi (x), denoted as Ai from here on, relates the observation x to the label yi . This term corresponds to the level set data energy (Eq. 2) that is inside or outside of the contour. The interaction term (1 − |yi − yj |) k vk fijk (x), denoted by Iij from here on, is a measure of how the labels at neighboring sites should interact. This term is analogous to the regularization energy (Eq. 2) in the level 1 set model. In particular, fijk (x) is defined as 1+|fik (x)−f . Because Eq. 1 is jk (x)| jointly log-concave in w and v, we can find the global maximum in terms of w and v. Level set methods define the evolving curve C in Ω ∈ R2 . C is implicitly represented as the zero-level of an embedding signed distance function φ : Ω → R. In particular, to segment an image into two disjoint regions Ω1 and Ω2 , the energy functional of a continuous model with embedded CRF has the following form: 1 −H(Φ)(wT f ) + (1 − H(Φ))(wT f ) + |∇H(Φ)| vk dx , E(Φ) =
1 + |∇f k| Ω k data term
regularization term
(2) where k is the index for each edge feature and vk ≥ 0. The associated EulerLagrange equation is:
∂Φ 1 ∇Φ T = δ (Φ) −2w f + div , (3) vk ∂t 1 + ∇fk |∇Φ| k
where δ is the regularized version of the delta function δ. The new continuous formulation takes the advantage of the embedded CRF model whose parameters {w, v} can be estimated by the supervised training in its discrete counterpart.
Level Set with Embedded CRF and Shape Priors
3
233
Adaptive Kernel Density Estimation and Embedded Shape Priors
In general, the discriminative model wT f works well when the test image can be well predicated through the supervised regressive training on sample images. However, when the variance of the difference between predicated and observed values is not well contained, the resulting segmentation can be ill-conditioned. This is especially the case for segmenting cells/nuclei from cytological smears because of the intensity inhomogeneity caused by the cell chromatin, the existence of dense intracellular as well as intercellular matter. In this paper, we propose to use a non-parametric density estimation approach to capture the local variation without imposing restrictive assumptions. In addition, we apply the top-down CRF model [8] for the shape constraint and derive its corresponding embedded term in the continuous space. Thus the weighting factor of shape priors can be estimated from the discrete CRF model. 3.1
Adaptive Kernel Density Estimation
Parzen window is a classical non-parametric kernel density estimation method. Nonetheless, the traditional approach of using a fixed window size h imposes negative effects on segmentation results when there are geometrically disjoint regions in the image, because not every data point makes an equal contribution to the final density estimation of a pixel i in the 2D space. For example, pixels located far away from pixel i should weigh less in the formulation. To have the bandwidth change with the spatial distance, we define the adaptive window size hij as:
14 d2ij 1 1 √ exp hij = , (4) N σ π σ2 where dij is the Euclidean distance between pixel i and j in the 2D space, σ is a user defined scale parameter, and N is the size of sample points. Thus, the new estimated intensity density function at pixel i is: N xi − xl 1 1 K pˆ(xi ) = , (5) N hil hil l=1
where K(·) is the Gaussian kernel, and xl is a sample drawn from population with density p(xi ). The accuracy of the estimated density is increased by using small bandwidth in regions that are near the pixel i since samples in the near neighborhood provide better estimation. On the other hand, a larger bandwidth is more appropriate for samples located far away. To be less sensitive to noise, we could convolve image x with a Gaussian kernel so that samples xl in Eq. 5 are replaced by a smoother version x ˜l . Discrete CRF model for segmenting an image into two disjoint regions Ω1 and Ω2 can now be re-written as: ⎛ ⎞ 1 p(y|x, w, v) = exp − ⎝ −wk log p(fi (x)|Ωk ) + Iij ⎠ k ∈ {1, 2} , (6) Z i∈N
i,j∈E
234
X. Wu and S.K. Shah
where −w log p(fi (x)|Ωk ) forms the association term Ai , Iij is the same as in Eq. 1, and p(fi (x)|Ωk ) is estimated using the adaptive kernel as:
Nk 1 1 fi (x) − f˜l (x) p(fi (x)|Ωk ) = K ∀l ∈ Ωk , (7) Nk hil hil l=1
where f˜l (x) is a smoother version of the node feature fl (x). Embedding CRF model of Eq. 6 into the functional in the continuous space, the new contour energy, Ecv , in the level set counterpart takes on the form: −H(Φ)w1 log p(f (x)|Ω1 ) − (1 − H(Φ))w2 log p(f (x)|Ω2 ) Ecv (Φ) = Ω
+|∇H(Φ)|
k
vk
1 dx , 1 + |∇fk |
(8)
whose parameters {w, v} are trained by the discrete CRF. Minimizing the energy in Eq. 8 is accomplished by solving the following Euler-Lagrange equation:
∂Φ 1 ∇Φ log p(f (x)|Ω ) 1 = δ (Φ) wT · . vk + div − log p(f (x)|Ω2 ) ∂t 1 + ∇fk |∇Φ| k (9) Fig. 2 shows that, compared to the local association model with Gaussian region statistics, the non-parametric approach introduced in Eq. 9 is less sensitive to the variation of intensity values. This results in improved segmentation even without the introduction of shape priors for non-occluding objects. We note that there has been a previous proposal, called local binary fitting energy (LBF) model, for fitting energy based intensity information in local regions [10]. Compared to the LBF model, the kernel function K used in Eq. 4 is a well-behaved kernel [11] and the asymptotic convergence of the kernel density estimator is guaranteed [12]. Unlike the LBF model, parameters of Eq. 8 are estimated systematically through the discrete CRF model.
(a)
(b)
(c)
Fig. 2. Comparative results for segmenting images with intensity inhomogeneity (a) Segmentation result using adaptive kernel density estimation. (b) Segmentation result using Gaussian model. (c) Intensity profile of the area inside of the nucleus.
Level Set with Embedded CRF and Shape Priors
3.2
235
Embedded Shape Priors
The occluded boundary of overlapping regions cannot be detected by algorithms that only use edge or region information. Statistical shape models have been developed to overcome these difficulties [5]. Inspired by the work of Levin et al. [8] and Cremers et al. [12], we propose an explicit integration of shape priors into the discrete CRF and derive embedding entities in the level set model. Therefore, parameters that weigh the influence from shape information can be estimated accurately by training the CRF model. Let an implicit representation of a shape be a binary image patch with labeling of {0, 1}. Let φc be a binary image patch representing a shape, φc (i) be the label of pixel i. The distance between two shapes a and b is defined N as d2 (φca , φcb ) = i=1 (φca (i) − φcb (i))2 , where N is the size of the image patch. Given a set of training shapes φci : i = 1...M , the probability density of a shape φc can be estimated as: M 1 1 2 c c p(φ ) = exp − 2 d (φ , φi ) , M i=1 2σ c
(10)
1 M 2 c c where σ 2 can be estimated by M i=1 minj=i d (φi , φj ) [12]. Let Su0 = c −z log p(φu0 ) be the potential associated with the image patch φc centered at u0 with binary label y and p(φcu0 ) is the probability density of a shape φc centered at u0 . Incorporating this into the discrete CRF model results in: ⎛ ⎞ 1 p(y|x, w, v, z) = exp − ⎝ Ai + Iij + Su0 ⎠ . (11) Z i∈N
i,j∈E
Ai and Iij are defined as in Eq. 6. Parameters {w, v, z} are estimated through the training of the discrete CRF model. To embed the shape prior into the level set, we re-define the implicit shape representation φ as the signed distance function. The distance between two shapes a and b in the level set formulation can be given by: 2 (H(φa (x)) − H(φb (x)))2 dx , (12) dL (φa , φb ) = Ω
where H(·) denotes the Heaviside function. The density of a shape in the level set formulation, represented by pL , is given as: M 1 1 2 pL (φu0 ) = exp − 2 dL (φ(x + u0 ), φi (x)) , M i=1 2σ
(13)
where u0 accounts for translation in continuous space. Let the shape energy Eshape = −z log(pL (φu0 )), then the total energy of the contour in the level set formulation is the sum of Ecv defined in Eq. 8 and Eshape , whose parameters {w, v, z} are estimated by the discrete CRF. Total energy is then minimized by:
236
X. Wu and S.K. Shah
∇Φ ∂Φ 1 log p(f (x)|Ω ) 1 vk + div = δ (Φ) wT · + − log p(f (x)|Ω2 ) ∂t 1 + ∇fk |∇Φ| k exp(− 2σ12 d2L (φ(x + u0 ), φi (x)))(H(φ(x + u0 )) − H(φi (x))) L z , (14) 2 σL exp(− 2σ12 d2L (φ(x + u0 ), φi (x))) L
M
1 2 2 is evaluated by M where σL i=1 minj=i dL (φi , φj ). During the computation, Φ is reinitialized to the signed distance function and we assume homogeneous Neumann boundary conditions.
3.3
Construction of Shape Priors
With the objective of segmenting nuclei, a set of contours of elliptical shape priors was generated according to the general parametric form of the ellipse: x(t) = a cos t cos θ − b sin t sin θ (15) y(t) = a cos t sin θ + b sin t cos θ , where t ∈ [0, 2π]. θ is the angle between the X-axis and the major axis of the ellipse. The ratio ab is a measure of how much the conic section deviates from being circular. In this paper, we construct shape prior with ab ∈ {0.4, 0.6, 0.8, 1.0}. The angle of the major axis, θ, was also varied from 0 to π in steps of π8 . Fig. 3 shows the array of elliptical training shapes. Each training shape is represented by a 32 × 32 patch. Rotation invariance is integrated into the estimation of the density of a shape (Eq. 10 or Eq. 13) by using this set of training shapes.
Fig. 3. Training shapes for elliptical objects with different eccentricities and rotation angles
4
Implementation and Experimental Results
Images used for training and testing are obtained from slides of cytological smears that are Papanicolaou or Diff-Quik stained for the purpose of differentiating pathologies in fine needle aspirated cells from thyroid nodules. Images
Level Set with Embedded CRF and Shape Priors
237
are acquired by using an optical microscope and a grayscale CCD camera having 768 × 512 pixels (9 × 9μm) at 8-bit digitization. Manual segmentation was performed to establish the ground truth. The segmentation accuracy is measured by global consistency error, a well-established standard measurement used in image segmentation problem [13]. To evolve the curve according to Eq. 14, an iteration scheme is implemented by discretizing the PDE [4]. The level set function is initialized as a step function that takes negative values inside the contour and positive values outside of the contour. The initial contour is obtained from the segmentation result using watershed and unsupervised clustering [14]. Since the watershed lines separate the image into non-overlapped regions, we can evolve each closed watershed line of a region independently. Four parameters are estimated by training the discrete CRF with shape priors [15]: w1 is related to image information inside of the contour; w2 is for image information outside of the contour; v governs the force defined by edge features; z balances the influences from the shape prior. We have selected 20 images for the training: ten images with Papanicolaou stain and ten images with Diff-Quik stain. Each of the image contains at least one pair of overlapping nuclei. Parameters are trained separately for different stains. Table. 1 lists the value of parameters trained for different stained images. Fig. 4 shows that if the process of curve evolution is dominated by edge features, the evolved contour can not arrive at the boundary of the nucleus of interest. Although a global optimal selection of parameters in terms of segmentation accuracy can not be guaranteed [15], parameters estimated by the proposed scheme can be treated as local optimal selections. To further evaluate this local optimal property of selected parameters obtained through the CRF training, we added a small perturbation to the value z 1 of these parameters by varying the ratios w w2 and v . Fig. 5(a) and fig. 5(b) show Table 1. Parameters Estimation Results From Training Stain w1 w2 v z Papanicolaou 1.0 1.0 3.25e3 8e5 Diff-Quik 1.0 2.0 1.73e3 9e5
(a)
(b)
Fig. 4. Comparative results for parameters estimation. (a) Contour of balanced forces; (b) Contour of weak constraints from shape prior.
238
X. Wu and S.K. Shah
(a)
(b) Fig. 5. Comparative segmentation results for parameters estimation. (a) Varying the 1 (b) Varying the ratio vz . ratio w w2
the segmentation accuracy across the entire training set under varying parameters. It can be seen that the selected parameters from CRF training have the best segmentation performance. To quantify the performance of the proposed method, a total of 40 testing images were selected. Half of these images were Papanicolaou stained while the other half are Diff-Quik stained. The performance of the proposed model is evaluated with two criteria, the accuracy of the segmentation in terms of the area occupied by nuclei and the number of nuclei correctly counted by each segmented region. First, we evaluated potential benefit of using non-parametric adaptive density estimation to model variations of intensities in cytological images. We found that the average segmentation accuracy using the proposed model integrated with adaptive density estimation is 81%, as opposed to 72% accuracy rate obtained by the using the Gaussian regional model integrated in the overall framework.
Level Set with Embedded CRF and Shape Priors
(a) Proposed method (b) Watershed
(c) CRF
(d) Proposed method (e) Watershed
(f) CRF
239
Fig. 6. Sample segmentation results using different models. (a)(b)(c) Papanicolaou stained sample. (d)(e)(f) Diff-Quik stained sample. Table 2. Comparison of accuracy of segmentation and nuclei counting Model Accuracy of segmentation Accuracy of nuclei counting Proposed method 81% 94% CRF 79% 52% Watershed 74% 94%
Second, we compared our method with two other popular models, watershed [14] and CRF [16]. Test results listed in Table. 2 show that the proposed model has the best overall performance in terms of segmentation and nuclei counting accuracy. The low accuracy of nuclei counting by using CRF model is expected due to its tendency to merge adjoining regions together as shown in fig. 6(c) and fig. 6(f). Compared to the CRF model, although watershed is more effective in differentiating overlapping nuclei, it has two drawbacks. First, the shape information of nuclei, a key morphological feature in cytological classification, could be distorted because of the unique labeling constraint. This is attributed to the fact that each pixel in the overlapping region could have two different labels when two nuclei overlap. Second, the model used for constructing watershed lines is dominated by the image intensity only, which is very sensitive to the intensity inhomogeneity. As a result, its segmentation accuracy in terms of the area occupied by the nuclei is lower than CRF, which models both local and neighborhood information. The proposed method inherits the modeling advantage of CRF and can model non-local dependencies during the curve evolution [7]. Because each contour evolves independently, pixels in overlapping areas can be assigned multiple labels as indicated in fig. 6(a) and fig. 6(d). Fig. 7 presents another example of segmentation results after applying the proposed method, watershed and CRF to Papanicolaou and Diff-Quik stained images, respectively.
240
X. Wu and S.K. Shah
(a) Proposed method
(b) Watershed
(c) CRF
(d) Proposed method
(e) Watershed
(f) CRF
Fig. 7. Segmentation results using different models. (a)(b)(c) Papanicolaou stained sample. (d)(e)(f) Diff-Quik stained sample.
5
Conclusion
Shape information is a key descriptor in analyzing cytological images. To recover the shape information for overlapping objects after segmentation, we extend the embedded CRF model by incorporating shape priors. In contrast to other approaches, parameters that govern the force defined by the image information and shape priors are estimated accurately through the training of discrete CRF. By evolving each contour independently, original shape information is recovered by assigning multiple labels to the common region shared by overlapping nuclei. In addition, we have presented an adaptive scheme for non-parametric density estimation based on intensity information in local regions. The proposed method is tested for the problem of segmenting overlapping nuclei with intensity inhomogeneity while preserving shape information of occluded objects.
References 1. Amrikachi, M., Ramzy, I., Rubenfeld, S., Wheeler, T.: Accuracy of fine-needle aspiration of thyroid. Arch. Pathol. Lab. Med. 125, 484–488 (2001) 2. Li, G., Liu, T., Nie, J., Guo, L., Chen, J., Zhu, J., Xia, W., Mara, A., Holley, S., Wong, S.: Segmentation of touching cell nuclei using gradient flow tracking. Journal of Microscopy 231, 47–58 (2008)
Level Set with Embedded CRF and Shape Priors
241
3. Fan, X., Bazin, P.L., Prince, J.L.: A multi-compartment segmentation framework with homeomorphic level sets. In: CVPR (2008) 4. Vese, L., Chan, T.: A multiphase level set framework for image segmentation using the mumford and shah model. IJCV 50, 271–293 (2002) 5. Rousson, M., Paragios, N.: Shape priors for level set representations. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 78–92. Springer, Heidelberg (2002) 6. Mosaliganti, K., Gelas, A., Gouaillard, A., Noche, R.: Detection of spatially correlated objects in 3d images using appearance models and coupled active contours. In: Yang, G.-Z., Hawkes, D., Rueckert, D., Noble, A., Taylor, C. (eds.) MICCAI 2009. LNCS, vol. 5762, pp. 641–648. Springer, Heidelberg (2009) 7. Cobzas, D., Schmidt, M.: Increased discrimination in level set methods with embedded conditional random fields. In: CVPR (2009) 8. Levin, A., Weiss, Y.: Learning to combine bottom-up and top-down segmentation. IJCV 81, 105–118 (2009) 9. Kumar, S., Hebert, M.: Discriminative random fields. IJCV 68, 179–201 (2006) 10. Li, C., Kao, C.Y., Gore, J.C., Ding, Z.: Implicit active contours driven by local binary fitting energy. In: CVPR (2007) 11. Mittal, A., Paragios, N.: Motion-based background substraction using adaptive kernel density estimation. In: CVPR (2004) 12. Cremers, D., Osher, S.J., Soatto, S.: Kernel density estimation and intrinsic alignment for shape priors in level set segmentation. IJCV 69, 335–351 (2006) 13. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV (2001) 14. Wu, X., Shah, S.: Comparative analysis of cell segmentation using absorption and color images in fine needle aspiration cytology. In: IEEE International Conference on Systems, Man and Cybernetics (2008) 15. Szummer, M., Kohli, P., Hoiem, D.: Learning cRFs using graph cuts. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 582– 595. Springer, Heidelberg (2008) 16. Wu, X., Shah, S.: A bottom-up and top-down model for cell segmentation using multispectral data. In: ISBI (2010)
Optimal Two-View Planar Scene Triangulation Kenichi Kanatani and Hirotaka Niitsuma Department of Computer Science, Okayama University, Okayama 700-8530, Japan {kanatani,niitsuma}@suri.cs.okayama-u.ac.jp
Abstract. We present a new algorithm for optimally computing from point correspondences over two images their 3-D positions when they are constrained to be on a planar surface. We consider two cases: the case in which the plane and camera parameters are known and the case in which they are not. In the former, we show how observed point correspondences are optimally corrected so that they are compatible with the homography between the two images determined by the plane and camera parameters. In the latter, we show how the homography is optimally estimated by iteratively using the triangulation procedure.
1
Introduction
Computing the 3-D position of a point from its projection in two images is called triangulation and is a fundamental tool of computer vision [4]. The basic principle is to compute the intersection of the rays starting from the camera lens center and passing through the corresponding image points. However, point correspondence detection using an image processing operation incurs errors to some extent, and the two rays may not intersect. A naive solution is to compute the midpoint of the shortest segment connecting the two rays (Fig. 1(a)), but Kanatani [7] and Hartley and Sturm [5] pointed out that for optimal estimation the corresponding points should be displaced so that the rays meet in the scene (Fig. 1(b)) in such a way that the sum of the square displaced distances, or reprojection error , is minimized. For this, Hartley and Sturm [5] presented an algorithm that reduces to solving a 6th degree polynomial, while Kanazawa and Kanatani [13] gave a first approximation in an analytical form. Later, Kanatani et al. [11] showed that the first approximation is sufficiently accurate and that a few iterations lead to complete agreement with the Hartley-Sturm solution with far more efficiency. Lindstrom [14] further improved this approach. The aim of this paper is to demonstrate that exactly the same holds when the points we are viewing are constrained to be on a planar surface. This is a common situation in indoor and urban scenes. If a 3-D point is constrained to be on a known plane, the corresponding points must be displaced so that the rays not merely intersect but also meet on that plane. We call this planar triangulation after Chum et al. [3]. A first approximation solution was given by Kanazawa and Kanatani [12], while Chum et al. [3] presented an algorithm that reduces to solving an 8th degree polynomial. The purpose of this paper is to demonstrate that a few iterations of the first approximation lead to an R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 242–253, 2011. Springer-Verlag Berlin Heidelberg 2011
Optimal Two-View Planar Scene Triangulation
(a)
243
(b)
Fig. 1. Triangulation. (a) The midpoint of the shortest segment connecting the rays. (b) The points are optimally corrected so that their rays intersect.
optimal solution. We consider two cases: the case in which the plane and camera parameters are known, and the case in which they are not. The algorithm of Chum et al. [3] deals with the former. The latter case could be solved using the bundle adjustment approach, as demonstrated by Bartoli and Sturm [1] for an arbitrary number of images, but we show that a much simpler method exists. In fact, we show that our optimal triangulation procedure for the former case can easily be extended to the latter. In Sec. 2, we summarize the fundamentals about planar projection and homographies. In Sec. 3 and 4, we present an iterative algorithm for optimal planar triangulation for known plane and camera parameters. In Sec. 5, we show that our optimal triangulation procedure can be straightforwardly extended to the case of unknown plane and camera parameters. In Sec. 6, we do numerical simulation and demonstrate that our algorithm and the first approximation of Kanazawa and Kanatani [12] give practically the same value.
2
Planar Surface and Homography
Consider a plane with a unit surface normal n at distance d from the origin of an XY Z coordinate system fixed to the scene (Fig. 2(a)). We take images of this plane from two positions. The ith camera, i = 1, 2, is translated from the coordinate origin O by ti after rotated by Ri (Fig. 2(b)). We call {ti , Ri } the motion parameters of the ith camera. We assume that by prior camera
n
X d
Z
t1 R1
t2
R2
O Y
(a)
(b)
Fig. 2. (a) Plane and camera configuration. (b) The points are optimally corrected so that their rays intersect on the plane.
244
K. Kanatani and H. Niitsuma
(x’, y’)
(x, y)
Fig. 3. Two images of the same planar surface are related by a homography
calibration the image coordinate origin is placed at the principal point and that the aspect ratio is 1 with no image skew. The image of the plane taken from the first position, call it the “first image”, and the image taken from the second position, call it the “second image”, are related by the following homography (Fig. 3) [4, 7]: x = f0
h11 x + h12 y + h13 f0 , h31 x + h32 y + h33 f0
y = f0
h21 x + h22 y + h23 f0 . h31 x + h32 y + h33 f0
(1)
Here, f0 is a scale factor of approximately the size of the image for stabilizing numerical computation with finite length. The 3 × 3 matrix H = (hij ) is determined by the parameter {n, d} of the plane, the motion parameters {Ri , ti } and the focal lengths fi , i = 1, 2, in the following form [4, 7]: H = diag(1, 1,
f0 t1 n t2 n f1 )(I + )R1 diag(1, 1, ). )R2 (I − f2 d d − (t1 , n) f0
(2)
Here, I is the unit matrix, and diag(a, b, c) denotes the diagonal matrix with diagonal elements a, b, and c in that order. Throughout this paper, we denote the inner product of vectors a and b by (a, b).
3
Triangulation for Known Plane and Cameras
We first consider the case in which we know {n, d}, {Ri , ti }, and fi , i = 1, 2, hence the homography H. In homogeneous coordinates, we can write (1) as follow [4, 7]: ⎛ ⎞ ⎛ ⎞ ⎞⎛ x /f0 h11 h12 h13 x/f0 ⎝ y /f0 ⎠ ∼ (3) = ⎝ h21 h22 h23 ⎠ ⎝ y/f0 ⎠ . 1 h31 h32 h33 1 The symbol ∼ = denotes equality up to a nonzero constant. We can equivalently write (3) as ⎞ ⎛ ⎞ ⎛ ⎞ ⎞⎛ ⎛ h11 h12 h13 0 x/f0 x /f0 ⎝ y /f0 ⎠ × ⎝ h21 h22 h23 ⎠ ⎝ y/f0 ⎠ = ⎝ 0 ⎠ . (4) 1 h31 h32 h33 1 0 The three components of this equation multiplied by f02 are (ξ (1) , h) = 0,
(ξ (2) , h) = 0,
(ξ (3) , h) = 0,
(5)
Optimal Two-View Planar Scene Triangulation
245
S p
p
Fig. 4. The point p is orthogonally projected on to p¯ on S in the 4-D joint space
where we define the 9-D vectors h, ξ (1) , ξ (2) , and ξ (3) by h = h11 h12 h13 h21 h22 h23 h31 h32 h33 , ξ (1) = 0 0 0 −f0 x −f0 y −f02 xy yy f0 y , ξ (2) = f0 x f0 y f02 0 0 0 −xx −yx −f0 x , ξ (3) = −xy −yy −f0 y xx yx f0 x 0 0 0 .
(6)
A corresponding point pair (x, y) and (x , y ) can be identified with a point p = (x, y, x , y ) in the 4-D xyx y joint space. Each of the three equations in (5) defines a hypersurface in this 4-D joint space. However, the identity x ξ (1) + y ξ (2) + f0 ξ (3) = 0 holds, so (5) defines a 2-D variety (algebraic manifold) S in the 4-D joint space. In the presence of noise, the point p is not necessarily on S. Optimal planar triangulation is to displace p to a point p¯ on S in such a way that the reprojection error ¯ 2, (7) E = p − p is minimized subject to ¯ h) = 0, (ξ (k) (p),
k = 1, 2, 3.
(8)
Geometrically, this means orthogonally projecting p onto the variety S in the 4-D joint space (Fig. 4). Once such a p¯ = (¯ x, y¯, x ¯ , y¯ ) is obtained, the corresponding 3-D position (X, Y, Z) is determined by solving ⎛ ⎞ ⎛ ⎞ ⎞ ⎞ ⎛ ⎛ X X /f x ¯/f0 x ¯ 0 ⎜Y ⎟ ⎜Y ⎟ ⎟ ⎟ ⎝ y¯/f0 ⎠ ∼ ⎝ y¯ /f0 ⎠ ∼ (9) = P1 ⎜ = P2 ⎜ ⎝ Z ⎠, ⎝Z ⎠, 1 1 1 1 where P1 and P2 are the 3 × 4 projection matrices defined as follows [4, 11]: P1 = diag(1, 1,
f0 f0 )R1 I −t1 , P2 = diag(1, 1, )R 2 I −t1 . f1 f2
(10)
Four linear equations in X, Y , and Z are obtained from (9), but because (8) is satisfied, a unique solution is obtained [11].
246
4
K. Kanatani and H. Niitsuma
Optimal Planar Triangulation
We now present a new procedure for minimizing (7) subject to (8). While unconstrained triangulation [11] involves a single constraint describing the epipolar geometry, planar triangulation is constrained by three equations in the form of (8), not mutually algebraically independent. For this, the first approximation has already been presented by Kanazawa and Kanatani [12]. We modify their method so that an optimal solution is obtained by iterations. The procedure is as follows (the derivation is omitted; it is a straightforward extension of the unconstrained triangulation in [11]): 1. Let E0 = ∞ (a sufficiently large number), and define the 4-D vectors ⎛
⎞ x ⎜y⎟ ⎟ p=⎜ ⎝ x ⎠ , y
⎛
⎞ x ˆ ⎜ yˆ ⎟ ⎟, pˆ = ⎜ ⎝x ˆ ⎠ yˆ
⎛
⎞ x ˜ ⎜ y˜ ⎟ ⎟, p˜ = ⎜ ⎝x ˜ ⎠ y˜
(11)
˜ = y˜ = x ˜ = y˜ = 0. where we let x ˆ = x, yˆ = y, x ˆ = x , yˆ = y , and x (1) (2) (3) 2. Compute the following 9 × 4 matrices T , T , and T : ⎛
⎞ ⎛ ⎛ ⎞ ⎞ 0 0 00 f0 0 0 0 −ˆ y 0 0 −ˆ x ⎜ 0 0 00⎟ ⎜ 0 f0 0 0 ⎟ ⎜ 0 −ˆ y ⎟ y 0 −ˆ ⎜ ⎟ ⎜ ⎜ ⎟ ⎟ ⎜ 0 0 00⎟ ⎜ 0 0 0 0⎟ ⎜ 0 0 0 −f0 ⎟ ⎜ ⎟ ⎜ ⎜ ⎟ ⎟ ⎜ −f0 0 0 0 ⎟ ⎜ 0 0 0 0⎟ ⎜ x ˆ 0 x ˆ 0 ⎟ ⎜ ⎟ ⎜ ⎜ ⎟ ⎟ (2) ⎜ (3) ⎜ ⎟ ⎟ ˆ yˆ 0 ⎟ T (1) =⎜ ⎜ 0 −f0 0 0 ⎟, T =⎜ 0 0 0 0 ⎟ , T =⎜ 0 x ⎟. ⎜ 0 0 00⎟ ⎜ 0 0 0 0⎟ ⎜ 0 0 f0 0 ⎟ ⎜ ⎟ ⎜ ⎜ ⎟ ⎟ ⎜ yˆ 0 0 x ⎜ −ˆ ⎜ 0 0 0 0 ⎟ ˆ⎟ x 0⎟ x 0 −ˆ ⎜ ⎟ ⎜ ⎜ ⎟ ⎟ ⎝ 0 yˆ 0 yˆ ⎠ ⎝ 0 −ˆ ⎝ 0 0 0 0 ⎠ y 0⎠ x −ˆ 0 0 0 f0 0 0 −f0 0 0 0 0 0
(12) 3. Compute the following ξ (1)∗ , ξ (2)∗ , and ξ (3)∗ : ˆ −f0 yˆ −f02 x ˆyˆ yˆyˆ f0 yˆ ˜ + T (1) p, ξ (1)∗ = 0 0 0 −f0 x (2)∗ (2) 2 ˆ f0 yˆ f0 0 0 0 −ˆ xx ˆ −ˆ yxˆ −f0 x ˆ ˜ ξ = f0 x + T p, yyˆ −f0 yˆ x ˆx ˆ yˆx ˆ f0 x ˆ 0 0 0 + T (3) p. ˜ xyˆ −ˆ ξ (3)∗ = −ˆ (kl)
4. Compute the following 9 × 9 matrices V0 (kl)
V0
(13)
[ξ]:
[ξ] = T (k) T (l) .
5. Compute the 3 × 3 matrix W = (W ) ⎛ ⎞− (11) (12) (13) (h, V0 [ξ]h) (h, V0 [ξ]h) (h, V0 [ξ]h) ⎜ ⎟ W = ⎝ (h, V0(21) [ξ]h) (h, V0(22) [ξ]h) (h, V0(23) [ξ]h) ⎠ , (31) (32) (33) (h, V0 [ξ]h) (h, V0 [ξ]h) (h, V0 [ξ]h) 2
(14)
(kl)
(15)
where ( · )− r denotes pseudoinverse constrained to rank r (the smallest eigenvalue is replaced by 0 in its spectral decomposition).
Optimal Two-View Planar Scene Triangulation
247
6. Update p˜ and pˆ as follows: p˜ =
3
W (kl) (ξ (l)∗ , h)T (k) h,
˜ pˆ ← p − p.
(16)
k,l=1
7. Evaluate the reprojection error E by ˜ 2. E = p
(17)
If E ≈ E0 , then return pˆ as p¯ and stop. Else, let E0 ← E and go back to Step 2. The use of the pseudoinverse constrained to rank 2 in (15) reflects the fact that only two of the three constraints in (5) are algebraically independent. This algorithm produces the same solution as that of Chum et al. [3], which solves an 8th degree polynomial. However, our algorithm involves only a few iterations of linear calculus without requiring any polynomial solver, which is sometimes inefficient and numerically unstable.
5
Triangulation for Unknown Plane and Cameras
Next, we consider the case in which {n, d}, {Ri , ti }, i = 1, 2, are unknown. As is well known [4, 7], these parameters can be estimated by computing the homography H between the two images, provided that the focal lengths fi are known; we assume that they are given by prior camera calibration. Since the analytical procedure for computing the plane and camera parameters from a homography H has been described by many researchers in different forms [6, 15, 18–20], the problem reduces to computing the homography H from point correspondences over the two images. The simplest method is to minimize the algebraic distance, which is known by many names such as least squares and DLT (Direct Linear Transformation) [4], but the accuracy is low in the presence of noise. A method known to be very accurate is what is called Sampson error minimization [4], and an iterative scheme was presented by Scoleri et al. [17]. However, Sampson error minimization does not necessarily compute an exactly optimal solution in the sense of maximum likelihood [9]. We now show that by combining the Sampson error minimization with the optimal planar triangulation described in Sec. 4, we can obtain an exactly optimal H. The basic principle is already presented by Kanatani and Sugaya [9], but their theory applies only for a single constraint, which is the case in ellipse fitting and fundamental matrix computation. Here, we extend their procedure to homographies constrained by multiple equations. Given N corresponding points (xα , yα ) and (xα , yα ), α = 1, ..., N , we compute the 9-D vector h that encodes the homography H. Omitting the derivation, we describe the procedure: 1. Let E0 = ∞ (a sufficiently large number), and give an initial guess of h in (6) using any method, say least squares.
248
K. Kanatani and H. Niitsuma
2. Let pα , pˆα , and p˜α be the vectors in (11) for the αth pair (xα , yα ), and (xα , yα ), α = 1, ..., N . (1) (2) (3) 3. Let Tα , Tα , and Tα be, respectively, the values of T (1) , T (2) , and T (3) in (12) for the αth pair, α = 1, ..., N . (kl) (k) (l) and the 3 × 3 matrices 4. Compute the 9 × 9 matrices V0 [ξα ] = Tα Tα (kl) (1)∗ (2)∗ Wα = (Wα ) in (15) for the αth pair, and let the 9-D vectors ξα , ξα , (3)∗ and ξα be the values of ξ (1)∗ , ξ (2)∗ , and ξ (3)∗ in (13) for the αth pair, α = 1, ..., N . 5. Compute the 9-D unit vector h that minimizes J=
N 3 1
Wα(kl) (ξα(k)∗ , h)(ξα(l)∗ , h). N α=1
(18)
k,l=1
6. Update p˜α and pˆα , α = 1, ..., N , as follows: p˜α =
3
Wα(kl) (ξα(l)∗ , h)Tα(k) h,
pˆα ← pα − p˜α .
(19)
k,l=1
N 7. Evaluate the reprojection error E = α=1 p˜α 2 . If E ≈ E0 , then return h and stop. Else, let E0 ← E and go back to Step 3. This procedure is identical to the optimal planar triangulation in Sec. 4 except for Step 5. The expression in (18) coincides with what is known as the Sampson (k)∗ (l)∗ (k) error [4] if ξα and ξα on the right-hand side are respectively replaced by ξα (l) and ξα (the values of (6) for the αth pair). It can be minimized by the scheme of Scoleri et al. [17], but here we use a much simpler reformulation of Kanatani et al. [16], which is a direct extension of the FNS (Fundamental Numerical Scheme) of Chojnacki et al. [2]. The procedure goes as follows: 1. Provide an initial value h0 for h, e.g., by least squares. 2. Compute the matrices M and L as follows: M=
N N N 1 (k) (l) (kl) 1 (kl) (k)∗ (l)∗ Wα ξα ξα , L = vα vα V0 [ξα ], N α=1 N α=1
(20)
k,l=1
where we define vα(k)
=
3
Wα(kl) (ξα(l)∗ , h).
(21)
l=1
3. Solve the eigenvalue problem (M − L)h = λh, and compute the unit eigenvector h for the smallest eigenvalue λ. 4. If h ≈ h0 , return h and stop. Else, let h0 ← h, and go back to Step 2.
(22)
Optimal Two-View Planar Scene Triangulation
249
For unconstrained triangulation, the optimal algorithm of Kanatani et al. [11], which assumes a given fundamental matrix, can be automatically converted to optimal fundamental matrix computation merely by inserting a Sampson error minimization step, as shown by Kanatani and Sugaya [8]. In contrast, the polynomial solving algorithm of Hartley and Sturm [5] cannot be so easily converted to optimal fundamental matrix computation. Similarly, the optimal planar triangulation in Sec. 4, which assumes a given homography, can be automatically converted to optimal homography computation merely by inserting a Sampson error minimization step. In contrast, the polynomial solving algorithm of Chum et al. [3] cannot be so easily converted to optimal homography computation.
6
Experiments
Figure 5(a) shows two images of a simulated grid. The image size is 500 × 500 pixels; the focal lengths are f1 = f2 = 600 pixels. We added Gaussian noise of mean 0 and standard deviation 1 pixel to the x and y coordinates of each of the N (= 121) grid points. Then, we reconstructed the 3-D position of each grid point by unconstrained triangulation [11] and by our planar triangulation. Figure 5(b) shows the 3-D positions of the grid points. We can see that by planar triangulation (in blue) all the points are on the specified plane but not by unconstrained triangulation (in red). For quantitative evaluation, we measured the root mean square reprojection error
N
1
yα − yα )2 + (ˆ xα − xα )2 + (ˆ yα − yα )2 , (ˆ xα − xα )2 + (ˆ (23) e= N α=1 xα , yˆα ) are the corrected positions of the observations (xα , yα ) where (ˆ xα , yˆα ) and (ˆ and (xα , yα ), respectively. We also evaluated the 3-D reconstruction error
N
1
D= rˆα − r¯α 2 , (24) N α=1 where rˆα is the reconstructed position of the αth point, and r¯α its true position. The table in Fig. 5 lists the values for unconstrained triangulation [11], the first approximation of the planar triangulation (the iteration is terminated after the first round), which corresponds to the result of Kanazawa and Kanatani [12], and the exact values computed by our method. From this table, we observe that the reprojection error e increases by assuming planarity. This is because the corresponding points need to be displaced so that the rays not simply intersect but also intersect on the specified plane. Statistical analysis [7] tells us that under maximum likelihood N e2 /σ 2 is subject to a χ2 distribution with N degree of freedom for unconstrained triangulation and with 2N degrees √of freedom for planar triangulation. Hence, e should be approximately σ and 2σ, respectively. The values in the table in Fig. 5 are very close to the prediction. However, the increase in the reprojection error e does
250
K. Kanatani and H. Niitsuma
(a)
(b)
reprojection theoretical 3-D error expectation error unconstrained 1.99503 2.00000 4.24466 planar 1st approx. 2.83127 — 0.95937 exact 2.83124 2.82843 0.95935 Fig. 5. (a) Simulated images of a planar grid taken from different places. (b) 3-D position of the reconstructed grid. Points reconstructed by planar triangulation (in blue) are on the specified plane, but those reconstructed by unconstrained triangulation [11] (in red) are not necessarily on it. The table below lists the reprojection error, its theoretical expectation, and the average 3-D reconstruction error. reprojection theoretical 3-D error expectation error known 2.83124 2.82843 0.01919 unknown 1st approx. 2.81321 — 0.13643 exact 2.81323 2.78128 0.13660 plane
Fig. 6. The reconstructed grid by estimating the plane and the camera positions (in red) and its true position (in black). The table lists the reprojection error of planar triangulation by estimating the plane and the camera positions, its theoretical expectation, and the average 3-D reconstruction error.
not mean the increase in the 3-D reconstruction error D. In fact, the 3-D reconstruction error D actually decreases with the knowledge of planarity. We also see that the exact values are very close to the first approximation of Kanazawa and Kanatani [12]. Next, we tested the case in which the plane and camera parameters are unknown. Since the absolute scale is indeterminate, we scaled the relative displacement between the cameras to unit length. Figure 6 shows the 3-D positions of the reconstructed grid (in red) and its true position (in black). Due to the error in estimating the plane, i.e., the homography, the computed position is slightly different from its true position. The table in Fig. 6 compares the reprojection error e and the 3-D reconstruction error D in the known and unknown plane cases. The values in the known plane case are the same as in the table in Fig 5 except the normalization t = 1.
Optimal Two-View Planar Scene Triangulation
251
We observe that the reprojection error e is smaller in the unknown plane case than in the known plane case. This is because the parameters of the plane are estimated so that the reprojection error is minimized. Statistical analysis [7] tells us that under maximum likelihood N e2 /σ 2 is subject to a χ2 distribution with 2N − 8 degrees of freedom and hence has expectation 2N − 8. This reflects the fact that the homography constraint has eight degrees of freedom with codimension two [7]. Consequently, the reprojection error e should approximately be 2(1 − 4/N )σ. The value in the table in Fig 5 is very close to the prediction. We also see that the first approximation (using only a single Sampson error minimization step) and the exact maximum likelihood value are very close to each other, as generally predicted in [9]. Again, the smaller reprojection error does not mean more accurate 3-D reconstruction. Rather, the 3-D reconstruction accuracy deteriorates because of the error in estimating the plane, as shown in the table in Fig. 5. Note that when the camera positions are unknown, the 3-D positions of the points cannot be reconstructed without the knowledge of planarity. If the points are in general position, their 3-D positions and the camera positions can be reconstructed from two views [10], but that computation fails if the points degenerate to be coplanar [4, 7].
7
Concluding Remarks
We have presented an optimal algorithm1 for computing the 3-D positions of points viewed from two images by using the knowledge that they are constrained to be on a planar surface. This is an extension of the unconstrained triangulation of Kanatani et al. [11] without assuming planarity. Our algorithm automatically encompass the case in which the plane and camera parameters are unknown; they are estimated merely by inserting a Sampson error minimization step. As a result, an exact maximum likelihood estimate is obtained for the homography between the two images. This is a complete parallel to the scheme of Kanatani and Sugaya [8] for computing an exact maximum likelihood estimate of the fundamental matrix between two images merely by inserting a Sampson error minimization step in the unconstrained triangulation of Kanatani et al. [11]. In contrast, the optimal triangulation of Hartley and Sturm [5], which solves a 6th degree polynomial, is not so easily converted to optimal fundamental matrix estimation. Similarly, the optimal planar triangulation of Chum et al. [3], which solves an 8th degree polynomial, is not so easily converted to produce an optimal homography. We have also confirmed experimentally that the first approximation is very close to the exact maximum likelihood estimate. Thus, we conclude that our optimal scheme is not really necessary in practice. In fact, that the Sampson error minimization solution is known to coincide with the exactly optimal solution up to several significant digits in many problems [9]. In our experiment, we used the least squares, also known as the DLT, to compute the initial value h to start the 1
The code is available at http://www.suri.cs.okayama-u.ac.jp/~kanatani/e/
252
K. Kanatani and H. Niitsuma
FNS procedure described in Sec. 5. We have observed that the Sampson error minimization iterations may not converge in the presence of extremely large noise and that the use of “HyperLS” or its “Taubin approximation” [16] can significantly extend the noise level range of convergence. Our optimal homography computation does not reach a global minimum of the reprojection error if the Sampson error minimization in Step 5 does not return a global minimum of the Sampson error. In fact, the FNS procedure described in Sec. 5 is not theoretically guaranteed to return a global minimum h of the Sampson error J, although it usually does. However, if the Sampson error J can be globally minimized, e.g., using branch and bound, in each iteration, we can obtain a global minimum solution h in the end, because the Sampson error J coincides with the reprojection error when the iterations have converged [9]. Acknowledgments. This work was supported in part by the Ministry of Education, Culture, Sports, Science, and Technology, Japan, under a Grant in Aid for Scientific Research (C 21500172).
References 1. Bartoli, A., Sturm, P.: Constrained structure and motion from multiple uncalibrated views of a piecewise planar scene. Int. J. Comput. Vis. 52(1), 45 (2003) 2. Chojnacki, W., Brooks, M.J., van den Hengel, A., Gawley, D.: On the fitting of surfaces to data with covariances. IEEE Trans. Patt. Anal. Mach. Intell. 22(11), 1294–1303 (2000) 3. Chum, O., Pajdla, T., Sturm, P.: The geometric error for homographies. Comput. Vis. Im. Understand. 97(1), 86–102 (2002) 4. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004) 5. Hartley, R.I., Sturm, P.: Triangulation. Comput. Vision Image Understand. 68(2), 146–157 (1997) 6. Hay, J.C.: Optical motions and space perception—An extension of Gibson’s analysis. Psych. Rev. 73(6), 550–565 (1966) 7. Kanatani, K.: Statistical Optimization for Geometric Computation: Theory and Practice. Elsevier, Amsterdam (1996); reprinted, Dover, York (2005) 8. Kanatani, K., Sugaya, Y.: Compact fundamental matrix computation. IPSJ Trans. Comp. Vis. Appl. 2, 59–70 (2010) 9. Kanatani, K., Sugaya, Y.: Unified computation of strict maximum likelihood for geometric fitting. J. Math. Imaging Vis. 38(1), 1–13 (2010) 10. Kanatani, K., Sugaya, Y., Kanazawa, Y.: Latest algorithms for 3-D reconstruction form two views. In: Chen, C.H. (ed.) Handbook of Pattern Recognition and Computer Vision, 4th edn., pp. 201–234. World Scientific, Singapore (2009) 11. Kanatani, K., Sugaya, Y., Niitsuma, H.: Triangulation from two views revisited: Hartley-Sturm vs. optimal correction. In: Proc. 9th British Machine Vis. Conf., September 2008, pp. 173–182 (2008) 12. Kanazawa, Y., Kanatani, Y.: Direct reconstruction of planar surfaces by stereo vision. IEICE Trans. Inf. & Syst. E78-D(7), 917–922 (1995) 13. Kanazawa, Y., Kanatani, K.: Reliability of 3-D reconstruction by stereo vision. IEICE Trans. Inf. & Syst. E78-D(10), 1301–1306 (1995)
Optimal Two-View Planar Scene Triangulation
253
14. Lindstrom, P.: Triangulation made easy. In: Proc. IEEE Conf. Comput. Vis. Patt. Recog. (June 2010) 15. Longuet-Higgins, H.C.: The reconstruction of a planar surface from two perspective projections. Proc. Royal Soc. Lond. B-227(1249), 399–410 (1986) 16. Niitsuma, H., Rangarajan, P., Kanatani, K.: High accuracy homography computation without iterations. In: Proc. 16th Symp. Sensing via Imaging Information (June 2010) 17. Scoleri, T., Chojnacki, W., Brooks, M.J.: A multi-objective parameter estimator for image mosaicing. In: Proc. 8th Int. Symp. Signal Processing and its Applications, August 2005, vol. 2, pp. 551–554 (2005) 18. Tsai, R.Y., Huang, T.S.: Estimation of three-dimensional motion parameters of a rigid planar patch. IEEE Trans. Acoustics Speech Signal Process. 29(6), 1147–1152 (1981) 19. Tsai, R.Y., Huang, T.S.: Estimation of three-dimensional motion parameters of a rigid planar patch III: Finite point correspondence and the three-view problem. IEEE Trans. Acoustics Speech Signal Process. 32(2), 213–220 (1984) 20. Tsai, R.Y., Huang, T.S., Zhu, W.-L.: Estimation of three-dimensional motion parameters of a rigid planar patch II: Singular value decomposition. IEEE Trans. Acoustics Speech Signal Process. 30(4), 525–544 (1982)
Pursuing Atomic Video Words by Information Projection Youdong Zhao1 , Haifeng Gong2 , and Yunde Jia1 1
School of Computer Science, Beijing Institute of Technology, Beijing 100081, China 2 GRASP Lab., University of Pennsylvania, Philadelphia, PA 19104, USA {zyd458,jiayunde}@bit.edu.cn,
[email protected]
Abstract. In this paper, we study mathematical models of atomic visual patterns from natural videos and establish a generative visual vocabulary for video representation. Empirically, we employ small video patches (e.g., 15×15×5, called video “bricks”) in natural videos as basic analysis unit. There are a variety of brick subspaces (or atomic video words) of varying dimensions in the high dimensional brick space. The structures of the words are characterized by both appearance and motion dynamics. Here, we categorize the words into two pure types: structural video words (SVWs) and textural video words (TVWs). A common generative model is introduced to model these two type video words in a unified form. The representation power of a word is measured by its information gain, based on which words are pursued one by one via a novel pursuit algorithm, and finally a holistic video vocabulary is built up. Experimental results show the potential power of our framework for video representation.
1
Introduction
There is a long history of research on texton or image primitive [1–3]. Image patches have played a significant role in recognition. In recent literature, video patches have attracted more and more attentions [4, 5]. However, they mainly focus on human action analysis by designing discriminative features. There lacks a systematic analysis and holistic view for the distribution of atomic motion patterns under a unified modeling framework. And it was never clear in vision what are the atomic video elements. In this paper, we study mathematical models of atomic visual patterns from natural videos and establish a generative visual vocabulary for video coding and representation. Empirically, we employ small video patches (e.g., 15 × 15 × 5, called video “bricks”) in natural videos as basic analysis unit. Fig. 1 shows some examples. From a mathematical perspective, these bricks embedded in the high (1125) dimensional space, form a variety of clusters of varying dimensions and structures. We adopt the notion “atomic video words” for these clusters which can be used as building blocks for video coding and representation. Our objective is two-fold: (i) Analyzing the space distribution of video bricks and dividing into 2 types under a common generative model; (ii) Studying a pursuit algorithm for R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 254–267, 2011. Springer-Verlag Berlin Heidelberg 2011
Pursuing Atomic Video Words by Information Projection sketch
flat
texture
Edge
Bar
N/A
N/A
water
grass/ leaves
crowded
trackable
motion
N/A
lighting
N/A
intrackable
still
appearance motion
255
Fig. 1. The types of appearance and motions of video bricks
selecting the 2 type video words step by step in the brick space and building up a visual vocabulary for video representation and modeling. These small video bricks are less affected by compositionality and thus remain “pure”. They are characterized by two factors: appearance and motion. Fig. 1 illustrates the catalogue of video bricks. According to appearance they are divided into three types: (i) flat patch; (ii) structural primitives, for object boundaries[1, 2]; and (iii) texture, for complex visual patterns which are often represented by MRF models [6]. Motion also divide them into three types: (i) still; (ii) trackable motion, such as moving attributed points [7], bars/edges [8], or articulated body [9] and (iii) intrackable motion, such as crowded flying birds and water flowing, which are often called dynamic texture [10, 11]. Another factor characterizing bricks is lighting changes, whose effect is low dimensional [12], e.g., an image patch under vehicle lights in surveillance video [13]. According to above categories of bricks, we categorize subspaces in the brick space into two types: structural video words (SVWs) and textural video words (TVWs). A SVW is spanned by one (or few) large base vectors, e.g. to represent a moving primitive, and a TVW is constrained by certain statistical properties– marginal histograms shared by all bricks in the subspace. As shown in Fig.2, bricks from a video sequence may fall in a small number of subspaces. That is, a video can be represented by some subspaces certainly. We introduce a common generative model for the two type words, which are pursued by information projection. The representation power of a word is measured by information gain, based on which words are pursued one by one, and finally a holistic video vocabulary is built up. This vocabulary could be applied for many video analysis tasks. Extensive experimental evaluations show the promising applications of the proposed framework. Our work is related to the work in [3], which maps the natural image patches into two type manifolds in terms of appearance. In our work, appearance and motion dynamics of spatiotemporal video patches are considered simultaneously.
256
Y. Zhao, H. Gong, and Y. Jia
Structural video brick
Brick Space
Textural video brick
Fig. 2. Map bricks from a video onto subspaces in the brick space: structural video words (SVWs) and textural video words (TVWs)
New modeling methods are introduced to capture the words efficiently. Our work is important for the rapidly increasing video analysis tasks. The proposed work is also related to dynamic texture [10], distributed oriented energy representation [14], and mixture of dynamic textures [15]. While they success in some video tasks, they all only employ one type model and do not account for the important structure information explicitly. We adopt two complementary models taking the informative structure part into account explicitly. The remainder of this paper is organized as follows: in Section 2, the formulation of the atomic video words is presented. After that we introduce how to model the two type words and pursue the video words in Section 3. The experimental results are presented in Section 4. We conclude in Section 5.
2
Formulation of the Atomic Video Word
We fix the size of brick, denoted by B, as 15 × 15 pixels in space, and 5 frames (about 200ms) in time, so that each brick contains a pure, not mixed, appearance or motion. Definition 1. A structural video word (SVW) is a subspace spanned by one (or few) basis functions, n αi Bi + }, (1) Ωsvw = {B : B = i=1
where the basis functions Bi are selected (pursued) from a spatio-temporal primitive bank Δsvw = {Bi }N i=1 , αi is the reconstruction coefficients and is the residual. The spatio-temporal primitive bank Δsvw is moving Gabor filters. We use large scale 2D Gabor filters at 8 orientations with size of 13 × 13 pixels. They move perpendicular to their orientations with speeds at 0, ±1, ±2, ±3, ±4, ±5, and 8 pixels. So we obtain N = 8 × 12 = 96 spatio-temporal basis functions.
Pursuing Atomic Video Words by Information Projection
257
Definition 2. A textural video word (TVW) is a subspace of brick that share a set of common constraints, Ωtvw = {B : Hi (B ∗ Fi ) = Hi∗ + , i = 1, 2, · · · , l}, (2) where Fi is selected from a filter bank Δtvw = {Fi }N i=1 , (· ∗ ·) denotes the convolution operator, Hi (B ∗ Fi ) is histogram of the filter responses over all pixels in a brick B, Hi∗ is mean histogram of the cluster, and is the fluctuation. We use small scale 2D Gabor filters at 8 orientations with size 7 × 7 in pixel. They move perpendicular to their orientations with speeds at 0, ±1, ±2, ±3, and 5 pixels. So we again obtain N = 8 × 8 = 64 spatio-temporal Gabor filters. In addition, the responses of filters are normalized against neighborhood for comparability. For each word Ω k ∈ {Ωtvw , Ωsvw } , there is an underlying distribution fk (B) to describe it. The objective of modeling is to learn a probability model pk (B) to approach the fk (B), which can be realized by minimizing the Kullback-Leibler divergence, fk (B) KL(fk pk ) = Efk log . (3) pk (B) Here, we give the common generative model for the two types of words: n 1 exp − λi ri (B) qk (B) , pk (B|λ, Δ) = Z(λ) i=1
(4)
n
where Δ = {Bi or Fi }i=1 are selected basis set, λ = (λ1 , · · · , λn ) are the corresponding parameters, Z (λ) is the partition function, and qk (B) is reference distribution. If the word is from Ωsvw , the selected basis set Δ = {Bi }ni=1 ⊂ Δsvw , ri (B) = max{Sigmoid(|B ∗ Bi |2 )} (i.e., ri (B) is the maximum of responses modulated by Sigmoid transform in B), and qk (B) usually is the distribution estimated offline from natural scene bricks. This model is similar to active basis model [16], with the difference in spatiotemporal filters. We can simply measure how well a brick B belongs to Ω k by Scoresvw = i ri (B). If the word is a TVW, the selected filter set Δ = {Fi }li=1 ⊂ Δtvw , ri (B) = Hi (B ∗ Fi ) − Hi∗ 2 , Hi∗ is the mean histogram pooled over the subspace, and qk (B) usually is the uniform distribution. How B belongs to Ω k could also be simply measured well a brick tvw ∗ 2 = i Hi − Hi . by Score
3
Atomic Video Words Pursuit
Based on the above definitions, in this section, we will show how to pursue the SVWs and TVWs in a common generative framework.
258
3.1
Y. Zhao, H. Gong, and Y. Jia
Pursuing the Model of a Word by Information Projection
Given a set of bricks sampled from a pure subspace (or word) Ω k , its probability model defined in Eq.(4) could be iteratively pursued in the following two steps: (I) Learning the parameter λ+ . As a new feature B+ or F+ is introduced, the corresponding parameter λ+ is computed by minimizing the KL divergence in Eq. (3). For notation simplicity, we denote the current model as p and the augmented model as p+ as illustrated in Fig.3. Then the optimized augmented model is p∗+ = arg min KL (f p+ ) , (5) p+
subject to the statistical constraint: k
|Ω | 1 Ep+ [r+ ] = Ef [r+ ] = r + = k r+ (Bi ), |Ω | i=1
(6)
where r+ is an abbreviation for r+ (B), which is the response of feature B+ or F+ on B. Solving the above constrained optimization, we obtain the augmented model: 1 p (B) exp {−λ+ r+ (B)} , (7) p+ (B) = Z (λ+ ) where λ+ is calculated so that the new constraint (Eq.(6)) is satisfied and Z(λ+ ) = Ep [exp{−λ+ r+ }]. If we keep B+ or F+ independent of the previously selected features, we have Ep [exp{−λ+ r+ }] = Eq [exp{−λ+ r+ }], where q is the reference distribution (mentioned in Section 2). Then, due to log-linear form of Eq.(7), λ+ can be solved by a look-up table of λ+ versus Ep+ [r+ ]. Z(λ+ ) also could be computed in a similar way. p Ð+ = fp : Ep[r+(B)] = Ef [r+(B)]g
p+
f KL(f jp) = KL(f jp+) + KL(p+jp)
Fig. 3. Sketch map for the feature pursuit process
(II) Select a new feature B+ or F+ . Among all the candidate features Δsvw = {Bi } or Δtvw = {Fi }, the best parameters λ of each feature are computed as discussed in above Step (I), and as shown in Fig.3 the best feature B+ or F+ is chosen so that it maximizes the the information gain KL(p+ p): p+ def ∗ ∗ B+ or F+ = arg max KL(p+ p) = arg max Ep+ log = gain(B+ or F+ ). p (8)
Pursuing Atomic Video Words by Information Projection
259
Because of the response of a new feature r+ is a one-dimensional constraint and the features are pursued one by one, the information gain can be computed as Ep+ [log pp+ ] = −λ+ Ep+ [r+ ]−log Z(λ+ ). For SVWs, λ is negative, so the above information gain suggests selecting the feature B+ with largest mean response Ep+ [r+ ]. For TVWs, λ is positive, the suggested feature F+ achieving the small∗ 2 est sample variance of Ep+ [H+ (B ∗ F+ ) − H+ ]. This learning process is called information projection. Each time, it projects the model to a new constrained space specified by Eq.(6) to augment the model. In practice, for computational efficiency of TVWs, instead of pursuing feature one by one, we simplify the modeling problem by using all the features in Δtvw to form a long histogram feature (called implicit representation) for each brick. And the distance from each brick to the mean histogram of the cluster is defined as |Δtvw | tvw Hi (B, Fi ) − Hi∗ 2 = H(B) − H ∗ 2 , (9) r (B) = i=1 t t (H1t , . . . , HΔ ) tvw
where H(B) = and H ∗ is the mean histogram. Then the new probability model for a TVW is pk (B) =
1 1 q(B) exp −λr tvw (B) = q(H) exp −λH − H ∗ 2 = pk (H). Z(λ) Z(λ) (10)
We simply assume that pk (H) is Gaussian, and that q(H) is uniform. pk (H) is a multivariate Gaussian, but we assume that the variance-covariance matrix of it is of such a low rank that we could essentially treat it as a univariate Gaussian along its major principal axis with variance σ 2 . So in this case, λ = √ |Ω k | k | i=1tvw H(Bi ) − H ∗ 2 . This 1/2σ 2 and Z(λ) = 2πσ 2 , where σ 2 = 1/|Ωtvw approximation is computational efficiency and reasonable, which is shown in Section 4.1. 3.2
Pursuing Atomic Video Words by Hybrid Modeling
Given millions of bricks sampled from a large number of natural scene videos, there are various TVWs and SVWs embedded in the high dimensional brick space. In terms of the above model definitions, we employ an EM-like algorithm to learn the mixture of TVWs and the mixture of SVWs over the whole samples respectively, which is initialized by a hierarchial k-means clustering scheme [17]. For TVWs, the learning process is called implicit modeling and 1 2 N , Ωtvw , . . . , Ωtvw }. the captured mixture of TVWs is denoted as Ωtvw = {Ωtvw And explicit modeling indicates the learning procedure of mixture of SVWs 1 2 M Ωsvw = {Ωsvw , Ωsvw , . . . , Ωsvw }. Ωtvw and Ωsvw are two candidate sets of atomic video words and there are heavy overlaps in space between them. Our goal is to learn a video vocabulary Ω = {Ω1 , Ω2 , . . . , ΩK } containing various atomic words with few overlapping and covering the whole brick space. This procedure is called hybrid modeling, which is discussed in following.
260
Y. Zhao, H. Gong, and Y. Jia
Let f (B) be the underlying true distribution governing the natural video bricks ensemble Ωnat .
Ksvw k Ktvw k Ωnat = Ωsvw , Ωtvw k=1 , Ω0 = {Ω0 , Ω1 , . . . , ΩK } , (11) k=1 where Ksvw + Ktvw = K and Ω0 is the remaining uniform distribution. The probability model for Ωnat is a mixture distribution: p (B|Λ, Δ) =
K
wk pk (B|Λ; Δ),
(12)
k=0
where wk and pk (B|Λ, Δ) are the weight and probability model (discussed above) of the k-th word, Λ is the parameter set, and Δ = {Δsvw , Δtvw }. We now learn the mixture model p (B|Λ, Δ) in Eq.(12) to approach f (B) by minimizing the Kullback-Leibler divergence: f (B) KL(f p) = Ef log (13) = −Ef [log p (B)] + Const. p (B) This minimization amounts to maximizing Ef [log p(B)] , which can be decom
posed as K wk pk (B) (14) arg max Ef [log p (B)] = arg max Ef log p
p
≥ arg max p
K k=0
k=0
wk Ef [log pk (B)] ≈ arg max p
K
wk EΩk [log pk (B)] ,
k=0
where wk is approximated by frequency, and the approximation holds because overlapping between subspaces is neglectable. The mean score EΩk [log pk (B)] over Ωk can be computed by pk (B) pk (B) EΩk [log pk (B)] = Epk log + log q(B) = Epk log + Const.(15) q(B) q(B) In order to avoid overfitting during minimizing KL(f p) that equivalents to maximum likelihood estimation, we introduce a constraint of model complexity − k αwk log wk in Eq.(14), where α is a positive parameters (e.g., α = 0.1). So the information gain of a SVW or TVW is pk (B) (16) − αwk log wk , lk = wk Epk log q(B) where
n pk (B) Epk log −λi Epk [ri ] − log Z(λi ). = q(B) i=1
(17)
As the number of words is small and the frequency of a word is very high, this introduced constraint term −αwk log wk is a monotonously decreasing functional w.r.t. wk and encourages splitting the word. On the contrary, it becomes a monotonously increasing functional and favors merging this word with others.
Pursuing Atomic Video Words by Information Projection
261
Algorithm 1. Video words pursuit by hybrid modeling 1 2 M 1 2 N Input: Ωsvw = {Ωsvw , Ωsvw , . . . , Ωsvw } and Ωtvw = {Ωtvw , Ωtvw , . . . , Ωtvw } Output: Ω = {Ω1 , Ω2 , . . . , ΩK }
1 2 3 4 5 6
7
Initalize Ω = ∅, K ← 0; repeat s s Select theΩsvw or Ωtvw with highest gain gmax calculated by Eqs. (16) and (17). for each Ω k ∈ Ωsvw , Ωtvw do Remove copies of Bi from all clusters if Bi ∈ Ω s ; Recalculate the model of clusters overlapping with Ω s in two manners and select the one with higher gain; until gmax < t, or the sample set is ∅, or Ωtvw and Ωsvw = ∅ ;
The video words pursuit algorithm is summarized in Algorithm 1. This algorithm essentially ranks all the words in Ωtvw and Ωsvw by their information gains, and removes overlaps along the way bay favoring the word that achieves a greater amount of information gain. This pursuit process may be viewed as seeking the words with the largest frequency and smallest entropy at each step. Through this stage, we obtain a genetic video dictionary for video coding and modeling. For example, we could use it for video representation, e.g., as a high dimensional word frequency vector, or segmentation of videos composed of various appearances and motions, which will be shown in the next experiment section. It is worth noting that the whole process of our framework is unsupervised and scalable for larger video dataset.
4
Experiments
To evaluate the proposed method, we construct three types of experiments: (i) Dynamic texture segmentation. We show the encouraging representation power of our implicit representation, which also indicates the simplification and Gaussian assumption in Section 3.1 are reasonable. (ii) Generic video vocabulary learning. By learning a generic video vocabulary using the hybrid modeling, we get some insights in the distribution of natural video bricks and show representation power of two type words by an embedding experiment. (iii) Video scene recognition. Inspired by the above promising representation, a real video scene recognition is conducted, which supplies a numerical comparison among related methods and shows the superiority of our method to others. 4.1
Dynamic Texture Segmentation by Implicit Representation
We first compare our implicit representation optical flow, pixel filter response and pixel synthetic sequences as shown in Fig. 4. The is a 64-dimensional L1-normalized histogram
with three others, i.e., traditional intensity, on two complementary implicit representation of a brick of filter responses pooled over the
262
Y. Zhao, H. Gong, and Y. Jia Frames
Our Results
Optical Flow
Filter Response
Pixel Intensity
r=0.9362
r=0.7480
r=0.5480
r=0.5304
r=0.9385
r=0.7186
r=0.5029
r=0.5007
Fig. 4. Segmentation by changing dynamics and texture orientation. In row 1, the two dynamic textures are identical in appearance, but differ in the dynamics (speeded up by a factor of 2). In row 2, segmentation of two dynamic textures which differ only in spatial orientation (rotated square area by 90 degrees) but share the same dynamics and general appearance. 0.06
0.06 Background Froeground
0.04 0.03 0.02 0.01 0
Background Foreground
0.05 Normalized Responses
Normalized Responses
0.05
0.04 0.03 0.02 0.01
0
10
20
30 40 Features
50
60
(a) Changing dynamics
70
0
0
10
20
30 40 Features
50
60
70
(b) Texture orientation
Fig. 5. Four mean histograms correspond to the segmentation results of our implicit representation in Fig.4. The foreground indicates the square region in the center. It is apparent while the bg/fg share many common statistics, they are distinct by the dynamic speeds or texture orientation.
small brick volume1 . Each dimension corresponds to a filter mentioned in Section 2. The pixel filter response means we utilize the 64 filter responses in each pixel to characterizing the pixel directly. The commonly used mean-shift algorithm [18] is adopted as the segmentation method. There is one mean-shift bandwidth parameter h needed to be tuned. In Fig. 4, while the same segmentation scheme is employed, only our implicit representation successes in separating the two sequences. Fig. 5 supplies an intuitive explanation that how the proposed implicit representation is capable of dynamic texture segmentation. We also compare our implicit representation with state-of-the-arts2 on a public dataset [15] as shown in Fig. 6. The results of DTM [15] on these examples require knowledge of the number of textures as well as hand contour initialization. The LDT [19] is a graphical model that has significantly larger computational 1 2
Note that you could also divide the brick volume into several spatiotemporal cells, compute histogram in each and get a longer description. The results compared are available at http://www.svcl.ucsd.edu/projects/layerdytex/synthdb/
Pursuing Atomic Video Words by Information Projection Frames
Our Results
LDT
DTM
263
GPCA
texture _010 r=0.8899
r=0.7454
r=0.8604
r=0.5060
r=0.5442
r=0.5028
texture _011 r=0.5001
r=0.5002
Fig. 6. Comparison with the state-of-the-art dynamic texture segmentation methods
(a)
(c)
(e)
(b)
(d)
(f)
(g)
t
Ă Normalized Responses
0.05 0.04 0.03 0.02 0.01
10
20
30 40 Features
50
60
Fig. 7. Test on two real challenging scenes
requirements. While the state-of-the-arts fail in separating the foreground/ background on two dynamic texture examples, the proposed method exhibits successful segmentation results. The implicit representation is also evaluated on two real scenes in Fig.7. Scene 1 contains rich dynamic textures with different spatial orientations, moving speeds and motion directions. Though there is no ground truth of segmentation for this scene, the result (in Fig.7(b)) is reasonable and meaningful. In scene 2, a human is passing a river with fast-flowing via a cableway. The human is separated from the fast-flowing river successfully in Fig.7(d). For comparison, we perform the hybrid modeling framework in this scene, and the results are illustrated in Fig. 7(e). The whole brick space of this sequence is modeled by two TVWs (fast-moving water and human body) and five SVWs (cable, human legs, etc.) as shown in Fig.7(f) and (g). It is apparent that the structural information lost by above mean-shift scheme is captured and modeled well in the hybrid framework. From this result, it seems that the learnt hybrid vocabulary is a better representation for the video scene in semantic and maybe competent for some high-level visual tasks, e,g., video scene recognition. 4.2
Video Representation by Hybrid Vocabulary
In this experiment, 200 scene video sequences are collected (samples shown in Fig. 1). All these sequence are compressed by Xvid coder with 25 fps, resized to be 310 × 190 and persisting 3 seconds at least. We use 100 videos for video
264
Y. Zhao, H. Gong, and Y. Jia
Fig. 8. Word types in the dictionary. Red indicates TVWs and blue indicates SVWs
1.5
3
1.5
TVP SVP
2.5
TVP SVP
2
TVP SVP
1
1
0.5
0.5
1.5 1 0.5 0
0
5
10
15
20
25
30
0
0
5
10
15
20
25
30
0
0
5
10
15
20
25
30
Fig. 9. Three video sequences and information gains of the top 30 words in them
vocabulary learning. For each sequence, 60 frames are selected and converted to gray, from which 138, 225 bricks are extracted. Totally, near 1.4 × 107 bricks are collected and used. Through the hybrid modeling, a video dictionary composed of 1983 words are obtained. Fig. 8 shows the word types in the dictionary, i.e., red color indicating the textural video words and blue color the structural video words. The TVWs are mainly distributed in the high information gain region, and the SVWs are converse. In the middle region the numbers of two type words are comparative. In the first 100 words, there are only four structural video words, which are static edge, bar, and two moving edges. The highest information gain word is a cluster in which the bricks are near flat. The distribution of information gains of these words is decreasing monotonously whereas the frequency is not. This indicates there exist clusters that distribute sparsely than others. Once we obtain a ranked video dictionary, a video sequence could be represented in the fashion of Bag-of-Words. Owning to the two complementary types of words, we call the representation using our vocabulary as Bag-of-2-Words (Bo2Ws). As in document analysis, the high frequency words in a document often imply the document type. Here, we analyze this phenomena in videos by the video dictionary. Fig.9 shows us three video sequences and corresponding statistics of the top30 words appearing in them respectively. In the crowded video (1st column), there are mainly TVWs and several SVWs appearing on the edges. The SVWs are rich in the third simple video. In the middle complex scene (2nd column), two type bricks appear in turn. The top30 words in each video simply classify them from each other. Here, we only take two type words into account in general but do not further consider more details in each word, e.g., texture type, structure type or motion direction. In order to illustrate the representation power of the video dictionary intuitively, we design a visualization experiment using Isomap algorithm [20]. We
Pursuing Atomic Video Words by Information Projection
265
4
7
x 10
52
6
133
5
54
4 35 71 51 78 140 126 103 107
3 2
72
134 45 9 141 6593 138 31 42 55 60 537429 4711933 18 115 130 117 68 36 161 129 108 731334 104 37 81 118 8156 16 94 12 152 158 67 145 153 92 3146 19 116 30 157 159 155 144 91 28 23 164 84 70 25 120 88 162 163 46 124 40 111 89 149 20 123 26 39 56 7150 151 64 66 17 50 160 113 44 106 57 125 90 114 15 109 59 1131 10 24 41 148 127 22 128 43 14 139 38 100 101 21 10 95 142 99 48 77 102 79 49 82 154 122 69 27 132 6 143 112 2 5 147 63137 121 76 8580 1 32 87 61 98 86 96 105 97 135 83 11 4 62 136
1 0
75 58
-1 -2 -3 -10
-8
-6
-4
-2
0
2
4 4
x 10
Fig. 10. Isomap visualization of distribution of test videos by our vocabulary
select 164 video sequences for testing. Some of them share the same scenes as training videos but at a different time interval. Each testing video is quantized as a frequency vector of 1, 983 dimensions. All the 164 videos are projected into a 2 dimensional space by Isomap algorithm (in Fig.10). The digit near each point is the number of the corresponding video. Samples of 9 video scenes are shown, which correspond to eight points (75, 34, 130, 155, 71, 54, 133, and 52) along the curve and the 136th point respectively. The video complexity is increasing along the curve and most of the videos are distributed in the middle region. The 136th video is a Marathon scene with camera still, and the translational motion of the crowded people is simpler than the random motion (e.g., in the 52nd video). This agrees with the fact that most of the testing videos are in middle complexity and few videos are in extremely low or high complexity. 4.3
Video Scene Recognition
In order to numerically study the power of vocabulary learnt by the proposed method, we conduct a video scene recognition task on the public Hollywood Scene Dataset established by Marszalk et al. [21]. They first evaluate on it. Their scene classifiers are based on a bag-of-words video representation and classification with SVM. Average precision (AP), which approximates the area under a recall-precision curve, is employed as the measurement of performance. They use simultaneously static (SIFT) and dynamic features (HOG/HOF) for video representation and show that both static and dynamic features are important for recognition of scenes. Following the evaluation framework in [21], we compare four different video vocabularies which are learnt by simple K-means clustering (as baseline), implicit modeling, explicit modeling and hybrid modeling. Specifically, we first extract spatiotemporal interesting points randomly. Then we learn video vocabularies by above four mentioned methods. For K-means clustering, we use the implicit
266
Y. Zhao, H. Gong, and Y. Jia
Average precision (AP)
0.8
0.6
Kmeans Implicit-Modeling Explicit-Modeling Mixed Modeling Marszalek's Method
0.4
0.2
0 EXT-house EXT-road INT-bed.
INT-car INT-hotel INT-kit.
INT-liv. INT-office INT-rest. INT-shop
Fig. 11. Results of video scene recognition on Hollywood Scene dataset Table 1. Mean Average Precisions of Five Methods over Ten Scenes Methods K-means Im-modeling Ex-modeling Hy-modeling Marszalk et al.’s Mean AP 0.336 0.358 0.321 0.415 0.351
representation as descriptions of bricks. Finally, SVM classifiers with χ2 -kernel are used to classify samples using one-against-rest strategy. The results are shown in Fig.11 and Table1. Note that due to the simplification in Section 3.1, implicit modeling is similar to Kmeans clustering method, and it is not surprising that they achieve similar results. For most of the scenes, implicit modeling using only TVWS obtains better results than explicit modeling except the “INT-Car” and “INT-LivingRoom”. This may be because in the two scenes there are more structural parts and the SVWs are more distinctive. The hybrid modeling combining two type words in a common generative framework achieves improved performance and is superior to Marszalk et al.’s method which also combines different type features. Our results further confirm that SVWs and TVWs are two complementary atomic video patterns and both important for video representation. This is consistent with the observations of Marszalk et al’s work. Note that different types of features are combined by a unified generative model in our method, unlike Marszalk et al.’s method that fuses different features in a SVM framework using multi-channel Gaussian kernel.
5
Summary
In this paper, we have presented a unified framework for characterizing various video bricks to generate a “panorama map” of words in the brick space. These words are “weighted” (by information gain) and pursued under an information projection framework. A video dictionary containing two complementary types of words is obtained for video coding and modeling. We show some applications of the video word representation and achieve promising results compared with the state-of-the-arts. These video words are atomic structures to construct higher level video patterns through composition. In future work, we will study how words are composed to form larger motion patterns, such as human actions, in larger image patches and longer time sequences.
Pursuing Atomic Video Words by Information Projection
267
Acknowledgement. This work was done at Lotus Hill Research Institute and was supported by the Natural Science Foundation of China (NO. 90920009 and NO. 60875005) and the Chinese High-Tech Program (NO. 2009AA01Z323).
References 1. Olshausen, B.A., Field, D.J.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 (1996) 2. Zhu, S.C., Guo, C.E., Wang, Y.Z., Xu, Z.J.: What are textons? IJCV (2005) 3. Shi, K., Zhu, S.C.: Mapping natural image patches by explicit and implicit manifolds. In: CVPR (2007) 4. Doll´ ar, P., Rabaud, V., Cottrell, G., Belongie, S.: Behavior recognition via sparse spatio-temporal features. In: VS-PETS (2005) 5. Shechtman, E., Irani, M.: Space-time behavior-based correlation. PAMI (2007) 6. Zhu, S.C., Wu, Y.N., Mumford, D.: Filters, random-fields and maximum-entropy (frame): Towards a unified theory for texture modeling. IJCV 27, 107–126 (1998) 7. Veenman, C., Reinders, M., Backer, E.: Resolving motion correspondence for densely moving points. PAMI (2001) 8. Olshausen, B.A.: Learning sparse, overcomplete representations of time-varying natural images. In: ICIP (2003) 9. Wu, Y., Hua, G., Yu, T.: Tracking articulated body by dynamic markov network. In: ICCV (2003) 10. Soatto, S., Doretto, G., Wu, Y.: Dynamic textures. In: ICCV (2001) 11. Wang, Y., Zhu, S.C.: Modeling textured motion: Particle, wave and sketch. In: ICCV (2003) 12. Belhumeur, P., Kriegman, D.: What is the set of images of an object under all possible illumination conditions? Int. Journal of Computer Vision 28, 245–260 (1998) 13. Zhao, Y.D., Gong, H., Lin, L., Jia, Y.: Spatio-temporal patches for night background modeling by subspace learning. In: ICPR (2008) 14. Derpanis, K.G., Wildes, R.P.: Early spatiotemporal grouping with a distributed oriented energy representation. In: CVPR (2009) 15. Chan, A.B., Vasconcelos, N.: Modeling, clustering, and segmenting video with mixtures of dynamic textures. PAMI 30, 909–926 (2008) 16. Wu, Y.N., Si, Z., Fleming, C., Zhu, S.C.: Deformable template as active basis. In: ICCV (2007) 17. Nister, D., Stewenius, H.: Scalable recognition with a vocabulary tree. In: CVPR (2006) 18. Comaniciu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. PAMI (2002) 19. Chan, A.B., Vasconcelos, N.: Layered dynamic textures. PAMI 31, 1862–1879 (2009) 20. Tenenbaum, J.B., de Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science (2000) 21. Marszalk, M., Laptev, I., Schmid, C.: Actions in context. In: CVPR (2009)
A Direct Method for Estimating Planar Projective Transform Yu-Tseh Chi1 , Jeffrey Ho1 , and Ming-Hsuan Yang2 1
2
Computer and Information Science and Engineering, University of Florida, Gainesville, FL 32611, U.S.A. {ychi,joh}@cise.ufl.edu Electrical Engineering and Computer Science, University of California at Merced, Merced, CA 95344, U.S.A.
[email protected]
Abstract. Estimating planar projective transform (homography) from a pair of images is a classical problem in computer vision. In this paper, we propose a novel algorithm for direct registering two point sets in R2 using projective transform without using intensity values. In this very general context, there is no easily established correspondences that can be used to estimate the projective transform, and most of the existing techniques become either inadequate or inappropriate. While the planar projective transforms form an eight-dimensional Lie group, we show that for registering 2D point sets, the search space for the homographies can be effectively reduced to a three-dimensional space. To further improve on the running time without significantly reducing the accuracy of the registration, we propose a matching cost function constructed using local polynomial moments of the point sets and a coarse to fine approach. The resulting registration algorithm has linear time complexity with respect to the number of input points. We have validated the algorithm using points sets collected from real images. Preliminary experimental results are encouraging and they show that the proposed method is both efficient and accurate.
1
Introduction
The importance of 2D projective transforms (planar homographies) in computer vision stems from the well-known fact that different images of a planar scene are related by planar projective transforms. For vision applications that deal with planar objects, examples of which include recognizing hand drawings and traffic signs, etc., estimating the projective transform between two images is required for normalizing the image variation due to different cameras. Accordingly, there are now well-established techniques for automatically recovering planar homography between images [1]. For these feature-based methods, the idea is to register two sets of interest points P = {p1 , · · · , pk } and Q = {q1 , · · · , qk } extracted from the two images using projective transforms. Intensity values (local intensity patterns) provide important clues on establishing correspondences across images, R. Kimmel, R. Klette, and A. Sugimoto (Eds.): ACCV 2010, Part II, LNCS 6493, pp. 268–281, 2011. Springer-Verlag Berlin Heidelberg 2011
A Direct Method for Estimating Planar Projective Transform
269
Fig. 1. Four examples for which a direct application of SIFT [2] fails to produce correct correspondences. The first two images are hand drawings on a whiteboard, and the others are images of street signs.
and Direct Linear Transform (DLT) requires only four correspondences in order to compute a homography. Implicitly, the algorithms define a discrete search space S that contains a large number of projective transforms (from all possible correspondences made across the images), and the image intensity pattern is used to prune away most part of S that are irrelevant, and the algorithms exhaustively test each remaining homography according to some matching cost function to obtain the optimal homography. In this paper, we study the 2D projective registration problem in its barest form: using only point sets without intensity values. While DLT requires only four correct correspondences, there are cases where it is difficult to correctly determine these correspondences automatically. Figure 1 shows four pairs of such images that cannot be registered satisfactorily using existing methods. The lack of salient local intensity patterns often confuses algorithms that match local feature descriptors. Furthermore, local geometric structure such as curvatures are not reliable features for computing correspondences since they could vary wildly under projective transforms. Nevertheless, we can still solve the problem by sampling a collection of points from each image and register the point sets using projective transforms1 . A set of four points (or more) can be chosen, and a homography search space S can be defined as consisting homographies generated by all correspondences for these four points. For each homography in S, its registration quality can be evaluated by a matching cost function such as Γ (H; P, Q) =
l j=1
min |qj − H(pi )|2 .
i=1,··· ,k
(1)
For point sets containing 500 points, S contains roughly 65 billion homographies. While this is by no mean a small number, it is still within the capability of modern hardware to exhaustively search S for the optimal homography. However, the quartic time complexity makes this approach unattractive since if 5000 points were sampled instead, the search space would increase enormously to the point that renders the algorithm impotent, even though the two problems do not different in any essential way other than the number of points (they have the same images). The approach advocated in this paper is to abandon the usual paradigm of defining the search space S using small number of correspondences, which is often impossible for us anyway. Instead, the search space S should be defined entirely 1
Or we can register the binary image obtained from the segmented image as in [3].
270
Y.-T. Chi, J. Ho, and M.-H. Yang
by the point sets in some global way. The main contribution of this paper is to show that, under a fairly general assumption on the variability of the homographies, we can define an effective three-dimensional (discrete) search space S for projective registration of two point sets in R2 . The size of S is independent of the sizes of the input point sets, and in many cases, it is possible exhaustively search for the optimal homography in S. Therefore, the time complexity of the algorithm depends on the complexity of evaluating the matching cost function. For Γ defined in Equation 1, the complexity is quadratic since it requires the computation of pairwise distances. However, by using a less expensive matching cost function constructed from local polynomial moments of the point sets, we show that it is possible to have an accurate projective registration algorithm with linear time-complexity. The projective registration method proposed in this paper is direct in its truest sense: the algorithm takes in two point sets, which can be readily generated from images using low-level image p