Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Moshe Y. Vardi Rice University, Houston, TX, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany
4843
Yasushi Yagi Sing Bing Kang In So Kweon Hongbin Zha (Eds.)
Computer Vision – ACCV 2007 8th Asian Conference on Computer Vision Tokyo, Japan, November 18-22, 2007 Proceedings, Part I
13
Volume Editors Yasushi Yagi Osaka University The Institute of Scientific and Industrial Research 8-1 Mihogaoka, Ibaraki, Osaka, 567-0047, Japan E-mail:
[email protected] Sing Bing Kang Microsoft Corporation 1 Microsoft Way, Redmond WA 98052, USA E-mail:
[email protected] In So Kweon KAIST School of Electrical Engineering and Computer Science 335 Gwahag-Ro Yusung-Gu, Daejeon, Korea E-mail:
[email protected] Hongbin Zha Peking University Department of Machine Intelligence Beijing, 100871, China E-mail:
[email protected]
Library of Congress Control Number: 2007938408 CR Subject Classification (1998): I.4, I.5, I.2.10, I.2.6, I.3.5, F.2.2 LNCS Sublibrary: SL 6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics ISSN ISBN-10 ISBN-13
0302-9743 3-540-76385-6 Springer Berlin Heidelberg New York 978-3-540-76385-7 Springer Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com © Springer-Verlag Berlin Heidelberg 2007 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12183654 06/3180 543210
Preface
It is our great pleasure to welcome you to the Proceedings of the Eighth Asian Conference on Computer Vision (ACCV07), which held November 18–22, 2007 in Tokyo, Japan. ACCV07 was sponsored by the Asian Federation of Computer Vision. We received 640 abstracts by the abstract submission deadline, 551 of which became full submissions. This is the largest number of submissions in the history of ACCV. Out of these 551 full submissions, 46 were selected for oral presentation and 130 as posters, yielding an acceptance rate of 31.9%. Following the tradition of previous ACCVs, the reviewing process was double blind. Each of the 31 Area Chairs (ACs) handled about 17 papers and nominated five reviewers for each submission (from 204 Program Committee members). The final selection of three reviewers per submission was done in such a way as to avoid conflict of interest and to evenly balance the load among the reviewers. Once the reviews were done, each AC wrote summary reports based on the reviews and their own assessments of the submissions. For conflicting scores, ACs consulted with reviewers, and at times had us contact authors for clarification. The AC meeting was held in Osaka on July 27 and 28. We divided the 31 ACs into 8 groups, with each group having 3 or 4 ACs. The ACs can confer within their respective groups, and are permitted to discuss with pre-approved “consulting” ACs outside their groups if needed. The ACs were encouraged to rely more on their perception of paper vis-a-vis reviewer comments, and not strictly based on numerical scores alone. This year, we introduced the category “conditional accept;” this category is targeted at papers with good technical content but whose writing requires significant improvement. Please keep in mind that no reviewing process is perfect. As with any major conference, reviewer quality and timeliness of reviews varied. To minimize the impact of variation of these factors, we chose highly qualified and dependable people as ACs to shepherd the review process. We all did the best we could given the large number of submissions and the limited time we had. Interestingly, we did not have to instruct the ACs to revise their decisions at the end of the AC meeting—all the ACs did a great job in ensuring the high quality of accepted papers. That being said, it is possible there were good papers that fell through the cracks, and we hope such papers will quickly end up being published at other good avenues. It has been a pleasure for us to serve as ACCV07 Program Chairs, and we can honestly say that this has been a memorable and rewarding experience. We would like to thank the ACCV07 ACs and members of the Technical Program Committee for their time and effort spent reviewing the submissions. The ACCV Osaka team (Ryusuke Sagawa, Yasushi Makihara, Tomohiro Mashita, Kazuaki Kondo, and Hidetoshi Mannami), as well as our conference secretaries (Noriko
VI
Preface
Yasui, Masako Kamura, and Sachiko Kondo), did a terrific job organizing the conference. We hope that all of the attendees found the conference informative and thought provoking. November 2007
Yasushi Yagi Sing Bing Kang In So Kweon Hongbin Zha
Organization
General Chair General Co-chairs
Program Chair Program Co-chairs
Workshop/Tutorial Chair Finance Chair Local Arrangements Chair Publication Chairs Technical Support Staff
Area Chairs
Katsushi Ikeuchi (University of Tokyo, Japan) Naokazu Yokoya (NAIST, Japan) Rin-ichiro Taniguchi (Kyuushu University, Japan) Yasushi Yagi (Osaka University, Japan) In So Kweon (KAIST, Korea) Sing Bing Kang (Microsoft Research, USA) Hongbin Zha (Peking University, China) Kazuhiko Sumi (Mitsubishi Electric, Japan) Keiji Yamada (NEC, Japan) Yoshinari Kameda (University of Tsukuba, Japan) Hideo Saito (Keio University, Japan) Daisaku Arita (ISIT, Japan) Atsuhiko Banno (University of Tokyo, Japan) Daisuke Miyazaki (University of Tokyo, Japan) Ryusuke Sagawa (Osaka University, Japan) Yasushi Makihara (Osaka University, Japan) Tat-Jen Cham (Nanyang Tech. University, Singapore) Koichiro Deguchi (Tohoku University, Japan) Frank Dellaert (Georgia Inst. of Tech., USA) Martial Hebert (CMU, USA) Ki Sang Hong (Pohang University of Sci. and Tech., Korea) Yi-ping Hung (National Taiwan University, Taiwan) Reinhard Klette (University of Auckland, New Zealand) Chil-Woo Lee (Chonnam National University, Korea) Kyoung Mu Lee (Seoul National University, Korea) Sang Wook Lee (Sogang University, Korea) Stan Z. Li (CASIA, China) Yuncai Liu (Shanghai Jiaotong University, China) Yasuyuki Matsushita (Microsoft Research Asia, China) Yoshito Mekada (Chukyo University, Japan) Yasuhiro Mukaigawa (Osaka University, Japan)
VIII
Organization
P.J. Narayanan (IIIT, India) Masatoshi Okutomi (Tokyo Inst. of Tech., Japan) Tomas Pajdla (Czech Technical University, Czech) Shmuel Peleg (The Hebrew University of Jerusalem, Israel) Jean Ponce (Ecole Normale Superieure, France) Long Quan (Hong Kong University of Sci. and Tech., China) Ramesh Raskar (MERL, USA) Jim Rehg (Georgia Inst. of Tech., USA) Jun Sato (Nagoya Inst. of Tech., Japan) Shinichi Sato (NII, Japan) Yoichi Sato (University of Tokyo, Japan) Cordelia Schmid (INRIA, France) Christoph Schnoerr (University of Mannheim, Germany) David Suter (Monash University, Australia) Xiaoou Tang (Microsoft Research Asia, China) Guangyou Xu (Tsinghua University, China)
Program Committee Adrian Barbu Akash Kushal Akihiko Torii Akihiro Sugimoto Alexander Shekhovtsov Amit Agrawal Anders Heyden Andreas Koschan Andres Bruhn Andrew Hicks Anton van den Hengel Atsuto Maki Baozong Yuan Bernt Schiele Bodo Rosenhahn Branislav Micusik C.V. Jawahar Chieh-Chih Wang Chin Seng Chua Chiou-Shann Fuh Chu-song Chen
Cornelia Fermuller Cristian Sminchisescu Dahua Lin Daisuke Miyazaki Daniel Cremers David Forsyth Duy-Dinh Le Fanhuai Shi Fay Huang Florent Segonne Frank Dellaert Frederic Jurie Gang Zeng Gerald Sommer Guoyan Zheng Hajime Nagahara Hanzi Wang Hassan Foroosh Hideaki Goto Hidekata Hontani Hideo Saito
Hiroshi Ishikawa Hiroshi Kawasaki Hong Zhang Hongya Tuo Hynek Bakstein Hyun Ki Hong Ikuko Shimizu Il Dong Yun Itaru Kitahara Ivan Laptev Jacky Baltes Jakob Verbeek James Crowley Jan-Michael Frahm Jan-Olof Eklundh Javier Civera Jean Martinet Jean-Sebastien Franco Jeffrey Ho Jian Sun Jiang yu Zheng
Organization
Jianxin Wu Jianzhuang Liu Jiebo Luo Jingdong Wang Jinshi Cui Jiri Matas John Barron John Rugis Jong Soo Choi Joo-Hwee Lim Joon Hee Han Joost Weijer Jun Sato Jun Takamatsu Junqiu Wang Juwei Lu Kap Luk Chan Karteek Alahari Kazuhiro Hotta Kazuhiro Otsuka Keiji Yanai Kenichi Kanatani Kenton McHenry Ki Sang Hong Kim Steenstrup Pedersen Ko Nishino Koichi Hashomoto Larry Davis Lisheng Wang Manabu Hashimoto Marcel Worring Marshall Tappen Masanobu Yamamoto Mathias Kolsch Michael Brown Michael Cree Michael Isard Ming Tang Ming-Hsuan Yang Mingyan Jiang Mohan Kankanhalli Moshe Ben-Ezra Naoya Ohta Navneet Dalal Nick Barnes
Nicu Sebe Noboru Babaguchi Nobutaka Shimada Ondrej Drbohlav Osamu Hasegawa Pascal Vasseur Patrice Delmas Pei Chen Peter Sturm Philippos Mordohai Pierre Jannin Ping Tan Prabir Kumar Biswas Prem Kalra Qiang Wang Qiao Yu Qingshan Liu QiuQi Ruan Radim Sara Rae-Hong Park Ralf Reulke Ralph Gross Reinhard Koch Rene Vidal Robert Pless Rogerio Feris Ron Kimmel Ruigang Yang Ryad Benosman Ryusuke Sagawa S.H. Srinivasan S. Kevin Zhou Seungjin Choi Sharat Chandran Sheng-Wen Shih Shihong Lao Shingo Kagami Shin’ichi Satoh Shinsaku Hiura ShiSguang Shan Shmuel Peleg Shoji Tominaga Shuicheng Yan Stan Birchfield Stefan Gehrig
Stephen Lin Stephen Maybank Subhashis Banerjee Subrata Rakshit Sumantra Dutta Roy Svetlana Lazebnik Takayuki Okatani Takekazu Kato Tat-Jen Cham Terence Sim Tetsuji Haga Theo Gevers Thomas Brox Thomas Leung Tian Fang Til Aach Tomas Svoboda Tomokazu Sato Toshio Sato Toshio Ueshiba Tyng-Luh Liu Vincent Lepetit Vivek Kwatra Vladimir Pavlovic Wee-Kheng Leow Wei Liu Weiming Hu Wen-Nung Lie Xianghua Ying Xianling Li Xiaogang Wang Xiaojuan Wu Yacoob Yaser Yaron Caspi Yasushi Sumi Yasutaka Furukawa Yasuyuki Sugaya Yeong-Ho Ha Yi-ping Hung Yong-Sheng Chen Yoshinori Kuno Yoshio Iwai Yoshitsugu Manabe Young Shik Moon Yunde Jia
IX
X
Organization
Zen Chen Zhifeng Li Zhigang Zhu
Zhouchen Lin Zhuowen Tu Zuzana Kukelova
Additional Reviewers Afshin Sepehri Alvina Goh Anthony Dick Avinash Ravichandran Baidya Saha Brian Clipp C´edric Demonceaux Christian Beder Christian Schmaltz Christian Wojek Chunhua Shen Chun-Wei Chen Claude P´egard D.H. Ye D.J. Kwon Daniel Hein David Fofi David Gallup De-Zheng Liu Dhruv K. Mahajan Dipti Mukherjee Edgar Seemann Edgardo Molina El Mustapha Mouaddib Emmanuel Prados Frank R. Schmidt Frederik Meysel Gao Yan Guy Rosman Gyuri Dorko H.J. Shim Hang Yu Hao Du Hao Tang Hao Zhang Hirishi Ohno Hiroshi Ohno Huang Wei Hynek Bakstein
Ilya Levner Imran Junejo Jan Woetzel Jian Chen Jianzhao Qin Jimmy Jiang Liu Jing Wu John Bastian Juergen Gall K.J. Lee Kalin Kolev Karel Zimmermann Ketut Fundana Koichi Kise Kongwah Wan Konrad Schindler Kooksang Moon Levi Valgaerts Li Guan Li Shen Liang Wang Lin Liang Lingyu Duan Maojun Yuan Mario Fritz Martin Bujnak Martin Matousek Martin Sunkel Martin Welk Micha Andriluka Michael Stark Minh-Son Dao Naoko Nitta Neeraj Kanhere Niels Overgaard Nikhil Rane Nikodem Majer Nilanjan Ray Nils Hasler
Nipun kwatra Olivier Morel Omar El Ganaoui Pankaj Kumar Parag Chaudhuri Paul Schnitzspan Pavel Kuksa Petr Doubek Philippos Mordohai Reiner Schnabel Rhys Hill Rizwan Chaudhry Rui Huang S.M. Shahed Nejhum S.H. Lee Sascha Bauer Shao-Wen Yang Shengshu Wang Shiro Kumano Shiv Vitaladevuni Shrinivas Pundlik Sio-Hoi Ieng Somnath Sengupta Sudipta Mukhopadhyay Takahiko Horiuchi Tao Wang Tat-Jun Chin Thomas Corpetti Thomas Schoenemann Thorsten Thormaehlen Weihong Li Weiwei Zhang Xiaoyi Yu Xinguo Yu Xinyu Huang Xuan Song Yi Feng Yichen Wei Yiqun Li
Organization
Yong MA Yoshihiko Kawai
Zhichao Chen Zhijie Wang
Sponsors Sponsor Technical Co-sponsors
Asian Federation of Computer Vision IPSJ SIG-CVIM IEICE TG-PRMU
XI
Table of Contents – Part I
Plenary and Invited Talks Less Is More: Coded Computational Photography . . . . . . . . . . . . . . . . . . . . Ramesh Raskar
1
Optimal Algorithms in Multiview Geometry . . . . . . . . . . . . . . . . . . . . . . . . . Richard Hartley and Fredrik Kahl
13
Machine Vision in Early Days: Japan’s Pioneering Contributions . . . . . . Masakazu Ejiri
35
Shape and Texture Coarse-to-Fine Statistical Shape Model by Bayesian Inference . . . . . . . . . Ran He, Stan Li, Zhen Lei, and ShengCai Liao
54
Efficient Texture Representation Using Multi-scale Regions . . . . . . . . . . . . Horst Wildenauer, Branislav Miˇcuˇs´ık, and Markus Vincze
65
Fitting Comparing Timoshenko Beam to Energy Beam for Fitting Noisy Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ili´c Slobodan A Family of Quadratic Snakes for Road Extraction . . . . . . . . . . . . . . . . . . . Ramesh Marikhu, Matthew N. Dailey, Stanislav Makhanov, and Kiyoshi Honda
75 85
Poster Session 1: Calibration Multiperspective Distortion Correction Using Collineations . . . . . . . . . . . . Yuanyuan Ding and Jingyi Yu
95
Camera Calibration from Silhouettes Under Incomplete Circular Motion with a Constant Interval Angle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Po-Hao Huang and Shang-Hong Lai
106
Mirror Localization for Catadioptric Imaging System by Observing Parallel Light Pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ryusuke Sagawa, Nobuya Aoki, and Yasushi Yagi
116
Calibrating Pan-Tilt Cameras with Telephoto Lenses . . . . . . . . . . . . . . . . . Xinyu Huang, Jizhou Gao, and Ruigang Yang
127
XIV
Table of Contents – Part I
Camera Calibration Using Principal-Axes Aligned Conics . . . . . . . . . . . . . Xianghua Ying and Hongbin Zha
138
Poster Session 1: Detection 3D Intrusion Detection System with Uncalibrated Multiple Cameras . . . . Satoshi Kawabata, Shinsaku Hiura, and Kosuke Sato Non-parametric Background and Shadow Modeling for Object Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tatsuya Tanaka, Atsushi Shimada, Daisaku Arita, and Rin-ichiro Taniguchi Road Sign Detection Using Eigen Color . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Luo-Wei Tsai, Yun-Jung Tseng, Jun-Wei Hsieh, Kuo-Chin Fan, and Jiun-Jie Li
149
159
169
Localized Content-Based Image Retrieval Using Semi-supervised Multiple Instance Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dan Zhang, Zhenwei Shi, Yangqiu Song, and Changshui Zhang
180
Object Detection Combining Recognition and Segmentation . . . . . . . . . . . Liming Wang, Jianbo Shi, Gang Song, and I-fan Shen
189
An Efficient Method for Text Detection in Video Based on Stroke Width Similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viet Cuong Dinh, Seong Soo Chun, Seungwook Cha, Hanjin Ryu, and Sanghoon Sull
200
Multiview Pedestrian Detection Based on Vector Boosting . . . . . . . . . . . . Cong Hou, Haizhou Ai, and Shihong Lao
210
Pedestrian Detection Using Global-Local Motion Patterns . . . . . . . . . . . . . Dhiraj Goel and Tsuhan Chen
220
Poster Session 1: Image and Video Processing Qualitative and Quantitative Behaviour of Geometrical PDEs in Image Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arjan Kuijper
230
Automated Billboard Insertion in Video . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hitesh Shah and Subhasis Chaudhuri
240
Improved Background Mixture Models for Video Surveillance Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chris Poppe, Ga¨etan Martens, Peter Lambert, and Rik Van de Walle
251
Table of Contents – Part I
XV
High Dynamic Range Scene Realization Using Two Complementary Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ming-Chian Sung, Te-Hsun Wang, and Jenn-Jier James Lien
261
Automated Removal of Partial Occlusion Blur . . . . . . . . . . . . . . . . . . . . . . . Scott McCloskey, Michael Langer, and Kaleem Siddiqi
271
Poster Session 1: Applications High Capacity Watermarking in Nonedge Texture Under Statistical Distortion Constraint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fan Zhang, Wenyu Liu, and Chunxiao Liu Attention Monitoring for Music Contents Based on Analysis of Signal-Behavior Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masatoshi Ohara, Akira Utsumi, Hirotake Yamazoe, Shinji Abe, and Noriaki Katayama View Planning for Cityscape Archiving and Visualization . . . . . . . . . . . . . Jiang Yu Zheng and Xiaolong Wang
282
292
303
Face and Gesture Synthesis of Exaggerative Caricature with Inter and Intra Correlations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chien-Chung Tseng and Jenn-Jier James Lien Pose-Invariant Facial Expression Recognition Using Variable-Intensity Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shiro Kumano, Kazuhiro Otsuka, Junji Yamato, Eisaku Maeda, and Yoichi Sato Gesture Recognition Under Small Sample Size . . . . . . . . . . . . . . . . . . . . . . . Tae-Kyun Kim and Roberto Cipolla
314
324
335
Tracking Motion Observability Analysis of the Simplified Color Correlogram for Visual Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Qi Zhao and Hai Tao
345
On-Line Ensemble SVM for Robust Object Tracking . . . . . . . . . . . . . . . . . Min Tian, Weiwei Zhang, and Fuqiang Liu
355
Multi-camera People Tracking by Collaborative Particle Filters and Principal Axis-Based Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wei Du and Justus Piater
365
XVI
Table of Contents – Part I
Poster Session 2: Camera Networks Finding Camera Overlap in Large Surveillance Networks . . . . . . . . . . . . . . Anton van den Hengel, Anthony Dick, Henry Detmold, Alex Cichowski, and Rhys Hill
375
Information Fusion for Multi-camera and Multi-body Structure and Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alexander Andreopoulos and John K. Tsotsos
385
Task Scheduling in Large Camera Networks . . . . . . . . . . . . . . . . . . . . . . . . . Ser-Nam Lim, Larry Davis, and Anurag Mittal
397
Poster Session 2: Face/Gesture/Action Detection and Recognition Constrained Optimization for Human Pose Estimation from Depth Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Youding Zhu and Kikuo Fujimura
408
Generative Estimation of 3D Human Pose Using Shape Contexts Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xu Zhao and Yuncai Liu
419
An Active Multi-camera Motion Capture for Face, Fingers and Whole Body . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eng Hui Loke and Masanobu Yamamoto
430
Tracking and Classifying Human Motions with Gaussian Process Annealed Particle Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Leonid Raskin, Michael Rudzsky, and Ehud Rivlin
442
Gait Identification Based on Multi-view Observations Using Omnidirectional Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kazushige Sugiura, Yasushi Makihara, and Yasushi Yagi
452
Gender Classification Based on Fusion of Multi-view Gait Sequences . . . . Guochang Huang and Yunhong Wang
462
Poster Session 2: Learning MAPACo-Training: A Novel Online Learning Algorithm of Behavior Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Heping Li, Zhanyi Hu, Yihong Wu, and Fuchao Wu
472
Optimal Learning High-Order Markov Random Fields Priors of Colour Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ke Zhang, Huidong Jin, Zhouyu Fu, and Nianjun Liu
482
Table of Contents – Part I
XVII
Hierarchical Learning of Dominant Constellations for Object Class Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nathan Mekuz and John K. Tsotsos
492
Multistrategical Approach in Visual Learning . . . . . . . . . . . . . . . . . . . . . . . . Hiroki Nomiya and Kuniaki Uehara
502
Poster Session 2: Motion and Tracking Cardiac Motion Estimation from Tagged MRI Using 3D-HARP and NURBS Volumetric Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jia Liang, Yuanquan Wang, and Yunde Jia
512
Fragments Based Parametric Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Prakash C., Balamanohar Paluri, Nalin Pradeep S., and Hitesh Shah
522
Spatiotemporal Oriented Energy Features for Visual Tracking . . . . . . . . . Kevin Cannons and Richard Wildes
532
Synchronized Ego-Motion Recovery of Two Face-to-Face Cameras . . . . . . Jinshi Cui, Yasushi Yagi, Hongbin Zha, Yasuhiro Mukaigawa, and Kazuaki Kondo
544
Optical Flow – Driven Motion Model with Automatic Variance Adjustment for Adaptive Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kazuhiko Kawamoto
555
A Noise-Insensitive Object Tracking Algorithm . . . . . . . . . . . . . . . . . . . . . . Chunsheng Hua, Qian Chen, Haiyuan Wu, and Toshikazu Wada
565
Discriminative Mean Shift Tracking with Auxiliary Particles . . . . . . . . . . . Junqiu Wang and Yasushi Yagi
576
Poster Session 2: Retrival and Search Efficient Search in Document Image Collections . . . . . . . . . . . . . . . . . . . . . . Anand Kumar, C.V. Jawahar, and R. Manmatha
586
Human Pose Estimation Hand Posture Estimation in Complex Backgrounds by Considering Mis-match of Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Akihiro Imai, Nobutaka Shimada, and Yoshiaki Shirai
596
Learning Generative Models for Monocular Body Pose Estimation . . . . . Tobias Jaeggli, Esther Koller-Meier, and Luc Van Gool
608
Human Pose Estimation from Volume Data and Topological Graph Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hidenori Tanaka, Atsushi Nakazawa, and Haruo Takemura
618
XVIII
Table of Contents – Part I
Matching Logical DP Matching for Detecting Similar Subsequence . . . . . . . . . . . . . . Seiichi Uchida, Akihiro Mori, Ryo Kurazume, Rin-ichiro Taniguchi, and Tsutomu Hasegawa
628
Efficient Normalized Cross Correlation Based on Adaptive Multilevel Successive Elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shou-Der Wei and Shang-Hong Lai
638
Exploiting Inter-frame Correlation for Fast Video to Reference Image Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arif Mahmood and Sohaib Khan
647
Poster Session 3: Face/Gesture/Action Detection and Recognition Flea, Do You Remember Me? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Grabner, Helmut Grabner, Joachim Pehserl, Petra Korica-Pehserl, and Horst Bischof
657
Multi-view Gymnastic Activity Recognition with Fused HMM . . . . . . . . . Ying Wang, Kaiqi Huang, and Tieniu Tan
667
Real-Time and Marker-Free 3D Motion Capture for Home Entertainment Oriented Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Brice Michoud, Erwan Guillou, Hector Brice˜ no, and Sa¨ıda Bouakaz
678
Tracking Iris Contour with a 3D Eye-Model for Gaze Estimation . . . . . . . Haiyuan Wu, Yosuke Kitagawa, Toshikazu Wada, Takekazu Kato, and Qian Chen
688
Eye Correction Using Correlation Information . . . . . . . . . . . . . . . . . . . . . . . Inho Choi and Daijin Kim
698
Eye-Gaze Detection from Monocular Camera Image Using Parametric Template Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ryo Ohtera, Takahiko Horiuchi, and Shoji Tominaga
708
An FPGA-Based Smart Camera for Gesture Recognition in HCI Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yu Shi and Timothy Tsui
718
Poster Session 3: Low Level Vision and Phtometory Color Constancy Via Convex Kernel Optimization . . . . . . . . . . . . . . . . . . . Xiaotong Yuan, Stan Z. Li, and Ran He
728
Table of Contents – Part I
XIX
User-Guided Shape from Shading to Reconstruct Fine Details from a Single Photograph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alexandre Meyer, Hector M. Brice˜ no, and Sa¨ıda Bouakaz
738
A Theoretical Approach to Construct Highly Discriminative Features with Application in AdaBoost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuxin Jin, Linmi Tao, Guangyou Xu, and Yuxin Peng
748
Robust Foreground Extraction Technique Using Gaussian Family Model and Multiple Thresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hansung Kim, Ryuuki Sakamoto, Itaru Kitahara, Tomoji Toriyama, and Kiyoshi Kogure Feature Management for Efficient Camera Tracking . . . . . . . . . . . . . . . . . . Harald Wuest, Alain Pagani, and Didier Stricker Measurement of Reflection Properties in Ancient Japanese Drawing Ukiyo-e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xin Yin, Kangying Cai, Yuki Takeda, Ryo Akama, and Hiromi T. Tanaka
758
769
779
Texture-Independent Feature-Point Matching (TIFM) from Motion Coherence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ping Li, Dirk Farin, Rene Klein Gunnewiek, and Peter H.N. de With
789
Where’s the Weet-Bix? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuhang Zhang, Lei Wang, Richard Hartley, and Hongdong Li
800
How Marginal Likelihood Inference Unifies Entropy, Correlation and SNR-Based Stopping in Nonlinear Diffusion Scale-Spaces . . . . . . . . . . . . . . Ram¯ unas Girdziuˇsas and Jorma Laaksonen
811
Poster Session 3: Motion and Tracking Kernel-Bayesian Framework for Object Tracking . . . . . . . . . . . . . . . . . . . . . Xiaoqin Zhang, Weiming Hu, Guan Luo, and Steve Maybank
821
Markov Random Field Modeled Level Sets Method for Object Tracking with Moving Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xue Zhou, Weiming Hu, Ying Chen, and Wei Hu
832
Continuously Tracking Objects Across Multiple Widely Separated Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yinghao Cai, Wei Chen, Kaiqi Huang, and Tieniu Tan
843
Adaptive Multiple Object Tracking Using Colour and Segmentation Cues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pankaj Kumar, Michael J. Brooks, and Anthony Dick
853
XX
Table of Contents – Part I
Image Assimilation for Motion Estimation of Atmospheric Layers with Shallow-Water Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ´ Nicolas Papadakis, Patrick H´eas, and Etienne M´emin
864
Probability Hypothesis Density Approach for Multi-camera Multi-object Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nam Trung Pham, Weimin Huang, and S.H. Ong
875
Human Detection AdaBoost Learning for Human Detection Based on Histograms of Oriented Gradients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chi-Chen Raxle Wang and Jenn-Jier James Lien
885
Multi-posture Human Detection in Video Frames by Motion Contour Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Qixiang Ye, Jianbin Jiao, and Hua Yu
896
A Cascade of Feed-Forward Classifiers for Fast Pedestrian Detection . . . . Yu-Ting Chen and Chu-Song Chen
905
Combined Object Detection and Segmentation by Using Space-Time Patches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yasuhiro Murai, Hironobu Fujiyoshi, and Takeo Kanade
915
Segmentation Embedding a Region Merging Prior in Level Set Vector-Valued Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ismail Ben Ayed and Amar Mitiche
925
A Basin Morphology Approach to Colour Image Segmentation by Region Merging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Erchan Aptoula and S´ebastien Lef`evre
935
Detecting and Segmenting Un-occluded Items by Actively Casting Shadows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tze K. Koh, Amit Agrawal, Ramesh Raskar, Steve Morgan, Nicholas Miles, and Barrie Hayes-Gill
945
A Local Probabilistic Prior-Based Active Contour Model for Brain MR Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jundong Liu, Charles Smith, and Hima Chebrolu
956
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
965
Less Is More: Coded Computational Photography Ramesh Raskar Mitsubishi Electric Research Labs (MERL), Cambridge, MA, USA
Abstract. Computational photography combines plentiful computing, digital sensors, modern optics, actuators, and smart lights to escape the limitations of traditional cameras, enables novel imaging applications and simplifies many computer vision tasks. However, a majority of current Computational Photography methods involve taking multiple sequential photos by changing scene parameters and fusing the photos to create a richer representation. The goal of Coded Computational Photography is to modify the optics, illumination or sensors at the time of capture so that the scene properties are encoded in a single (or a few) photographs. We describe several applications of coding exposure, aperture, illumination and sensing and describe emerging techniques to recover scene parameters from coded photographs.
1 Introduction Computational photography combines plentiful computing, digital sensors, modern optics, actuators, and smart lights to escape the limitations of traditional cameras, enables novel imaging applications and simplifies many computer vision tasks. Unbounded dynamic range, variable focus, resolution, and depth of field, hints about shape, reflectance, and lighting, and new interactive forms of photos that are partly snapshots and partly videos are just some of the new applications found in Computational Photography. In this paper, we discuss Coded Photography which involves encoding of the photographic signal and post-capture decoding for improved scene analysis. With filmlike photography, the captured image is a 2D projection of the scene. Due to limited capabilities of the camera, the recorded image is a partial representation of the view. Nevertheless, the captured image is ready for human consumption: what you see is what you almost get in the photo. In Coded Photography, the goal is to achieve a potentially richer representation of the scene during the encoding process. In some cases, Computational Photography reduces to ‘Epsilon Photography’, where the scene is recorded via multiple images, each captured by epsilon variation of the camera parameters. For example, successive images (or neighboring pixels) may have a different exposure, focus, aperture, view, illumination, or instant of capture. Each setting allows recording of partial information about the scene and the final image is reconstructed from these multiple observations. In Coded Computational Photography, the recorded image may appear distorted or random to a human observer. But the corresponding decoding recovers valuable information about the scene. ‘Less is more’ in Coded Photography. By blocking light over time or space, we can preserve more details about the scene in the recorded single photograph. In this paper we look at four specific examples. Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 1–12, 2007. © Springer-Verlag Berlin Heidelberg 2007
2
R. Raskar
(a) Coded Exposure: By blocking light in time, by fluttering the shutter open and closed in a carefully chosen binary sequence, we can preserve high spatial frequencies of fast moving objects to support high quality motion deblurring. (b) Coded Aperture Optical Heterodyning: By blocking light near the sensor with a sinusoidal grating mask, we can record 4D light field on a 2D sensor. And by blocking light with a mask at the aperture, we can extend the depth of field and achieve full resolution digital refocussing. (c) Coded Illumination: By observing blocked light at silhouettes, a multi-flash camera can locate depth discontinuities in challenging scenes without depth recovery. (d) Coded Sensing: By sensing intensities with lateral inhibition, a gradient sensing camera can record large as well as subtle changes in intensity to recover a highdynamic range image. We describe several applications of coding exposure, aperture, illumination and sensing and describe emerging techniques to recover scene parameters from coded photographs. 1.1 Film-Like Photography Photography is the process of making pictures by, literally, ‘drawing with light’ or recording the visually meaningful changes in the light leaving a scene. This goal was established for film photography about 150 years ago. Currently, ‘digital photography’ is electronically implemented film photography, refined and polished to achieve the goals of the classic film camera which were governed by chemistry, optics, mechanical shutters. Film-like photography presumes (and often requires) artful human judgment, intervention, and interpretation at every stage to choose viewpoint, framing, timing, lenses, film properties, lighting, developing, printing, display, search, index, and labelling. In this article we plan to explore a progression away from film and film-like methods to something more comprehensive that exploits plentiful low-cost computing and memory with sensors, optics, probes, smart lighting and communication. 1.2 What Is Computational Photography? Computational Photography (CP) is an emerging field, just getting started. We don’t know where it will end up, we can’t yet set its precise, complete definition, nor make a reliably comprehensive classification. But here is the scope of what researchers are currently exploring in this field. – Computational photography attempts to record a richer visual experience, captures information beyond just a simple set of pixels and makes the recorded scene representation far more machine readable. – It exploits computing, memory, interaction and communications to overcome long-standing limitations of photographic film and camera mechanics that have persisted in film-style digital photography, such as constraints on dynamic
Less Is More: Coded Computational Photography
3
range, depth of field, field of view, resolution and the extent of scene motion during exposure. – It enables new classes of recording the visual signal such as the ‘moment’ [Cohen 2005], shape boundaries for non-photorealistic depiction [Raskar et al 2004] , foreground versus background mattes, estimates of 3D structure, ‘relightable’ photos and interactive displays that permit users to change lighting, viewpoint, focus, and more, capturing some useful, meaningful fraction of the ‘light field’ of a scene, a 4-D set of viewing rays. – It enables synthesis of impossible photos that could not have been captured at a single instant with a single camera, such as wrap-around views (‘multiple-centerof-projection’ images [Rademacher and Bishop 1998]), fusion of time-lapsed events [Raskar et al 2004], the motion-microscope (motion magnification [Liu et al 2005]), video textures and panoramas [Agarwala et al 2005]. They also support seemly impossible camera movements such as the ‘bullet time’ (Matrix) sequence recorded with multiple cameras with staggered exposure times. – It encompass previously exotic forms of scientific imaging and data gathering techniques e.g. from astronomy, microscopy, and tomography. 1.3 Elements of Computational Photography Traditional film-like photography involves (a) a lens, (b) a 2D planar sensor and (c) a processor that converts sensed values into an image. In addition, the photography may involve (d) external illumination from point sources (e.g. flash units) and area sources (e.g. studio lights).
Computational Photography
Novel Illumination Light Sources
Novel Cameras
Modulators Generalized Optics
Generalized
Sensor
Processing Ray Reconstruction
Generalized
Optics
4D Incident Lighting
4D Ray Bender
Upto 4D Ray Sampler
4D Light Field Display
Recreate 4D Lightfield
Scene: 8D Ray Modulator
Fig. 1. Elements of Computational Photography
4
R. Raskar
Computational Photography generalizes these four elements. (a) Generalized Optics: Each optical element is treated as a 4D ray-bender that modifies a light field. The incident 4D light field for a given wavelength is transformed into a new 4D lightfield. The optics may involve more than one optical axis [Georgiev et al 2006]. In some cases the perspective foreshortening of objects based on distance may be modified using wavefront coded optics [Dowski and Cathey 1995]. In recent lensless imaging methods [Zomet and Nayar 2006] and coded-aperture imaging [Zand 1996] used for gamma-ray and X-ray astronomy, the traditional lens is missing entirely. In some cases optical elements such as mirrors [Nayar et al 2004] outside the camera adjust the linear combinations of ray bundles that reach the sensor pixel to adapt the sensor to the viewed scene. (b) Generalized Sensors: All light sensors measure some combined fraction of the 4D light field impinging on it, but traditional sensors capture only a 2D projection of this lightfield. Computational photography attempts to capture more; a 3D or 4D ray representation using planar, non-planar or even volumentric sensor assemblies. For example, a traditional out-of-focus 2D image is the result of a capture-time decision: each detector pixel gathers light from its own bundle of rays that do not converge on the focused object. But a Plenoptic Camera [Adelson and Wang 1992, Ren et al 2005] subdivides these bundles into separate measurements. Computing a weighted sum of rays that converge on the objects in the scene creates a digitally refocused image, and even permits multiple focusing distances within a single computed image. Generalizing sensors can extend their dynamic range [Tumblin et al 2005] and wavelength selectivity as well. While traditional sensors trade spatial resolution for color measurement (wavelengths) using a Bayer grid or red, green or blue filters on individual pixels, some modern sensor designs determine photon wavelength by sensor penetration, permitting several spectral estimates at a single pixel location [Foveon 2004]. (c) Generalized Reconstruction: Conversion of raw sensor outputs into picture values can be much more sophisticated. While existing digital cameras perform ‘demosaicking,’ (interpolate the Bayer grid), remove fixed-pattern noise, and hide ‘dead’ pixel sensors, recent work in computational photography can do more. Reconstruction might combine disparate measurements in novel ways by considering the camera intrinsic parameters used during capture. For example, the processing might construct a high dynamic range scene from multiple photographs from coaxial lenses, from sensed gradients [Tumblin et al 2005], or compute sharp images a fast moving object from a single image taken by a camera with a ‘fluttering’ shutter [Raskar et al 2006]. Closed-loop control during photography itself can also be extended, exploiting traditional cameras’ exposure control, image stabilizing, and focus, as new opportunities for modulating the scene’s optical signal for later decoding. (d) Computational Illumination: Photographic lighting has changed very little since the 1950’s: with digital video projectors, servos, and device-to-device communication, we have new opportunities to control the sources of light with as much sophistication as we use to control our digital sensors. What sorts of spatiotemporal modulations for light might better reveal the visually important contents
Less Is More: Coded Computational Photography
5
of a scene? Harold Edgerton showed high-speed strobes offered tremendous new appearance-capturing capabilities; how many new advantages can we realize by replacing ‘dumb’ the flash units, static spot lights and reflectors with actively controlled spatio-temporal modulators and optics? Already we can capture occluding edges with multiple flashes [Raskar 2004], exchange cameras and projectors by Helmholz reciprocity [Sen et al 2005], gather relightable actor’s performances with light stages [Wagner et al 2005] and see through muddy water with coded-mask illumination [Levoy et al 2004]. In every case, better lighting control during capture to builds richer representations of photographed scenes.
2 Sampling Dimensions of Imaging 2.1 Epsilon Photography for Optimizing Film-Like Camera Think of film cameras at their best as defining a ‘box’ in the multi-dimensional space of imaging parameters. The first, most obvious thing we can do to improve digital cameras is to expand this box in every conceivable dimension. This effort reduces Computational Photography to ‘Epsilon Photography’, where the scene is recorded via multiple images, each captured by epsilon variation of the camera parameters. For example, successive images (or neighboring pixels) may have different settings for parameters such as exposure, focus, aperture, view, illumination, or the instant of capture. Each setting allows recording of partial information about the scene and the final image is reconstructed from these multiple observations. Epsilon photography is thus concatenation of many such boxes in parameter space; multiple film-style photos computationally merged to make a more complete photo or scene description. While the merged photo is superior, each of the individual photos is still useful and comprehensible on its own, without any of the others. The merged photo contains the best features from all of them. (a) Field of View: A wide field of view panorama is achieved by stitching and mosaicking pictures taken by panning a camera around a common center of projection or by translating a camera over a near-planar scene. (b) Dynamic range: A high dynamic range image is captured by merging photos at a series of exposure values [Debevec and Malik 1997, Kang et al 2003] (c) Depth of field: All-in-focus image is reconstructed from images taken by successively changing the plane of focus [Agrawala et al 2005]. (d) Spatial Resolution: Higher resolution is achieved by tiling multiple cameras (and mosaicing individual images) [Wilburn et al 2005] or by jittering a single camera [Landolt et al 2001]. (e) Wavelength resolution: Traditional cameras sample only 3 basis colors. But multi-spectral (multiple colors in the visible spectrum) or hyper-spectral (wavelengths beyond the visible spectrum) imaging is accomplished by taking pictures while successively changing color filters in front of the camera, using tunable wavelength filters or using diffraction gratings. (f) Temporal resolution: High speed imaging is achieved by staggering the exposure time of multiple low-framerate cameras. The exposure durations of individual cameras can be non-overlapping ) [Wilburn et al 2005] or overlaping [Shechtman et al 2002].
6
R. Raskar
Taking multiple images under varying camera parameters can be achieved in several ways. The images can be taken with a single camera over time. The images can be captured simultaneously using ‘assorted pixels’ where each pixel is a tuned to a different value for a given parameter [Nayar and Narsimhan 2002]. Simultaneous capture of multiple samples can also be recorded using multiple cameras, each camera having different values for a given parameter. Two designs are currently being used for multi-camera solutions: a camera array [Wilburn et al 2005] and single-axis multiple parameter (co-axial) cameras [Mcguire et al 2005]. Coded Exposure
Temporal 11-D broadband code
Coded Aperture
Spatial 2-D broadband code
Fig. 2. Blocking light to achieve Coded Photography. (Left) Using a 1-D code in time to block and unblock light over time, a coded exposure photo can reversibly encode motion blur (Raskar et al 2006). (Right) Using a 2-D code in space to block parts of the light via a masked aperture, a coded aperture photo can reversibly encode defocus blur (Veeraraghavan et al 2007).
2.2 Coded Photography But there is much more beyond the ‘best possible film camera’. We can virtualize the notion of the camera itself if we consider it as a device that collects bundles of rays, each ray with its own wavelength spectrum and exposure duration. Coded Photography is a notion of an ‘out-of-the-box’ photographic method, in which individual (ray) samples or data sets may or may not be comprehensible as ‘images’ without further decoding, re-binning or reconstruction. Coded aperture techniques, inspired by work in astronomical imaging, try to preserve high spatial frequencies so that out of focus blurred images can be digitally re-focused [Veeraraghavan07]. By coding illumination, it is possible to decompose radiance in a scene into direct and global components [Nayar06]. Using a coded exposure technique, one can rapidly flutter open and close the shutter of a camera in a carefully chosen binary sequence, to capture a single photo. The fluttered shutter encoded the motion in the scene in the observed blur in a reversible way. Other examples include confocal images and techniques to recover glare in the images [Talvala07].
Less Is More: Coded Computational Photography
7
We may be converging on a new, much more capable ‘box’ of parameters in computational photography that we don’t yet recognize; there is still quite a bit of innovation to come! In the rest of the article, we survey recent techniques that exploit exposure, focus, active illumination and sensors. Coding in Time
Coding in Space
Coded Illumination
Coded Sensing
Exposure
Aperture
Inter-View
Gradient Sensor (Differential Encoding)
[Raskar et al 2006]
[Veeraraghavan et al 07]
[Raskar et al 2004]
[Tumblin et al 2005]
Mask, Optical Heterodyning
Intra-view
[Veeraraghavan et al 07]
[Nayar et al 2006]
Fig. 3. An overview of projects. Coding in time or space, coding the incident active illumination and coding the sensing pattern.
3 Coded Exposure In a conventional single-exposure photograph, moving objects or moving cameras cause motion blur. The exposure time defines a temporal box filter that smears the moving object across the image by convolution. This box filter destroys important high-frequency spatial details so that deblurring via deconvolution becomes an illposed problem. We have proposed to flutter the camera’s shutter open and closed during the chosen exposure time with a binary pseudo-random sequence, instead of leaving it open as in a traditional camera [Raskar et al 2006]. The flutter changes the box filter to a broad-band filter that preserves high-frequency spatial details in the blurred image and the corresponding deconvolution becomes a well-posed problem. Results on several challenging cases of motion-blur removal including outdoor scenes, extremely large motions, textured backgrounds and partial occluders were presented. However, the authors assume that PSF is given or is obtained by simple user interaction. Since changing the integration time of conventional CCD cameras is not feasible, an external ferro-electric shutter is placed in front of the lens to code the exposure. The shutter is driven opaque and transparent according to the binary signals generated from PIC using the pseudo-random binary sequence.
8
R. Raskar
Fig. 4. The flutter shutter camera. The coded exposure is achieved by fluttering the shutter open and closed. Instead of a mechanical movement of the shutter, we used a ferro-electric LCD in front of the lens. It is driven opaque and transparent according to the desired binary sequence.
4 Coded Aperture and Optical Heterodyning Can we capture additional information about a scene by inserting a patterned mask inside a conventional camera? We use a patterned attenuating mask to encode the light field entering the camera. Depending on where we put the mask, we can effect desired frequency domain modulation of the light field. If we put the mask near the lens aperture, we can achieve full resolution digital refocussing. If we put the mask near the sensor, we can recover a 4D light field without any additional lenslet array.
Fig. 5. Encoded Blur Camera, i.e. with mask in the aperture, can preserve high spatial images frequencies in the defocus blur. Notice the glint in the eye. In the misfocused photo, on the left, the bright spot appears blurred with the bokeh of the chosen aperture (shown in the inset). In the deblurred result, on the right, the details on the eye are correctly recovered.
Less Is More: Coded Computational Photography
9
Ren et al. have developed a camera that can capture the 4D light field incident on the image sensor in a single photographic exposure [Ren et al. 2005]. This is achieved by inserting a microlens array between the sensor and main lens, creating a plenoptic camera. Each microlens measures not just the total amount of light deposited at that location, but how much light arrives along each ray. By re-sorting the measured rays of light to where they would have terminated in slightly different, synthetic cameras, one can compute sharp photographs focused at different depths. A linear increase in the resolution of images under each microlens results in a linear increase in the sharpness of the refocused photographs. This property allows one to extend the depth of field of the camera without reducing the aperture, enabling shorter exposures and lower image noise. Our group has shown that it is also possible to create a plenoptic camera using a patterned mask instead of a lenslet array. The geometric configurations remains nearly identical [Veeraraghavan2007]. The method is known as ‘spatial optical heterodyning’. Instead of remapping of rays in 4D using microlens array so that they can be captured on a 2D sensor, spatial optical heterodyning remaps frequency components of the 4D lightfield so that the frequency components can be recovered from Fourier transform of the captured 2D image. In microlens array based design, each pixel effectively records light along a single ray bundle. With patterned masks, each pixel records a linear combination multiple ray-bundles. By carefully coding the linear combination, the coded heterodyning method can reconstruct the values of individual ray-bundles. This is reversible modulation of 4D light field by inserting a patterned planar mask in the optical path of a lens based camera. We can reconstruct the 4D light field from a 2D camera image. The patterned mask attenuates light rays inside the camera instead of bending them, and the attenuation recoverably encodes the ray on the 2D sensor. Our mask-equipped camera focuses just as a traditional camera might to capture conventional 2D photos at full sensor resolution, but the raw pixel values also hold a modulated 4D light field. The light field can be recovered by rearranging the tiles of the 2D Fourier transform of sensor values into 4D planes, and computing the inverse Fourier transform. Mask?
Mask
Sensor
Coded Aperture for Full Resolution Digital Refocusing
Sensor
Mask
Sensor
Heterodyne Light Field Camera
Fig. 6. Coding Light Field entering a camera via a mask
10
R. Raskar
5 Coded Illumination By observing blocked light at silhouettes, a multi-flash camera can locate depth discontinuities in challenging scenes without depth recovery. We used a multi-flash camera to find the silhouettes in a scene [Raskar et al 2004]. We take four photos of an object with four different light positions (above, below, left and right of the lens). We detect shadows cast along the depth discontinuities are use them to detect depth discontinuities in the scene. The detected silhouettes are then used for stylizing the photograph and highlighting important features. We also demonstrate silhouette detection in a video using a repeated fast sequence of flashes. Bottom Flash
Top Flash
Left Flash
Right Flash
Ratio images showing shadows and traversal to find edges
Photo
Shadow-Free
Depth Edges
Depth Edges
Fig. 7. Multi-flash Camera for Depth Edge Detection. (Left) A camera with four flashes. (Right) Photos due to individual flashes, highlighted shadows and epipolar traversal to compute the single pixel depth edges.
6 High Dynamic Range Using a Gradient Camera A camera sensor is limited in the range of highest and lowest intensities it can measure. To capture the high dynamic range, one can adaptively exposure the sensor so that the signal to noise ratio is high over the entire image, including in the the dark and brightly lit regions. One approach for faithfully recording the intensities in a high dynamic range scenes is to capture multiple images using different exposures, and then to merge these images. The basic idea is that when longer exposures are used, dark regions are well exposed but bright regions are saturated. On the other hand, when short exposures are used, dark regions are too dark but bright regions are well imaged. If exposure varies and multiple pictures are taken of the same scene, value of a pixel can be taken from those images where it’s neither too dark nor saturated. This type of approach is often referred to as exposure bracketing. At the sensor level, various approaches have also been proposed for high dynamic range imaging. One type of approach is to use multiple sensing elements with different sensitivities within each cell [Street 1998, Handy 1986, Wen 1989, Hamazaki 1996]. Multiple measurements are made from the sensing elements, and they are combined
Less Is More: Coded Computational Photography
11
on-chip before a high dynamic range image is read out from the chip. Spatial sampling rate is lowered in these sensing devices, and spatial resolution is sacrificed. Another type of approach is to adjust the well capacity of the sensing elements during photocurrent integration [Knight 1983, Sayag 1990, Decker 1998] but this gives higher noise. By sensing intensities with lateral inhibition, a gradient sensing camera can record large as well as subtle changes in intensity to recover a high-dynamic range image. By sensing different between neighboring pixels instead of actual intensities, our group has shown that a ‘Gradient Camera’ can record large global variations in intensity [Tumblin et al 2005]. Rather than measure absolute intensity values at each pixel, this proposed sensor measures only forward differences between them, which remain small even for extremely high-dynamic range scenes, and reconstructs the sensed image from these differences using Poisson solver methods. This approach offers several advantages: the sensor is nearly impossible to over- or under-expose, yet offers extremely fine quantization, even with very modest A/D convertors (e.g. 8 bits). The thermal and quantization noise occurs in the gradient domain, and appears as low frequency ‘cloudy’ noise in the reconstruction, rather than uncorrelated highfrequency noise that might obscure the exact position of scene edges.
7 Conclusion As these examples indicate, we have scarcely begun to explore the possibilities offered by combining computation, 4D modeling of light transport, and novel optical systems. Nor have such explorations been limited to photography and computer graphics or computer vision. Microscopy, tomography, astronomy and other optically driven fields already contain some ready-to-use solutions to borrow and extend. If the goal of photography is to capture, reproduce, and manipulate a meaningful visual experience, then the camera is not sufficient to capture even the most rudimentary birthday party. The human experience and our personal viewpoint is missing. Computational Photography can supply us with visual experiences, but can’t decide which one’s matter most to humans. Beyond coding the first order parameters like exposure, focus, illumination and sensing, maybe the ultimate goal of Computational Photography is to encode the human experience in the captured single photo.
Acknowledgements We wish to thank Jack Tumblin and Amit Agrawal for contributing several ideas for this paper. We also thank co-authors and collaborators Ashok Veeraraghavan, Ankit Mohan, Yuanzen Li, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew Turk. We thank Shree Nayar and Marc Levoy for useful comments and discussions.
References Raskar, R., Tan, K., Feris, R., Yu, J., Turk, M.: Non-photorealistic Camera: Depth Edge Detection and Stylized Rendering Using a Multi-Flash Camera. SIGGRAPH 2004 (2004) T umblin, J., Agrawal, A., Raskar, R.: Why I want a Gradient Camera. In: CVPR 2005, IEEE, Los Alamitos (2005)
12
R. Raskar
Raskar, R., Agrawal, A., Tumblin, J.: Coded exposure photography: motion deblurring using fluttered shutter. ACM Trans. Graph 25(3), 795–804 (2006) Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., Tumblin, J.: Dappled Photography: Mask-Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing. ACM Siggraph (2007) Nayar, S.K., Narasimhan, S.G.: Assorted Pixels: Multi-Sampled Imaging With Structural Models. In: ECCV. Europian Conference on Computer Vision, vol. IV, pp. 636–652 (2002) Debevec, Malik.: Recovering high dynamic range radiance maps from photographs. In: Proc. SIGGRAPH (1997) Mann, Picard.: Being ’undigital’ with digital cameras: Extending dynamic range by combining differently exposed pictures. In: Proc. IS&T 46th ann. conference (1995) McGuire, M., Matusik, Pfister, Hughes, Durand.: Defocus Video Matting, ACM Transactions on Graphics. Proceedings of ACM SIGGRAPH 2005 24(3) (2005) Adelson, E.H., Wang, J.Y.A.: Single Lens Stereo with a Plenoptic Camera. IEEE Transactions on Pattern Analysis and Machine Intelligence 14(2) (1992) Ng, R.: Fourier Slice Photography, SIGGRAPH (2005) Morimura. Imaging method for a wide dynamic range and an imaging device for a wide dynamic range. U.S. Patent 5455621 (October 1993) Levoy, M., Hanrahan, P.: Light field rendering. In: SIGGRAPH, pp. 31–42 (1996) Dowski Jr., E.R., Cathey, W.T.: Extended depth of field through wave-front coding. Applied Optics 34(11), 1859–1866 (1995) Georgiev, T., Zheng, C., Nayar, S., Salesin, D., Curless, B., Intwala, C.: Spatio-angular Resolution Trade-Offs in Integral Photography. In: proceedings, EGSR 2006 (2006)
Optimal Algorithms in Multiview Geometry Richard Hartley1 and Fredrik Kahl2 Research School of Information Sciences and Engineering, The Australian National University National ICT Australia (NICTA) Centre for Mathematical Sciences, Lund University, Sweden
1
2
Abstract. This is a survey paper summarizing recent research aimed at finding guaranteed optimal algorithms for solving problems in Multiview Geometry. Many of the traditional problems in Multiview Geometry now have optimal solutions in terms of minimizing residual imageplane error. Success has been achieved in minimizing L2 (least-squares) or L∞ (smallest maximum error) norm. The main methods involve Second Order Cone Programming, or quasi-convex optimization, and Branch-andbound. The paper gives an overview of the subject while avoiding as far as possible the mathematical details, which can be found in the original papers. J.E.Littlewood: The first test of potential in mathematics is whether you can get anything out of geometry. G.H.Hardy: The sort of mathematics that is useful to a superior engineer, or a moderate physicist has no esthetic value and is of no interest to the real mathematician.
1
Introduction
In this paper, we describe recent work in geometric Computer Vision aimed at finding provably optimal solutions to some of the main problems. This is a subject which the two authors of this paper have been involved in for the last few years, and we offer our personal view of the subject. We cite most of the relevant papers that we are aware of and apologize for any omissions. There remain still several open problems, and it is our hope that more researchers will be encouraged to work in this area. Research in Structure from Motion since the start of the 1990s resulted in the emergence of a dominant accepted technique – bundle adjustment [46]. In this method, a geometric problem is formulated as a (usually) non-linear optimization problem, which is then solved using an iterative optimization algorithm.
NICTA is funded by the Australian Government’s Backing Australia’s Ability initiative, in part through the Australian Research Council.
Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 13–34, 2007. c Springer-Verlag Berlin Heidelberg 2007
14
R. Hartley and F. Kahl
Generally, the bundle adjustment problem is formulated as follows. One defines a cost function (also called an objective function) in terms of a set of parameters. Solving the problem involves finding the set of parameters that minimize the cost. Generally, the parameters are associated with the geometry that we wish to discover. Often they involve parameters of a set of cameras, as well as parameters (such as 3D point coordinates) describing the geometry of the scene. The cost function usually involves image measurements, and measures how closely a given geometric configuration (for instance the scene geometry) explains the image measurements. Bundle adjustment has many advantages, which account for its success and popularity. 1. It is quite general, and can be applied to a large range of problems. 2. It is very easy to augment the cost function to include other constraints that the problem must satisfy. 3. We can “robustify” the cost function by minimizing a robust cost function, such as the Huber cost function. 4. Typically, the estimation problem is sparse, so sparse techniques can be used to achieve quite fast run times. One of the issues with bundle adjustment is that it requires a relatively accurate initial estimate of the geometry in order to converge to a correct minimum. This requirement led to one of the other main themes of research in Multiview Geometry through the 1990s - finding reliable initial solutions usually through so-called algebraic techniques. Most well known among such techniques is the 8point algorithm [28] for estimation of the essential or fundamental matrix, which solves the two-view relative orientation problem. Generally, in such methods, one defines an algebraic condition that must hold in the case of a noise-free solution, and defines an algebraic cost function that expresses how far this condition is from being met. Unfortunately, the cost function often is not closely connected with the geometry, and the cost function may be meaningless in any geometric or statistical sense. Multiple minima. One of the drawbacks of bundle adjustment is the possibility of converging to a local, rather than a global minimum of the cost function. The cost functions that arise in multiview optimization problems commonly do have multiple local minima. As an example, in Fig 1 we show graphs of the cost functions associated with the two-view triangulation problem (described later) and the autocalibration problem. The triangulation problem with more than two views is substantially more complex. It has been shown in [43] that the triangulation problem with three views involves solving a polynomial of degree 47, and hence the cost function potentially may have up to 24 minima. For higher numbers of views, the degree of the polynomial grows cubically. Stew´enius and Nist´er have calculated the sequence of degrees of the polynomial to be 6, 47, 148, 336, 638, 1081 for 2 to 7 view triangulation, which implies a large number of potential local minima of the cost function.
Optimal Algorithms in Multiview Geometry
15
Fig. 1. Top: Two-view triangulation cost function, showing up to three minima. The independent variable (x-axis) parametrizes the epipolar plane. On the left, an example with three local minima (two of them with equal cost). On the right, an example with global solution with zero cost (perfect noise-free triangulation), yet having a further local minimum. The graphs are taken from [12]. Bottom: 2-dimensional cross-section of a cost associated with an autocalibration problem. The graphs are taken from [9]. Left to right: top view, side-view, contour plot. Note the great complexity of the cost function, and the expected difficulties in finding a global minimum.
As for autocalibration, progress has been made on this problem in [5], which finds optimal methods of carrying out the projective - affine - Euclidean upgrade, the “stratified” approach to autocalibration. However, this is not quite the same as an optimal solution to autocalibration, which remains an open (very difficult) problem. Optimal methods. The difficulties and uncertainties associated with bundle adjustment, and algebraic methods has led to a theme of research in the last few years that aims at finding guaranteed provably optimal methods for solving these problems. Although such methods have not been successful for all geometric problem, the number of problems that can be solved using such optimal methods continues to grow. It is the purpose of this paper to survey progress in this area.
2
What Is Meant by an Optimal Solution?
In this section, we will argue that what is meant by an optimal solution to a geometric problem is not clearly defined. We consider a problem in which a set of measurements are made and we need to fit a parametrized model of some kind to these measurements. Optimization involves finding the set of parameters of the model that best fit the purpose. The optimization problem is defined in terms of a defined cost function which must be minimized over the set of
16
R. Hartley and F. Kahl
meaningful parameters. However, what particular cost functions merit being called “optimal” will be considered in the rest of this section. To set up the problem, consider a set of measurements, xi to which we wish to ˆ i which are fit a parametrized model. The model gives rise to some model values x defined in some way in terms of the parametrization. The set of residuals δ i are defined as δi = xi −ˆ xi where · represents a suitable norm in the measurement space. For image measurements, this is most reasonably the distance in the image ˆ i . Finally, denote by Δ the vector with components δi . This between xi and x may be called the vector of residuals. 2.1
L2 Error and ML Estimation
Often, it is argued that the optimal solution is the least squares solution, which minimizes the cost function ˆ i 2 , xi − x Δ 2 = i
namely the L2 -norm of the vector of residuals.1 The argument for the optimality of this solution is as follows. If we assume that the measurements are derived from actual values, corrupted with Gaussian noise with variance σ 2 , then the ˆ i is given by probability of the set of measurements xi given true values x ˆ i 2 /(2σ 2 ) . P ({xi } | {ˆ xi }) = K exp −xi − x i
where K is a normalizing constant. Taking logarithms and minimizing, we see that the modelled data that2 maximizes the probability of the measurements ˆ i . Thus, the least-squares solution is the maximialso minimizes i xi − x mum likelihood (ML) estimate, under an assumption of Gaussian noise in the measurements. This is the argument for optimality of the least-squares solution. Although useful, this argument does rely on two assumptions that are open to question. 1. It makes an assumption of Gaussian noise. This assumption is not really justified at all. Measurement errors in a digital image are not likely to satisfy a Gaussian noise model. 2. Maximum likelihood is not necessarily the same as optimal, a term that we have not defined. One might define optimal to mean maximum likelihood, but this is a circular argument. 2.2
L∞ Error
An alternative noise model, perhaps equally justified for image point measurements is that of uniform bounded noise. Thus, we assume that all measurements 1
In future the symbol · represents the 2-norm of a vector.
Optimal Algorithms in Multiview Geometry
17
less than a given threshold distance from the true value are equally likely, but measurements beyond this threshold have probability zero. In the case of a discrete image, measurements more accurate than one pixel from the true value are difficult to obtain, though in fact they may be achieved in carefully controlled situations. If we assume that the measurement error probability model is ˆ ) = K exp (−(x − x ˆ /σ)p ) P (x | x
(1)
where K is a normalizing factor, then as before, the ML estimate is the one ˆ i p . Letting p increase to infinity, the probability that minimizes i xi − x distribution (1) converges uniformly (except at σ) to a uniform distribution for ˆ . Now, taking the p-th root, we see that minimizing (1) is x within distance σ of x equivalent to normalizing the p-norm of the vector Δ of residuals. Furthermore, as p increases to infinity, Δp converges to the L∞ norm Δ∞ . In this way, minimizing the L∞ error corresponds to an assumption of uniform distribution of measurement errors. Looked at another way, under the L∞ norm, all sets of measurements that are within a distance σ of the modelled values are equally likely, whereas a set of measurements where one of the values exceeds the threshold σ has probability zero. Then L∞ optimization finds the smallest noise threshold for which the set of measurements is possible, and determines the ML estimate for this minimum threshold. Note that the L∞ norm of a vector is simply the largest component of the vector, in absolute value. Thus, ˆi , min Δ∞ = min max xi − x i
where the minimization is taken over the parameters of the model. For this reason, L∞ minimization is sometimes referred to as minimax optimization. 2.3
Other Criteria for Optimality
It is not clear that the maximum likelihood estimate has a reasonable claim to being the optimal estimate. It is pointed out in [13] that the maximum likelihood estimate may be biased, and in fact have infinite bias even for quite simple estimation problems. Thus, as the noise level of measurements increases, the average (or expected) estimate drifts away from the true value. This is of course undesirable. In addition, a different way of thinking of the estimation problem is in terms of the risk of making a wrong estimate. For example consider the triangulation problem (discussed in more detail later) in which several observers estimate a bearing (direction vector) to a target from known observation points. If the bearing directions are noisy, then where is the target? In many cases, there is a cost associated with wrongly estimating the position of the target. (For instance, if the target is an incoming ballistic missile, the
18
R. Hartley and F. Kahl
cost of a wrong estimate can be quite high.) A reasonable procedure would be to choose an estimate that minimizes the expected cost. As an example, if the cost of an estimate is equal to the square of the distance between the estimate and the true value, then the expected cost is equal to the mean of the posterior probability distribution of the parameters, P (θ | {xi })2 More discussion of these matters is contained in [13], Appendix 3. We are not, however, aware of any literature in multiview geometry finding estimates of this kind. What we mean by optimality. In this survey, we will consider the estimates that minimize the L2 or L∞ norms of the residual error vector, with respect to a parametrized model as being the optimal. This is reasonable in that it is related in either case to a specific geometric noise model, tied directly to the statistics of the measurements.
3
Polynomial Methods
One approch for obtaining optimal solutions to multiview problems is to compute all stationary points of the cost function and then check which of these is the global minimum. From a theoretical point of view, any structure and motion problem can be solved in this manner as long as the cost function can be expressed as a rational polynomial function in the parameters. This will be the case for most cost functions encountered (though not for L∞ cost functions, which are not differentiable). The method is as follows. A minimum of the cost function must occur at a point where the derivatives of the cost with respect to the parameters vanish. If the cost function is a rational polynomial function, then the derivatives are rational as well. Setting the derivatives to zero leads to a system of polynomial equations, and the solutions of this set of equations define the stationary points of the cost function. These can be checked one-by-one to find the minimum. This method may also be applied when the parameters satisfy certain constraints, such as a constraint of zero determinant for the fundamental matrix, or a constraint that a matrix represents a rotation. Such problems can be solved using Lagrange multipliers. Although this method is theoretically generally applicable, in practice it is only tractable for small problems, for example the triangulation problem. A solution to the two-view triangulation problem was given in [12], involving the solution of a polynomial of degree 6. The three-view problem was addressed in [43]; the solution involves the solution of a polynomial of degree 47. Further work (unpublished) by Stew´enius and Nist´er has shown that the triangulation problem for 4 to 7 views can be solved by finding the roots of a polynomial of degree 148, 336, 638, 1081 respectively, and in general, the degree grows cubically. Since 2
This assumes that the parameters θ are in a Euclidean space, which may not always be the case. In addition, estimation of the posterior probability distribution may require the definition of a prior P (θ).
Optimal Algorithms in Multiview Geometry
19
solving large sets of polynomials is numerically difficult, the issue of accuracy has been addressed in [4]. The polynomial method may also be applied successfully in many minimalconfiguration problems. We do not attempt here to enumerate all such problems considered in the literature. One notable example, however, is the relative orientation (two-view reconstruction) problem with 5 point correspondences. This has long been known to have 10 solutions [7,16].3 The second of these references gives a very pleasing derivation of this result. Recent simple algorithms for solving this problem by finding the roots of a polynomial have been given in [31,26]. Methods for computing a polynomial solution need not result in a polynomial of the smallest possible degree. However recent work using Galois theory [32] gives a way to address this question, showing that the 2-view triangulation and the relative orientation problem essentially require solution of polynomials of degree 6 and 10 respectively.
4
L∞ Estimation and SOCP
In this section, we will consider L∞ optimization, and discuss its advantages vis-a-vis L2 optimization. We show that there are many problems that can be formulated in L∞ and give a single solution. This is the main advantage, and contrasts with L2 optimization, which may have many local minima, as was shown in Fig 1. 4.1
Convex Optimization
We start by considering convex optimization problems. First, a few definitions. Convex set. A subset S of Rn is said to be convex if the line segment joining any two point in S is contained in S. Formally, if x0 , x1 ∈ S, then (1−α)x0 +αx1 ∈ S for all α with 0 ≤ α ≤ 1. Convex function. A function f : Rn → R is convex if its domain is a convex set and for all x, y ∈ domain(f ), and α with 0 ≤ α ≤ 1, we have f ((1 − α)x0 + αx1 ) ≤ (1 − α)f (x0 ) + αf (x1 ). Another less formal way of defining a convex function is so say that a line joining two points on the graph of the function will always lie above the function. This is illustrated in Fig 2. 3
These papers find 20 solutions, not 10, since they are solving for the number of possible rotations. There are 10 possible essential matrices each of which gives two possible rotations, related via the twisted pair ambiguity (see [13]). Only one of these two rotations is cheirally correct, corresponding to a possible realizable solution.
20
R. Hartley and F. Kahl
Fig. 2. Left. Examples of convex and non-convex sets. Middle. The definition of a convex function; the line joining two points lies above the graph of the function. Right. Convex optimization.
Convex optimization. A convex optimization problem is as follows: – Given a convex function f : D → R, defined on a convex domain D ⊂ Rn , find the minimum of f on D. A convex function is always continuous, and given reasonable conditions that ensure a minimum of the function (for instance D is compact) such a convex optimization problem is solvable by known algorithms4 . A further desirable property of a convex problem is that it has no local minima apart from the global minimum. The global minimum value is attained at a single point, or at least on a convex set in Rn where the function takes the same minimum value at all points. For further details, we refer the reader to the book [3]. Quasi-convex functions. Unfortunately, although convex problems are agreeable, they do not come up often in multiview geometry. However, interestingly enough, certain other problems do, quasi-convex problems. Quasi-convex functions are defined in terms of sublevel sets as follows. Definition 1. A function f : D → R is called quasi-convex if its α-sublevel set, Sα = {x ∈ D | f (x) ≤ α} is convex for all α. Examples of quasi-convex and non-quasi-convex functions are shown in Fig 3. Quasi-convex functions have two important properties. 1. A quasi-convex function has no local minima apart from the global minimum. It will attain its global minimum at a single point or else on a convex set where it assumes a constant value. 4
The most efficient algorithm to use will depend on the form of the function f , and the way the domain D is specified.
Optimal Algorithms in Multiview Geometry
21
Fig. 3. Quasi-convex functions. The left two functions are quasi-convex. All the sublevel sets are convex. The function on the right is not quasi-convex. The indentation in the function graph (on the left) means that the sublevel-sets are not convex. All convex functions are quasi-convex, but the example on the left shows that the converse is not true.
2. The pointwise maximum of a set of quasi-convex functions is quasi-convex. This is illustrated in Fig 4 for the case of functions of a single variable. The general case follows directly from the following observation concerning sublevel sets. Sδ (fi ) Sδ (max fi ) = i
i
which is convex, since each Sδ (fi ) is convex.
Fig. 4. The pointwise maximum of a set of quasi-convex functions is quasi-convex
A quasi-convex optimization problem is defined in the same way as a convex optimization problem, except that the function to be minimized is quasi-convex. Nevertheless, quasi-convex optimization problems share many of the pleasant properties of convex optimization. Why consider quasi-convex functions? The primary reason for considering such functions is that the residual of a measured image point x with respect to the projection of a point X in space is a quasi-convex function. In other words, f (X) = d(x, PX) is a quasi-convex function of X. Here, PX is the projection of a point X into an image, and d(·, ·) represents distance in the image.
22
R. Hartley and F. Kahl
Fig. 5. The triangulation problem: Assuming that the maximum reprojection error is less than some value δ, the sought point X must lie in the intersection of a set of cones. If δ is set too small, then the cones do not have a common intersection (left). If δ is set too large, then the cones intersect in a convex region in space, and the desired solution X must lie in this region (right). The optimal value of δ lies between these two extremes, and can be found by a binary search (bisection) testing successive values of δ. For more details, refer to the text.
Specifically, the sublevel set Sδ (f (X)) is a convex set, namely a cone with vertex the centre of projection, as will be discussed in more detail shortly (see Fig 5). As the reader may easily verify by example, however, the sum of quasi-convex functions is not in general quasi-convex. If we take several image measurements then the sum of squares of the residuals will not in general be a quasi-convex 2 function. In other words, an L2 cost function of the form N i=1 fi (X) will not in general be a quasi-convex function of X, nor have a single minimum. On the other hand, as remarked above, the maximum of a set of quasiconvex functions is quasi-convex, and hence will have a single minimum. Specifically, maxi fi (X) will be quasi-convex, and have a single minimum with respect to X. For this reason, it is typically easier to solve the minimax problem minX maxi fi (X) than the corresponding least-squares (L2 ) problem. Example: The triangulation problem. The triangulation problem is the most simple problem in multiview geometry. Nevertheless, in the L2 formulation, it still suffers from the problem of local minima, as shown in Fig 1. In this problem, we have a set of known camera centres Oi and a set of direction vectors vi which give the direction of a target point X from each of the camera centres. Thus, nominally, vi = (X − Oi )/X − Oi . The problem is to find the position of the point X. We choose to solve this problem in L∞ norm, in other words, we seek the point X that minimizes the maximum error (over all i) between vi and the directions given by X − Oi . Consider Fig 5. Some simple notation is required. Define ∠(X − Oi , vi ) to be the angle between the vectors X − Oi and vi . Given a value δ > 0, the set of points Cδ (Oi , vi ) = {X | ∠(X − Oi , vi ) ≤ δ} forms a cone in R3 with vertex Oi , axis vi and angle determined by δ.
Optimal Algorithms in Multiview Geometry
23
We begin by hypothesizing that there exists a solution X to the triangulation problem for which the maximum error is δ. In this case, the point X must lie inside cones in R3 with vertex Oi , axis vi and angle determined by δ. If the cones are too narrow, they do not have a common intersection and there can be no solution with maximum error less than δ. On the other hand, if δ is sufficiently large, then the cones intersect, and the desired solution X must lie in the intersection of the cones. The optimal value of δ is found by a binary search over values of δ to find the smallest value such that the cones Cδ (Oi , vi ) intersect in at least one point. The intersection will be a single point, or in special configurations, a segment of a line. The problem of determining whether a set of cones have non-empty intersection is solved by a convex optimization technique called Second Order Cone Programming (SOCP), for which open source libraries exist [44]. We make certain observations about this problem: 1. Each cone Cδ (Oi , vi ) is a convex set, and hence their intersection is convex. 2. If we define a cost function Cost∞ (X) = max ∠(X − Oi , vi ), i
then the sublevel set Sδ (Cost∞ ) is simply the intersection of the cones Cδ (Oi , vi ), which is convex for all δ. This by definition says that Cost∞ (X) is a quasi-convex function of X. 3. Finding the optimum min Cost∞ (X) = min max ∠(X − Oi , vi ) X
X
i
is accomplished by a binary search over possible values of δ, where for each value of δ we solve an SOCP feasibility problem, (determine whether a set of cones have a common intersection). Such a problem is known as a minimax or L∞ optimization problem. Generally speaking, this procedure generalizes to arbitrary quasi-convex optimization problems; they may be solved by binary search involving a convex feasibility problem at each step. If we have a set of individual cost functions fi (X), perhaps each associated with a single measurement, and each of them quasi-convex, then the maximum of these cost functions maxi fi (X) is also quasiconvex, as illustrated in Fig 4. In this case, the minimax problem of finding minX maxi fi (X) is solvable by binary search. Reconstruction with known rotations. Another problem that may be solved by very similar means to the triangulation problem is that of Structure and Motion with known rotations, which is illustrated in Fig 6. The role of cheirality. It is important in solving problems of this kind to take into account the concept of “cheirality”, which means the requirement that
24
R. Hartley and F. Kahl
Fig. 6. Structure and motion with known rotations. If the orientations of several cameras are all known, then image points correspond to direction vectors in a common coordinate frame. Here, blue (or grey) circles represent the positions of cameras, and black circles the positions of points. The arrows represent direction vectors (their length is not known) from cameras to points. The positions of all the points and cameras may be computed (up to scale and translation) using SOCP. Abstractly, this is the embedding problem for a bi-partite graph in 3D, where the orientation of the edges of the graph is known. The analogous problem for an arbitrary (not bi-partite) graph was applied in [41] to solve for motion of the cameras without computing the point positions.
points visible in an image must lie in front of the camera, not behind. If we subdivide space by planes separating front and back of the camera, then there will be at least one local minimum of the cost function (whether L∞ or L2 ) in each region of space. Since the number of regions separated by n planes grows cubically, so do the number of local minima, unless we constrain the solution so that the points lie in front of the cameras. Algorithm acceleration. Although the bisection algorithm using SOCP has been the standard approach to L∞ geometric optimization problems, there has been recent work for speeding up the computations [39,40,34]. However, it has been shown that the general structure and motion (with missing data) is N P -hard no matter what criterion of optimality of reprojection error is used [33]. 4.2
Problems Solved in L∞ Norm
The list of problems that can be solved globally with L∞ estimation continues to grow and by now it is quite long, see Table 1. In [21], an L∞ estimation algorithm serves as the basis for solving the leastmedian squares problem for various geometric problems. However, the extension to such problems is essentially based on heuristics and it has no guarantee of finding the global optimum.
Optimal Algorithms in Multiview Geometry
25
Table 1. Geometric reconstruction problems that can be solved globally with the L∞ or L2 norm L∞ -norm − Multiview triangulation − Camera resectioning (uncalibrated case) − Camera pose (calibrated case) − Homography estimation − Structure and motion recovery with known camera orientation − Reconstruction by using a reference plane − Camera motion recovery − Outlier detection − Reconstruction with covariance-based uncertainty − Two-view relative orientation − 1D retinal vision L2 -norm − Affine reconstruction from affine cameras − Multiview triangulation − Camera resectioning (uncalibrated case) − Homography estimation − 3D – 3D registration and matching − 3D – 3D registration and matching (unknown pairing)
5
References [11,18,20,6] [18,20] [10,47] [18,20] [11,18,20] [18] [41] [42,20,25,47] [41,21] [10] [2] References [23,45] [1,29] [1] [1] [14] [27]
Branch-and-Bound Theory
The method based on L∞ optimization is not applicable to all problems. In this section, we will describe in general terms a different method that has been used with success in obtaining globally optimal solutions. Branch and bound algorithms are non-heuristic methods for global optimization in non-convex problems. They maintain a provable upper and/or lower bound on the (globally) optimal objective value and terminate with a certificate proving that the solution is within of the global optimum, for arbitrarily small . Consider a non-convex, scalar-valued objective function Φ(x), for which we seek a global minimum over a domain Q0 . For a subdomain Q ⊆ Q0 , let Φmin (Q) denote the minimum value of the function Φ over Q. Also, let Φlb (Q) be a function that computes a lower bound for Φmin (Q), that is, Φlb (Q) ≤ Φmin (Q). An intuitive technique to determine the solution would be to divide the whole search region Q0 into a grid with cells of sides δ and compute the minimum of a lower bounding function Φlb defined over each grid cell, with the presumption that each Φlb (Q) is easier to compute than the corresponding Φmin (Q). However, the number of such grid cells increases rapidly as δ → 0, so a clever procedure must be deployed to create as few cells as possible and “prune” away as many of these grid cells as possible (without having to compute the lower bounding function for these cells). Branch and bound algorithms iteratively subdivide the domain into subregions (which we refer to as rectangles) and employ clever strategies to prune away as many rectangles as possible to restrict the search region.
26
R. Hartley and F. Kahl
Φ(x)
x
l
(a)
u
Φ(x)
Φ(x)
Φ(x)
q1∗
q1∗
q1∗ q2∗
x
l
u
(b)
x
l
u
(c)
x
l
u
(d)
Fig. 7. This figure illustrates the operation of a branch and bound algorithm on a one dimensional non-convex minimization problem. Figure (a) shows the the function Φ(x) and the interval l ≤ x ≤ u in which it is to be minimized. Figure (b) shows the convex relaxation of Φ(x) (indicated in yellow/dashed), its domain (indicated in blue/shaded) and the point for which it attains a minimum value. q1∗ is the corresponding value of the function Φ. This value is the current best estimate of the minimum of Φ(x), and is used to reject the left subinterval in Figure (c) because the minimum value of the convex relaxation is higher than q1∗ . Figure (d) shows the lower bounding operation on the right sub-interval in which a new estimate q2∗ of the minimum value of Φ(x) is found.
A graphical illustration of the algorithm is presented in Fig 7. Computation of the lower bounding functions is referred to as bounding, while the procedure that chooses a domain and subdivides it is called branching. The choice of the domain picked for refinement in the branching step and the actual subdivision itself are essentially heuristic. Although guaranteed to find the global optimum (or a point arbitrarily close to it), the worst case complexity of a branch and bound algorithm is exponential. However, in many cases the properties offered by multiview problems lead to fast convergence rates in practice.
6
Branch-and-Bound for L2 Minimization
The branch-and-bound method can be applied to find L2 norm solutions to certain simple problems. This is done by a direct application of branch-andbound over the parameter space. Up to now, methods used for branching have been quite simple, consisting of simple subdivision of rectangles in half, or in half along all dimensions. In order to converge as quickly as possible to the solution, it is useful to confine the region of parameter space that needs to be searched. This can be conveniently done if the cost function is a sum of quasi-convex functions. For instance, suppose the cost is C2 (X) = i fi (X)2 , and the optimal value is denoted by Xopt . If a good initial estimate X0 is available, with C2 (X0 ) = δ 2 , then C2 (Xopt ) = fi (Xopt )2 ≤ C2 (X0 ) = δ 2 . i
This implies that each fi (Xopt ) ≤ δ, and so Xopt ∈ i Sδ (fi ), which is a convex set enclosing both X0 and Xopt . One can find a rectangle in parameter space that
Optimal Algorithms in Multiview Geometry
27
encloses this convex region, and begin the branch-and-bound algorithm starting with this rectangle. This general method was used in [1] to solve the L2 multiview triangulation problem and the uncalibrated camera resection (pose) problem. In that paper fractional programming (described later in section 8.2) was used to define the convex sub-envelope of the cost function, and hence provide a cost lower bound on each rectangle. The same branch-and-bound idea, but with a different bounding method, was described in [29] to provide a simpler solution to the triangulation problem. The triangulation problem for other geometric features, more specifically lines and conics, was solved in [17] using the same approach.
7 7.1
Branch-and-Bound in Rotation Space Essential Matrix
All the problems that we have so-far considered, solvable using quasi-convex optimization or SOCP, involved no rotations. If there are rotations, then the optimization problem is no longer quasi-convex. An example of this type of problem is estimation of the essential matrix from a set of matching points xi ↔ xi . A linear solution to this problem was given in 1981 by Longuet-Higgins [28]. From the Essential matrix E, that satisfies the defining equation xi Exi = 0 for all i, we can extract the relative orientation (rotation and translation) of the two cameras. To understand why this is not a quasi-convex optimization problem, we look at the minimal problem involving 5 point correspondences. It is well know that with 5 points, there are 10 solutions for the relative orientation. (Recent algorithms for solving the 5-point orientation are given in [31,26]). However, if there are many possible discrete solutions, then the problem can not be quasi-convex or convex – such problems have a unique solution. This is so whether we are seeking an L∞ or L2 solution, since the 5-point problem has an exact solution, and hence the cost is zero in either norm. Many algorithms for estimating the essential matrix have been given, without however any claim to optimality. Recently, an optimal solution, at least in L∞ norm was given in [10]. To solve the essential matrix problem optimally (at least in L∞ norm) we make the following observation. If the rotation of the two cameras were somehow known, then the problem would reduce to the one discussed in Fig 6, where the translation of the cameras can be estimated optimally (in L∞ norm) given the rotations. The residual cost of this optimal solution may be found, as a function of the assumed rotation. To solve for the relative pose (rotation and translation) of the two cameras, we merely have to consider all possible rotations, and select the one that yields the smallest residual. The trick is to do this without having to look at an infinite number of rotations. Fortunately, branch-and-bound provides a means of carrying out this search. A key to the success of branch-and-bound is that the optimal cost (residual) estimated for one value of the rotation constrains the optimal cost for “nearby”
28
R. Hartley and F. Kahl
rotations. This allows us to put a lower bound on the optimal cost associated with all rotations in a region of rotation space. The branch-and-bound algorithm carries out a search of rotation space (a 3-dimensional space). The translation is not included in the branch-and-bound search. Instead, for a given rotation, the optimal translation may be computed using SOCP, and hence factored out of the parameter-space search. A similar method of search over rotation space is used in [10] to solve the calibrated camera pose problem. This is the problem of finding the position and orientation of a camera given known 3D points and their corresponding image points. An earlier iterative algorithm that addresses this problem using L∞ norm is given in [47]. The algorithm appears to converge to the global optimum, but the author states that this is unproven. 7.2
General Structure and Motion
The method outlined in section 7.1 for solving the structure and motion problem for two views could in principle be extended to give an optimal solution (albeit in L∞ norm) to the complete structure and motion problem for any number of views. This would involve a search over the combined space of all rotations. For the two camera problem there is only one relative rotation, and hence the branchand-bound algorithm involves a search over a 3-dimensional parameter space. In the case of n cameras, however the parameter space is 3(n − 1)-dimensional. Had we but world enough and time a branch-and-bound search over the combined rotation parameter space would yield an optimal L∞ solution to structure and motion with any number of cameras and points. Unfortunately, in terms of space and time requirements, this algorithm would be vaster than empires and more slow[30]. 7.3
1D Retinal Vision
One-dimensional cameras have proven useful in several different applications, most prominently for autonomous guided vehicles (see Fig 8), but also in ordinary vision for analysing planar motion and the projection of lines. Previous results on one-dimensional vision are limited to classifying and solving of minimal cases, bundle adjustment for finding local minima to the structure and motion problem and linear algorithms based on algebraic cost functions. A method for finding the global minimum to the structure and motion problem using the max norm of reprojection errors is given in [2]. In contrast to the 2Dcase which uses SOCP, the optimal solution can be computed efficiently using simple linear programming techniques. It is assumed that neither the positions of the objects nor the positions and orientations of the cameras are known. However, it is assumed that the correspondence problem is solved, i.e., it is known which measured bearings correspond to the same object. The problem can formally be stated as follows. Given n bearings from m different positions, find the camera positions and 2D points
Optimal Algorithms in Multiview Geometry
29
2
reflector
1.5
1
0.5
0
−0.5 −2
−1.5
−1
−0.5
0
0.5
1
1.5
2
2.5
Fig. 8. Left: A laser guided vehicle. Middle: A laser scanner or angle meter. Right: Calculated structure and motion for the icehockey experiment.
in the plane, such that the reprojected solution has minimal residual errors. The norm for measuring the errors will be the L∞ norm. The basic idea of the optimization scheme is to first consider optimization with fixed camera orientations - which is a quasi-convex problem - and then use branch-and-bound over the space of possible orientations, similar to that of section 7.1. Hockey rink data. By combining optimal structure and motion with optimal resection and intersection it is possible to solve for arbitrarily many cameras and views. We illustrate this with the data from a real set of measurements performed at an ice-hockey rink. The set contains 70 images of 14 points. The result is shown in the right of Fig 8. 7.4
3D – 3D Alignment
A similar method of doing branch-and-bound in rotation space was used in [27] to find an optimal solution for the problem of aligning two sets of points in 3D with unknown pairing. The solution consists of a specified pairing of the two point sets, along with a rotation and translation to align the paired points. The algorithm relies on the fact that if the rotation is known, then the optimal pairing can be computed directly using the Hungarian algorithm [36]. This enables the problem to be addressed by a branch-and-bound search over rotations. The problem is solved for L2 norm in [27] using a bounding method based on Lipschitz bounds. Though the L∞ problem is not specifically addressed in the paper, it would probably also yield to a similar approach. 7.5
An Open Problem: Optimal Essential Matrix Estimation in L2 Norm
The question naturally arises of whether we can use similar techniques to section 7.1 to obtain the optimal L2 solution for the essential matrix. At present, we have no solution to this problem. Two essential steps are missing.
30
R. Hartley and F. Kahl
1. If the relative rotation R between the two cameras is given, can we estimate the optimal translation t. This is simple in L∞ norm using SOCP, or in fact linear programming. In L2 norm, a solution has been proposed in [15], but still it is iterative, and it is not clear that it is guaranteed to converge. For n points, this seems to be a harder problem than optimal n-view L2 triangulation (for which solutions have been recently given [1,29]). 2. If we can find the optimal residual for a given rotation, how does this constrain the solution for nearby rotations? Loose bounds may be given, but they may not be sufficiently tight to allow for efficient convergence of the branch-and-bound algorithm.
8 8.1
Other Methods for Minimizing L2 Norm Convex Relaxations and Semidefinite Programming
Another general approach for solving L2 problems was introduced in [19] based on convex relaxations (underestimators) and semidefinite programming. More specifically, the approach is based on a hierarchy of convex relaxations to solve non-convex optimization problems. Linear matrix inequalities (LMIs) are used to construct the convex relaxations. These relaxations generate a monotone sequence of lower bounds of the minimal value of the objective function and it is shown how one can detect whether the global optimum is attained at a given relaxation. Standard semidefinite programming software (like SeDuMi [44]) is extensively used for computing the bounds. The technique is applied to a number of classical vision problems: triangulation, camera pose, homography estimation and epipolar geometry estimation. Although good results are obtained, there is no guarantee of achieving the optimal solution and the sizes of the problem instances are small. 8.2
Fractional Programming
Yet another method was introduced in [1,35]. It was the first method to solve the n-view L2 triangulation problem with a guarantee of optimality. Other problem applications include camera resectioning (that is, uncalibrated camera pose), camera pose estimation and, homography estimation. In its most general form, fractional programming seeks to minimize/maximize the sum of fractions subject to convex constraints. Our interest from the point of view of multiview geometry, however, is specific to the minimization problem min x
p fi (x) i=1
gi (x)
subject to x ∈ D
where fi : Rn → R and gi : Rn → R are convex and concave functions, respectively, and the domain D ⊂ Rn is a convex compact set. This is because the residual of the projection of a 3D point into an image may be written in this form. Further, it is assumed that both fi and gi are positive with lower
Optimal Algorithms in Multiview Geometry
31
and upper bounds over D. Even with these restrictions the above problem is N P -complete [8], but practical and reliable estimation of the global optimum is still possible for many multiview problems through an iterative algorithm that solve an appropriate convex optimization problem at each step. The procedure is based on branch and bound. Perhaps the most important observation made in [1] is that many multiview geometry problems can be formulated as a sum of fractions where each fraction consists of a convex over a concave function. This has inspired a new more efficient ways of computing the L2 -minimum for n-view triangulation, see [29].
9
Applications
There have been various application papers that have involved this type of optimization methodology, though they can not be said to have found an optimal solution to the respective problems. In [38] SOCP has been used to solve the problem of tracking and modelling a deforming surface (such as a sheet of paper) from a single view. Results are shown in Fig 9.
Fig. 9. Modelling a deforming surface from a single view. Left: the input image, with automatically overlaid grid. Right: the computed surface model viewed from a new viewpoint. Image features provide cone constraints that constrain the corresponding 3D points to lie on or near the corresponding line-of-sight, namely a ray through the camera centre. Additional convex constraints on the geometry of the surface allow the shape to be determined unambiguously using SOCP.
In another application-inspired problem, SOCP has been applied (see [22]) to the odometry problem for a vehicle with several rigidly mounted cameras with almost non-overlapping fields of view. Although the algorithm in [22] is tested on laboratory data, it is motivated by its potential use with vehicles such as the one shown in Fig 10. Such vehicles are used for urban-mapping. Computation of individual essential matrices for each of the cameras reduces the computation of the translation of the vehicle to a multiple view triangulation problem, which is solved using SOCP.
32
R. Hartley and F. Kahl
Fig. 10. Camera and car mount used for urban mapping. Images from non-overlapping cameras on both sides of the car can be used to do odometry of the vehicle. An algorithm based on triangulation and SOCP is proposed in [22]. (The image is used with permission of the UNC Vision group).
10
Concluding Remarks
The application of new optimization methods to the problems of Multiview Geometry have led to the development of reliably and provably optimal solutions under different geometrically meaningful cost functions. At present these algorithms are not as fast as standard methods, such as bundle adjustment. Nevertheless the run times are not wildly impractical. Recent work on speeding up the optimization process is yielding much faster run times, and further progress is likely. Such optimization techniques are also being investigated in other areas of Computer Vision, such as discrete optimization. A representative paper is [24]. For 15 years or more, geometric computer vision has relied on a small repertoire of optimization methods, with Levenberg-Marquardt [37] being the most popular. The benefit of using new methods such as SOCP and other convex and quasi-convex optimization methods is being realised.
References 1. Agarwal, S., Chandraker, M.K., Kahl, F., Kriegman, D.J., Belongie, S.: Practical global optimization for multiview geometry. In: European Conf. Computer Vision, Graz, Austria, pp. 592–605 (2006) 2. ˚ Astr¨ om, K., Enqvist, O., Olsson, C., Kahl, F., Hartley, R.: An L∞ approach to structure and motion problems in 1d-vision. In: Int.Conf. Computer Vision, Rio de Janeiro, Brazil (2007) 3. Boyd, S., Vanderberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004) 4. Byr¨ od, M., Josephson, K., ˚ Astr¨ om, K.: Improving numerical accuracy in gr¨ obner basis polynomial equation solvers. In: Int. Conf.Computer Vision, Rio de Janeiro, Brazil (2007) 5. Chandraker, M.K., Agarwal, S., Kriegman, D.J., Belongie, S.: Globally convergent algorithms for affine and metric upgrades in stratified autocalibration. In: Int. Conf. Computer Vision, Rio de Janeiro, Brazil (2007)
Optimal Algorithms in Multiview Geometry
33
6. Farenzena, M., Fusiello, A., Dovier, A.: Reconstruction with interval constraints propagation. In: Proc. Conf. Computer Vision and Pattern Recognition, New York City, USA, pp. 1185–1190 (2006) 7. Faugeras, O.D., Maybank, S.J.: Motion from point matches: Multiplicity of solutions. Int. Journal Computer Vision 4, 225–246 (1990) 8. Freund, R.W., Jarre, F.: Solving the sum-of-ratios problem by an interior-point method. J. Glob. Opt. 19(1), 83–102 (2001) 9. Hartley, R., de Agapito, L., Hayman, E., Reid, I.: Camera calibration and the search for infinity. In: Proc. 7th International Conference on Computer Vision, Kerkyra, Greece, September 1999, pp. 510–517 (1999) 10. Hartley, R., Kahl, F.: Global optimization through searching rotation space and optimal estimation of the essential matrix. Int. Conf. Computer Vision (2007) 11. Hartley, R., Schaffalitzky, F.: L∞ minimization in geometric reconstruction problems. In: Conf. Computer Vision and Pattern Recognition, Washington DC, USA, vol. I, pp. 504–509 (2004) 12. Hartley, R., Sturm, P.: Triangulation. Computer Vision and Image Understanding 68(2), 146–157 (1997) 13. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004) 14. Horn, B.K.P.: Closed form solution of absolute orientation using unit quaternions. J. Opt. Soc. America 4(4), 629–642 (1987) 15. Horn, B.K.P.: Relative orientation. Int. Journal Computer Vision 4, 59–78 (1990) 16. Horn, B.K.P.: Relative orientation revisited. J. Opt. Soc. America 8(10), 1630–1638 (1991) 17. Josephson, K., Kahl, F.: Triangulation of points, lines and conics. In: Scandinavian Conf. on Image Analysis, Aalborg, Denmark (2007) 18. Kahl, F.: Multiple view geometry and the L∞ -norm. In: Int. Conf. Computer Vision, Beijing, China, pp. 1002–1009 (2005) 19. Kahl, F., Henrion, D.: Globally optimal estimates for geometric reconstruction problems. Int. Journal Computer Vision 74(1), 3–15 (2007) 20. Ke, Q., Kanade, T.: Quasiconvex optimization for robust geometric reconstruction. In: Int. Conf. Computer Vision, Beijing, China, pp. 986–993 (2005) 21. Ke, Q., Kanade, T.: Uncertainty models in quasiconvex optimization for geometric reconstruction. In: Conf. Computer Vision and Pattern Recognition, New York City, USA, pp. 1199–1205 (2006) 22. Kim, J.H., Hartley, R., Frahm, J.M., Pollefeys, M.: Visual odometry for nonoverlapping views using second-order cone programming. In: Asian Conf. Computer Vision (November 2007) 23. Koenderink, J.J., van Doorn, A.J.: Affine structure from motion. J. Opt. Soc. America 8(2), 377–385 (1991) 24. Kumar, P., Torr, P.H.S., Zisserman, A.: Solving markov random fields using second order cone programming relaxations. In: Conf. Computer Vision and Pattern Recognition, pp. 1045–1052 (2006) 25. Li, H.: A practical algorithm for L-infinity triangulation with outliers. In: CVPR, vol. 1, pp. 1–8. IEEE Computer Society, Los Alamitos (2007) 26. Li, H., Hartley, R.: Five-point motion estimation made easy. In: Int. Conf. Pattern Recognition, pp. 630–633 (August 2006) 27. Li, H., Hartley, R.: The 3D – 3D registration problem revisited. In: Int. Conf. Computer Vision (October 2007) 28. Longuet-Higgins, H.C.: A computer algorithm for reconstructing a scene from two projections. Nature 293, 133–135 (1981)
34
R. Hartley and F. Kahl
29. Lu, F., Hartley, R.: A fast optimal algorithm for l2 triangulation. In: Asian Conf. Computer Vision (November 2007) 30. Marvell, A.: To his coy mistress. circa (1650) 31. Nist´er, D.: An efficient solution to the five-point relative pose problem. IEEE Trans. Pattern Analysis and Machine Intelligence 26(6), 756–770 (2004) 32. Nist´er, D., Hartley, R., Stew´enius, H.: Using Galois theory to prove that structure from motion algorithms are optimal. In: Conf. Computer Vision and Pattern Recognition (June 2007) 33. Nist´er, D., Kahl, F., Stew´enius, H.: Structure from motion with missing data is N P -hard. In: Int. Conf. Computer Vision, Rio de Janeiro, Brazil (2007) 34. Olsson, C., Eriksson, A., Kahl, F.: Efficient optimization of L∞ -problems using pseudoconvexity. In: Int. Conf. Computer Vision, Rio de Janeiro, Brazil (2007) 35. Olsson, C., Kahl, F., Oskarsson, M.: Optimal estimation of perspective camera pose. In: Int. Conf. Pattern Recognition, Hong Kong, China, vol. II, pp. 5–8 (2006) 36. Papadimitriou, C., Steiglitz, K.: Combinatorial Optimization: Algorithms and Complexity. Prentice-Hall, Englewood Cliffs (1982) 37. Press, W., Flannery, B., Teukolsky, S., Vetterling, W.: Numerical Recipes in C. Cambridge University Press, Cambridge (1988) 38. Salzman, M., Hartley, R., Fua, P.: Convex optimization for deformable surface 3D tracking. In: Int. Conf. Computer Vision (October 2007) 39. Seo, Y., Hartley, R.: A fast method to minimize L∞ error norm for geometric vision problems. In: Int. Conf. Computer Vision (October 2007) 40. Seo, Y., Hartley, R.: Sequential L∞ norm minimization for triangulation. In: Asian Conf. Computer Vision (November 2007) 41. Sim, K., Hartley, R.: Recovering camera motion using the L∞ -norm. In: Conf. Computer Vision and Pattern Recognition, New York City, USA, pp. 1230–1237 (2006) 42. Sim, K., Hartley, R.: Removing outliers using the L∞ -norm. In: Conf. Computer Vision and Pattern Recognition, New York City, USA, pp. 485–492 (2006) 43. Stew´enius, H., Schaffalitzky, F., Nist´er, D.: How hard is three-view triangulation really? In: Int. Conf. Computer Vision, Beijing, China, pp. 686–693 (2005) 44. Sturm, J.F.: Using SeDuMi 1.02, a Matlab toolbox for optimization over symmetric cones. Optimization Methods and Software 11(12), 625–653 (1999) 45. Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: A factorization approach. Int. Journal Computer Vision 9(2), 137–154 (1992) 46. Triggs, W., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.: Bundle adjustment for structure from motion. In: Vision Algorithms: Theory and Practice, pp. 298–372. Springer, Heidelberg (2000) 47. Zhang, X.: Pose estimation using L∞ . In: Image and Vision Computing New Zealand (2005)
Machine Vision in Early Days: Japan’s Pioneering Contributions Masakazu Ejiri R & D Consultant in Industrial Science, formerly at Central Research Laboratory, Hitachi, Ltd.
Abstract. The history of machine vision started in the mid-1960s by the efforts of Japanese industry researchers. A variety of prominent visionbased systems was made possible by creating and evolving real-time image processing techniques, and was applied to factory automation, office automation, and even to social automation during the 1970-2000 period. In this article, these historical attempts are briefly explained to promote understanding of the pioneering efforts that opened the door and formed the bases of today’s computer vision research. Keywords: Factory automation, office automation, social automation, real-time image processing, video image analysis, robotics, assembly, inspection.
1
Introduction
There is an old saying, “knowing the old brings you a new wisdom for tomorrow,” that originated with Confucius (a Chinese philosopher, 551BC-479BC). This is the basic idea underlying this article, and its purpose is to enlighten young researchers on old technologies rather than new ones. In the 1960s, one of the main concerns of researchers in the field of information science was the realization of intelligence by using a conventional computer, which had been used mainly for numerical computing. At that time, a hand-eye system was thought to be an excellent research tool to visualize intelligence and demonstrate its behavior. The hand-eye system was, by itself, soon recognized as an important research target, and it became known as the “intelligent robot.” One of the core technologies of the intelligent robot was, of course, vision, and people started to call this vision research area “computer vision.” However, the academic research on computer vision was apt to be stagnant. Its achievements stayed at the level of simulated tasks and could not surpass our expectations because of its intrinsic difficulty and the limitation of computing power in those days. On the other hand, practical vision technology was eagerly anticipated in industry, particularly in Japan, as one of the core technologies towards attaining flexible factory automation. Research was initiated in the mid-1960s, typically at our group of Hitachi’s Central Research Laboratory. In contrast to the word “computer vision,” we used the word “machine vision” for representing a more Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 35–53, 2007. c Springer-Verlag Berlin Heidelberg 2007
36
M. Ejiri
pragmatic approach towards “useful” vision systems, because the use of computers was less essential in the pragmatic approach. Most of the leading companies followed this lead, and they all played an important role in the incubation and development of machine vision technology. Currently, computer vision is regarded as a fundamental and scientific approach to investigate the principles that underlie vision and how artificial vision can be best achieved, in contrast to the more pragmatic, needs-oriented machine vision approach. We have to note, however, that there is no difference in the ultimate goals of these two approaches. Though the road to machine vision was not smooth, Japanese companies fortunately achieved some key successes. In this article, we briefly introduce these pioneering vision applications and discuss the history of machine vision.
2
Prehistoric Machine Vision
Our first attempt at vision application, in 1964, was to automate the assembly process (i.e., wire-bonding process) of transistors. In this attempt, we used a very primitive optical sensor by combining a microscope and a rotating-drum type scanner with two slits on its surface. By detecting the reflection from the transistor surface with photo-multipliers, the position and orientation of transistor chips were determined with about a 95% success rate. However, this percentage was still too low to enable us to replace human workers; thus, our first attempt failed and was eventually abandoned after a two-year struggle. What we learned from this experience was the need for reliable artificial vision comparable to a human’s pattern recognition capability, which quickly captures the image first, and then reduces the information quantity drastically until the positional information is firmly determined. Our slit-type optical-scanning method inherently lacked the right quantity of captured information; thus, the recognition result was apt to be easily affected by reflective noises. In those days, microprocessors had not yet been developed and the available computers were still too expensive, bulky, and slow, particularly for image processing and pattern recognition. Moreover, memory chips were extremely costly, so the use of full-frame image memories was prohibitive. Though there was no indication that these processors would soon improve, we started seminal research on flexible machines in 1968. A generic intelligent machine conceived at that time was the one that consisted of three basic functions: understanding of a human’s instruction or intention to clarify the goal of the task, understanding of objects or environment to clarify the start of the task, and decision-making to find the optimum route between the start and the goal. Based on this conception, a prototype intelligent robot was developed in 1970. The configuration of this intelligent robot is shown in Fig.1. In this robot, the image of a trihedral plan drawing was captured by one of the cameras and was analyzed to clarify the goal assemblage as well as the components of the assemblage. Another camera looked at the real blocks scattered on a table, and found their positions and postures. From these blocks, the
Machine Vision in Early Days: Japan’s Pioneering Contributions
37
Fig. 1. Intelligent robot (1970)
computer recognized the component blocks needed to complete this assembly task, and made a plan to assemble these blocks. For this assembly planning, backward reasoning was used to find the route from the goal to the start, not from the start to goal. That is, the route was found by analyzing the disassembly task from the goal assemblage to each component. The assembly sequence was determined as the reverse of this disassembly sequence. Thus, the robot could assemble blocks into various forms by responding to the objectives presented macroscopically by an assembly drawing [2]. This research formed part of Japan’s national project on PIPS (Pattern Information Processing System), initiated in the following year.
3
Factory Applications
Our project on the prototype intelligent robot in 1970 revealed many basic problems underlying “flexible machines” and gave us useful insights into future applications of robotics. One significant problem we were confronted with was the robot’s extremely slow image-processing speed in understanding objects. Our next effort was therefore focused on developing high-speed dedicated hardware for image processing with the minimum use of memory, instead of using rather slow and expensive computers. One of the core ideas was to adaptively threshold the image signal into a binary form by responding to the signal behavior and to input it into a shift-register-type local memory that dynamically stored the latest pixel data of several horizontal scan-lines. This local-paralleltype configuration enabled us to simultaneously extract plural pixel data from a 2-D local area in synchronization with image scanning. By designing the logic circuit connected to this 2-D local area according to the envisaged purpose, the processing hardware could be adapted to many applications. One useful yet simple method using this local-parallel-type image processing was windowing. This method involved setting up a few particular window areas
38
M. Ejiri
Fig. 2. Bolting robot for piles/poles (1972)
in the image plane, and the pixels in the windows were selectively counted to find the background area and the object area occupying the windows. In 1972 a bolting robot applying windowing was developed in order to automate the molding process of concrete piles and poles [3]. It became the first application of machine vision to moving objects. Note that this paper [3] was published at a later time. This was likely to happen because publication was not the first priority for industry researchers. Another effective method based on local-parallel architecture was erosion/ dilation of patterns, which was executed by simple AND/OR logic on a 2-D local area. This method could detect defects in printed circuit boards (PCBs), and formed one of the bases of today’s morphological processing. This defectdetection machine in 1972 also became the first application of machine vision to the automation of visual inspection [4]. These two pioneering applications are illustrated in Figs. 2 and 3. Encouraged by the effectiveness of these machine-vision systems in actual production lines, we again started to develop a new assembly machine for transistors that was, this time, based fully on image processing. A multiple local pattern matching method was extensively studied for this purpose. In this method, each local pattern position in a transistor image was found by matching to a standard pattern. The distance and the angle between a pair of detected local pattern positions were sequentially checked to see if these local patterns were correctly detected. The electrode positions for wiring were then calculated from the coordinates of the first detected correct pair. By basing a local-parallel-type image processor on this matching, we finally developed fully automatic transistor assembly machines in 1973 [5]. This successful development was the result of a ten-year effort since our first failed attempt. The developed assembly machines were recognized as the world’s first image-based machines for fully automatic assembly of semiconductor devices. These machines and their configuration are shown in Fig. 4.
Machine Vision in Early Days: Japan’s Pioneering Contributions
39
Fig. 3. PCB inspection machine (1972)
Fig. 4. Transistor assembly machine (1973)
After this development, our efforts were then focused on expanding machinevision applications from transistors to other semiconductor devices, such as integrated circuits (ICs), hybrid ICs, and large-scale integrated circuits (LSIs). Consequently, the automatic assembly of all types of semiconductor devices was completed by 1977. With the export of this automatic assembly technology to a US manufacturer as a start, the technology gained widespread attention from semiconductor manufacturers worldwide and expanded quickly into industry. As a result, the semiconductor industry as a whole prospered by virtue of higher speed production of higher quality products with more uniform performance than had been achieved previously. Encouraged by the success of semiconductor assembly, our efforts were further broadened to other industrial applications in the mid-1970s to early 1980s. Examples of such applications during this period are a hose-connecting robot
40
M. Ejiri
Fig. 5. Wafer inspection machines (1984-1987)
for pressure testing in pump production lines, a reading machine of 2-D objectcode for an intra-factory physical distribution system, and a quality inspection machine for marks and characters printed on electronic parts [6][7][8]. Machines for inspecting photo-masks in semiconductor fabrication and CRT black-matrix fabrication were other examples [9] (by Toshiba) and [10] (by Hitachi). Machines for classifying medical tablets and capsules [11] (by Fuji Electric) and machines for classifying agricultural products and fish [12][13][14] (by Mitsubishi Electric) were also unique and epoch-making achievements in those days. These examples show that the key concept representing those years seemed to be the realization of a “productive society” through factory automation, and the objectives of machine vision were mainly position detection for assembly, shape detection for classification, and defect detection for inspection. In 1980, the PIPS project finished after a 10-year effort by Japanese industry. A variety of recognition systems were successfully prototyped for hand-written Kanji characters, graphics, drawings, documents, color pictures, speeches, and three-dimensional objects. One particular outcome among others was the development of high-speed, general-purpose image processors [15], which in turn served as the basis of subsequent research and development in industry. The most difficult but rewarding development in the mid-1980s was an inspection machine for detecting defects in semiconductor wafers [16]. It was estimated that even the world’s largest super-computer available at that time would require at least one month of computing for finishing the defect detection of an 8-inch single wafer. We therefore had to develop special hardware for lowering the processing time to less than 1 hour/wafer. The resulting hardware was a network of local-parallel-type image processors that use a “design pattern referring method,” shown in Fig. 5. In this machine, hardware-based knowledge processing, in which each processor was regarded as a combination of IF-part and THEN-part logical circuits, was first attempted [17].
Machine Vision in Early Days: Japan’s Pioneering Contributions
41
Meanwhile, the processing speed of microprocessors improved considerably since their appearance in the mid-1970s, and the memory capacity drastically increased without excessively increasing costs. These improvements facilitated the use of gray-scale images instead of binary ones, and dedicated LSI chips for image processing were developed in the mid-1980s [18]. These developments all contributed to achieving more reliable, microprocessor-based general-purpose machine vision systems with full-scale buffers for gray-level images. As a result, applications of machine vision soon expanded from circuit components, such as semiconductors and PCBs, to home-use electronic equipment, such as VCRs and color TVs. Currently, machine vision systems are found in various areas such as electronics, machinery, medicine, and food industries.
4
Office Applications
Besides the above-described machine-vision systems for factory applications, there was extensive research on character recognition in the area of office automation. For example, in the mid-1960s, a FORTRAN program reader was developed to replace key-punching tasks. Mail-sorting machines for post offices were developed in the late 1960s to automatically read handwritten postal codes (by Toshiba et al.). Another topical developmental effort started in 1974 for automatic classification of fingerprint patterns, and in 1982 the system was first put in use at a Japanese police office with great success, and later at US police offices (by NEC). Our first effort to apply machine-vision technology to areas other than factory automation was the automatic recognition of monetary bills in 1976. This recognition system was extremely successful in spurring the development of automated teller machines (ATMs) for banks (see Fig. 6). Due to the processing time limitation, the entire image of a bill was not captured, but by combining several partial images obtained from optical sensors with those from magnetic sensors, so-called sensor-fusion was first attempted and thus resulted in highaccuracy bill recognition with a less than 1/1015 theoretical error rate. Early ATM models for domestic use employed vertical safes, but in the later models, the horizontal safes were extensively used for increasing spatial efficiency and for facilitating use in Asian countries having a larger number of bill types. Our next attempt, in the early 1980s, was the efficient handling of a large amount of graphic data in the office [19]. The automatic digitization of paperbased engineering drawings and maps was first studied. The recognition of these drawings and maps was based on a vector representation technique, such as that shown in Fig. 7. The recognition was usually executed by spatially-paralleltype image processors, in which each processor was designated to a specific image area. Currently, geographic information systems (GIS) based on these digital maps gained popularity and are being used by many service companies and local governments to manage their electric power supply, gas supply, water supply facilities, and sewage service facilities (see Fig. 8). The use of digital maps was then extended to car navigation systems and more recently to various
42
M. Ejiri
Fig. 6. ATM: automated teller machines (1976-1995)
Fig. 7. Automatic digitizer for maps and engineering drawings (1982)
other information service systems via the Internet. Machine-vision technology contributed, mainly in the early developmental stage of these systems, to the digitization of original paper-based maps into electronic form until these digital maps began to be produced directly from measured data through computer-aided map production. Spatially divided parallel processing was also useful for large-scale images such as those from satellite data. One of our early attempts in this area was the recognition of wind vectors, back in 1972, by comparing two simulated satellite images with a 30-minute interval. This system formed a basis of weather forecasting using Japan’s first meteorological geo-stationary satellite “Himawari,” launched a few years later. Also, an algorithm for deriving a sea temperature contour map from infra-red satellite images was built for environmental study and fisheries.
Machine Vision in Early Days: Japan’s Pioneering Contributions
43
Fig. 8. GIS: geographic information systems(1986)
Research on document understanding also originated as part of machine-vision research in the mid-1980s [20]. During those years, electronic document editing and filing became popular owing to the progress in word-processing technology for over 4000 Kanji and Kana characters. The introduction of an electronic patent-application system in Japan in 1990 was an important stimulus for further research on office automation. We developed dedicated workstations and a parallel-disk-type distributed filing system for the use of patent examiners. This system enabled examiners to efficiently retrieve and display the images of past documents for comparison. The recognition of handwritten postal addresses was one of the most challenging topics in machine-vision applications. In 1992, a decision was made by a government committee (to which the author served as a member) to adopt a new 7-digit postal code system in Japan beginning in 1998. To this end, three companies (Hitachi, Toshiba and NEC) developed new automatic mail-sorting machines for post offices in 1997. An example of the new sorting machines is shown in Fig. 9. In those machines, hand-written/printed addresses in Kanji characters are fully read together with the 7-digit postal codes; both results are then matched for consistency; and the recognized full address is printed on each letter as a transparent barcode consisting of 20-digit data. The letters are then dispatched to other post offices for delivery. In a subsequent process, only these barcodes are read, and prior to home delivery the letters are arranged by the new sorting machine in such a way that the order of the letters corresponds to the house order on the delivery route. In these postal applications, the recognition of all types of printed fonts and hand-written Kanji characters was made possible by using a multi-microprocessor type image processing system. A mail image is sent to one of the unoccupied processors, and this designated processor analyzes the image. The address recognition by a designated single processor usually requires 1.0 to 2.5 seconds, depending on the complexity of the address image. As up to 32 microprocessors are used
44
M. Ejiri
Fig. 9. New mail sorting machine (1997)
in parallel for successively flowing letters, the equivalent recognition time of the whole system is less than 0.1 seconds/letter, producing a maximum processing speed of 50,000 letters per an hour. The office applications of vision technology described above show that the key concept representing those years seemed to be the realization of an “efficient society” through office automation, and the objectives of machine vision were mainly efficient handling of large-scale data and also high-precision, high-speed recognition and handling of paper-based information. Recent progress in network technology has also increased the importance of office automation. To secure the reliability of information and communication systems, a variety of advanced image processing technologies will be expected. These will include more effective and reliable compression, encryption, scrambling, and watermarking technologies of image data.
5
Social Applications
In recent years, applications to social automation have become increasingly important. Social automation here means “making the systems designed for social use more intelligent,” and it includes systems for traffic and for environmental use. The technologies used in these systems are, for example, surveillance, monitoring, flow control and security assurance. The earliest attempt at social automation was probably our elevator-eye project in 1977, in which we tried to implement machine vision in an elevator system in order to control the human traffic in large-scale buildings. The elevator hall on each floor was equipped with a camera to observe the hall, and a vision system to which these cameras were connected surveyed all floors in a time-sharing manner and estimated the number of persons waiting for an elevator. The vision system then designated an elevator cage to quickly serve the crowded floor [21]. The configuration of this system is shown in Fig. 10.
Machine Vision in Early Days: Japan’s Pioneering Contributions
45
Fig. 10. Elevator and other traffic applications (1977-1986)
In this elevator system, a robust change-finding algorithm based on edge vectors was used in order to cope with the change in the brightness of the surroundings. In this algorithm, the image plane was divided into several blocks, and the edge-vector distribution in each block was compared with that of the background image, which was updated automatically by new image data when no motion was observed and thus nobody was in the elevator hall. This system could minimize the average waiting time for the elevator. Though a few systems were put into use in Tokyo area in the early 1980s, there has not been enough market demand to continue to develop the system further. More promising applications of image recognition seemed to be for monitoring road traffic, where license plates, traffic jams, and illegally parked cars were identified so that traffic could be controlled smoothly and parking lots could be automatically allocated [22]. Charging tolls automatically at toll gates without stopping cars, by means of a wireless system with IC card, is now popular on highways as a result of the ITS (Intelligent Transport System) project. The system will be further improved if the machine vision can be effectively combined to quickly recognize other important information such as license plate numbers and even driver’s faces and other identities. A water-purity monitoring system using fish behavior [23] was in operation for at least 10 years at a river control center in a local city in Japan since the river water was accidentally polluted by toxicants. A schematic diagram of the system is shown in Fig. 11. The automatic observation of algae in water in sewage works was also studied. Volcanic lava flow was continuously monitored at the base of Mt. Fugendake in Nagasaki, Japan, during the eruption period in 1993. To optically send images from unmanned remote observation posts to the central control station, laser communication routes were planned by using 3-D undulation data derived from GIS digital contour maps. A GIS was also constructed to assist in restoration after the “Hanshin-Awaji” earthquake in Kobe, Japan, in 1995. Aerial photographs after the earthquake were analyzed by
46
M. Ejiri
Fig. 11. Environmental use (1990-1995)
matching them with digital 3-D urban maps containing additional information on the height of buildings. Buildings with damaged walls and roofs could thus be quickly detected and given top priority for restoration [24]. Intruder detection is also becoming important in the prevention of crimes and in dangerous areas such as those around high-voltage electric equipment. Railroad crossings can also be monitored intensively by comparing the vertical line data in an image with that in a background image updated automatically [25]. Arranging the image differences in this vertical window gives a spatiotemporal image of objects intruding onto the crossing. In almost all of these social applications, real-time color-image processing is becoming increasingly important for reliable detection and recognition. As mentioned before, the application of image processing to communications is increasingly promising as multimedia and network technologies improve. Humanmachine interfaces will be greatly improved if the machine is capable of recognizing every media used by humans. Human-to-human communication assisted by intelligent machines and networks is also expected. Machine vision will contribute to this communication in such fields as motion capturing, face recognition, facial expression understanding, gesture recognition, sign language understanding, and behavior understanding. In addition, applications of machine vision to the field of human welfare, medicine, and environment improvement will become increasingly important in the future. Examples of these applications are rehabilitation equipment, medical surgery assistance, and water purification in lakes. Thus, the key concept representing the future seems to be the realization of a calm society, in which all uneasiness will be relieved through networked social automation, and the important objectives of machine vision will typically be the realization of the following two functions: 24-hour/day “abnormality monitoring” via networks and reliable “personal identification” via networks.
Machine Vision in Early Days: Japan’s Pioneering Contributions
6
47
Key Technologies
In most of the future applications, dynamic image processing will be a key to success. There are various approaches already for analyzing incoming video images in real-time by using smaller-scale personal computers. One typical example is the “Mediachef” system, which automatically cuts video images into a set of scenes by finding significant changes between consecutive image frames [26]. The principle of the system is shown in Fig. 12. This is one of the essential technologies for video indexing and video-digest editing. To date, this technology has been put into use in the video inspection process in a broadcasting company so that subliminal advertising can be detected before the video is on the air.
Fig. 12. Key technologies: “Mediachef” for video indexing and editing (1990)
For the purpose of searching scenes, we developed a real-time video coding technique that uses an average color in each frame and represents its sequence by a “run” between frames. This method can compress 24-hour video signals into a memory capacity of only 2 MB. This video-coding technology can be applied to automatically detect the time of broadcast of a specific TV commercial by continuously monitoring TV signals by means of a compact personal computer. It therefore allows manufacturers to monitor their commercials being broadcast by an advertising company and, thus, provides evidence of a broadcast. The technology called “Cyber BUNRAKU,” in which human facial expressions are monitored by small infrared-sensitive reflectors put on a performer’s face, is also noteworthy. By combining the facial expressions thus obtained with the limb motions of a 19-jointed “Bunraku doll” (used in traditional Japanese theatrical performance), a 3-D character model in a computer can be animated in real-time to create video images [27], as shown in Fig. 13. This technology can create TV animation programs much faster than through traditional handdrawing methods. Another example of dynamic video analysis is “Tour into the picture (TIP)” technology. As shown in Fig. 14, a 2-D picture is scanned and interpreted into
48
M. Ejiri
Fig. 13. Key technologies: Cyber BUNRAKU(1996)
Fig. 14. Key technologies: Tour into the picture (1997)
three-dimensional data by manually fitting vanishing lines on the displayed picture. The picture can then be looked at from different angles and distances [28]. A motion video can thus be generated from a single picture and viewers can feel as if they were taking a walk in an ancient city when an old picture of the city is available. A real-time creation of panoramic pictures is also an important application of video-image processing [29]. A time series of each image frame from a video camera during panning and tilting is spatially connected in real-time into a single still picture (i.e. image mosaicing), as shown in Fig. 15. Similarly, by connecting all the image frames obtained during the zooming process, a high-resolution picture (having higher resolution in the inner areas) can be obtained as shown in Fig. 16.
Machine Vision in Early Days: Japan’s Pioneering Contributions
49
Fig. 15. Key technologies: Panoramic view by panning and tilting(1998)
Fig. 16. Key technologies: Panoramic view by zooming (1999)
As mentioned already, one important application of machine vision is personal identification in social use. Along these lines, there have been a few promising developments. These include the personal identification systems by means of fingerprint patterns [30] (by NEC, 1996), of iris patterns (by Oki Electric, 1997), of finger vein patterns (by Hitachi, 2000, see Fig. 17) and of palm vein patterns (by Fujitsu, 2002). These are now finding the wide use in security systems, including the application to automated teller machines (ATMs). We have given a few examples of real-time image processing technologies, which will be key technologies applicable to wide variety of systems in the future. The most difficult technical problem that social automation is likely to face, however, is how to make robust machine-vision systems that can be used day or night in all types of weather conditions. To cope with the wide changes in illumination, the development of a variable-sensitivity imaging device with a wide dynamic range is still a stimulating challenge. Artificial retina chips [31] (by Mitsubishi, 1998) and high-speed vision chips (by Fujitsu and University of Tokyo, 1999) are expected to play an important role along these lines.
50
M. Ejiri
Fig. 17. Key technologies: Biometrics based on finger vein patterns (2000)
7
Summaries
The history of machine vision and its applications in Japan was briefly reviewed by focusing on the efforts in industry, and is summarized roughly in Fig. 18 as a chronological form.
Fig. 18. History of machine vision research
Details of topics of industrial activities are listed on a year-to-year basis in Table 1, together with the various topics in other related fields for easier understanding of each period. The history is also summarized in Table. 2 as a list form characterizing each developmental stage. As indicated, we can see that, in addition to factory automation, office automation and social automation have been greatly advanced in those years by the evolution of machine-vision technology, owing to the progress of processor and
Machine Vision in Early Days: Japan’s Pioneering Contributions
51
Table 1. History of machine vision research (1961-2000)
memory technologies. However, it is also a fact that one of the most prominent contributions of machine vision technology was in the production of semiconductors. The semiconductor industry, and thus our human life, would not have been able to enjoy prosperity without machine vision technology. This article was prepared from the view point of the old saying; “knowing the old brings you a new wisdom for tomorrow” by Confucius. The author will be extremely pleased if this article is read widely by young researchers, as it would give them some insights to this field, and would encourage them to get into, and play a great role in, this seemingly simple but actually difficult research field.
52
M. Ejiri Table 2. History of machine vision research
The closing message from the author to young researchers is as follows: Lift your sights, raise your spirits, and get out into the world!
References 1. Ejiri, M., Miyatake, T., Sako, H., Nagasaka, A., Nagaya, S.: Evolution of realtime image processing in practical applications. In: Proc. IAPR MVA, Tokyo, pp. 177–186 (2000) 2. Ejiri, M., Uno, T., Yoda, H., Goto, T., Takeyasu, K.: A prototype intelligent robot that assembles objects from plan drawings. IEEE Trans. Comput. C-21(2), 161–170 (1972) 3. Uno, T., Ejiri, M., Tokunaga, T.: A method of real-time recognition of moving objects and its application. Pattern Recognition 8, 201–208 (1976) 4. Ejiri, M., Uno, T., Mese, M., Ikeda, S.: A process for detecting defects in complicated patterns, Comp. Graphics & Image Processing 2, 326–339 (1973) 5. Kashioka, S., Ejiri, M., Sakamoto, Y.: A transistor wire-bonding system utilizing multiple local pattern matching techniques. IEEE Trans. Syst. Man & Cybern. SMC-6(8), 562–570 (1976) 6. Ejiri, M.: Machine vision: A practical technology for advanced image processing. Gordon & Breach Sci. Pub, New York (1989) 7. Ejiri, M.: Recent image processing applications in industry. In: Proc. 9th SCIA, Uppsala, pp. 1–13 (1995) 8. Ejiri, M.: A key technology for flexible automation. In: Proc. of Japan-U.S.A. Symposium on Flexible Automation, Otsu, Japan, pp. 437–442 (1998) 9. Goto, N.: Toshiba Review 33, 6 (1978) (in Japanese) 10. Hara, Y., et al.: Automatic visual inspection of LSI photomasks. In: Proc. 5th ICPR (1980) 11. Haga, K., Nakamura, K., Sano, Y., Miyamori, N., Komuro, A.: Fuji Jiho 52(5), pp.294–298 (1979) (in Japanese) 12. Nakahara, S., Maeda, A., Nomura, Y.: Denshi Tokyo. IEEE Tokyo Section 18, 46–48 (1979)
Machine Vision in Early Days: Japan’s Pioneering Contributions
53
13. Nomura, Y., Ito, S., Naemura, M.: Mitsubishi Denki Giho 53(12), 899–903 (1979) (in Japanese) 14. Maeda, A., Shibayama, J.: Pattern measurement, ITEJ Technical Report, 3, 32 (in Japanese) (1980) 15. Mori, K., Kidode, M., Shinoda, H., Asada, H.: Design of local parallel pattern processor for image processing. In: AFIP Conf. Proc., vol. 47, pp. 1025–1031 (1978) 16. Yoda, H., Ohuchi, Y., Taniguchi, Y., Ejiri, M.: An automatic wafer inspection system using pipelined image processing techniques, IEEE Trans. Pattern Analysis & Machine Intelligence. PAMI-10 1 (1988) 17. Ejiri, M., Yoda, H., Sakou, H.: Knowledge-directed inspection for complex multilayered patterns. Machine Vision and Applications 2, 155–166 (1989) 18. Fukushima, T., Kobayashi, Y., Hirasawa, K., Bandoh, T., Ejiri, M.: Architecture of image signal processor, Trans. IEICE, J-66C 12, 959–966 (1983) 19. Ejiri, M., Kakumoto, S., Miyatake, T., Shimada, S., Matsushima, H.: Automatic recognition of engineering drawings and maps. In: Proc. Int. Conf. on Pattern Recognition, Montreal, Canada, pp. 1296–1305 (1984) 20. Ejiri, M.: Knowledge-based approaches to practical image processing. In: Proc. MIV-89, Inst. Ind. Sci, Univ. of Tokyo, pp. 1–8. Tokyo (1989) 21. Yoda, H., Motoike, J., Ejiri, M., Yuminaka, T.: A measurement method of the number of passengers using real-time TV image processing techniques, Trans. IEICE, J-69D 11, 1679–1686 (1986) 22. Takahashi, K., Kitamura, T., Takatoo, M., Kobayashi, Y., Satoh, Y.: Traffic flow measuring system by image processing. In: Proc. IAPR MVA, Tokyo, pp. 245–248 (1996) 23. Yahagi, H., Baba, K., Kosaka, H., Hara, N.: Fish image monitoring system for detecting acute toxicants in water. In: Proc. 5th IAWPRC, pp. 609–616 (1990) 24. Ogawa, Y., Kakumoto, S., Iwamura, K.: Extracting regional features from aerial images based on 3-D map matching, Trans. IEICE, D-II 6, 1242–1250 (1998) 25. Nagaya, S., Miyatake, T., Fujita, T., Itoh, W., Ueda, H.: Moving object detection by time-correlation-based background judgment. In: Li, S., Teoh, E.K., Mital, D., Wang, H. (eds.) Recent Developments in Computer Vision. LNCS, vol. 1035, pp. 717–721. Springer, Heidelberg (1996) 26. Nagasaka, A., Miyatake, T., Ueda, H.: Video retrieval method using a sequence of representative images in a scene. In: Proc. IAPR MVA, Kawasaki, pp. 79–82 (1994) 27. Arai, K., Sakamoto, H.: Real-time animation of the upper half of the body using a facial expression tracker and an articulated input device, Research Report 96CG-83, Information Processing Society of Japan (in Japanese), 96, 125, pp. 1–6 (1996) 28. Horry, Y., Anjyo, K., Arai, K.: Tour into the picture: Using a spidery mesh interface to make animation from a single image. In: Proc. ACM SIGGRAPH 1997, pp. 225– 232 (1997) 29. Nagasaka, A., Miyatake, T.: A real-time video mosaics using luminance-projection correlation, Trans. IEICE, J82-D-II 10, 1572–1580 (1999) 30. Kamei, T., Shinbata, H., Uchida, K., Sato, A., Mizoguchi, M., Temma, T.: Automated fingerprint classification, IEICE Technical Report, Pattern Recognition and Understanding, 95(470), 17–24 (in Japanese) (1996) 31. Ui, H., Arima, Y., Murao, F., Komori, S., Kyuma, K.: An artificial retina chip with pixel-wise self-adjusting intensity response, ITE Technical Report, 23(30), pp.29–33 (in Japanese) (1999)
Coarse-to-Fine Statistical Shape Model by Bayesian Inference Ran He, Stan Li, Zhen Lei, and ShengCai Liao Institute of Automation, Chinese Academy of Sciences, Beijing, China
[email protected]
Abstract. In this paper, we take a predefined geometry shape as a constraint for accurate shape alignment. A shape model is divided in two parts: fixed shape and active shape. The fixed shape is a user-predefined simple shape with only a few landmarks which can be easily and accurately located by machine or human. The active one is composed of many landmarks with complex shape contour. When searching an active shape, pose parameter is calculated by the fixed shape. Bayesian inference is introduced to make the whole shape more robust to local noise generated by the active shape, which leads to a compensation factor and a smooth factor for a coarse-to-fine shape search. This method provides a simple and stable means for online and offline shape analysis. Experiments on cheek and face contour demonstrate the effectiveness of our proposed approach. Keywords: Active shape model, Bayesian inference, statistical image analysis, segmentation.
1 Introduction Shape analysis is an important area in computer vision. A common task of shape analysis is to recover both pose parameters and low-dimensional representation of the underlying shape from an observed image. Applications of shape analysis spread from medical image processing, face recognition, object tracking and etc. After the pioneering work on active shape model (ASM) put forward by Cootes and Taylor [1,2], various shape models have been developed for shape analysis, which mainly focus on two parts: (1) statistic framework to estimate the shape and pose parameters and (2) optimal features to accurately model appearance around landmarks. For parameter estimation, Zhou, Gu, and Zhang [3] propose a Bayesian tangent shape model to estimate parameters more accurately by Bayesian inference. Liang et al. [4] adopt Markov network to find an optimal shape which is regularized by the PCA based shape prior through a constrained regularization algorithm. Li and Ito [5] use AdaBoosted histogram classifiers to model local appearances and optimize shape parameters. Thomax Brox et al. [6] integrated 3D shape knowledge into a variational model for pose estimation and image segmentation. For optimal features, van Ginneken et al. [7] propose a non-linear ASM with Optimal Features (OF-ASM), which allows distributions of multi-modal intensities and uses a k-nearest neighbors classifier for local textures classification. Federico Sukno et al. [8] further develop Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 54–64, 2007. © Springer-Verlag Berlin Heidelberg 2007
Coarse-to-Fine Statistical Shape Model by Bayesian Inference
55
this non-linear appearance model, incorporating a reduced set of differential invariant features as local image descriptors. A Cascade structure containing multiple ASMs is introduced in [9] to make location of landmarks more accurate and robust. However, these methods will lose their efficiency when dealing with complicated geometry of shapes or large texture variations. Can we utilize some accurate information to simplify ASM algorithm and make shape parameters estimation more robust? For example, we can utilize face detection algorithm to detect the coordinates of eyes and mouth or manually label these coordinates when we want to find a facial contour for further analysis. In this paper, the problem of shape analysis is addressed from three aspects. Firstly, we present geometry constrained active shape model (GCASM) and divide it in two parts: fixed shape and active shape. The fixed shape is a user-predefined shape with only a few points and lines. Those points could be easily and accurately located by machine or human. The active one is a user's desired shape and is composed of many landmarks with a complex contour. It will be located automatically with the help of the fixed shape. Secondly, Bayesian inference is introduced to make parameter estimation more robust to local noise generated by the active shape, which leads to a compensation factor and a smooth factor to perform a coarse-to-fine shape search. Thirdly, optimal features are selected as local image descriptors. Since the pose parameters can be calculated by the fixed shape, classifiers are trained for each landmark without scarifying performance. The rest of the paper is organized as follows: In Section 2, we begin with a brief review of ASM. Section 3 describes our proposed algorithm and Bayesian inference. Experimental results are provided in Section 4. Finally, we draw the conclusions in Section 5.
2 Active Shape Models This section briefly reviews the ASM segmentation scheme. We follow the description and notation of [2]. An object is described by points, referred as landmark points. The landmark points are (manually) determined in a set of N training images. From these collections of landmark points, a point distribution model (PDM) [10] is constructed as follows. The landmark points (x1, y1, … , xn, yn) are stacked in shape vectors.
x = ( x1 , y1 ,..., xn , yn )T .
(1)
Principal component analysis (PCA) is applied to the shape vectors x by computing the mean shape, covariance and eigensystem of the covariance matrix.
x=
1 N
∑
N
x and S =
i =1 i
N N ( xi − x )( xi − x )T . ∑ i =1 N −1
(2)
The eigenvectors corresponding to the k largest eigenvalues λ j are retained in a matrix Φ = (φ1 | φ2 | ⋅ ⋅ ⋅ | φk ) . A shape can now be approximated by x ≈ x + Φb .
(3)
56
R. He et al.
Where b is a vector of k elements containing the shape parameters, computed by b = ΦT ( x − x ) .
(4)
When fitting the model to a set of points, the values of b are constrained to lie within a range | b j |≤ ±c λ j .
(5)
where c usually has a value between two and three. Before PCA is applied, the shapes can be aligned by translating, rotating and scaling so as to minimize the sum of squared distances between the landmark points. We can express the initial estimate x of a shape as a scaled, rotated and translated version of original shape
x = M ( s,θ )[ x] + t .
(6)
Where M ( s,θ ) and t are pose parameters (See [1] for details). Procrustes analysis [11] and EM algorithm [3] are often used to estimate the pose parameters and align the shapes. This transformation and its inverse are applied both before and after projection of the shape model. The alignment procedure makes the shape model independent of the size, position, and orientation of the objects.
3 Coarse-to-Fine Statistical Shape Model 3.1 Geometry Constrained Statistical Shape Model
To make use of the user-predefined information, we extend PDM to two parts: active shape and fixed shape. The active shape is a collection of landmarks to describe an object in the basic PDM. It is composed of many points with a complex contour. The fixed shape is a predefined simple shape accurately marked by user or machine. It is composed of several connected lines between these points which can be easily and accurately marked by machine or human. Considering there are tremendous points in a line, we present a line with several equidistant points. Thus the extended PDM is constructed as follows. The landmarks (x1, y1, … , xm, ym) are stacked in active shape vectors, and landmarks (xm+1, ym+1, … , xn, yn) are stacked in fixed shape vectors. x = ( x1 , y1 ,..., xm , ym , xm +1 , ym +1 ,..., xn , yn )T .
(7)
As in PDM, a shape can now be approximated by x ≈ x + Φb .
(8)
When aligning shapes during training, the pose parameters of a shape (scaling, rotation and translation) are estimated by the fixed shape. An obvious reason is that the fixed shape is simpler and more accurate than the active one. Taking cheek contour as an example, the active shape is composed of landmarks in a cheek contour and the fixed shape is composed of 13 landmarks derived from three manual labeled points: left eye center, right eye center and mouth center. Five
Coarse-to-Fine Statistical Shape Model by Bayesian Inference
57
landmarks are added equidistantly between two eyes center to represent horizontal connected line. And five landmarks are inserted equidistantly in the vertical line passing the mouth center and perpendicular to the horizontal line. (See left graph of Fig.1 for details) During training, two shapes are aligned according to the points between two eyes only. Each item of b reflects a specific variation along the corresponding principle component (PC) axis. Shape variation along first three PCs is shown in right graph of Fig.1. The interpretation of these PCs is straight forward. The first PC describes left-right head rotations. The second PC accounts for face variation in vertical direction: long or short. And the third one explains a human face fat or thin.
Fig. 1. The fixed shape and shapes reconstructed by the first three PCs. The thirteen white circles in left image are points of the fixed shape. In right image, the middle one in each row is the mean shape.
3.2 Bayesian Inference
When directly calculating shape parameter b by formula (4), there is an offset between the reconstructed fixed shape and the given fixed shape. But the fixed shape is supposed to be accurate. This noise comes from reconstruction error of the active shape. Inspired by paper [3], we associate PCA with a probabilistic explanation. An isotropic Gaussian noise item is added to both fixed and active shape; thereby we can compute the poster of model parameters. The model can be written as: y = x + Φb + ε .
(9)
y − x − Φb = ε .
(10)
Where the shape parameter b is a n-dimensional vector distributed as multivariate Gaussian N (0, Λ ) and Λ = diag (λ1 ,..., λk ) . ε denotes an isotropic noise on the whole shape. It is a n-dimensional random vector which is independent with b and distributes as
p(ε ) ~ exp{− || ε ||2 / 2( ρ ) 2 } .
(11)
ρ = ∑ i =1α i || yiold − yi ||2 .
(12)
n
Where yold is the shape estimated in the last iteration and y is an observed shape in the current iteration. ai is classification confidence related to a classifier used in locating a
58
R. He et al.
landmark. When ai is 0, which implies that classifier can perfectly predict shape’s boundary; when ai is 1, which means classifier fails to predict the boundary. Combing (10) and (11) we obtain the likelihood of model parameters: 1 P(b | y ) = constP ( y | b) P (b) ~ exp( − [( y − x − Φb)T ( y − x − Φb) / ρ + bT Λ −1b]) 2
Let
(13)
∂ (ln P (b | y )) = 0 , we get: ∂b
b j = (λ j \ (λ j + ρ ))φ Tj ( y − x ) .
(14)
Combining (4), we obtain: b j = (λ j \ (λ j + ρ ))b j .
(15)
It is obvious that value of bj will become smaller after updating of (15) ( ρ ≥ 0 ). This updating will slow down search speed. Hence, a compensation factor p1 is introduced to make shape variation along eigenvectors corresponding to large eigenvalues more aggressive (see formula 18). If p1 is equal to (λmax + ρ ) / λmax , we get b j = ((λmax + ρ ) \ λmax ) × (λ j \ (λ j + ρ )) × b j .
(16)
Formula (16) shows that the parameter bj corresponding to a larger eigenvalue will receive a small punishment. And the parameter bj corresponding to a small eigenvalue will become smaller after updating. Moreover, we expect a smooth shape contour and neglect details in the first several iterations. A smooth factor p2 (see formula 18) is introduced to further punish the parameter bj. It is noticed that ρ is smaller than the largest eigenvalue and will become smaller. The p2 regularizes the parameters by enlarging the punishment. As in Fig.2, the reconstructed shape’s contour by Bayesian inference is smoother than the one by PCA in regions pointed by the black arrows. Although the PCA reconstruction can remove some noise, the reconstructed shape is still unstable when the image is noisy. Formula (18) makes the parameter estimation more robust to local noise.
Fig. 2. Shapes reconstructed from PCA and Bayesian Inference. Left shape is mean shape after desired movements; middle shape is reconstructed by PCA; right shape is reconstructed by Bayesian Inference. The black arrows highlight the regions to be compared.
Coarse-to-Fine Statistical Shape Model by Bayesian Inference
59
3.3 Optimal Features
Recently, optimal features are applied in ASMs and have drawn more and more attentions. [5, 6, 7] Experimental results show that optimal features can make shape segmentation more accurate. But a main drawback of optimal features method is that it takes ASMs more time to find the desired landmarks due to extract optimal features in each iteration. An efficient speed-up strategy is to select a subset of the available features for all landmarks. [6, 7] It is clear that textures around different landmarks are different. It is impossible for a single subset of optimal features to describe various textures around all the landmarks. In GCASM, the pose parameters of scale, rotation and translation can be calculated by the fixed shape. All landmarks can be categorized into several groups, for each of which we select the same discriminate features. When search a shape, the image is divided into several areas according to the categories. For each area, the same optimal features are extracted to determine movement. Optimal features are features reported in both paper [6] and [7]. Fig.3 shows classification results for each landmark. The Mean classification accuracy is 76.67%. We can learn about that landmarks near jaw and two ears have low classification accuracy, and the landmarks near cheek have high classification accuracy. Considering this classification error, we introduce Bayesian Inference and ai of formula (12) to make shape estimation more robust.
Fig. 3. Classification results for each cheek landmark. Classification accuracy stands for a classifier’s ability to classify whether a point near the landmark is in or outside of the shape. The points around the indices of 4 and 22 are close to ears and the points around the index of 13 are close to jaw.
3.4 Coarse-to-Fine Shape Search
During image search, main differences between GCASM and ASM lie in twofold. One is that since the pose parameters of GCASM have been calculated by the fixed shape, we needn’t to think about the pose variation during iterative updating procedure. The other is that the fixed shape is predefined accurately in GCASM. After reconstruction from the shape parameters, the noise will make the reconstructed fixed shape leave away from the given fixed shape. Because the fixed shape is supposed to be accurate, it should be realigned to the initial points. The iterative updating procedure of GCASM and ASM are shown in Fig.4. We use formula (17) to calculate shape parameter b=[b1,…,bk]T and normalize b by formula (18).
60
R. He et al.
b j = ΦTj ( y − x ) .
(17)
b j = ( p1λ j /(λ j + p2 ρ ))b j
(18)
Where 1 ≤ p1 ≤ (λmax + p2 ρ ) / λmax , p2 ≥ 1 . We call the parameter p1 compensation factor which makes shape variation in a more aggressive way. The parameter p2 is a smooth factor which gives a penalty to the shape parameter when shape has a large variation. The compensation factor and smoother factor give more emphasis on shape parameters corresponding to large eigenvalues. This can adjust a shape along major PCs and neglect shape’s local detail in initial several iterations. When the algorithm converges ( ρ → 0 ), p1λ j /(λ j + p2 ρ ) is equal to 1. Hence, the compensation factor and smoother factor lead a coarse-to-fine shape searching. Here, we simply set p1 = (λmax + p2 ρ ) / λmax , α i = 1 and p2 = 4 . Obviously, the formula (18) can also be used in ASMs to normalize the shape parameter.
Fig. 4. Updating rules of ASM and GCASM. The left block diagram is the basic ASM’s updating rule and the right block diagram is GCASM updating rule.
4 Experiments In this section, our proposed method is tested on two experiments: cheek contour search and facial contour search. A total of 100 face images are randomly taken from the XM2VTS face database. [12] Each image is aligned by coordinates of two eyes. The average distance between two eyes is 80 pixels. Three points of the fixed shape including two eyes and mouth center are manually labeled. The fixed shape takes a shape of letter ‘T’. Hamarneh’s ASM source code [13] is taken as the standard ASM without modification. Optimal features are collected from features reported in both paper [7] and [8]. The number of optimal features is reduced by sequential feature
Coarse-to-Fine Statistical Shape Model by Bayesian Inference
61
selection [14]. In this work, all the points near the landmarks are classified by linear regression to predict whether they lie in or out of a shape. 4.1 Experiments on Cheek Contour
A designed task to directly search a cheek contour without eyes, brow, mouth, and nose is presented to validate our method. A total of 25 cheek landmarks are labeled manually on each image. The PCA thresholds are set to 99% for every ASMs. The fixed shape is composed of points between two eyes and mouth. As in Fig.3, it is difficult to locate points around landmarks near ears and jaw. When a contour shape is simple and textures around landmarks are complex, the whole shape will be dragged off from the right position if there are several inaccurate points. It is clear that the cheek shape can be accurately located with the help of the fixed shape.
Fig. 5. Comparison of different algorithms’ cheek searching results: Shapes in first column are results of ASM searching; Shapes in second column are results of simple OF-ASM; Shapes in third column are results of the basic GCASM; Shapes in fourth column are results of GCASM with optimal features; Shapes in fifth column are results of GCASM with optimal features and Bayesian inference
As in Fig.5, first two columns are the searching results of ASM and OF-ASM. It is clear that the searching results miss desired position because of local noise. Several inaccurate landmarks will drag the shape from desired position. It also illustrates that optimal features can model contour appearance more accurately. As illustrated in the last three columns in Fig.5, searching results are well trapped in a local area when the fixed shape is introduced. Because the fixed shape is accurate without noise, reconstructed shape will fall into a local area around the fixed shape even if some landmarks are inaccurate. Every landmark will find a local best matched point instead of a global one. Comparing the third and fourth column, we can learn about that optimal features can locate landmarks more accurately. But optimal
62
R. He et al.
features couldn’t keep local contour detail very well. There is still some noise in searching results. Looking at the fifth column of Fig.5, it is clear that borders of the shapes become smoother. The Bayesian inference can further improve the accuracy. 4.2 Experiments on Facial Contour
A total of 96 face landmarks are labeled manually on each image. The PCA thresholds are set to 95% for every ASMs. Three landmarks are inserted into two eyes to present horizontal connected line. And three landmarks are inserted between mouth and horizontal line to present the vertical line. For the sake of simplicity, optimal features don’t used in this subsection. The results are shown in table 1. Table 1. Comparison results of traditional ASM and our method without optimal features
ASM Our algorithm Improvement
Face 7.74 4.68 39.5%
F.S.O 6.45 4.41 31.6%
Cheek Contour 11.4 5.47 52.0%
Where F.S.O. means five sense organs. Location error is measured in pixel. It is clear that our algorithm is much more accurate than ASM.
Fig. 6. Comparison results of ASM and GCASM with Bayesian inference. The first row is ASM results, and the second row is our results.
Fig.6 shows a set of searching results of basic ASM and GCASM with Bayesian inference. In the case, there are wrinkles and shadings on the facial contour or other facial sub-parts. It is clear that our method can recover the shape from local noise. A direct reason is that the shape variation is restricted in a local area when combining accurate information in ASM. The Bayesian inference holds the whole shape and smoothes the shape border.
Coarse-to-Fine Statistical Shape Model by Bayesian Inference
63
5 Conclusion This work focuses on an interesting topic how to combine some accurate informationgiven by user or machine to further improve shape alignment accuracy. The PDM is extended by adding a fixed shape which is generated from given information. After PCA reconstruction, local noise of the active shape will make the whole shape unsmooth. Hence Bayesian inference is proposed to further normalize parameters of the extended PDM. Both compensate factor and smooth factor lead a coarse-to-fine shape adjustment. Comparisons of our algorithm and the ASM algorithms demonstrate the effectiveness and efficiency.
Acknowledgements This work was supported by the following funding resources: National Natural Science Foundation Project #60518002, National Science and Technology Supporting Platform Project #2006BAK08B06, National 863 Program Projects #2006AA01Z192 and #2006AA01Z193, Chinese Academy of Sciences 100 people project, and the Authen-Metric Collaboration Foundation.
References 1. Cootes, T.F., Taylor, C.J., Cooper, D., Graham, J.: Active shape models-Their training and application. Comput. Vis. Image Understanding 61(1), 38–59 (1995) 2. Cootes, T.F., Taylor, C.J.: Statistical models of appearance for computer vision, Wolfson Image Anal. Unit, Univ. Manchester, Manchester, U.K., Tech. Rep (1999) 3. Zhou, Y., Gu, L., Zhang, H.-J.: Bayesian tangent shape model: Estimating shape and pose parameters via Bayesian inference. In: IEEE Conf. on Computer Vision and Pattern Recognition, Madison, WI (June 2003) 4. Liang, L., Wen, F., Xu, Y.Q., Tang, X., Shum, H.Y.: Accurate Face Alignment using Shape Constrained Markov Network. In: Proc. CVPR (2006) 5. Li, Y.Z., Ito, W.: Shape parameter optimization for Adaboosted active shape model. In: ICCV, pp. 259–265 (2005) 6. Brox, T., Rosenhahn, B., Weickert, J.: Three-Dimensional Shape Knowledge for Joint Image Segmentation and Pose Estimation. In: Kropatsch, W.G., Sablatnig, R., Hanbury, A. (eds.) Pattern Recognition. LNCS, vol. 3663, pp. 109–116. Springer, Heidelberg (2005) 7. Ginneken, B.V., Frangi, A.F., Staal, J.J., ter Har Romeny, B.M., Viergever, M.A.: Active shape model segmentation with optimal features. IEEE Transactions on Medical Imaging 21(8), 924–933 (2002) 8. Sukno, F., Ordas, S., Butakoff, C., Cruz, S., Frangi, A.F.: Active shape models with invariant optimal features IOF-ASMs. In: Kanade, T., Jain, A., Ratha, N.K. (eds.) AVBPA 2005. LNCS, vol. 3546, pp. 365–375. Springer, Heidelberg (2005) 9. Zhang, S., Wu1, L.F., Wang, Y.: Cascade MR-ASM for Locating Facial Feature Points. The 2nd International Conference on Biometrics (2007) 10. Dryden, I., Mardia, K.V.: The Statistical Analysis of Shape. Wiley, London, U.K (1998)
64
R. He et al.
11. Goodall, C.: Procrustes methods in the statistical analysis of shapes. J.Roy. Statist. 53(2), 285–339 (1991) 12. Messer, K., Matas, J., Kittler, J., Luettin, J., Maitre, G.: XM2VTSDB: The extended M2VTS database. In: Proc. AVBPA, pp. 72–77 (1999) 13. Hamarneh, G.: Active Shape Models with Multi-resolution, http://www.cs.sfu.ca/~hamarneh/ software/asm/index.html 14. Kudo, M., Sklansky, J.: Comparison of algorithms that select features for pattern classiers. Pattern Recognition, 25–41 (2000)
Efficient Texture Representation Using Multi-scale Regions Horst Wildenauer1 , Branislav Miˇcuˇs´ık1,2, and Markus Vincze1 Automation and Control Institute Institute of Computer Aided Automation, PRIP Group, Vienna University of Technology, Austria 1
2
Abstract. This paper introduces an efficient way of representing textures using connected regions which are formed by coherent multi-scale over-segmentations. We show that the recently introduced covariancebased similarity measure, initially applied on rectangular windows, can be used with our newly devised, irregular structure-coherent patches; increasing the discriminative power and consistency of the texture representation. Furthermore, by treating texture in multiple scales, we allow for an implicit encoding of the spatial and statistical texture properties which are persistent across scale. The meaningfulness and efficiency of the covariance based texture representation is verified utilizing a simple binary segmentation method based on min-cut. Our experiments show that the proposed method, despite the low dimensional representation in use, is able to effectively discriminate textures and that its performance compares favorably with the state of the art.
1
Introduction
Textures and structured patterns are important cues towards image understanding, pattern classification and object recognition. The analysis of texture properties and their mathematical and statistical representation is attracting the interest of researchers since many years; with the primary goal of finding low dimensional and expressive representations that allow for reliable handling and classification of texture patterns. Texture representations, which have been successfully applied to image segmentation tasks, include steerable filter responses [1], color changes in a pixel’s neighborhood [2], covariance matrices of gradients, color, and pixel coordinates [3], Gaussian Mixture Models (GMM) computed from color channels [4,5], color histograms [6], or multi-scale densities [7,8]. Since textures ”live” at several scales, a scale-dependent discriminative treatment should be aimed for. In this paper, we explore the possibility to refine coarse texture segmentation by matching textures between adjacent scales, taking
The research has been supported by the Austrian Science Foundation (FWF) under the grant S9101, and the European Union projects MOVEMENT (IST-2003-511670), Robots@home (IST-045350), and MUSCLE (FP6-507752).
Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 65–74, 2007. c Springer-Verlag Berlin Heidelberg 2007
66
H. Wildenauer, B. Miˇcuˇs´ık, and M. Vincze
(a)
(c)
(b)
(d)
(e)
Fig. 1. Segmentation results using the min-cut algorithm [12] with different texture representations. (a) Input image with user specified foreground and background markers. (b) The proposed multi-scale texture representation. (c) Color Histograms [6]. (d) GrabCut [4] using GMMs. (e) Color changes in the pixel neighbourhoods [2].
advantage of spatial and statistical properties which persist across scale. We show that texture segments can be efficiently treated in a multi-scale hierarchy similarly to [8], however building on superpixels. In our approach, textures are represented by covariance matrices, for which an effective similarity measure based on the symmetric generalized eigenproblem was introduced in [9]. In contrast to the rectangular windows used in [3], covariance matrices are computed from irregular structure-coherent patches, found at different scales. In order to allow for an efficient image-partitioning into scalecoherent regions we devised a novel superpixel method, utilizing watershed segmentations imposed by extrema of an image’s mean curvature. However, the suggested framework of multi-scale texture representation is generally applicable for other superpixel methods, such as [10,11], depending on accuracy and time complexity constraints imposed by the application domain. We verify the feasibility and meaningfulness of the multi-scale covariance based texture representation by a binary segmentation method based on the min-cut algorithm [12]. Figure 1 shows an example of how different types of texture descriptors influence the min-cut segmentation of a particularly challenging image, consisting of textured regions with highly similar color characteristics. The remainder of the paper is organized as follows. We present the details of the proposed method in Section 2. Section 3 reports experimental results and compares them to the results obtained using state-of-the-art methods. The paper is concluded with a discussion in Section 4.
Efficient Texture Representation Using Multi-scale Regions
2 2.1
67
Our Approach Superpixels
Probably one of the most commonly used blob detectors is based on the properties of the Laplacian of Gaussians (LoG) or its approximation, the Difference of Gaussians (DoG) [13]. Given a Scale-Space representation L(t) obtained by repeatedly convolving an input image by Gaussians of increasing sizes t, the shape of the intensity surface around a point x at scale t can be described using the Hessian matrix L (x, t) Lxy (x, t) . (1) H(x, t) = xx Lxy (x, t) Lyy (x, t) The LoG corresponds to the trace of the Hessian: ∇2 L(x, t) = Lxx (x, t) + Lyy (x, t),
(2)
and equals the mean intensity curvature multiplied by two. The LoG’s computation results in strong √ negative or positive responses for bright and dark blob-like structures of size t respectively. Using this, the position and characteristic scale of blobs can by found by detecting Scale-Space extrema of scale normalized LoG responses [14]. In our approach, we do not directly search for blob positions and scales, but rather use spatial response extrema as starting points for a watershed-based oversegmentation of an image’s mean curvature surface. Specifically, we proceed as follows: √ 1. Computation of LoG responses at scales of t = 2m/3 , with m = 1 . . . M , where M denotes a predefined number of scales. I.e., we calculate 3 scale levels per Scale-Space octave. 2. Watershed-segmentation: (a) Detection of spatial response extrema at all scales. Extrema with low contrast, i.e. those with a minimum absolute difference to adjacent pixels smaller than a predefined threshold, are discarded. (b) At each scale, segment the image into regions assigned to positive or negative mean curvature. This is achieved by applying the watershed to the negative absolute Laplacian −|∇2 L(x, t)| using the seeds from (a). The majority of watersheds thus obtained follow the zero-crossings of the Laplacian; i.e., the edges where the mean curvature of the intensity surface changes its sign. Though, for irregularly shaped blobs, which exhibit significant variations in mean curvature, usually several seed-points are detected. This results in an over-segmentation of regions with otherwise consistent curvature signs. Figure 2 shows a direct comparison of the superpixels produced by our method at a single scale and the normalized-cut based superpixels suggested in [10]. Another method for image over-segmentation, which is partially favoured for its speed, utilizes the Minimum Spanning Tree [11]. However, for larger superpixels, which are needed to stably compute the covariance-based descriptor
68
H. Wildenauer, B. Miˇcuˇs´ık, and M. Vincze
Fig. 2. Left: Superpixels obtained by the proposed method. Right: Superpixels obtained by the method of Ren et al. [10].
on, the regions obtained by this method are highly irregular and often do not align well with object boundaries. Figure 3 shows the effect of using different superpixels in conjunction with the method proposed in this paper. As one can see, our method gives acceptable results compared to the normalized-cut based approach, which needs more than 100 times longer to compute the segmentation. The outlined approach is similar in spirit to the watershed segmentation of principal curvature images proposed by Deng et al. [15]. In their approach, the first principle curvature image (i.e., the image of the larger eigenvalue of the Hessian matrix) is thresholded near zero and either the positive, or negative remainder is flooded starting from the resulting zero-valued basins. Hence, as opposed to our method, the watersheds follow the ridges of the image’s principal curvature surface. In experiments we found that this approach was not suitable for our purposes since it tends to under-segment images, aggressively merging regions with same-signed principal curvature. 2.2
Covariance-Based Texture Similarity
Recently, Tuzel et al. [3] have introduced region covariance matrices as potent, low-dimensional image patch descriptors, suitable for object recognition and texture classification. One of the authors’ main contributions was the introduction of an integral-image like preprocessing stage, allowing for the computation of covariances from image features of an arbitrarily sized rectangular window in constant time. However, in the presented work covariances are directly obtained from irregularly shaped superpixels, the aforementioned fast covariance computation is not applicable. We proceed to give a brief description of the covariance-based texture descriptor in use. The sample covariance matrix of feature vectors collected inside a superpixel is give by: M=
N 1 (z n − μ)(z n − μ) , N − 1 n=1
(3)
Efficient Texture Representation Using Multi-scale Regions
69
Fig. 3. Effect of superpixels on the final image segmentation. From left to right: Superpixels through a color-based Minimum Spanning Tree [11]. The proposed approach. Superpixels based on normalized-cuts using combined color and textured gradient [10].
where μ denotes the sample mean, and {z n }n=1...N are the d-dimensional feature vectors extracted at N pixel positions. In our approach, these feature vectors are composed of the values of the RGB color channels R, G, and B and the absolute values of the first derivatives of the Intensity I at the n-th pixel ∂I ∂I (4) z n = Rn , Gn , Bn , , . ∂x ∂y The resulting 5×5 covariance matrix gives a very compact texture representation with the additional advantage of exhibiting a certain insensitivity to illumination changes. And, as will be shown experimentally, offers sufficient discriminative power for the segmentation task described in the remainder of the paper. To measure the similarity ρ(M i , M j ) of two covariance matrices M i and M j we utilize the distance metric initially proposed by F¨ orstner [9]: d ρ(M i , M j ) = ln λ2k (M i , M j ), (5) k=1
where the {λk }k=1...d are the eigenvalues obtained by solving the generalized eigenvalue problem (6) M i ek = λk M j ek , k = 1 . . . d with ek = 0 denoting the generalized eigenvectors. The cost for computing ρ is in the order of O(d3 ) flops which, due to the low dimensionality of the representation, leads to speed advantages compared to histogram matching methods. For a detailed discussion among the topic, other choices of feature combinations as well as the useful properties of region covariances see [3]. 2.3
Foreground and Background Codebooks
From the covariance-based descriptors proposed in Subsection 2.2 we compute representative codebooks for foreground and background regions. These are used later on to drive the image segmentation.
70
H. Wildenauer, B. Miˇcuˇs´ık, and M. Vincze
Sink
Source
Fig. 4. Two-layer MRF with superpixels detected at two different scales. To avoid clutter not all superpixels are connectd to sink/source nodes.
For the foreground and background codebook, we require user-specified markB ers, as shown in Figure 1(a), computing the covariance matrices M F i and M i from all points under the marker region. Usually, the background contains more textures and cluttered areas requiring more seeds to be establish. Moreover, in applications like object detection or recognition, the background can significantly vary across images while objects of interest usually remain quite consistent in appearance. To somewhat alleviate the burden of manually selecting many seeds, we propose to avoid the need of background markers by following a simple strategy: We take a rim at the boundary of the image and feed all superpixels under the rim into a hierarchical clustering method with a predefined stopping distance threshold, with the distance between superpixels given by Equation 5. After clustering we take the K most occupied clusters and compute the mean covariance matrix for each cluster out of all covariance matrices belonging to the cluster. For efficiency reasons, we do not calculate the mean covariance matrix by polling over all participating feature vectors, but use the method described in [16,3], which has its roots in formulations of symmetric positive definite matrices lying on connected Riemannian manifolds. Using this procedure, we arrive at the background codebook matrices M B i . Of course, the applicability of this ad-hoc technique is limited to cases where the object of interest touches the boundary, or when the rim is not representative enough. However, in most cases the approach lead to background codebooks with sufficient explanatory power for a successful segmentation. 2.4
Multi-scale Graph-Cut
In order to verify the validity of the covariance-based texture representation, taking into account the superpixel behaviour across different scales we adopted a binary segmentation method based on the min-cut algorithm [12]. Suppose that the image at a certain scale t is represented by a graph Gt = Vt , Et , where Vt is a set of all vertices representing superpixels, and Et is a set of all intrascale edges connecting spatially adjacent vertices. To capture the Scale-Space behaviour we connect the graphs by interscale edges forming a set of edges S. We form the entire graph G = V, E, S consisting of the union of
Efficient Texture Representation Using Multi-scale Regions
71
all vertices Vt , and all intrascale Et and interscale edges S. For more clarity, the resulting graph structure is depicted in Figure 4. The binary segmentation of the graph G is achieved by finding the minimumcut [12], minimizing the Gibbs energy, Edata (xi , M i ) + λ δ(xi , xj ) Esm im (M i , M j )+ E(x) = i∈V
(i,j)∈E
+γ
δ(xi , xj ) Esm
sc (M i , M j ),
(7)
(i,j)∈C
where x = [x0 , x1 , . . .] corresponds to a vector with label xi for each vertex. We concentrate on a bi-layer segmentation where the label xi is either 0 (background) or 1 (foreground). M i corresponds to the measurement in the i-th graph vertex, i.e., to a covariance matrix for a given superpixel. The weight constants λ, γ control the influence of the image (intrascale), and interscale smoothness terms respectively; δ denotes the Kronecker delta. The data term. describes how likely the superpixel is foreground or background. The data term for the foreground is defined as l(M i , F ) , (8) l(M i , F ) + l(M i , B)
where l(Mi , F ) = exp − mink=1...|F | ρ(Mi , MkF )/(2σ12 ) stands for the foreground likelihood of the superpixel i. M F k denotes the k-th covariance matrix from a foreground codebook set F , and σ1 is an experimentally determined parameter. As the derivation of the background terms and likelihoods follows analogously, we will omit its description. Edata (xi = 1, M i ) =
The smoothness term. describes how strongly neighborhood pixels are bound together. There are two types of the smoothness terms, see Equation (7), one for intrascale neighborhoods, Esm im , one for interscale neighborhoods, Esm sc . The intrascale smoothness term using α blending is defined as Esm im (Mi , Mj ) = α exp − ρ(M i , M j )/(2σ22 ) +
2 2 + (1 − α) exp − l(M i , F ) − l(M j , F ) /(2σ3 ) , (9) where σ2 and σ3 are pre-defined parameters. The interscale smoothness term is only defined for edges between two vertices from neighboring scales when the corresponding superpixels share at least one same image pixel. The weight on the edge between superpixels i and j from consecutive scales is set to Esm sc (Mi , Mj ) = β
area(i ∩ j) + area(i)
2 2 + (1 − β) exp − l(M i , F ) − l(M j , F ) /(2σ3 ) . (10)
72
H. Wildenauer, B. Miˇcuˇs´ık, and M. Vincze
Fig. 5. Importance of inter-scale graph edges. From left to right: Only one lower scale used. Only one higher scale used. Three consecutive scales used.
The second term in both Equations (9), (10) increases the dependency of smoothness terms on the foreground likelihood, making it more robust as originally suggested by [8]. However, we rely on this term only partially through the interpolation parameteres α, β, since a full dependency on the likelihood often resulted in compact, but otherwise incomplete segmentations. Figure 5 shows how the use of multiple scales and inter-scale edges improves the segmentation compared to segmentation performed separately for given scales.
3
Experimental Results
We performed segmentation tests on images from the Berkeley dataset1 . We compare the result to the recent approach proposed by Micusik&Pajdla [2]. Their method looks at color changes in the pixel neighbourhood, yielding superior results on textured images compared to other methods. For both methods the same manually established foreground and background markers were used. To guarantee a fair comparison, the automatic background codebook creation proposed in Section 2.3 was omitted. We present some results where our proposed method performs superior or comparable to [2]. These images typically contain textures with similar colors and are, as stated in [2], the most crucial for their texture descriptor. One must realize that covariance based texture description cannot cope reliably with homogenoeus color regions, see the missing roof of the hut in Figure 6. This should be kept in mind, and use such a descriptor complementary with some color features. Overall, as experiments show, the newly proposed technique performs very well on textures. The advantage over methods, e.g. [6,4,2], is computational efficienty. Moreover, using more accurate superpixels, e.g. [10], improve the accuracy of the result for the price of higher time consumption.
4
Summary and Conclusion
We present an efficient way of representing textures using connected regions, formed by coherent multi-scale over-segmentations. We show the favourable 1
http://www.cs.berkeley.edu/projects/vision/grouping/segbench
Efficient Texture Representation Using Multi-scale Regions
(a)
(b)
73
(c)
Fig. 6. Segmentation comparison. (a) Input image with user marked seeds. (b) The method from [2]. (c) Our approach.
74
H. Wildenauer, B. Miˇcuˇs´ık, and M. Vincze
performance on segmentation of textures images. However, our primary goal is not to segment images accurately, but to demonstrate the feasibility of the covariance matrix based descriptor used in a multi-scale hierarchy built on superpixels. The method is aimed at a further use in recognition and an image understanding systems where so accurate segmentation is not required.
References 1. Malik, J., Belongie, S., Leung, T., Shi, J.: Contour and texture analysis for image segmentation. IJCV 43(1), 7–27 (2001) 2. Miˇcuˇs´ık, B., Pajdla, T.: Multi-label image segmentation via max-sum solver. In: Proc. CVPR (2007) 3. Tuzel, O., Porikli, F., Meer, P.: Region covariance: A fast descriptor for detection and classification. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 589–600. Springer, Heidelberg (2006) 4. Rother, C., Kolmogorov, V., Blake, A.: ”grabcut”: interactive foreground extraction using iterated graph cuts. In: Proc. ACM SIGGRAPH, pp. 309–314. ACM Press, New York (2004) 5. Kolmogorov, V., Criminisi, A., Blake, A., Cross, G., Rother, C.: Probabilistic fusion of stereo with color and contrast for bi-layer segmentation. PAMI 28(9), 1480–1492 (2006) 6. Boykov, Y., Jolly, M.P.: Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images. In: Proc. ICCV, pp. 105–112 (2001) 7. Hadjidemetriou, E., Grossberg, M., Nayar, K.S.: Multiresolution histograms and their use for recognition. PAMI 26(7), 831–847 (2004) 8. Turek, W., Freedman, D.: Multiscale modeling and constraints for max-flow/mincut problems in computer vision. In: Proc. CVPR Workshop, vol. 180 (2006) 9. F¨ orstner, W., Moonen, B.: A metric for covariance matrices. Technical report, Dpt. of Geodesy and Geoinformatics, Stuttgart University (1999) 10. Ren, X., Malik, J.: Learning a classification model for segmentation (2003) 11. Felzenszwalb, P., Huttenlocher, D.: Efficient graph-based image segmentation. IJCV 59(2), 167–181 (2004) 12. Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. PAMI 26(9), 1124–1137 (2004) 13. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004) 14. Lindeberg, T.: Scale-Space Theory in Computer Vison. Kluwer Academic Publishers, Dordrecht (1994) 15. Deng, H., Zhang, W., Diettrich, T., Shapiro, L.: Principal curvature-based region detector for object recognition. In: Proc. CVPR (2007) 16. Pennec, X., Fillard, P., Ayache, N.: A riemannian framework for tensor computing. International Journal of Computer Vision 66(1), 41–66 (2006)
Comparing Timoshenko Beam to Energy Beam for Fitting Noisy Data Ili´c Slobodan Deutsche Telekom Laboratories Berlin University of Technology Ernst-Reuter Platz 7, 14199 Berlin, Germany
[email protected]
Abstract. In this paper we develop highly flexible Timoshenko beam model for tracking large deformations in noisy data. We demonstrate that by neglecting some physical properties of Timoshenko beam, classical energy beam can be derived. The comparison of these two models in terms of their robustness and precision against noisy data is given. We demonstrate that Timoshenko beam model is more robust and precise for tracking large deformations in the presence of clutter and partial occlusions. The experiments using both synthetic and real image data are performed. In synthetic images we fit both models to noisy data and use Monte Carlo simulation to analyze their performance. In real images we track deformations of the pole vault, the rat whiskers and the car antenna.
1 Introduction In this paper, we develop true physical 2D Timoshenko beam model and use it for tracking large deformations in noisy image data. Timoshenko beam relays on shear deformation to account for non-linearities. We derive from it physically based energy beam, by neglecting shear deformation. The models which closely approximate real physics we call true physical models, in this case Timoshenko beam, while the models which are designed to retain some physical properties we call physically based models, in this case energy beam. Physically based models introduced almost twenty years ago [1,2,3,4] have demonstrated their effectiveness in the Computer Vision problems. However, they typically rely on simplifying assumptions to yield easy to minimize energy functions and ignore the complex non-linearities that are inherent to large deformations present in highly flexible structures. To justify the use of complex true physical models over simplified physically based models we compare Timoshenko beam to energy beam. Both models were fitted to noisy synthetic data and real images in presence of clutter and partial occlusions. We demonstrate that using fully non-linear Timoshenko beam model, which approximates the physical behavior more closely, yields to more robust and precise fitting to noisy data and tracking of large deformations in the presence of clutter and partial occlusions.
The rat whiskers images shown in this paper were obtained at Harvard University’s School of Engineering and Applied Sciences by Robert A. Jenks.
Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 75–84, 2007. c Springer-Verlag Berlin Heidelberg 2007
76
I. Slobodan
The fitting algorithm used for both beam representations is guided by the image forces. Since the image forces are proportional to the square distance of the model to the image data, they are usually not sufficient to deform Timoshenko beam model of known material properties, so that it fits the data immediately. The image forces only move the model toward the image observations. To fit the model to the data we repeat Gauss-Newton minimization in several steps. We stop when the distance of the model to the data of two consecutive Gauss-Newton minimizations is smaller then some given precision. We use Levenberg-Marquardt optimizer to fit quadratic energy function of the energy beam. In this case image forces are sufficient to deform the beam in a single minimization because the model energy, being only the approximation of the real model strain energy, does not impose realistic physical restrictions to the beam deformations. In the reminder of the paper we give a brief overview of the know physically based techniques, then introduce non-linear Timoshenko planar beam model, derive energy beam from it, then describe our fitting algorithm and finally present the results.
2 Related Work Recovering model deformations from images requires deformable models to constrain the search space and make the problem tractable. In recent decade deformable models were exploited in Computer Graphics [5], Medical Imaging [6] and Computer Vision [7]. There are several important categories of physically based models: mass-spring models [8], finite element models (FEM) [9,10], snake like models [1,3,4] and models obtained from FEM, to reduce number of dofs, by modal analysis [11,2,12,13]. In this paper we are particularly interested in physical models, especially those based on FEM. FEM are known to be precise and to produce physically valid deformations. However, because of their mathematical complexity the FEM were mainly developed for small linear deformations [14] where the model stiffness matrix is constant. In case of large deformations the stiffness matrix and the applied forces become the function of the displacement. Such non-linear FEM were used by [15] to recover material parameters from images of highly elastic shell like models. The model deformation was measured form 3D model scan and then given to the Finite Element Analysis (FEA) software. By contract we develop non-linear beam equations and recover the model deformations automatically through optimization. In computer vision physically based models, based on the continuous energy function were extensively used. The original ones [1] were 2D and have been shown to be effective for 2D deformable surface registration [16]. They were soon adapted for 3D surface modeling purposes by using deformable superquadrics [3,4], triangulated surfaces [2], or thin-plate splines [6]. In this framework, modeling generic structures often requires many degrees of freedom that are coupled by regularization terms. In practice, this coupling implicitly reduces the number of degrees of freedom, which makes these models robust to noise and is one of the reasons for their immense popularity. In this paper we compare complex true physical 2D Timoshenko beam model to 2D elastic energy beam in terms of their robustness against noisy data. We reveal, in spite of their complexity, the real benefits of true physical models.
Comparing Timoshenko Beam to Energy Beam for Fitting Noisy Data
77
3 Plane Timoshenko Beam Model The beam represents the most common structural component in civil and mechanical engineering. A beam is a rod-like — meaning that one dimension is considerably larger then the other two — structural element that resists transversal loading applied between its supports. Timoshenko beam, we develop, assumes geometrically large deformations under small strains with the linearly elastic materials. Timoshenko beam theory [17] accounts for nonlinear effect, such as shear assuming that the cross-section does not remains normal to the deformed longitudinal axis. The beam is divided to a number of finite elements. The beam deformation is defined by 3 dof per node being the axial displacement uX (X), the transverse displacement uY (X) and the cross-section rotation θ(X), where X is longitudinal coordinate in the reference configuration as shown in Fig 1. The undeformed initial configuration is referred as reference and the deformed one as current configuration. The parameters describing beam geometry and the material properties are A0 cross-sectional area, L0 element length in reference configuration, L element length in deformed configuration, I0 second moment of inertia, E Young modulus of elasticity, and G is shear modulus. The material remains linearly elastic. The beam rotation is defined by the angle ψ also equal to the rotation of the cross-section. The angle γ¯ is a shear angle for which the cross section deviates from its normal position defined by the angle ψ. The total rotation of a beam cross section becomes θ = ψ − γ¯ which is exactly one of dofs defined above. To describe beam kinematics we consider the motion of the point P0 (X, Y ) in reference configuration to the point P (x, y) in the current configuration. We keep the assumption that the cross section dimensions do not change and that the shear rotation
(a)
(b)
Fig. 1. (a) Plane Timoshenko beam kinematics notation. (b)Synthetic example of fitting the plane beam, initially aligned along the x-axis , to the synthetic image data shown as magenta dots. The intermediate steps shown in blue are the output of number of repeated Gauss-Newton optimizations driven by the image forces.
78
I. Slobodan
is small γ¯ 1 so that cosγ ≈ 1. The Lagrangian representation of the motion relating points P0 (X, Y ) and P (x, y) is then given by x = X +uX −Y sin θ, y = uY +Y cos θ. The displacement of any point on the beam element can be than represented by a vec tor w = uX (X) uY (X) θ(X) . In the FEM formulation for 2-node C 0 element it is natuaral to express the displacements and rotation functions of w as linear 2 2 2combination of node displacements uX = i=1 Ni uXi , vX = i=1 Ni vXi , θX = i=1 Ni θXi or in matrix form w = Nu where N1 = 12 (1 − ξ), N2 = 12 (1 + ξ) are the linear element shape functions. The strain is a measure of the change of the object shape, in this case the length, before and after the deformation caused by some applied load. The stress is the internal distribution of force per unit area that balances and reacts to external loads applied to a body. We have three different stress components per beam element: e axial strain measuring the beam relative extension, γ shear strain measuring the relative angular change between any two lines in a body before and after the deformation and κ measuring the curvature change. They can be computed from the deformation gradient of motion F =
∂x ∂x ∂X ∂Y ∂y ∂y ∂X ∂Y 1 T 2 (F F
. Green-Lagrange(GL) strain tensor describing the model strain
becomes e = − I). After the derivation the only nonzero elements are axial strain eXX and the shear strain 2eXY = eXY + eY X . Under small strain assumption we can finally express strain vector as: eXX e−Yκ e1 = = (1) e= e2 2eXY γ where three strain quantities introduced above are e axial strain, γ shear strainand κ curvature.These can be collected in the generalized strain vector hT = e γ κ . Because of the assumed linear variations in X of uX (X), uY (Y ) and θ(X), e and γ depend on θ and κ is constant over the element depending only on rotation angles at element end nodes. e and γ can be expressed in a geometrically invariant form: e=
L L L L cos γ¯ − 1 = cos (θ − ψ) − 1, γ = sin γ¯ = sin (θ − ψ) L0 L0 L0 L0
(2)
These geometrically invariant strain quantities can be used for the beam in arbitrary reference configuration. The variations of δe, δγ and δκ with respect to the nodal displacement variations are required for derivations of strain-displacement relation δh = Bδu. To form strain-displacement matrix B we take partial derivatives of e, γ and κ with respect to the node displacements and collect them into matrix B: ⎤ ⎡ cω sω L0 N2 γ −cω −sω L0 N1 γ (3) B = ⎣ sω −cω −L0 N1 (1 + e) −sω cω −L0 N2 (1 + e)⎦ 0 0 −1 0 0 1 where ω = θ + ψ, cω = cos ω and sω = sin ω. We introduce pre-stress resultants N 0 , V 0 and M 0 which define the axial forces, transverse shear forces and bending moments respectively, in the reference configuration. We also define the stress resultants in the current configuration using linear elastic 0 0 equation to be N = N 0 + EA0 e, V = V + GA 0 γ and M = M + EI0 κ, and collect them into stress-resultant vector z = N V M .
Comparing Timoshenko Beam to Energy Beam for Fitting Noisy Data
79
The internal model strain energy along the beam, under zero pre-stress resultants N 0 = V 0 = M 0 = 0, can be expressed as the length integral: 1 1 1 zT hdX = EA0 e2 dX + GA0 γ 2 dX + EI0 κ2 dX (4) U= 2 2 2 L0 L0 L0 L0 where L0 is the beam length in reference configuration. The internal force vector can be obtained by taking the fist variation of the strain energy with respect to the nodal displacement: ∂U = BT (u)zdX. (5) p= ∂u L0 This expression we evaluate by reduced Gauss integration in order to eliminate shear locking which overstiff the model deformation making a shear energy to dominate. In addition we use residual bending flexibility (RBF) correction and replace GA0 for shear energy of Eq. 4 by 12EI0 /L20 . Finally, the first variation of the internal force defines the tangent stiffness matrix: ∂B ∂z ∂p = + z)dX = (KM + KG ). (BT (6) KT = ∂u ∂u ∂u L0 where KM is material stiffness and KG is geometric stiffness. The material stiffness is constant and identical to the linear stiffness matrix of the C 1 Euler-Bernoulli beam. The geometric stiffness comes the variation of B while stress resultants are kept fixed and caries the beam nonlinearity responsible for large geometric deformations.
4 Energy Beam Model The model energy of the energy beam can be derived directly from the Timoshenko beam strain energy of Eq. 4. Let us neglect the shear deformation by putting shear anγ − 1 ≈ LL0 − 1, gle to be zero γ¯ . The strain quantities of Eq. 2 become e = LL0 cos¯ γ = LL0 sin γ¯ ≈ 0. In this way the shear energy is eliminated and only the axial energy and bending energy are left. Also, since shear is eliminated the rotational dof θ(X) disappears, and only displacements in X and Y directions are taken to form new displacement vector w = [uX uY ]. Since we deal with discrete beam its energy can be expressed as: U=
1 ws 2
(
(i,j)∈1..n
vi − vj 1 − 1)2 + wb L0 2
2vj − vi − vk
2
(7)
(i,j,k)∈1..n
where i, j are pairs of element nodes, and i, j, k are triplets of element nodes necessary to define curvature at the j t h beam node. Derived energy can be considered as physically based since it directly comes from the realistic physical beam model. The weight coefficients ws and wb can be considered proportional to the Young modulus of elasticity E. However, we will show that, in practice, they significantly change behavior of the fitting algorithm.
80
I. Slobodan
5 Model Fitting The general approach in mechanical simulations is that some external load f is applied to the model and the displacement u is computed. This can be done through energy minimization, where the total potential energy Π of the system is computed as the sum of the system internal strain energy U and the energy caused by the external load P . The minimum of the energy in respect to the displacement u can be found by derivation: ∂U ∂P ∂Π = + ⇒ r(u) = p(u) + f = 0 (8) ∂u ∂u ∂u The r(u) is a force residual, p(u) is internal force of Eq. 5 and f is the external load. This is a nonlinear system of equations and is usually solved by using Newton-Raphson method. It is incremental method and at each iteration we solve for the displacement update du by solving linear system KT du = −(f + p(u)). In classical mechanical simulation the external forces f are a priori given. In our case we do not know them. To compute model displacements we solve Eq. 8, using image T forces. We create a vector of image observations F(u) = d1 (u) d2 (u) . . . dN (u) where di (u) are distances of the image observations from the beam segments. We use the edges in the image, obtained using Canny edge detector, to be our observations. In practice we sample every beam segment and then search for the multiple observations in the direction of the beam normal. The external image energy becomes PI = 12 FT (u)F(u). The images forces are obtained as a derivative of the energy in respect to the displacement fI = ∇FT (u)F(u). The force residual of Eq. 8 becomes: r(u) = p(u) + fI (u) = 0. We derive the displacement increment by developing the residual in a Taylor series around the current displacement u as follows: r(u + du) = p(u + du) + fI (u + du) = 0 ∇FT F + (∇2 FT F + ∇FT ∇F)du + p(u) + KT du = 0 (KT + ∇2 FT F + ∇FT ∇F)du = −(∇FT F + p(u)) ≈ (KT + ∇FT ∇F)du = −(∇FT F + p(u))
(9)
we obtain Gauss-Newton optimization step. We neglect the second order term ∇2 FT F of Eq. 9. To make it more robust we use Tukey robust estimator ρ of [18]. This is simply done by weighting the residuals di (u) of the image observation vector F(u) at each Newton iteration: Each di (u) is replaced by di (u) = wi di (u) such that: (di )2 (u) = (wi )2 d2i (u), therefore the weight is chosen to be: wi = ρ(di (u))1/2 /di (u). We then create a weighting matrix W = diag(. . . , wi , . . .). We then solve in each step: (KT + ∇FT W∇F)du = −(∇FT WF + p(u))
(10)
By solving the Eq. 9 we compute the displacement caused by the image forces. Since the image forces are proportional to the square distance of the model to the image edge points, they are not sufficient to deform the model so that it fits the data. They only move the model toward the image observations. To obtain the exact fit of the model to the data we repeatedly fit the model to the data performing Gauss-Newton method in several steps. We stop when the distance of the model to the data of two consecutive Gauss-Newton minimizations becomes smaller then some given precision. The
Comparing Timoshenko Beam to Energy Beam for Fitting Noisy Data
81
optimization algorithm is illustrated on the synthetic example of Fig. 1(b). We obtain the total displacement uT as a sum of all intermediate displacements. For fitting energy beam we slightly modify Eq. 10 by adding λI to the left side of Eq. 10 such that the we obtain Levenberg-Marquart optimizer. In practice it turned out to be more suitable for optimizing energy function of Eq. 7. The computational complexity of our method is quadratic and corresponds to the complexity of the Gauss-Newton minimization.
6 Results We fit the Timoshenko beam and energy beam to synthetic data and compare their performance in presence of different amount of added noise. We then run our experiments on real images in three different cases: the deformation of a car antenna, pole vault and the deformation of the rat whiskers. 6.1 Fitting Synthetic Data We generate the synthetic data clouds around two given ground truth positions of the deformed beams depicted in Fig. 1(b) by adding certain amount of Gaussian noise around them. The amount of noise is controlled by the variance σ ∈ {0.01, 0.1, 0.5, 1.0, 2.0}. We perform a Monte-Carlo simulation such that for each value of σ we fit, in a number of trials, Timoshenko and energy beam to randomly generated data clouds. The number of trials is 100 for every value of σ. We measure mean square error of the fitting result to the ground truth position of the beam. Error measures from ground trought for different values of Young modulus
Error measures from ground trought for different energy weights
0.25 E=1E2 E=1E3 E=1E4 E=1E5 E=1E6
0.06
2.5 ws=1E3,wb=1E1 2
ws=1E1,wb=1E3 ws=1E4,wb=1E2 ws=1E4,wb=1E2
1.5
ws=1E4,wb=1E4
1
0.05
0.04
0.03
0.02
0.15
0.1
0.05
0.5
0.01
0
0.2
0.4
0.6
0.8 1 1.2 1.4 Noise standard derivation σ
1.6
1.8
0
2
0
0.5
1 1.5 Noise standard derivation σ
2
0
2.5
Average mean square error
ws=1E4,wb=1E2 ws=1E4,wb=1E4
3
2
0.8 1 1.2 1.4 Noise standard derivation σ
1.6
1.8
2
Physics Beam Energy Beam
0.16
0.15
0.1
0.05
1
0.6
0.18
Average mean square error
0.2
ws=1E4,wb=1E2 4
0.4
0.2 E=1E2 E=1E3 E=1E4 E=1E5 E=1E6 E=1E7
ws=1E3,wb=1E1 ws=1E1,wb=1E3
0.2
Error in respect to the ground trought vs. different noise levels
0.25 ws=1E0,wb=1E0
5
0
Error measures from ground trought for different values of Young modulus
Error measures from ground trought for different energy weights 6
Average mean square error
Physics Beam Energy Beam 0.2 Average mean square error
Average mean square error
Average mean square error
ws=1E0,wb=1E0
0
Error in respect to the ground trought vs. different noise levels
0.07
3
0.14 0.12 0.1 0.08 0.06 0.04 0.02
0
0
0.2
0.4
0.6
0.8 1 1.2 1.4 Noise standard derivation σ
(a)
1.6
1.8
2
0
0
0.2
0.4
0.6
0.8 1 1.2 1.4 Noise standard derivation σ
(b)
1.6
1.8
2
0
0
0.2
0.4
0.6
0.8 1 1.2 1.4 Noise standard derivation σ
1.6
1.8
2
(c)
Fig. 2. Mean square error measured for number of fittings using Monte Carlo simulation from the ground truth in respect to the different amount of noise. (a) Energy beam fitting errors for different values of energy weights. (b) Timoshenko beam fitting errors for different values of Young modulus of elasticity. (c) Comparison of fitting errors for energy beam shown in red, and Timoshenko beam shown in blue.
82
I. Slobodan
(a)
(b)
(c)
(d)
Fig. 3. Failiour examples using energy beam with different energy weighting coefficients. (a) For wb = 104 , ws = 102 stays smooth but changes its length producing failiour in 5th frame. (b) For wb = 102 , ws = 104 tries to retains its length but no the smoothness producing failiour in 3rd frame. (c,d) The rat whisker fails in the 10th frame because of the occluded ear, and the energy coefficients are also wb = 103 , ws = 10 and ws = 102 , wb = 10 respectively.
Fig. 4. Tracking the car antenna using Temoshenko beam. Selected frames from the tracking sequence with the recovered model shown in white.
Fig. 5. Tracking the pole in a pole vault using Timoshenko beam. Because of the moving camera the image frames are warped to one reference frame using robustly estimated homography. Selected frames from the tracking sequence with the recovered model shown in yellow.
Initially we perform fittings for different values of energy weights ws and wb for energy beam and different values of Young modulus E for Timoshenko beam as shown in Fig. 2(a,b) respectively. The error differs for the different values of the energy weights. We take those values for which the error is minimal and refit the beams to the noisy data with different values of σ. Usually a good balance between stretching and bending energy is required for reasonable fitting performance of the energy beam. For Timoshenko
Comparing Timoshenko Beam to Energy Beam for Fitting Noisy Data
83
Fig. 6. Tracking the deformation of the rat whisker using Timoshenko beam. Selected frames from the tracking sequence with the recovered model are shown in white.
beam the small values of Young modulus ranging from 102 to 103 are unrealistic for materials with the small strains to which the Timoshenko theory applies. It means that small values of Young modulus are suitable for elastic materials with the large strains, while big values of Young modulus are suitable for elastic materials which have small strains, i.e tend to retain their length but can have large rotations. For that reasons we obtained the best fitting performance for values of E being 105 and 106 . The errors in respect to the ground truth for both beams in two synthetic examples are shown in Fig. 2(c). Timoshenko beam retains the same error measure with the increase of the amount of noise, while the error for the energy beam increases with the increase of the amount of added noise. This indicates that Timoshenko beam is more robust when fitted to noisy data. The same is proven bellow during tracking in real images. 6.2 Real Images In real images we chose to track highly flexible structures such as a car antenna, pole vault and the deformation of the rat whiskers. The car antenna example of Fig. 4 has simple background and both Timoshenko and energy beam track it with no problems as can be seen in supplementary videos. More complex pole vault and rat whiskers were successfully tracked using Timoshenko beam while they failed when energy beam was used. The failiour examples are depicted in Fig. 3. The selected frames form successful tracking using Timoshenko beam are depicted in Fig. 5 and Fig 6. In all examples the initialization was done manually in the first frame and then the frame to frame fitting was done. The energy beam has tendance to attach to the strong edges regardless the combination of the energy weights as depicted by Fig. 3, while the Timoshenko beam overcomes this problem because its naturally imposed physical constrains implicitly contained in the model description.
7 Conclusion In this paper we investigated true physical Timoshenko beam model to track large non-linear deformations in images. We compared it to physically based energy beam
84
I. Slobodan
approach which uses simplifying physical assumptions to create the model energy, similar to most physically based models used in computer vision. These approaches ignore the complex non-linearities that are inherent to large deformations. We discovered that using Timoshenko beam, which approximates the physical behavior more closely, contributed to robust fitting to noisy data and efficient tracking of large deformations in the presence of clutter and partial occlusions.
References 1. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active Contour Models. International Journal of Computer Vision 1(4), 321–331 (1988) 2. Cohen, L., Cohen, I.: Deformable models for 3-d medical images using finite elements and balloons. In: Conference on Computer Vision and Pattern Recognition, pp. 592–598 (1992) 3. Terzopoulos, D., Metaxas, D.: Dynamic 3D models with local and global deformations: Deformable superquadrics. IEEE Transactions on Pattern Analysis and Machine Intelligence 13, 703–714 (1991) 4. Metaxas, D.T.D.: Constrained deformable superquadrics and nonrigid motion tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence 15(6), 580–591 (1993) 5. Gibson, S., Mirtich, B.: A survey of deformable modeling in computer graphics. Technical report, Mitsubishi Electric Research Lab, Cambridge, MA (1997) 6. McInerney, T., Terzopoulos, D.: Deformable models in medical images analysis: a survey. Medical Image Analysis 1(2), 91–108 (1996) 7. Metaxas, D.: Physics-Based Deformable Models: Applications to Computer Vision, Graphics, and Medical Imaging. Kluwer Academic Publishers, Dordrecht (1996) 8. Lee, Y., Terzopoulos, D., Walters, K.: Realistic modeling for facial animation. In: SIGGRAPH 1995. Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pp. 55–62. ACM Press, New York (1995) 9. Essa, I., Sclaroff, S., Pentland, A.: Physically-based modeling for graphics and vision. In: Martin, R. (ed.) Directions in Geometric Computing. Information Geometers, U.K (1993) 10. Sclaroff, S., Pentland, A.P.: Physically-based combinations of views: Representing rigid and nonrigid motion. Technical Report 1994-016 (1994) 11. Pentland, A.: Automatic extraction of deformable part models. International Journal of Computer Vision 4(2), 107–126 (1990) 12. Delingette, H., Hebert, M., Ikeuchi, K.: Deformable surfaces: A free-form shape representation. SPIE Geometric Methods in Computer Vision 1570, 21–30 (1991) 13. Nastar, C., Ayache, N.: Frequency-based nonrigid motion analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 18(11) (1996) 14. O’Brien, J.F., Cook, P.R., Essl, G.: Synthesizing sounds from physically based motion. In: Fiume, E. (ed.) SIGGRAPH 2001. Computer Graphics Proceedings, pp. 529–536 (2001) 15. Tsap, L., Goldgof, D., Sarkar, S.: Fusion of physically-based registration and deformation modeling for nonrigid motion analysis (2001) 16. Bartoli, A., Zisserman, A.: Direct Estimation of Non-Rigid Registration. In: British Machine Vision Conference, Kingston, UK (2004) 17. Timoshenko, S., MacCullogh, G.: Elements of Strength in Materials, 3rd edn., van Nostrand. New York (1949) 18. Lepetit, V., Fua, P.: Monocular model-based 3d tracking of rigid objects: A survey. Foundations and Trends in Computer Graphics and Vision 1(1), 1–89 (2005)
A Family of Quadratic Snakes for Road Extraction Ramesh Marikhu1 , Matthew N. Dailey2 , Stanislav Makhanov3 , and Kiyoshi Honda4 Information and Communication Technologies, Asian Institute of Technology Computer Science and Information Management, Asian Institute of Technology 3 Sirindhorn International Institute of Technology, Thammasat University 4 Remote Sensing and GIS, Asian Institute of Technology
1 2
Abstract. The geographic information system industry would benefit from flexible automated systems capable of extracting linear structures from satellite imagery. Quadratic snakes allow global interactions between points along a contour, and are well suited to segmentation of linear structures such as roads. However, a single quadratic snake is unable to extract disconnected road networks and enclosed regions. We propose to use a family of cooperating snakes, which are able to split, merge, and disappear as necessary. We also propose a preprocessing method based on oriented filtering, thresholding, Canny edge detection, and Gradient Vector Flow (GVF) energy. We evaluate the performance of the method in terms of precision and recall in comparison to ground truth data. The family of cooperating snakes consistently outperforms a single snake in a variety of road extraction tasks, and our method for obtaining the GVF is more suitable for road extraction tasks than standard methods.
1
Introduction
The geographic information system industry would benefit from flexible automated systems capable of extracting linear structures and regions of interest from satellite imagery. In particular, automated road extraction would boost the productivity of technicians enormously. This is because road networks are among the most important landmarks for mapping, and manual marking and extraction of road networks is an extremely slow and laborious process. Despite years of research and significant progress in the computer vision and image processing communities (see, for example, [1,2] and Fortier et al.’s survey [3]), the methods available thus far have still not attained the speed and accuracy necessary for practical application in GIS tools. Among the most promising techniques for extraction of complex objects like roads are active contours or snakes, originally introduced by Kass et al. [4]. Since the seminal work of Kass and colleagues, techniques based on active contours have been applied to many object extraction tasks [5] including road extraction [6]. Rochery et al. have recently proposed higher-order active contours, in particular quadratic snakes, which hold a great deal of promise for extraction of linear Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 85–94, 2007. c Springer-Verlag Berlin Heidelberg 2007
86
R. Marikhu et al.
structures like roads [7]. The idea is to use a quadratic formulation of the contour’s geometric energy to encourage anti-parallel tangents on opposite sides of a road and parallel tangents along the same side of a road. These priors increase the final contour’s robustness to partial occlusions and decrease the likelihood of false detections in regions not shaped like roads. In this paper, we propose two heuristic modifications to Rochery et al.’s quadratic snakes, to address limitations of a single quadratic snake and to accelerate convergence to a solution. First, we introduce the use of a family of quadratic snakes that are able to split, merge, and disappear as necessary. Second, we introduce an improved formulation of the image energy combining Rochery et al.’s oriented filtering technique [7] with thresholding, Canny edge detection, and Xu and Prince’s Gradient Vector Flow (GVF) [8]. The modified GVF field created using the proposed method is very effective at encouraging the quadratic snake to snap to the boundaries of linear structures. We demonstrate the effectiveness of the family of snakes and the modified GVF field in a series of experiments with real satellite images, and we provide precision and recall measurements in comparison with ground truth data. The results are an encouraging step towards the ultimate goal of robust, fully automated road extraction from satellite imagery. As a last contribution, we have developed a complete GUI environment for satellite image manipulation and quadratic snake evolution, based on the Matlab platform. The system is freely available as open source software [9].
2 2.1
Experimental Methods Quadratic Snake Model
Here we provide a brief overview of the quadratic snake proposed by Rochery et al. [7]. An active contour or snake is parametrically defined as T γ(p) = x(p) y(p) ,
(1)
T where p is the curvilinear abscissa of the contour and the vector x(p) y(p) defines the Cartesian coordinates of the point γ(p). We assume the image domain Ω to be a bounded subset of R2 . The energy functional for Rochery et al.’s quadratic snake is given by Es (γ) = Eg (γ) + λEi (γ),
(2)
where Eg (γ) is the geometric energy and Ei (γ) is the image energy of the contour γ. λ is a free parameter determining the relative importance of the two terms. The geometric energy functional is defined as β tγ (p) · tγ (p ) Ψ (γ(p) − γ(p )) dp dp , (3) Eg (γ) = L(γ) + αA(γ) − 2
A Family of Quadratic Snakes for Road Extraction
87
where L(γ) is the length of γ in the Euclidean metric over Ω, A(γ) is the area enclosed by γ, tγ (p) is the unit-length tangent to γ at point p, and Ψ (z), given the distance z between two points on the contour, is used to weight the interaction between those two points (see below). α and β are constants weighting the relative importance of each term. Clearly, for positive β, Eg (γ) is minimized by contours with short length and parallel tangents. If α is positive, contours with small enclosed area are favored; if it is negative, contours with large enclosed area are favored. The interation function Ψ (z) is a smooth function expressing the radius of the region in which parallel tangents should be encouraged and anti-parallel tangents should be discouraged. Ψ (z) incorporates two constants: d, the expected road width, and , the expected variability in road width. During snake evolution, weighting by Ψ (z) in Equation 3 discourages two points with anti-parallel tangents (the opposite sides of a putative road) from coming closer than distance d from each other. The image energy functional Ei (γ) is defined as Ei (γ) =
nγ (p) · ∇I(γ(p)) dp − tγ (p) · tγ (p ) ∇I(γ(p)) · ∇I(γ(p )) Ψ (γ(p) − γ(p )) dp dp , (4)
where I : Ω → [0, 255] is the image and ∇I(γ(p)) denotes the 2D gradient of I evaluated at γ(p). The first linear term favors anti-parallel normal and gradient vectors, encouraging counterclockwise snakes to shrink around or clockwise snakes to expand to enclose dark regions surrounded by light roads.1 The quadratic term favors nearby point pairs with two different configurations, one with parallel tangents and parallel gradients and the other with anti-parallel tangents and anti-parallel gradients. After solving the Euler-Lagrange equations for minimizing the energy functional Es (γ) (Equation 2), Rochery et al. obtain the update equation nγ (p) ·
+ 2λ
1
∂Es (p) = −κγ (p) − α − λ∇I(γ(p))2 + G(γ(p)) ∂γ + β r (γ(p), γ(p )) · nγ (p ) Ψ (γ(p) − γ(p )) dp
r (γ(p), γ(p )) · nγ (p ) (∇I(γ(p)) · ∇I(γ(p ))) Ψ (γ(p) − γ(p )) dp + 2λ ∇I(γ(p )) · (∇∇I(γ(p)) × nγ (p )) Ψ (γ(p) − γ(p )) dp , (5)
For dark roads in light background, we negate all the terms involving image, including G(γ(p)) in Equation 5. In the rest of the paper, we assume light roads on a dark background.
88
R. Marikhu et al.
where κγ (p) is the curvature of γ at γ(p) and G(γ(p)) is the “specific energy,” γ(p)−γ(p ) evaluated at point γ(p) (Section 2.2). r (γ(p), γ(p )) = γ(p)−γ(p ) is the unit vector pointing from γ(p) towards γ(p ). ∇∇I(γ(p)) is the Hessian of I evaluated at γ(p). α, β, and λ are free parameters that need to be determined experimentally. d and are specified a priori according to the desired road width. Following Rochery et al., we normally initialize our quadratic snakes with a rounded rectangle covering the entire image. 2.2
Oriented Filtering
We use Rochery’s oriented filtering method [10] to enhance linear edges in our satellite imagery. The input image is first convolved with oriented derivativeof-Gaussian filters at various orientations. Then the minimum (most negative) filter response over the orientations is run through a ramp function equal to 1 for low filter values and −1 for high filter values. The thresholds are user-specified. An example is shown in Fig. 1(b). 2.3
GVF Energy
Rather than using the oriented filtering specific image energy G(x) from Section 2.2 for snake evolution directly, we propose to combine the oriented filtering approach with Xu and Prince’s Gradient Vector Flow (GVF) method [8]. T The GVF is a vector field V GVF (x) = u(x) v(x) minimizing the energy functional GVF E(V )= μ(u2x (x) + u2y (x) + vx2 (x) + vy2 (x)) (6) Ω 2 2 ˜ ˜ V (x) − ∇I(x) dx, + ∇I(x) ∂u ∂v ∂v ˜ where ux = ∂u ∂x , uy = ∂y , vx = ∂x , vy = ∂y , and I is a preprocessed version of image I, typically an edge image of some kind. The first term inside the integral encourages a smooth vector field whereas the second term encourages fidelity to ˜ μ is a free parameter controlling the relative importance of the two terms. ∇I. Xu and Prince [8] experimented with several different methods for obtaining ˜ We propose to perform Canny edge detection on G (the result of oriented ∇I. filtering and thresholding, introduced in Section 2.2) to obtain a binary image I˜ for GVF, then to use the resulting GVF V GVF as an additional image energy for quadratic snake evolution. The binary Canny image is ideal because it only includes information about road-like edges that have survived sharpening by oriented filters. The GVF field is ideal because during quadratic snake evolution, it points toward road-like edges, pushing the snake in the right direction from a long distance away. This speeds evolution and makes it easier to find suitable parameters to obtain fast convergence. Fig. 1 compares our method to alternative GVF formulations based on oriented filtering or Canny edge detection alone.
A Family of Quadratic Snakes for Road Extraction
89
Fig. 1. Comparison of GVF methods. (a) Input image. (b) G(x) obtained from oriented filtering on I(x). (c) Image obtained from G(x) using threshold 0. (d) Canny edge detection on (c), used as I˜ for GVF. (e-f) Zoomed views of GVFs in region delineated ˜ (f) in (d). (e) Result of using the magnitude of the gradient ∇(Gσ ∗ I) to obtain I. ˜ Result of using Canny edge detection alone to obtain I. (g) GVF energy obtained using our proposed edge image. This field pushes most consistently toward the true road boundaries.
2.4
Family of Quadratic Snakes
A single quadratic snake is unable to extract enclosed regions and multiple disconnected networks in an image. We address this limitation by introducing a family of cooperating snakes that are able to split, merge, and disappear as necessary. In our formulation, due to the curvature term κγ (p) and the area constant α in Equation 5, specifying the points on γ in a counterclockwise direction creates a shrinking snake and specifying the points on γ in a clockwise direction creates a growing snake. An enclosed region (loop or a grid cell) can be extracted effectively by initializing two snakes, one shrinking snake covering the whole road network and another growing snake inside the enclosed region. On the one hand, our method is heuristic and dependent on somewhat intelligent user initialization, but it is much simpler than level set methods for the same problem [7], and, assuming a constant number of splits and merges per iteration, it does not increase the asymptotic complexity of the quadradic snake’s evolution. Splitting a Snake. We split a snake into two snakes whenever two of its arms are squeezed too close together, i.e. when the distance between two snake points is less than dsplit and those two points are at least k snake points from each other in both directions of traversal around the contour. dsplit should be less than 2η, where η is the maximum step size.
90
R. Marikhu et al.
Merging Two Snakes. Two snakes are merged when they have high curvature points within a distance dmerge of each other, the two snakes’ order of traversal (clockwise or counterclockwise) is the same, and the tangents at the two high curvature points are nearly antiparallel. High curvature points are those with where κmax is the maximum curvature for any point on γ. High κγ (p) > 0.6κmax γ γ curvature points are taken to ensure merging only occurs if two snakes have the semi-circular tip of their arms facing each other. Filtering out the low curvature points necessitates computing angle between the tangents at two points only for the high curvature points. When these conditions are fulfilled, the two snakes are merged by deleting the high curvature points and joining the snakes into a single snake while preserving the direction of traversal for the combined snake. Deleting a Snake. A snake γ is deleted if it has low compactness ( 4πA(γ) L(γ)2 ) and delete . a perimeter less than L 2.5
Experimental Design
We analyze extraction results on different types of road networks using the single quadratic snake proposed by Rochery et al. [7] and the proposed family of cooperating snakes. The default convergence criterion is when the minimum Es (γ) has not improved for some number of iterations. Experiments have been performed to analyze the extraction of tree-structured road networks and those with loops, grids and disconnected networks. We then analyze the effectiveness of GVF energy obtained from the proposed edge image in Experiment 4. For all the experiments, we digitize the images manually to obtain the ground truth data necessary to compute precision and recall.
3
Results
We have obtained several parameters emperically. For splitting a snake, dsplit should be less than d. k to be chosen depending on how far the two splitting points should be to ensure that the snakes formed after splitting have at least k points. In order to ensure that merging of snakes takes place only among the arms with the semi-circular tips facing each other, the tangents at the high curvature points are checked for antiparallel threshold of 130π/180.. The compactness should be greater than 0.2 to ensure that linear structured contours are not deleted. 3.1
Experiment 1: Simple (Tree-Structured) Road Networks
A single quadratic snake is well suited for tree-structured road networks as the snake will not need to change its topology during evolution (Figure 2). A family of snakes enable faster and better road extraction as non-road regions are eliminated using splitting and deletion of snakes.
A Family of Quadratic Snakes for Road Extraction
91
Fig. 2. Evolution of quadratic snake on roads with tree structure. Each column displays an image with initial contour in red and the extracted road network below it.
Fig. 3. Evolution of quadratic snake on roads with loops and disconnected networks. Each column displays an image with initial contour in red and the extracted road network below it.
92
3.2
R. Marikhu et al.
Experiment 2: Road Networks with Single Loop and Multiple Disconnected Networks
The family of quadratic snakes are able to extract disconnected networks with high accuracy (Figure 3) but are not able to extract enclosed regions automatically as the snakes are not able to develop holes inside it in the form of growing snakes. 3.3
Experiment 3: Complex Road Networks
A road network is considered complex if it has multiple disconnected networks and enclosed regions and large number of branches. With the appropriate user initialization (Figure 4), the snakes are able to extract the road networks with high accuracy and in less time. 3.4
Experiment 4: GVF Energy to Enable Faster Evolution
The Gradient Vector Flow Field [8] boosts the evolution process as we can see from the number of iterations required for each evolution in Experiment 4 with and without the use of GVF energy. From the evolution in the fifth column, we see that the snake was able to extract the network with greater detail. Also, from the evolution in the last column, we see that it is necessary for the quadratic image energy to enable robust extraction and thus the GVF weight and λ need to be balanced appropriately.
Fig. 4. Evolution of quadratic snake on roads with enclosed regions. Each column displays an image with initial contour in green and the extracted road network below it.
A Family of Quadratic Snakes for Road Extraction
93
Fig. 5. Evolution of quadratic snake on roads with enclosed regions. Each column displays an image with initial contour in green and the extracted road network below it.
4
Discussion and Conclusion
In Experiment 1, we found that the our modified quadratic snake is able to move into concavities to extract entire tree-structured road networks with very high accuracy. Experiment 2 showed that the family of quadratic snakes is effective at handling changes in topology during evolution, enabling better extraction of road networks. Currently, loops cannot be extracted automatically. We demonstrated the difficulty in extracting complex road networks with multiple loops and grids in Experiment 3. However, user initialization of a family of contours enable extraction of multiple closed regions and help the snake to avoid road-like regions. The level set framework could be used to handle change in topology enabling effective extraction of enclosed regions. Rochery et al. [10] evolved the contour using the level set methods introduced by Osher and Sethian. However, our method is faster, conceptually simpler, and a direct extension of Kass et al.’s computational approach. In Experiment 4, we found that faster and robust extraction is achieved using oriented filtering and GVF energy along with image energy of the quadratic snakes. Our proposed edge image obtained from oriented filtering is effective for computing GVF energy to enhance the process of extraction. We also found that our method for obtaining the GVF outperforms standard methods. Finally, we have developed a complete GUI environment for satellite image manipulation and quadratic snake evolution, based on the Matlab platform. The system is freely available as open source software [9].
94
R. Marikhu et al.
Future work will focus on possibilities to automate the extraction of enclosed regions. Digital elevation models could be integrated with image energy for increased accuracy.
Acknowledgments This research was supported by Thailand Research Fund grant MRG4780209 to MND. RM was supported by a graduate fellowship from the Nepal High Level Commission for Information Technology.
References 1. Fischler, M., Tenenbaum, J., Wolf, H.: Detection of roads and linear structures in low-resolution aerial imagery using a multisource knowledge integration technique. Computer Graphics and Image Processing 15, 201–223 (1981) 2. Geman, D., Jedynak, B.: An active testing model for tracking roads in satellite images. IEEE Transactions on Pattern Analysis and Machine Intelligence 18(1), 1–14 (1996) 3. Fortier, A., Ziou, D., Armenakis, C., Wang, S.: Survey of work on road extraction in aerial and satellite images. Technical Report 241, Universit´e de Sherbrooke, Quebec, Canada (1999) 4. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active contour models. International Journal of Computer Vision 1(4), 321–331 (1987) 5. Cohen, L.D., Cohen, I.: Finite-element methods for active contour models and balloons for 2-D and 3-D images. IEEE Transactions on Pattern Analysis and Machine Intelligence 15(11), 131–147 (1993) 6. Laptev, I., Mayer, H., Lindeberg, T., Eckstein, W., Steger, C., Baumgartner, A.: Automatic extraction of roads from aerial images based on scale space and snakes. Machine Vision and Applications 12(1), 23–31 (2000) 7. Rochery, M., Jermyn, I.H., Zerubia, J.: Higher order active contours. International Journal of Computer Vision 69(1), 27–42 (2006) 8. Xu, C., Prince, J.L.: Gradient Vector Flow: A new external force for snakes. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 66–71 (1997) 9. Marikhu, R.: A GUI environment for road extraction with quadratic snakes Matlab software (2007), available at http://www.cs.ait.ac.th/∼mdailey/snakes. 10. Rochery, M.: Contours actifs d’order sup´erieur et leur application ` a la d´etection de lin´eiques dans des images de t´el´ed´etection. PhD thesis, Universit´e de Nice, Sophia Antipolis — UFR Sciences (2005)
Multiperspective Distortion Correction Using Collineations Yuanyuan Ding and Jingyi Yu Department of Computer and Information Sciences University of Delaware Newark, DE 19716, USA {ding,yu}@eecis.udel.edu
Abstract. We present a new framework for correcting multiperspective distortions using collineations. A collineation describes the transformation between the images of a camera due to changes in sampling and image plane selection. We show that image distortions in many previous models of cameras can be effectively reduced via proper collineations. To correct distortions in a specific multiperspective camera, we develop an interactive system that allows users to select feature rays from the camera and position them at the desirable pixels. Our system then computes the optimal collineation to match the projections of these rays with the corresponding pixels. Experiments demonstrate that our system robustly corrects complex distortions without acquiring the scene geometry, and the resulting images appear nearly undistorted.
1
Introduction
A perspective image represents the spatial relationships of objects in a scene as they would appear from a single viewpoint. Recent developments have suggested that alternative multiperspective camera models [5,16] can combine what is seen from several viewpoints into a single image. These cameras provide potentially advantageous imaging systems for understanding the structure of observed scenes. However, they also exhibit multiperspective distortions such as the curving of lines, apparent stretching and shrinking, and duplicated projections of a single point [12,14]. In this paper, we present a new framework for correcting multiperspective distortions using collineations. A collineation describes the transformation between the images of a camera due to changes in sampling and image plane selection. We show that image distortions in many previous cameras can be effectively reduced via proper collineations. To correct distortions in a specific multiperspective camera, we develop an interactive system that allows users to select feature rays from the camera and position them at the desirable pixels. Our system then computes the optimal collineation to match the projections of these rays with the corresponding pixels. Compared with classical distortion correction methods [12,2,11], our approach does not require prior knowledge on scene geometry and it can handle highly Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 95–105, 2007. c Springer-Verlag Berlin Heidelberg 2007
96
Y. Ding and J. Yu
complex distortions. We demonstrate the effectiveness of our technique on various synthetic and real multiperspective images, including the General Linear Cameras [14], catadioptric mirrors, and reflected images from arbitrary mirror surfaces. Experiments show that our method is robust and reliable, thus the resulting images appear nearly undistorted.
2
Previous Work
In recent years, there has been a growing interest in designing multiperspective cameras which capture rays from different viewpoints in space. These multiperspective cameras include pushbroom cameras [5], which collect rays along parallel planes from points swept along a linear trajectory, the cross-slit cameras [8,16], which collect all rays passing through two lines, and the oblique cameras [7], in which each pair of rays are oblique. The recently proposed General Linear Cameras (GLC) uniformly model these multiperspective cameras as 2D linear manifolds of rays (Fig. 1). GLCs produce easily interpretable images, which are also amenable to stereo analysis [9]. However, these images exhibit multiperspective distortions [14]. In computer vision, image-warping has been commonly used to reduce distortions. Image-warping computes an explicit pixel-to-pixel mapping to warp the original image onto a nearly perspective image. For cameras that roughly maintain a single viewpoint [6], simple parametric functions are sufficient to eliminate perspective, radial, and tangential distortions [2,3]. However, for complex imaging systems, especially those exhibiting severe caustic distortions [12], the warping function is difficult to model and may not have a closed-form solution. Image-based rendering algorithms have also been proposed to reduce image distortions [10,4]. There, the focus has been to estimate the scene structure from a single or multiple images. Swaminathan and Nayar [13] have shown that simple geometry proxies, such as the plane, sphere, and cylinder, are often sufficient to reduce caustic distortions on catadioptric mirrors, provided that the prior on scene structure is known. We present a third approach based on multiperspective collineations. A collineation describes the transformation between the images of a camera due to changes in sampling and image plane selection. For many multiperspective cameras such as the pushbroom [5] and the cross-slit [8], collineations can be uniformly modeled using the recently proposed General Linear Cameras (GLC) [15]. 2.1
GLC Collineation
In the GLC framework, every ray is parameterized by its intersections with the two parallel planes, where [u, v] is the intersection with the first and [s, t] the second, as shown in Fig. 1(a). This parametrization is often called a two-plane parametrization (2PP) [4,15]. We can reparameterize each ray by substituting σ = s − u and τ = t − v. In this paper, we will use this [σ, τ, u, v] parametrization to simplify our analysis. We also assume the default uv plane is at z = 0 and st plane at z = 1. Thus [σ, τ, 1] represents the direction of the ray.
Multiperspective Distortion Correction Using Collineations
97
r3 r2 t
(s2, t2)
(s3, t3)
(u2, v2)
s
r3
r1
(u3, v2) z
(s1, t1)
r3
r2
v
u
(u1, v1)
r2
r1
α⋅ r 1 + β ⋅ r 2 + (1 −α−β ) ⋅ r 3
(a)
Π1
r1
Π2
Π3
Π4
C
(b)
(c)
(d)
(e)
Fig. 1. General Linear Camera Models. (a) A GLC collects radiance along all possible affine combination of three rays. The rays are parameterized by their intersections with two parallel planes. The GLC model unifies many previous cameras, including the pinhole (b), the orthographic (c), the pushbroom (d), and the cross-slit (e).
A GLC is defined as the affine combination of three rays parameterized under 2PP: r = α[σ1 , τ1 , u1 , v1 ] + β[σ2 , τ2 , u2 , v2 ] + (1 − α − β)[σ3 , τ3 , u3 , v3 ], ∀α, β
(1)
Many well-known multiperspective cameras, such as pushbroom, cross-slit, linear oblique cameras are GLCs as shown in Fig. 1. If we assume uv is the image plane, we can further choose three special rays with [u, v] coordinates [0, 0], [1, 0], and [0, 1] to form a canonical GLC as: r[σ, τ, u, v] = (1 − α − β) · [σ1 , τ1 , 0, 0]+α · [σ2 , τ2 , 1, 0] + β · [σ3 , τ3 , 0, 1]
(2)
It is easy to see that α = u, β = v, and σ and τ are linear functions in u and v. Therefore, under the canonical form, every pixel [u, v] maps to a ray r(u, v) in the GLC. A GLC collineation maps every ray r(u, v) to a pixel [i, j] on the image plane Π[p, ˙ d1 , d2 ], where p˙ specifies the origin and d1 , and d2 specify the two spanning directions of Π. For every ray r[σ, τ, u, v], we can intersect r with Π to compute [i, j]: [u, v, 0] + λ[σ, τ, 1] = p˙ + id1 + jd2
(3)
Solving for i, j, and λ gives:
where
i=
y x z x (τ dz2 −dy 2 )(u−px )+(d2 −σd2 )(v−py )−(σd2 −τ d2 )pz γ
j=
y z z x x (dy 1 −τ d1 )(u−px )+(σd1 −d1 )(v−py )−(τ d1 −σd1 )pz γ
x x d1 d2 −σ y y γ = d1 d2 −τ dz1 dz2 −1
(4)
(5)
For a canonical GLC, since σ and τ are both linear functions in u and v, γ must be linear in u and v. Therefore, we can rewrite i and j as:
98
Y. Ding and J. Yu
i= j=
a1 u2 +b1 uv+c1 v 2 +d1 u+e1 v+f1 a3 u+b3 v+c3 2
(6)
2
a2 u +b2 uv+c2 v +d2 u+e2 v+f2 a3 u+b3 v+c3
˜ Π (u, v) of a GLC from the uv image plane to a new Thus, the collineation Col image plane Π is a quadratic rational function. Fig. 2 shows the images of a GLC under different collineations. It implies that image distortions may be reduced using a proper collineation. Π2
Π1 GLC
GLC
(a)
(b)
(c)
(d)
Fig. 2. The image of a cross-slit GLC (d) under collineation (c) appear much less distorted than the image (b) of the same camera under collineation (a)
3
Correct Distortions in GLCs
Given a specific GLC, our goal is to find the optimal collineation to minimize its distortions. Similar to previous approaches [12,11], we assume the rays captured by the camera are known. We have developed an interactive system to allow users to design their ideal undistorted images. Our system supports two modes. In the first mode, the user can select feature rays from the camera and position them at desirable pixels in the target images. In the second mode, the user can simply provide a reference perspective image. Our system then automatically matches the features points. Finally, the optimal collineation is estimated to fit the projections of the feature rays with the target pixels. 3.1
Interactive Distortion Correction
Given a canonical GLC, the user can first select n feature rays (blue crosses in Fig. 3(a)) from the source camera and then position them at desirable pixels (red crosses in Fig. 3(b)) on the target image. Denote [uk , vk ] as the uv coordinate of each selected ray rk in the camera and [ik , jk ] as the desired pixel coordinate ˙ d1 , d2 ] that of rk on the target image, we want to find the collineation Π[p, maps [u, v] as close to [i, j] as possible. We formalize it as a least squares fitting problem: n ˜ Π (uk , vk ) − [ik , jk ]||2 min ||Col (7) Π
k=1
Since each collineation Π[p, ˙ d1 , d2 ] has 9 variables, we need a minimal number of five ray-pixel pairs. This is not surprising because four pairs uniquely
Multiperspective Distortion Correction Using Collineations
99
determine a projective transformation, a degenerate collineation in the case of perspective cameras. Recall that the GLC collineations are quadratic rational functions. Thus, finding the optimal Π in Equation (7) requires using non-linear optimizations. To solve this problem, we use the Levenberg-Marquardt method. A common issue with the Levenberg-Marquardt method, however, is that the resulting optimum depends on the initial condition. To avoid getting trapped in a local minimum, we choose a near optimal initial condition by sampling different spanning directions of Π. We rewrite the spanning directions as: di = ηi · [cos(φi )cos(θi ), cos(φi )sin(θi ), sin(φi )],
i = 1, 2
(8)
We sample several θ1 , θ2 , φ1 , and φ2 and find the corresponding p, ˙ η1 , and η2 as the initial conditions. Finally, we choose the one with the minimum error. This preconditioned optimization robustly approximates a near optimal collineation that significantly reduces distortions as shown in Fig. 3(b).
(a)
(b)
Fig. 3. Interactive Distortion Correction. (a) The user selects feature rays (blue crosses) and positions them at desirable pixels (red crosses). (b) shows the new image under the optimal collineation. The distortions are significantly reduced. The green crosses illustrate the final projections of the feature rays.
3.2
Automatic Distortion Correction
We also present a simple algorithm to automatically reduce distortions. Our method consists of two steps. First, the user provides a target perspective image that captures the same scene. Next, we automatically select the matched features between the source camera and the target image and compute the optimal collineation by minimizing Equation (7). Recall that a GLC captures rays from different viewpoints in space and hence, its image may appear very different from a perspective image. To match the feature points, we use Scale Invariant Feature Transform (SIFT) to preprocess the two images. SIFT robustly handles image distortion and generates transformation-invariant features. We then perform global matching to find the potential matching pairs. Finally, we prune the outliers by using RANSAC with the homography model. To tolerate parallax, we use a loose inlier threshold of 20 pixels. In Fig. 4, we show our automatic distortion correction results on various GLCs including the pushbroom, the cross-slit, and the pencil cameras. The user inputs
100
Y. Ding and J. Yu Original Pushbroom
Original Cross-slit
Original Pencil
Target
(b) Corrected Pushbroom
(c) Corrected Cross-slit
(d) Corrected Pencil
(a)
(e)
(f)
(g)
Fig. 4. Automatic Distortion Correction. (a) Perspective reference image; (b),(c), and(d) are distorted images captured from pushbroom camera, cross-slit camera, and pencil camera. (e), (f), and (g) are the distortion corrected results of (b), (c), and (d) using the automatic algorithm.
a perspective image (Fig. 4(a)) and the corrected GLC images appear nearly undistorted using the optimal collineations (bottom row of Fig. 4).
4
Correcting Distortions on Catadioptric Mirrors
Next, we show how to correct multiperspective distortions on catadioptric mirrors. Conventional catadioptric mirrors place a pinhole camera at the focus of a hyperbolic or parabolic surface to synthesize a different pinhole camera with a wider field of view [6]. When the camera moves off the focus, the reflection images exhibit complex caustic distortions that are generally difficult to correct [12]. We apply a similar algorithm using multiperspective collineations. Our method is based on the observation that, given any arbitrary multiperspective imaging system that captures smoothly varying set of rays, we can map the rays onto a 2D ray manifold in the 4D ray space. The characteristics of this imaging system, such as its projection, collineation, and image distortions can be analyzed by the 2-D tangent ray planes, i.e., the GLCs [14]. This implies that a patch on an arbitrary multiperspective image can be locally approximated as a GLC. We first generalize the GLC collineation to arbitrary multiperspective imaging systems. Notice that not all rays in these systems can be parameterized as [σ, τ, u, v] (e.g., some rays may lie parallel to the parametrization plane). Thus, we use the origin o˙ and the direction l to represent each ray r. ˙ l] to a pixel [i, j] as: The collineation Π[p, ˙ d1 , d2 ] maps r[o, [ox , oy , oz ] + λ[lx , ly , lz ] = p˙ + id1 + jd2
(9)
Solving for i, j in Equation (9) gives: i=
x x z x x z y y x y y x z z (ly dz2 −lz dy 2 )(o −p )+(l d2 −l d2 )(o −p )+(l d2 −l d2 )(o −p ) γ∗
j=
y z x x x z z x y y y x x y z z (lz dy 1 −l d1 )(o −p )+(l d1 −l d1 )(o −p )+(l d1 −l d1 )(o −p ) γ∗
(10)
Multiperspective Distortion Correction Using Collineations
(a)
101
(b) (e)
(f) (c)
(d)
(g)
Fig. 5. Selecting different feature rays ((a) and (c)) produces different distortion correction results ((b) and (d)). (f) shows the automatic feature matching between a region (blue rectangle) on the spherical mirror and a perspective image. (g) is the final distortion corrected image. The holes are caused by the under-sampling of rays.
where
x x x d1 d2 −l ∗ γ = dy1 dy2 −ly dz1 dz2 −lz
(11)
˜ Π (o, ˙ l). We abbreviate Equation (10) as [i, j] = Col The user then selects n feature rays from the catadioptric mirror and positions them at target pixels [ik , jk ], k = 1 . . . n. Alternatively, they can provide a target perspective image (Fig. 5(f)) and our system will automatically establish feature correspondences using the SIFT-RANSAC algorithm. We then use the Levenberg-Marquardt method (equation (7)) with sampled initial conditions to ˜ Π. find the optimal collineation Col In the case of catadioptric mirrors, if the selected patch is too large, the resulting image may depend on which rays-pixel pairs are selected. In the kitchen scene example (Fig. 5(a)), selecting the rays from the right side of the spherical mirror produces different results than selecting the rays from the middle part, although distortions are reduced in both cases. This is because the rays inside the patch cannot be approximated as a single GLC model.
5
Results
We have experimented our system on various multiperspective images. We modify the PovRay [18] ray tracer to generate both GLC images and reflected images on catadioptric mirrors. Fig. 3 shows an image of a cross-slit camera in
102
Y. Ding and J. Yu
(b)
(c)
(a)
(d)
(e)
Fig. 6. Correcting distortions on a spherical mirror. The user selects separate regions on the sphere (a) to get (b) and (d). (c) and (e) are the resulting images by matching the selected features (blue) and target pixels (red) in (b) and (d) using collineations.
which the two slits form an acute angle. The user then selects feature rays (blue) from the GLC image and positions them at desirable pixels (red). Our system estimates the optimal collineation and re-renders the image under this collineation as shown in Fig. 3(b). The distortions in the resulting image are significantly reduced. Next, we apply our algorithm to correct reflection distortions on a spherical mirror shown in Fig. 6. It has been shown [14] that more severe distortions occur near the boundary of the mirror than at the center. Our algorithm robustly corrects both distortions in the center region and near the boundary. In particular, our method is able to correct the highly curved silhouettes of the refrigerator (Fig. 6(d)). The resulting images are rendered by intersecting the rays inside the patch with the collineation plane, thus, containing holes due to the undersampling of rays. Our algorithm can further correct highly complex distortions on arbitrary mirror surfaces. In Fig. 7, we render a reflective horse model of 48, 000 triangles at two different poses. Our system robustly corrects various distortions such as stretching, shrinking, and duplicated projections of scene points in the reflected image, and the resulting images appear nearly undistorted. We have also experimented our automatic correction algorithm on both the GLC models and catadioptric mirrors. In Fig. 4, the user inputs a target perspective image 4(a) and our system automatically matches the feature points between the GLC and the target image. Even though the ray structures in the GLCs are significantly different from a pinhole camera, the corrected GLC images appear close to perspective. In Fig. 5(f), a perspective image of a kitchen scene is used to automatically correct distortions on a spherical mirror. This
Multiperspective Distortion Correction Using Collineations
(a)
(b)
(c)
(d)
(e)
(f)
103
Fig. 7. Correcting complex distortions on a horse model. We render a reflective horse model under two different poses (a) and (d) and then select regions (b) and (e). (c) and (f) are the resulting images by matching the selected features (blue) and target pixels (red) in (b) and (e) using collineations.
(a)
(b)
(c)
(d)
Fig. 8. Correcting reflection distortions. (a) and (c) are two captured reflected images on a mirror sphere. Our algorithm not only reduces multiperspective distortions but also synthesizes strong perspective effects (b) and (d).
implies that our collineation framework has the potential for benefiting automatic catadioptric calibrations. Finally, we have applied our algorithm on real reflected images of a mirror sphere in a deep scene. We position the viewing camera far away from the sphere so that it can be approximated as an orthographic camera. We then calculate the corresponding reflected ray for each pixel and use our collineation algorithm to correct the distortions. Our system not only reduces multiperspective distortions but also synthesizes strong perspective effects as shown in Fig. 8.
104
6
Y. Ding and J. Yu
Discussions and Conclusion
We have presented a new framework for correcting multiperspective distortions using collineations. We have shown that image distortions in many previous cameras can be effectively reduced via proper collineations. To find the optimal collineation for a specific multiperspective camera, we have developed an interactive system that allows users to select feature rays from the camera and position them at the desirable pixels. Our system then computes the optimal collineation to match the projections of these rays with the corresponding pixels. Experiments demonstrate that our system robustly corrects complex distortions without acquiring the scene geometry, and the resulting images appear nearly undistorted.
(a)
(b)
(c)
(d)
Fig. 9. Comparing collineations with the projective transformation. The user selects feature rays (blue) and target pixels (red). (c) is the result using the optimal collineation. (d) is the result using the optimal projective transformation.
It is important to note that a collineation computes the mapping from a ray to a pixel whereas image warping computes the mapping from a pixel to a pixel. One limitation of using collineations is that we cannot compute the inverse mapping from pixels to rays. Therefore, if the rays in the source camera are undersampled, e.g., in the case of a fixed-resolution image of the catadioptric mirrors, the collineation algorithm produces images with holes. As for future work, we plan to explore using image-based rendering algorithms such as the push-pull method [4] to fill in the holes in the ray space. We have also compared our collineation method with the classical projective transformations. In Fig. 9, we select the same set of feature points (rays) from a reflected image on the horse model. Fig. 9(c) computes the optimal projective transformation and Fig. 9(d) computes the optimal collineation, both using the Levenberg-Marquardt method for fitting the feature points. The optimal collineation result is much less distorted and is highly consistent with the pinhole image while the projective transformation result remains distorted. This is because multiperspective collineation describes a much broader class of warping functions than the projective transformation.
Acknowledgement This work has been supported by the National Science Foundation under grant NSF-MSPA-MCS-0625931.
Multiperspective Distortion Correction Using Collineations
105
References 1. Chahl, J., Srinivasan, M.: Reflective surfaces for panoramic imaging. Applied Optics 37(8), 8275–8285 (1997) 2. Chen, S.E.: QuickTime VR – An Image-Based Approach to Virtual Environment Navigation. Computer Graphcs 29, 29–38 (1995) 3. Derrien, S., Konolige, K.: Approximating a single viewpoint in panoramic imaging devices. International Conference on Robotics and Automation, 3932–3939 (2000) 4. Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: ’The Lumigraph. SIGGRAPH 1996, 43–54 (1996) 5. Gupta, R., Hartley, R.I.: Linear Pushbroom Cameras. IEEE Trans. Pattern Analysis and Machine Intelligence 19(9), 963–975 (1997) 6. Nayar, S.K.: Catadioptric Omnidirectional Cameras. In: Proc. CVPR, pp. 482–488 (1997) 7. Pajdla, T.: Stereo with Oblique Cameras. Int’l J. Computer Vision 47(1/2/3), 161–170 (2002) 8. Pajdla, T.: Geometry of Two-Slit Camera, Research Report CTU–CMP–2002–02, March (2002) 9. Seitz, S., Kim, J.: The Space of All Stereo Images. In: Proc. ICCV, pp. 26–33 (July 2001) 10. Shum, H., He, L.: Rendering with concentric mosaics. Computer Graphcs 33, 299– 306 (1999) 11. Stein, G.P.: Lens distortion calibration using point correspondences. In: Proc. CVPR, pp. 143–148 ( June 1997) 12. Swaminathan, R., Grossberg, M.D., Nayar, S.K.: Caustics of Catadioptric Cameras. In: Proc. ICCV, pp. 2–9 (2001) 13. Swaminathan, R., Grossberg, M.D., Nayar, S.K.: A Perspective on Distortions. In: Proc. IEEE Computer Vision and Pattern Recognition, Wisconsin (June 2003) 14. Yu, J., McMillan, L.: Multiperspective Projection and Collineation. In: Sebe, N., Lew, M.S., Huang, T.S. (eds.) Computer Vision in Human-Computer Interaction. LNCS, vol. 3766, Springer, Heidelberg (2005) 15. Yu, J., McMillan, L.: Modelling Reflections via Multiperspective Imaging. In: Proc. IEEE Computer Vision and Pattern Recognition, San Diego (June 2005) 16. Zomet, A., Feldman, D., Peleg, S., Weinshall, D.: Mosaicing New Views: The Crossed-Slits Projection. IEEE Trans. on PAMI, 741–754 (2003) 17. Zorin, D., Barr, A.H.: Correction of Geometric Perceptual Distortions in Pictures. Computer Graphics 29, 257–264 (1995) 18. POV-Ray: The Persistence of Vision Raytracer, http://www.povray.org/
Camera Calibration from Silhouettes Under Incomplete Circular Motion with a Constant Interval Angle Po-Hao Huang and Shang-Hong Lai Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan {even,lai}@cs.nthu.edu.tw
Abstract. In this paper, we propose an algorithm for camera calibration from silhouettes under circular motion with an unknown constant interval angle. Unlike previous silhouette-based methods based on surface of revolution, the proposed algorithm can be applied to sparse and incomplete image sequences. Under the assumption of circular motion with a constant interval angle, epipoles of successive image pairs remain constant and can be determined from silhouettes. A pair of epipoles formed by a certain interval angle can provide a constraint on the angle and focal length. With more pairs of epipoles recovered, the focal length can be determined from the one that most satisfies the constraints and determine the interval angle concurrently. The rest of camera parameters can be recovered from image invariants. Finally, the estimated parameters are optimized by minimizing the epipolar tangency constraints. Experimental results on both synthetic and real images are shown to demonstrate its performance. Keywords: Circular Motion, Camera Calibration, Shape Reconstruction.
1 Introduction Reconstructing 3D model from image sequences has been studied for decades [1]. In real applications, for instance 3D object digitization in digital museum, modeling from circular motion sequences is a practical and widely used approach in computer vision and computer graphic communities. Numerous methods, which focus on circular motion, have been proposed and they can be classified into two camps; namely, the feature-based [2,3,4,5] and silhouette-based [6,7,8,9] approaches. In the feature-based approaches, Fitzgibbon et al. [2] proposed a method that makes use of the fundamental matrices and trifocal tensors to uniquely determine the rotation angles and determine the reconstruction up to a two-parameter family. Jiang et al. [3,4] further developed a method that avoids the computation of multiview tensors to recover the circular motion geometry by either fitting conics to tracked points in at least five images or computing a plane homography from minimally two points in four images. Cao et al. [5] aimed at the problem of varying focal lengths under circular motion with a constant but unknown rotation angle. However, it is difficult to establish accurate feature correspondences from the image sequences for objects of texture-less, semi-transparent, or reflective materials, such as jade. Instead of feature correspondences, the silhouette-based approach Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 106–115, 2007. © Springer-Verlag Berlin Heidelberg 2007
Camera Calibration from Silhouettes Under Incomplete Circular Motion
107
integrates the object contours to recover the 3D geometry. In [6], Mendonca and Cipolla addressed the problem of estimating the epipolar geometry from apparent contours under circular motion. Under the assumption of constant rotation angle, there are only two common epipoles of successive image pairs needed to be determined. The relation between the epipolar tangencies and the image of rotation axis is used to define a cost function. Nevertheless, the initialization of the epipole positions can influence the final result and make the algorithm converge to a local minimum. In [7], Mendonca et al. exploited the symmetry properties of the surface of revolution (SoR) swept out by the rotating object to obtain an initial guess of image invariants, followed by several one-dimensional searching steps to recover the epipolar geometry. Zhang et al. [8] further extended this method to achieve auto-calibration. The rotation angle is estimated from three views which sometimes results in inaccurate estimation. In [9], they formulated the circular motion as 1D camera geometry to achieve more robust motion estimation. Most of the silhouette-based methods are based on the SoR to obtain an initial guess of image invariants, thus making them infeasible when the image sequence is sparse (interval angle larger than 20 degree [7]) or incomplete. In this paper, we propose an algorithm for camera calibration from silhouettes of an object under circular motion with a sparse and incomplete sequence. In our approach, we first use the same cost function, as proposed in [6], to determine the epipoles of successive image pairs from silhouettes. Thus, constant interval angle is the main assumption of our algorithm. In addition, we propose a method for initializing the