Harbour Protection Through Data Fusion Technologies (NATO Science for Peace and Security Series C: Environmental Security)

Harbour Protection Through Data Fusion Technologies

NATO Science for Peace and Security Series This Series presents the results of scientific meetings supported under the NATO Programme: Science for Peace and Security (SPS). The NATO SPS Programme supports meetings in the following Key Priority areas: (1) Defence Against Terrorism; (2) Countering other Threats to Security and (3) NATO, Partner and Mediterranean Dialogue Country Priorities. The types of meeting supported are generally "Advanced Study Institutes" and "Advanced Research Workshops". The NATO SPS Series collects together the results of these meetings. The meetings are coorganized by scientists from NATO countries and scientists from NATO's "Partner" or "Mediterranean Dialogue" countries. The observations and recommendations made at the meetings, as well as the contents of the volumes in the Series, reflect those of participants and contributors only; they should not necessarily be regarded as reflecting NATO views or policy. Advanced Study Institutes (ASI) are high-level tutorial courses intended to convey the latest developments in a subject to an advanced-level audience Advanced Research Workshops (ARW) are expert meetings where an intense but informal exchange of views at the frontiers of a subject aims at identifying directions for future action Following a transformation of the programme in 2006 the Series has been re-named and re-organised. Recent volumes on topics not related to security, which result from meetings supported under the programme earlier, may be found in the NATO Science Series. The Series is published by IOS Press, Amsterdam, and Springer, Dordrecht, in conjunction with the NATO Public Diplomacy Division. Sub-Series A. B. C. D. E.

Chemistry and Biology Physics and Biophysics Environmental Security Information and Communication Security Human and Societal Dynamics

http://www.nato.int/science http://www.springer.com http://www.iospress.nl

Series C : Environmental Security

Springer Springer Springer IOS Press IOS Press

Harbour Protection Through Data Fusion Technologies

edited by

Elisa Shahbazian OODA Technologies Inc. Montreal, Quebec, Canada

Galina Rogova Encompass Consulting, Honeoye Falls, NY, U.S.A. and

Michael J. DeWeert BAE Systems Spectral Solutions, Sensor Systems - Identification and Surveillance ASW/MD Directorate Honolulu, Hawaii

Published in cooperation with NATO Public Diplomacy Division

Proceedings of the NATO Advanced Research Workshop on Data Fusion Technologies for Harbour Protection Tallinn, Estonia 27 June – 1 July 2005

Library of Congress Control Number: 2008932229

ISBN 978-1-4020-8882-7 (PB) ISBN 978-1-4020-8881-0 (HB) ISBN 978-1-4020-8883-4 (e-book)

Published by Springer, P.O. Box 17, 3300 AA Dordrecht, The Netherlands. www.springer.com

Printed on acid-free paper

All Rights Reserved © Springer Science + Business Media B.V. 2009 No part of this work may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, microfilming, recording or otherwise, without written permission from the Publisher, with the exception of any material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work.

PREFACE

An Advanced Research Workshop (ARW) “Data Fusion Technologies for Harbour Protection” was held in Tallinn, Estonia 27 June–1 July, 2005. This workshop was organized by request of the NATO Security Through Science Programme and the Defence Investment Division. An ARW is one of many types of funded group support mechanisms established by the NATO Science Committee to contribute to the critical assessment of existing knowledge on new important topics, to identify directions for future research, and to promote close working relationships between scientists from different countries and with different professional experiences. The NATO Science Committee was approved at a meeting of the Heads of Government of the Alliance in December 1957, subsequent to the 1956 recommendation of “Three Wise Men” – Foreign Ministers Lange (Norway), Martino (Italy) and Pearson (Canada) on Non-Military Cooperation in NATO. The NATO Science Committee established the NATO Science Programme in 1958 to encourage and support scientific collaboration between individual scientists and to foster scientific development in its member states. In 1999, following the end of the Cold War, the Science Programme was transformed so that support is now devoted to collaboration between Partner-country and NATO-country scientists or to contributing towards research support in Partner countries. Since 2004, the Science Programme was further modified to focus exclusively on NATO Priority Research Topics (i.e. Defence Against Terrorism or Countering Other Threats to Security) and also preferably on a Partner country priority area. The objective of this ARW was to bring together harbour security domain experts (administrators, policy makers, etc.) with a group of technology experts of international standing in Data/Information Fusion and detection technology in order to provide a mutual understanding of needs and capabilities related to Harbour protection. Thanks to the combined efforts of personnel from the NATO Security Through Science Programme, the NATO Defence Investment Division, the NATO Committee of National Armaments Directors (Defence Investment Division) and the ARW organizing committee, 61 participants in all pertinent domains (security and harbour officials, sensor technology, and fusion technology) were selected. The selection criteria included the technical complementarity of the area of expertise, the country of origin as well as the requirement to include a sufficient number of participants from each pertinent domain. Data/Information Fusion is a very broad interdisciplinary technology domain. It provides techniques and methods for processing and integrating information from multiple sources at different levels of abstraction to support effective decisionmaking and includes

v

vi

PREFACE

1. Estimation and prediction of the state of single objects (level 1) 2. Estimation and prediction of the state of the environment by analyzing relationships between estimated objects within a specific context (level 2 and 3) 3. Controlling/refining the fusion processes (level 4 fusion) It was apparent that most of the harbour security domain experts had very little understanding of fusion technology, and some understanding of specific types of sensors used for harbour security in their respective countries. While technology experts had some exposure to the problems and specific methods related to harbour security, they did not have full understanding of many aspects of Harbour operations. To facilitate information exchange and a better understanding of mutual problems, the ARW included both presentation and brainstorming sessions in the form of Working Group discussions. Participants were distributed into working groups so that each group included experts in each of the pertinent domains and that the groups were multinational. The program and working group participation can be found at http://arwestonia2005.com/program.htm. The contents of this volume are structured consistently with this program. Section 1 includes papers of domain experts describing policies, implementations of security measures and strategies in various countries, Section 2 discusses existing and future technologies for harbour security, Section 3 is devoted to the discussion of methodology for situation and threat assessment and decision support, Section 4 presents data fusion methods for maritime surveillance, and Section 5 includes the description of possible algorithms that can be applicable to designing port security systems. The volume concludes with a summary of the findings of the workshop presented at the SPIE Defence and Security Symposium 2006 (17 April 2006, Orlando (Kissimmee), FL, USA). The workshop identified information and processing necessary for developing a fusion system capable of enhancing the decision-making capabilities of security experts. There is a significant on-going effort in many countries aimed at designing data and information fusion systems in the military domain. At the same time, much less work has been done to understand the domain requirements and developing systems dealing with harbour protection. The workshop identified numerous information/data fusion related research issues to be investigated specifically for harbour security, and made the first step towards addressing these issues. In addition, other technologies and processing required for designing such systems were discussed. Among important research problems identified by the ARW participants were: – Methods of expert knowledge elicitation – Methods on comprehensive and computationally tractable domain representation e.g., methods of effective representation of all procedures and regulations related to ships, cargo, and people movements

PREFACE

–

–

– –

vii

Means to support cooperation between various agencies and technology enabling secure information exchange between different agencies and databases Methods of situation and threat assessment including methods of event detection, normalcy modeling, modeling of intent and capability of terrorist organizations Methods for information quality assessment Emerging sensor technologies

Participants representing Belgium, Bulgaria, Canada, Estonia, France, Germany, Israel, Italy, Jordan, Norway, Poland, Portugal, Romania, Russia, UK, and USA contributed in this ARW. A distinguished group of experts was assembled and the technical program was organized with the generous and very capable assistance of the Organizing Committee composed of Dr. Elisa Shahbazian (Canada), Mr. Aivar Usk (Estonia), Dr. Maria Rey (Canada), Dr. Galina Rogova (USA), and Dr. Albena Tchamova (Bulgaria). The value to be gained from any ARW depends on participants who devote so much of their time and talents to make the ARW successful. As the reader of these proceedings will see, this ARW was particularly honoured with an exceptional group of experts to whom the organizers and participants offer their deep appreciation. Due to the broad interdisciplinary nature of the subject, as well as the intentionally very disparate backgrounds of participants, the editors of this volume were faced with difficult decisions, such as: – – – –

Dealing with structural inconsistencies in papers provided for the proceedings between domain experts and technologists Dealing with different levels of maturity of technology presented by representatives of different countries Accepting papers that discussed technology methods without providing explicit discussion of how they may be applied for harbour security Accepting some differing interpretations of the technology that are not always consistent with the views of the editors

We are grateful to a number of organizations for providing the financial assistance that made the workshop possible. Foremost is the NATO Security Through Science Programme and the Defence Investment Division which initiated the workshop, provided important financial support, helped to involve very pertinent experts as well as provided valuable organizational support. In addition, the following sources made significant contributions: the U.S. Office of Naval Research, Defence Research and Development Canada in Valcartier, Cybernetica AS company of Estonia, Centre of Research in Mathematics of the University of Montreal, and Lockheed Martin Canada. We would like to thank the management of Hotel ATHENA located on the scenic Viimsi peninsula for ensuring that all the needs of the ARW were satisfied.

viii

PREFACE

We would specifically like to thank Aivar Usk from the Department of Navigation Systems of Cybernetica AS (the Estonian member of the Organizing Committee) without whom it would be very hard to ensure the ARW’s success. A very special acknowledgement goes to Ani Shahbazian who undertook the very challenging task of first performing the English Language editing of all the lecturers’ manuscripts and then re-formatting all lectures after the technical editing was complete, producing a camera-ready document for the publisher. And, finally, all of our thanks go to the people of Estonia, who certainly displayed, in every way, their warmth and hospitality. December 2007 Elisa Shahbazian Montreal, Canada Galina Rogova Rochester, USA Michael J. Deweert Honolulu, USA

ACKNOWLEDGEMENTS

The authors would like to thank the NATO Security Through Science Program and the Defence Investment Division for suggesting this ARW, sponsoring it as well as helping to involve many important domain experts into this ARW. The authors would also like to thank Canada's DRDC for their active participation, technical contributions and sponsorship as well as the US ONR for their financial contribution. x The authors thank all participants for their very informative presentations and hard work as members of the working groups. x Finally we would like to thank Aivar Usk from Cybernetica AS in Estonia, who chose and made arrangements for a most successful venue in Estonia, in addition to his technical contribution to the ARW.

ix

CONTENTS

Preface ..................................................................................................................... v Acknowledgements................................................................................................ ix List of Participants ............................................................................................... xv CEP and Transport PB&Cs .................................................................................. 1 Andrzej Jacuch Implementation of the ISPS Code in Norwegian Ports ...................................... 7 Erik Ørbeck Practical Implementation of the ISPS Code in the French Seaports .............. 13 Bernard Delsupexhe PBIST and Its Projects, with Focus on Port Security....................................... 17 Roberto Bertini Implementation of the ISPS Code in the Russian Federation: Ships and Ports ..................................................................................................... 23 Sergei Novikov Steps to Better Waterside Port and Harbor Security: The Development of Regional Maritime Safety Systems in the Russian Federation.................... 27 Alexander Sokolov Police National Maritime Security Strategy ...................................................... 33 James Douglass Harbour Protection in the Jordanian Port of Aqaba ....................................... 37 Bassam Kakish Feasibility Study on Surveillance and Interdiction Technologies for the Port of Halifax .......................................................................................... 43 Francine Desharnais, Dale Blodgett, Robert Bush, Robin Durnford and Bill Yakamovich Maritime Surveillance Information Availability in Estonia ............................ 53 Rein Haavel, Monika Oit and Aivar Usk xi

xii

CONTENTS

Case Study: The Reality of DF Technology Transition for Maritime Domain Awareness with a Focus on Container Security ................................. 61 Michael Moskal Cerberus ................................................................................................................ 69 Andrew B. Webb and William J. Borrowdale Emerging Sensor Technologies and Metrics for Harbour Security ................ 71 Ruth M. Woodward Low Power Modular Sonar Systems .................................................................. 79 Mart Rüütel An Unconstrained View of the Employment and Exploitation of Defence Data Fusion Technologies for Harbour Protection........................................... 87 James Llinas An Approach to Threat Assessment ................................................................... 95 Alan N. Steinberg Harbour Protection and Higher Level Fusion: Issues and Approaches ....... 109 Galina L. Rogova Perceptual Reasoning Managed Situation Assessment for Harbour Protection ............................................................................................................ 119 Ivan Kadar Understanding Military Information Processing – An Approach to Support Intelligence in Defence and Security ............................................. 127 Joachim Biermann Applying Decision Support and Data/Information Fusion/Management Concepts for Emergency Response in the Context of Harbour Protection .. 139 Éloi Bosse and Jean Roy Analyses of the Concept of Trust in Information Fusion and Situation Assessment .......................................................................................................... 161 Hasmik Atoyan and Elisa Shahbazian Issues with Developing Situation and Threat Assessment Capabilities ........ 171 Jean Couture and Eric Menard

CONTENTS

xiii

Distributed Data Fusion and Maritime Domain Awareness for Harbour Protection ............................................................................................................ 181 Simon J. Julier and Ranjeev Mittu Intelligent Geo-Information Systems and Harbour Protection ..................... 191 Ruslan P. Sorokin Using Modeling and Simulation as a Framework for Testing New Solutions Devoted to Securing Global Flows of Goods and People ...... 199 Agostino Bruzzone and Chiara Briano Fusion of Information from Disparate Electro-Optical Imagers for Maritime Surveillance.................................................................................. 209 Michael J. DeWeert Automatic Video Surveillance of Harbour Structures ................................... 223 Lauro Snidaro, Gian Luca Foresti and Claudio Piciarelli Heterogeneous Sensors Data Fusion Issues for Harbour Security................ 233 Stefano Piva, Matteo Gandetto, Reetu Singh and Carlo S. Regazzoni Ship Detection and Characterization Using Polarimetric SAR Data............ 243 Yannick Allard, Mickael Germain and Olivier Bonneau Quick Joint Detection and Fusion Applications in Passive Surveillance Systems ................................................................................................................ 251 Pierre Blanc-Benon, Nathalie Giordana and Yves Chocheyras The Quiet Interlude Processing System (QuIPS) – An Automated Data Fusion Capability for Littoral Environments.................................................. 259 James H. Wilson and Arvid Melkevik Sensor Data Processing for Tracking Underwater Threats Using Terascale Optical Core Devices ........................................................................ 267 Jacob Barhen and Neena Imam A New Neural Approach for Pattern Recognition in Space Imagery ........... 283 Victor-Emil Neagoe and Armand-Dragos Ropot Concurrent Self-Organizing Maps – A Powerful Artificial Neural Tool for Biometric Technology .................................................................................. 291 Victor-Emil Neagoe and Armand-Dragos Ropot

xiv

CONTENTS

Sensor-Fusion in Neural Networks................................................................... 299 Gunther Palm and Friedhelm Schwenker Target Identification Based on DSmT.............................................................. 307 Albena Tchamova and Jean Dezert Particle Filter Application to Localization....................................................... 317 Evgeni Kiriy, Hannah Michalska and Guy Michaud Restrictive Estimation in Tracking Problems ................................................. 329 Andrey Makshanov Findings of the NATO Workshop on Data Fusion Technologies for Harbour Protection ...................................................................................... 337 Elisa Shahbazian, Michael J. Deweert and Galina Rogova Subject Index ...................................................................................................... 353

LIST OF PARTICIPANTS

Kiril Alexiev Institute for Parallel Processing Bulgarian Academy of Sciences 25-A Acad. G. Bonchev Str. Sofia 1113 Bulgaria [email protected] Yannick Allard 4577 Ave de Lésplanade Montreal, Québec H2T 2Y6 Canada [email protected] Yval Alon G.S. SEALS Ltd. (Associate of the International Policy Institute for Counter Terrorism (ICT) 15 Tzahal Road, Be’er Ya’acov 70300, P.O.B. 275 Israel [email protected] Hasmik Atoyan Department of Mathematics and Industrial Engineering École Polytechnique Université de Montréal P.O. Box 6079, St. Centre-ville Montréal, Québec H3C 3A7 Canada [email protected]; [email protected] Jacob Barhen Oak Ridge National Laboratory Oak Ridge, TN 37831-6016

USA [email protected] Pierre Blanc-Benon Thales Underwater Systems CS 43814 F-29238 BREST CEDEX 3 France [email protected]. com Roberto Bertini Planning Board Inland Surface Transport PBIST-WG Via G. Pelosi 1 00143 Rome Italy [email protected] Joachim Biermann FGAN-FKIE Neuenhrerstrasse 20 53343 Wachtberg Germany [email protected] William J. Borrowdale Horsefeathers, Bpdle Street Green, Sussex, England, BN27 4QY UK [email protected] Eloi Bosse Defence R&D Canada (DRDC) Valcartier 2459 Pie-XI Blvd North Val-Bélair, Québec G3J 1X5 Canada [email protected]

xv

xvi

LIST OF PARTICIPANTS

Agoustino Bruzzone MISS-McLeod Institute of Simulation Sciences DIPTEM - University of Genoa via Opera Pia 15 16145 Genova Italy [email protected] Dominique Cant P&O Ports Europe N.V. Nieuwe Westweg, Haven 742 - 2040 Antwerp Belgium [email protected] JOSÉ J.P. de Centeno Portuguese Maritime Authority Capitania Do Porto De Lisboa Alcantara Mar 1350-352 Lisboa Portugal [email protected] Tom Cousins DRDC Ottawa 3701 Carling Avenue Ottawa, Ontario K1A 0Z4 Canada [email protected] Jean Couture 6111 Royalmount Avenue Montreal, Québec H4P-1K6 Canada [email protected] Bernard Delsupexhe Ministry for Transport, Direction of Maritime Transport and Ports 92055 LA DEFENSE CEDEX France

Bernard.Delsupexhe@equipement. gouv.fr Francine Desharnais (paper/not present) DRDC Atlantic 9 Grove Street P.O. Box 1012, Dartmouth, Nova Scotia B2Y 3Z7 Canada [email protected] Mike DeWeert BAE Systems, Sensor Systems Identification and Surveillance ASW/MD Directorate 999 Bishop Street Suite 2700, Honolulu HI 96813 USA [email protected] James Douglass Room 20, PO Box 61701 London SW1H 0XN UK [email protected] ov.uk Jean-Rémi Duquet 6111 Royalmount Avenue Montreal, Québec H4P-1K6 Canada [email protected] Chris Fendt MPS Marine Support Unit, Wapping Police Station 98 Wapping High St London E1W 2NE UK [email protected]

LIST OF PARTICIPANTS

Matteo Gandetto University of Genova Department of Biophysical and Electrical Engineering via Opera Pia 16145 Genova Italy [email protected] Krzysztof Gromadowski Port of Gdynia Authority S.A. Rotterdamska 9 St. 81-337 Gdynia Poland [email protected] Rein Haavel Cybernetica AS Akadeemia tee 21 12618 Tallinn Estonia [email protected] Mati Iila Emergency Management Department, Ministry of Economic Affairs and Communications Tallinn Estonia [email protected] Ulo Jaaksoo Cybernetica AS Akadeemia tee 21 12618 Tallinn Estonia [email protected] Andrzej Jacuch NATO HQ Blvd Leopold III 1110 Brussels

xvii

Poland [email protected] Bruce Jones Euroülikooli, Mustamäe tee, Tallinn, Estonia [email protected] Simon Julier Department of Computer Science University College London London WC1E 6BT UK [email protected] Ivan Kadar Interlink Systems Sciences Inc. 1979 Marcus Avenue, ste 210 Lake Success NY 11042 USA [email protected] Bassam Kakish P.O. Box 9081 11191 Amman Jordan [email protected] Jaanus Kaugerand Estonian Navy, Navy staff, N3. Juhkendali 58 15007 Tallinn Estonia [email protected] Evgeni Kiriy McGill University 3480University St. Rm. 342 Montreal, Québec H3A-2A7 Canada [email protected]

xviii

LIST OF PARTICIPANTS

Jim Llinas State University of New York at Buffalo 62 Meadowview Lane Williamsville, New York 14221 USA [email protected] Aivar Lõhmus Estonian Board of Border Guards Uus-Sadama 14 Tallinn, Estonia Estonia [email protected] Andrey Makshanov 14 Linia, 39, SPIIRAS 199178 St. Petersburg Russia [email protected] Arvid Melkevik Norwegian Defence Research Establishment (FFI) P. O. Box 25 NO-2027 Kjeller Norway [email protected] Mike Moskal CUBRC, Inc. 4455 Genesee St. Buffalo, NY USA [email protected] Roman Nawra Ministry of Infrastructure/PBIST Chalubinskiego 4/6 00-928 Warsaw Poland [email protected]

Victor-Emil Neagoe POLITEHNICA University of Bucharest Ale. Baia de Aries 3, bloc 5B, sc. C, et. 4, apart. 105, sector 6, 060805 Bucharest Romania [email protected] Eric Noor Estonian Maritime Administration/Maritime Safety Division/ Port Supervision Department, Lume 9 10416 Tallinn Estonia [email protected] Sergey Novikov Transas Eurasia 54-4 Maly pr., V.O. 192019 St. Petersburg Russia [email protected] Monika Oit Cybernetica AS Akadeemia tee 21 12618 Tallinn Estonia [email protected] Erik Ørbeck Norwegian National Coastal Administration Moerveien 16, 1430 As P.O. Box 625 4809 Arendal Norway [email protected]

LIST OF PARTICIPANTS

xix

Marc Owen SPAWARSYSCEN San Diego, 53560 Hull Street Code 2731, San Diego CA 92152-5001 USA [email protected]

Carlo Regazzoni DIBE - Department of Biophysical and Electronic Engineering Via Opera Pia 11A, I-16145, Genova Italy [email protected]

Gunter Palm Institute of Neural Information Processing University of Um Germany [email protected]

Maria Rey DRDC CORA 101 Colonel By Drive 6CBS Ottawa, Ontario K1A 0K2 Canada [email protected]

Jiri Patera Department of Mathematics and Statistics Université de Montréal Pavillon André-Aisenstadt 2920, chemin de la Tour Montréal, Québec H3T 1J4 Canada [email protected] Ervin Piikmann Estonian Border Guard Süsta tänav 15 Tallinn Estonia [email protected] John Regan US Customs and Border Protection Field Operations Academy Federal Law Enforcement Training Center 1131 Chapel Crossing Road Glynco GA 31524 USA [email protected]

Carlos Rodolfo AFCEA Portugal AIP – EC Junqueira Praça das Indústrias 1300-307 Lisboa Portugal [email protected] Galina Rogova 9 Country Meadows drive Honeoye Falls NY 14472 USA [email protected] Mart Rüütel R-Süsteemid OÜ Mustamäe tee 5 EE10616 Tallinn, Estonia Estonia [email protected] Elisa Shahbazian 4891 Grosvenor ave Montreal, Quebec H3W-2M2 Canada [email protected]

xx

LIST OF PARTICIPANTS

Lauro Snidaro Department of Mathematics and Computer Science University of Udine via delle Scienze 206 33100 Udine Italy [email protected] Alexander Sokolov Transas Eurasia 54-4 Maly pr. V.O. 192019 St. Petersburg Russia [email protected] Ruslan Sorokin 14 Linia, 39, SPIIRAS 199178 St. Petersburg Russia [email protected] Alan Steinberg 72 Wesbury Lane Palm Coast FL 32164 USA [email protected] Toomas Sullakatko Estonian Naval HQ Juhkentali 58 EE 15007 Tallinn Estonia [email protected] Albena Tchamova Institute for Parallel Processing Bulgarian Academy of Sciences 25-A Acad. G. Bonchev Str. 1113 Sofia Bulgaria [email protected]

Nathan Tonnisson Ministry of Economic Affairs and Communications Harju 11 15072 Tallinn Estonia [email protected] Aivar Usk Cybernetica AS Akadeemia tee 21 12618 Tallinn Estonia [email protected] Jim Wilson Planning Systems, Inc. 3834 Vista Azul San Clemente, CA 92672 USA [email protected] Ruth Woodward HT Consultants Ltd. Magdalen Centre Robert Robinson Avenue The Oxford Science Park Oxford OX4 4GA UK [email protected]

CEP AND TRANSPORT PB&Cs

ANDRZEJ JACUCH NATO HQ CEP/OPS, Brussels

Abstract. This paper introduces Civil Emergency Planning (CEP) in NATO and three transport Planning Boards and Committees. Keywords: Civil emergency planning, planning boards & committees, planning board for inland surface transport

1 Introduction NATO has been concerned with Civil Emergency Planning (CEP) matters from its inception. It has provided a forum to the member states, and later also to Partners, for the exchange of information on CEP issues and for work carried out jointly by NATO and Partner nations towards the improved protection of the civilian population and the use of civil resources in crisis. An integral part of CEP is the importance of the continuity of key functions of society, which is also an important part of Critical Infrastructure Protection. What does NATO mean by “Civil Emergency Planning”? In broad terms, it is the development of collective plans for the systematic and effective use of civil resources at NATO- and national levels in support of Alliance strategy and arrangements for the protection of populations. How is it organized? In NATO, everything is done by committee, and for CEP, that committee is the Senior Civil Emergency Planning Committee (SCEPC). There are eight planning boards and committees under the SCEPC. The SCEPC advises the Council on relevant issues, drafts policy regarding CEP and related activities for the NAC’s approval, reviews progress by nations in CEP, provides guidance for, and monitors, the work of the eight Planning Boards and Committees (PB&Cs) including three transport PB&Cs. The transport PB&Cs date back to the early 1950s and cover inland surface transport, sea transport and air transport. They are: the Planning Board for Inland Surface Transport (PBIST), the Planning Board for Ocean Shipping (PBOS) and the Civil Aviation Planning Committee (CAPC). PBIST is responsible for Port issues. All PB&Cs develop and maintain adequate crisis management arrangements including a pool of selected highly qualified experts. In the three transport PB&Cs E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

1

2

A. JACUCH

there exist approximately 150 civil experts from different sectors of transport industry, and in some cases, national Ministries. The experts’ primary job is to provide advice to NATO in their area of specialty. This is normally accomplished from their normal place of business, but it could also require their presence at NATO HQs or some other location as agreed.

2 Civil emergency planning in NATO CEP in NATO includes planning for the protection of civil populations, measures to support Crisis Response Operations, including Peace Support, Humanitarian and Disaster response operations, and advice in the areas of civil protection, civil communications, industrial planning, food and agriculture, medical, and the three modes of transportation, air, sea and surface. The focal point for CEP in NATO is the SCEPC. It reports to NAC and it is supported by eight PB&Cs: Civil Aviation Planning Committee (CAPC); Civil Communication Planning Committee (CCPC); Civil Protection Committee (CPC); Food and Agriculture Planning Committee (FAPC); Industrial Planning Committee (IPC); Planning Board for Inland Surface Transport (PBIST); Planning Board for Ocean Shipping (PBOS). Civil emergency planning in NATO is guided first and foremost by the 1999 Strategic Concept and the NAC agreed guidance document for civil emergency planning, which was formulated on the basis of the new Strategic Concept and NATO’s experiences with crisis response operations in the Balkans. This Council document defines five roles for civil emergency planning. They are: x x x x

Support for Alliance military operations under Article 5 Support for non-Article 5 Crisis Response Operations Support for national authorities in civil emergencies Support for national authorities in the protection of populations against the effects of WMD x Cooperation with Partners The planning boards and committees are structured to respond to each of these roles. In addition to this overarching guidance, more specific and frequent direction is given by Ministerial Guidance every two years. On the basis of this guidance, the eight planning boards and committee under the SCEPC each develop and implement their individual programs. As a result of the terrorist attacks of 11 September 2001, the SCEPC has also developed a CEP Action Plan as a living document in response to the threat of terrorism. This document contains a list of actions and initiatives within the CEP community to improve civil preparedness for possible terrorist attacks with chemical, biological, radiological, or nuclear agents (CBRN). It includes such projects as Critical Infrastructure Protection, Non-binding Guidance, and Minimum Standards.

CEP AND TRANSPORT PB&Cs

3

How is civil emergency planning carried out in the Alliance? In accordance with their work programmes and the CEP Action Plan, the planning boards and committees accomplish planning functions and also develop, or help nations develop their relevant arrangements. The PB&Cs have existed since the early 1950s, and while their basic function remains the same, that of providing advice, and the roles and missions of NATO, have changed significantly. To accomplish this task, all PB&Cs develop and maintain adequate crisis management arrangements including a pool of selected highly qualified experts. The expert positions are open to nominations from all 46 Euro-Atlantic partnership nations. Experts provide advice to nations, to the NATO Military Authorities and to various other NATO bodies. Advice to nations is provided typically in areas of civil protection, civil infrastructure protection, standardization, crisis communications and CBRN response. Advice to the NATO Military Authorities is concentrated in the areas of standardization and interoperability, and commercial market information for deployment and sustainment. Advice is also given to military planners for their long-term and contingency planning and to military commanders during the execution phase of an operation. The other NATO bodies to which advice is given include the Council, the Euro-Atlantic Partnership Council, the SCEPC, and other committees such as the Military Committee, the NC3 Board, the Science Committee, and the SNLC. In addition to the eight PB&Cs under the SCEPC, CEP is also accomplished by the Euro-Atlantic Disaster Response Coordination Centre (EADRCC). The EADRCC was established in 1998 by the Euro-Atlantic Partnership Council. Its role is to act as a clearinghouse for disaster assistance among NATO and Partner nations in case of natural and technological disasters or terrorist attacks.

3 The three transport planning boards and committees While each of the PB&Cs is important to achieving the five roles of CEP, I will focus only on the three transport PB&Cs for the rest of my presentation. There are approximately 150 civil experts from the industry and, in some case, from the national Ministries in the transport PB&Cs. They are nominated by their nations and they compete for expert positions. When selected, they receive security Clearances as would any national planner. Their primary job is to provide advice to NATO in the area of their specialty. This is normally accomplished from their normal place of business, but it could also require their presence at NATO headquarters, or at some other location, as agreed. Some examples of what the three transport PB&Cs do: x CAPC provides advice on legal aspects of aviation issues faced by NATO, evaluates safety and security issues associated with the use of civil resources,

4

A. JACUCH

monitors the market for the availability of civil aviation assets that could be used for a future NATO operation. x PBOS advises NATO on the protection of shipping against acts of terrorism, analyzes the work of the International Maritime Organization and International Labor Organization in the area of security, evaluates the civil shipping market and advises both civil and military planners as to current market conditions. x PBIST analyzes national and International Organizations (IOs) work in the area of inland surface transport (IST), including port security, studies how IST security can be improved in the Euro-Atlantic region, conducts a survey on the protection of critical infrastructure and also assesses the commercial market for the availability of assets and infrastructure if needed. The transport planning boards and committees work hard to remain relevant to the many customers in NATO as they assist in providing commercial input to many transport issues facing NATO today.

4 Final remarks The events of September 11, 2001 stressed the urgency of cooperation in preparing for possible terrorist attacks. The Partnership Action Plan against Terrorism, issued at the Prague Summit in November 2002, encourages the sharing of related information and the participation in civil emergency planning to assess risks and reduce the vulnerability of civilian populations to terrorism and weapons of mass destruction. The transportation system is featured as a Critical Infrastructure in all lists drafted by nations or International Organizations. It persists as an attractive target for terrorist activity due to its relative accessibility and the large number of people reliant on it. Moreover, the growing sophistication of the transport industry increases its vulnerability. Amongst areas of major concern we should cite ports, the intermodal terminals, the choke points in the transport network, the communications-information services, the support facilities of fuel and other energy sources. Terrorists can choose between innumerable vulnerabilities in the infrastructures. These vulnerabilities are both physical and cyber. Countries and organizations must identify and protect the points or systems that when targeted can lead to the worst consequences for society. Terrorism using Chemical Radiological Biological Nuclear and High Explosives (CBRNE) poses a major threat to societies everywhere. One example of a response to these new threats is the ISPS Code which aims at enhancing maritime security on board ships and at the ship/port interface by providing an international framework involving co-operation between SOLAS Contracting Governments, Government agencies, local administrations and the shipping and port industries to detect security threats and take preventative measures against security incidents affecting ships or port facilities.

CEP AND TRANSPORT PB&Cs

5

Strategies to better deal with vulnerabilities in critical infrastructures, in IST and in harbors, ranging from reforms of traditional instruments that enhance the protection of national assets to methodological approaches that emphasize analysis and consequence management, should be sought and any contribution to this effort, even the slightest, is important. The PBIST is ready to contribute in any possible and useful way to this undertaking.

IMPLEMENTATION OF THE ISPS CODE IN NORWEGIAN PORTS

ERIK ØRBECK National Coastal Administration, Norway

Abstract. This paper offers a brief overview of the maritime security authorities, international port, and trade and traffic structure, while illustrating the challenge of implementing the ISPS Code in Norway. It also outlines the minimum security standards for ISPS terminals, rapidly presents approved port facility plans and considers the challenges ahead. Keywords: ISPS security authorities, port traffic, terminal structure, security measures in Norwegian ports, port security plans

1 Introduction The SOLAS CHAP. Xl-2 and the ISPS Code entered into force in Norwegian ports as of July 1, 2004. The Ministry of Fisheries and Coastal Affairs is the Contracting Government. The National Coastal Administration is the designated authority for all port facilities; it is also responsible for authorizing Recognized Security Organizations (RSOs). Currently, 21 organizations are authorized as RSOs in Norway. Among these are a few port authorities with the necessary competence to make assessments and plans for their own port facilities. The rest are private companies in the consulting industry. Security management for ships is handled by the Norwegian Maritime Directorate under the Ministry of Trade and Industry. National authorities responsible for setting security levels and coordinating rescue centers operate under the Ministry of Justice and Police. This paper focuses on the port sector and port authorities. The main objective of the ISPS Code implementation was to ensure that necessary and sufficient measures would be put into force in Norwegian port facilities while simultaneously allowing waterborne commerce and ship traffic between Norway and foreign countries to continue normally after July 1, 2004, as it did prior to that date.

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

7

8

E. ØRBECK

2 Challenges Norway is a relatively small country with a very long coastline, measuring approximately 2,650 km. Apart from the offshore fisheries and oil and gas activities, most of the mainland’s industrial activities are situated in or close to the coastal zone. This makes for a large number and wide diversity of ports and maritime terminals along the entire coastline. With the exception of the main 10 to 15 national and regional public ports, and a few big industrial terminals located inside or outside public ports, most ports are composed of small local terminals. In addition, every year a great number of international cruise ships visit the fiords and the land of the midnight sun. Most terminals and quays are owned and run by private companies, especially the liquid and dry bulk terminals. The public interest is mostly directed toward general cargo terminals. These terminals usually serve both national and international traffic. All ports wish to maintain their marked position; nobody wants to lose trade or vessels calling their ports. Therefore, most facility owners have been eager to implement the new security regime in their terminals, and, so far, it has not been necessary for the national authorities to instruct the local port authorities to carry out the implementation. Cargo vessels and tankers dominate the international traffic in Norwegian ports, although the importance of passenger traffic cannot be ignored. Offshore and fishing vessels are also a typical element of the Norwegian port. Most of these vessels weigh less than 10,000 gross register tons (GRT). However, the biggest tankers, cargo and cruise vessels also call the ports. The average vessel in international traffic weighs about 4,000 gross tons (GT). The oil and gas industry dominate foreign trade through the ports, with more than 70 million tons traded per year. Dry bulk cargo trade amounts to 35 million tons per year, while general cargo amounts to 13 million tons per year.

3 Security standards The ISPS Code establishes the Maritime authorities’ responsibility to set security levels and provides guidance for protection from security incidents. Some of the requirements are mandatory, demanding appropriate operational and physical measures in all relevant port facilities, while some are described more as recommendations. Nevertheless, even though these recommendations are not strictly mandatory, they must be regarded as very strong guidance. As the designated authority for all port facilities, the National Coastal Administration has stated that, as a principal rule, the given recommendations should also be treated as mandatory requirements. The combination of a great number of ports with various functions, diverse actors, and many types and sizes of vessels calling ports, implies the need for

IMPLEMENTATION OF THE ISPS CODE IN NORWEGIAN PORTS

9

varied security measures in the port sector. However, the national authorities have observed different choices and enforcements of security measures even within comparable terminals. Although alternative measures may provide equivalent levels of protection, the authorities have been trying to establish and enforce a uniform national security regime in comparable terminals; harmonizing and standardizing the security measures and the enforcement of the security system in the port sector is a necessity. Determining a set of minimum security standards for all ISPS terminals, along with common guidelines and directions, was an important step in the national ISPS Code implementation process. Based on the Code and the national policy, a list of minimum requirements for the implementation and exercise of these security measures was drawn up by the National Coastal Administration. The list, in its present form, is not exhaustive. However, it does concern certain areas where the need for harmonization seems urgent. These security measures for port facilities shall be controlled by the National Coastal Administration as part of the approval of plans, and subsequent verifications and inspections. 3.1 ACCESS CONTROL x A physical barrier must be placed at ISPS-callings. x All access points should be locked up or controlled by authorized personnel, at ISPS-callings. x Only controlled persons and vehicles are allowed in the terminal at ISPS-callings. 3.2 BARRIER x Fencing and gates must be positioned according to national security class 2, or higher. x Exceptions may be made in approving equivalent solutions. 3.3 IDENTIFICATION x All terminals should have an updated record on controlled personnel. x All persons in the terminal at ISPS-callings must wear a visible identity card, issued and accepted by the terminal. 3.4 CHECKING THE IDENTITY OF PERSONS, VEHICLES AND CARGO x All persons, vehicles, cargo and ship stores entering the terminal may be controlled. x On security level 1, a minimum of 5% of all persons, vehicles and cargo, which have not been controlled in advance, shall be spot checked.

10

E. ØRBECK

x On security level 2, a minimum of 20% of all persons, vehicles and cargo shall be spot checked. x On security level 3, 100% of all persons, vehicles and cargo shall be checked. 3.5 RESTRICTED AREAS x Restricted areas shall be secured by a barrier or other types of perimeter security, according to the requirements in §2 Barrier. x If a terminal consists of more than one restricted area, an updated record for all admitted persons to all restricted areas shall be available at the terminal. x The identification system shall make a distinction between persons with permission and persons without permission being admitted to these areas. 3.6 HANDLING OF CARGO x All ISPS-terminals shall have systems for the identification of ISPS-cargo which will be able to distinguish ISPS-cargo from other cargo. x Only authorized personnel shall have access to ISPS-cargo. x As long as ISPS-cargo stays in an ISPS-terminal or in a restricted area of an ISPS-terminal, the ISPS-terminal shall have security measures to prevent cargo tampering. 3.7 IDENTIFICATION OF SHIP PERSONNEL x If ship personnel are allowed ashore, all ISPS-terminals shall have an updated crew list for all vessels calling the terminal. x As long as ship personnel remains on the terminal’s premises, the personnel shall be identifiable according to said crew lists. x Ship personnel shall wear either the identification card issued by the ship or a visitor identification card issued by the terminal. 3.8 SIGNS AND MARKS FOR ISPS TERMINALS x All access points to ISPS-terminals, including waterside access points, shall be marked with readable signs in both Norwegian and English (for example, they will be marked with: “NO ADMITTANCE”). x The seaside markings shall have good readability from vessels approaching the terminal.

4 Port facility security assessments and security plans One of the Norwegian Coastal Administration’s main tasks, besides the approval/ authorization of Recognized Security Organizations (RSOs), was to approve first

IMPLEMENTATION OF THE ISPS CODE IN NORWEGIAN PORTS

11

the Port Facility Security Assessments (PFSA) and then the Port Facility Security Plans (PFSP). The approval of the Port Facility Security Plans will be ready before terminals can begin the implementation of physical security measures. Verifications are based on inspections of port facilities, meetings and interviews with the Harbor master, the Port Facility Security Officer, and others. Verification is a condition for issuing the “Statement of Compliance.” Port facilities with approved Port Facility Security Plans have been reported to the International Maritime Organization (IMO) and registered in the IMO database.

5 Status for ISPS implementation Status as of June 2005: x 538 port facilities have an approved PFSA and PFSP. x 71 port facilities have confirmed that they are ready for verification. x 214 port facilities are verified and have received the “Statement of Compliance.” Most of the approved terminals (60%) have only one single function, while the others are multifunctional facilities. The most common terminal functions are general cargo handling (35%) and dry bulk (33%) handling. Only 12% of the terminal facilities serve passengers. The plan of progress for the National Coastal Administration projects that by the summer of next year all port facilities in Norway serving international vessel traffic will have established a security plan based on security assessments. After a verification process they will receive the “Statement of Compliance.”

6 Conclusion As this presentation has demonstrated, there is a great variety of Norwegian port facilities with a diversity of characteristics. There are terminals for passengers, general cargo, containers, bulk cargo, oil, etc. These terminals are not comparable and all require different security measures and enforcements. However, even for comparable terminals, different security regimes have been developed. The challenge for authorities is to ensure uniform and fair security enforcement in comparable port facilities. x To secure high level security systems in all Norwegian ports matching international standards. x Port users must find the security regimes predictable and cost effective.

12

E. ØRBECK

x A weak security regime/system must never have a marked advantage in the Norwegian port industry. To summarize, Norwegian authorities are responsible for ensuring that port security regimes are in compliance with SOLAS and the ISPS Code, while simultaneously ensuring that the competitiveness of seaborne transport is not unnecessary reduced.

PRACTICAL IMPLEMENTATION OF THE ISPS CODE IN THE FRENCH SEAPORTS

BERNARD DELSUPEXHE French Ministry of Transports and Equipment, France

Abstract. The ISPS code must be implemented in all port facilities servicing ships engaged on international voyages. This article gives an example of the practical implementation of this code in the French seaports. Keywords: ISPS code, port facility, video surveillance, Heartbeat Detector, Millimetre Microwave sensor, container monitoring

1 Introduction The security of the maritime lines of communication is mandatory to the development of world economy. Indeed, the world geopolitical evolution and the rise of terrorism sensitized the entire international community to the importance of security and the maritime field cannot be excused from this tendency. In response to this issue, a new chapter of the SOLAS1 convention, which intends to enhance maritime security in association with the ISPS2 code, is being promulgated. The European Union, by Regulation (EC) 725/2004, applicable to all Member States, established a transposition of this code, thus reinforcing it. The Commission will also promulgate in the weeks to come a directive enhancing port security.

2 Port security Examples of the application of the ISPS code in two very different French seaports will be presented. Calais is the first passenger port of the European continent with nearly 14 million passengers each year, and Marseilles, the first Mediterranean port with a combined traffic of 95 million tons of goods. A definition must first be given for what, in France and in many countries, is referred to as a “port facility.” The definition varies as per each government’s 1 2

SOLAS: Safety of Life at Sea. ISPS: International Ships and Port facilities Security.

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

13

14

B. DELSUPEXHE

choice. In France, however, a port facility and terminal are one and the same. This distinction seems important where shared responsibilities for security are concerned: each stakeholder of a port facility is responsible for the establishment of a port facility security plan and its implementation, yet the ports belong to the State and each port will have to establish, according to the upcoming European directive, its own security plan for the areas not covered by the port facility security plans. A total of 360 port facilities are identified in France, which is double the number of facilities in Rotterdam’s only port. Only 43 port facilities are declared by the United States. The ISPS code stipulates that the security plans of the ships and the port facilities envisage measures allowing: x Access control (ISPS A/7.2 and A/14.2.1). x Prevention of unauthorized access in restricted areas (ISPS A/9.4.2 and A/14.2.4). x Prevention of the introduction of unauthorized weapons, dangerous substances and devices intended for use against people, ships or ports (ISPS A/9.4.1 and A/16.3.1). 2.1 CALAIS

2.1.1 Perimeter control In the port of Calais, the first issue to be considered is safeguarding against infiltration of unverified individuals. For this purpose, each facility is completely enclosed; particularly, the cross-channel facility, which, as required by the code, is an almost entirely restricted access area. There exist separate control fields for trucks, private cars, and loading zones. Each week, approximately 50 people are stopped after having crossed the port facility’s external fences – a good indication of strong migratory pressure towards the United Kingdom. These external fences, referred to as ‘instrumented fences,’ indicate with precision the clandestine’ crossing point. Where possible, complementary stoppings of concertina wire are added. A video surveillance system is set up over the entire facility, supervised by headquarters dedicated to security. The surveillance system delivers images to all parties responsible for security – stakeholders, ship owners, police, customs, etc. In order to perfect peripheral control, system field trials are conducted, for instance, by erecting hyper frequency barriers and waterproof active infrared barriers wherever fencing is unfeasible. Port facilities await the development of innovative equipment offering better effectiveness at lower cost.

PRACTICAL IMPLEMENTATION OF THE ISPS CODE

15

2.1.2 Flow control Up to 10,000 cars and 2,500 trucks may embark on some days at Calais. Flow control is thus just as vital as perimeter control. However, it stands to reason that the vocation of a port is to support commercial activity, and consequently to have maximum flow. The inspection of vehicles is limited by both manpower restrictions and the problem of effectively searching fully laden vehicles without disrupting traffic flow. Flows of trucks and cars are kept separate, and various methods are applied in the search for clandestines. Trucks are subjected to inspection by a Passive Millimetre Microwave sensor or a Heart Beat Detector supplemented by a CO2 detector. These systems were set up by the British administration and subsequently exploited by French harbour personnel. The Heartbeat Detector detects the presence of persons hidden inside vehicles. Using data from special seismic sensors, the system finds the shock wave generated by the beating heart, which couples to any surface or object with which the body is in contact. It is simple, quick to use, therefore extremely effective. The Passive Millimetre Microwave sensor is used for scanning soft-sided heavy goods vehicles (HGVs) or plastic-sided refrigerated lorries. When a lorry drives between the mobile unit and the reflector, an image is formed showing the interior of the lorry. This is then displayed on a computer screen, whereupon various display and processing options are available to the operator. The Passive Millimetre Microwave sensor or Heart Beat Detector can be supplemented by CO2 detection carried out either by port personnel or by the administration. Trucks must then pass an inspection from the maritime companies, then jointly from the French and British immigration services and customs, the whole followed by a final inspection by the company of navigation. These filters prevent between 150 and 300 candidates from emigrating per week. The U.K. is indeed a destination of choice for refugees from the Eastern hemisphere. 2.2 MARSEILLES The port of Marseilles spans 45 km from east to west, with a total surface exceeding 100 km². The 40 port facilities deal with oil, gas, ores and various goods, as well as passengers and containers. It is thus both larger in size and can receive a much higher volume of traffic than the port of Calais. The eastern section of the port, which dates back more than two millennia, is set against the town of Marseilles and is protected on the seaside by a 6 km long pier. This allows for a completely enclosed area including port facilities. As a result, perimeter and flow control can be performed simultaneously and at two different levels: in the port and in port facilities. The western section of the port, in the Fos district, is set on a vast plain and is wide open to the sea. Each port facility is in charge of its own protection.

16

B. DELSUPEXHE

Let us take a closer look at container security procedures: the American customs launched in 2002 a container security initiative, or CSI. The American authorities fear in particular the introduction of weapons of mass destruction on their territory by means of containers. The port of Marseilles is among the 35 world ports complying with this initiative, which consists in moving back the American borders in our ports, and vice versa. Moreover, American customs officers are present in the two French ports taking part in the CSI, Le Havre and Marseilles. The first step of container monitoring is intelligence. A particular cell of French customs is in charge of this task. External control is executed by stakeholders of the port facility. The identification number of the container, the license plate number of the tractor and the identity of the driver with his badge and photograph are noted at each passing. If suspicions of intrusion occur at any given time on the port, it is possible to have the container inspected by customs using a mobile X-ray system. This autonomous equipment allows the verification of a few tens of containers per day. The main difficulty when using this equipment consists in maintaining a radiological zone of safety that must be created around the vehicle. If any doubt persists, the container is emptied and then inspected manually.

3 Conclusion The control of human and commercial flows transiting by sea requires reliable monitoring systems, which must be increasingly sophisticated and impede commercial operations as little as possible. Technical and technological solutions are applicable on several levels. x Equipment: development of innovative equipment offering a decisive advantage in terms of operational capacity, ownership cost of sensors and means of intervention. x System: networking the systems and sensors, thus fusing information and automating the operations and the decision-making aid. Two major issues must now be addressed. Firstly, how can ports ensure the detection of explosive charges likely to be embarked and that are considered a major threat when this requires that 400 vehicles be controlled effectively while respecting a time of loading not exceeding 30 minutes due to commercial needs? Secondly, in the case of Marseilles-Fos, a water surface of 100 km2 is wide-open to the sea and travelled by trading ships, fishermen, and yachtsmen, allowing access to port facilities. How can this traffic be controlled effectively, including possible underwater infiltrations, by means which are not extensible ad infinitum? These concerns must be urgently addressed. If not, I am afraid the future of the maritime trade will not be so quiet.

PBIST AND ITS PROJECTS, WITH FOCUS ON PORT SECURITY

ROBERTO BERTINI Planning Board for Inland Surface Transport (PBIST), Rome

Abstract. This paper summarizes the tasks and responsibilities of the NATO PBIST, with a particular focus on the Board’s studies on Critical Infrastructure Protection (CIP) and Seaport projects. Keywords: Critical infrastructure protection, threats, transport of dangerous goods, security initiatives

1 Introduction The major responsibilities of the NATO Planning Board for Inland Surface Transport (PBIST) are to ensure that inland surface transport serves NATO’s civil and military needs with maximum effectiveness by developing, reviewing and advising on Alliance policies, arrangements and procedures. This includes Railroad, Road and Inland water transport as well as the use of seaports and emergency anchorages. The PBIST is also called upon to pursue new emerging technologies and assets suitable for meeting the evolving requirements of the Alliance as far as Inland Surface Transport is concerned and to make recommendations thereupon. The PBIST is supported by a permanent Working Group which actually conducts and develops the activities and studies as required. The Board reports to the Senior Civil Emergency Planning Committee (SCEPC) and, through it, to the Council. All member and Partner nations are entitled to participate in PBIST’s activities. NATO military representatives also normally attend its meetings in order to maintain the necessary co-ordination of projects.

2 General projects Drawing from a vast array of technical expertise, the PBIST engages in several studies and activities. Some are purely technical, such as the recent volume on the “Technical aspects of the rail transportation of military material” while others are of a more general nature, such as the development of an “Agreement on the E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

17

18

R. BERTINI

facilitation of vital civil cross border transport” to ease border crossing procedures in case of disaster relief operations. It goes without saying that Inland Surface Transport is always involved in Civil Emergency Planning activities. The PBIST therefore has an active role in activities and studies performed by other NATO bodies and Planning Boards and Committees, insofar as Inland Surface Transport is involved. Lately, in particular after September 11, issues related to Critical Infrastructure Protection (CIP) have gained greater significance amongst our projects.

3 The PBIST and critical infrastructure protection Broadly speaking, the necessity of protecting critical infrastructure is not something new. Natural disasters or human errors have always been a concern for governments, International Organizations and populations. However, throughout the years we have witnessed an evolution of the concept of critical infrastructure protection which, after September 11, finally developed into a new and broader approach. The common understanding that in the quest for protection modern society must be modest has also been reached. We no longer believe that we can protect any asset 100%. Even though there is a general common understanding of what can be considered as a critical infrastructure, the definitions can vary from country to country and from one International Organization to another. The transportation system however, features as Critical Infrastructure in all lists drawn up by Nations or International Organizations. It persists, in effect, as an attractive target for terrorist activity due to, inter alia, its relative accessibility and the large numbers of people using it. Moreover, the growing sophistication of the Inland Surface Transport industry increases its vulnerability. Areas of major concern are, to name a few, seaports, intermodal terminals, choke points in the transport network, communications-information services, support facilities for fuel and other energy sources. The PBIST, in addressing the problems, prospects, issues, and strategies of Critical Infrastructure Protection, has stressed the need for a detailed understanding of threats and vulnerabilities as a prerequisite for the formulation of any counter-measure. The debate revolved moreover on the question of what we really want to protect: static infrastructure (an essential bridge, a tunnel which, if obstructed, could severely limit transport, etc.) or services, the physical and electronic information flows, their role and function for society, the core values that are delivered through the infrastructure? From what perspective should we approach CIP: from a technical perspective, a business perspective – with its preoccupation of business continuity (which is not merely focused on the system level but includes organizational and human factors) – or finally, from a national security perspective involving government

PBIST AND ITS PROJECTS, WITH FOCUS ON PORT SECURITY

19

officials from different agencies, representatives of the private sector and of the general public? Following deliberations, the PBIST, considering the various security problems to be contended with by all Inland Surface Transport modes of transport (Rail, Road, Inland water, Intermodality, Port activities), opted to select at least one International Expert – who would also be conversant with security issues – in each of the above mentioned fields of expertise. The PBIST also issued the EAPC Nations with a detailed questionnaire in order to take stock of how different nations, with different economic and political backgrounds, perceive the general problem of CIP. The questionnaire also included queries on risk assessment analysis and the existence of consequence management plans to be implemented after a catastrophic event has occurred. Nations are particularly willing to participate in this study because they surmise that, even though CIP is at the core of an integrated homeland security concept, a coordinated approach to security is essential, both within governments and among countries in a multilateral framework. Joint activities and coalitions between research, government, and private actors at the national and international levels would very much help in addressing Critical Infrastructure Protection, with particular emphasis on the transport chain as a whole. However, this effort towards a coordinated approach is sometimes hampered by the sensitiveness of some of the information solicited from Nations. In our case, for example, the final compilation of National inputs would have to be classified NATO/EAPC Confidential.

4 Seaport projects The PBIST is also responsible for Seaport activities which, together with Railroad activities, represent the crown jewels of our Board. Seaports are a perfect example of Intermodal Hubs, where maritime transport unites with the more specific Inland Surface Transport modes. All at one go, we can find rail transport, road transport, intermodal transport and inland water transport. Denial of access to/exit from the port/terminal, whether from the water or from onward routes, strikes at the basis of the service. The PBIST is particularly keen on focusing on Seaport problems. As far as Seaport activities are concerned, the PBIST and its WG are currently actively following a series of issues, outlined below: x Several studies in support of the NATO Military Authorities such as: –

A better utilization of commercial practices in the development of Reception, Staging and Onward Movement procedures. The Hub Port Concept, developed in cooperation with the Planning Board for Ocean Shipping (PBOS) is of particular note. It applies an old commercial notion to military deployment operations: the utilization of large and well

20

R. BERTINI

–

–

equipped ports in peaceful and well organized Regions, as close as possible to the areas of operation, to discharge the modern enormous Container-ships and Roll on-Roll off ships. Smaller feeder vessels would then be utilized to reach the final Seaports of Debarkation; these may have a smaller capacity, may be more dangerous, and simpler as far as logistical equipment is concerned. The population of NATO databases, with real-life information on capabilities and capacities of Seaports in areas of interest to the military. The aim is to have a set of real-life information readily usable by NATO Military Planners to help them in preparing preliminary movement plans. The identification of the limitations affecting the traffic of ammunition and hazardous material in Seaports. This too is crucial information for NATO Military Authorities when preparing contingency plans for the deployment and sustainment of their operations. The international agreement in which all other dangerous goods controls are grounded is the United Nations Recommendations on the Transport of Dangerous Goods (UNRMTDG), also known as the ‘Orange Book’. However, Nations may grant special temporary waivers of the requirements of the UN Agreement. This somewhat confuses matters, whereupon the PBIST was asked to make a general survey of varying national situations.

x Studies on existing conventions governing security procedures in Seaports. As concerns the task imparted by the Senior Civil Emergency Planning Committee, the PBIST continues to monitor work and activities of international and national organizations or authorities which are responsible for developing security standards in the field of Inland Surface Transport. Setting aside discussions on the International Ship and Port Facility Security (ISPS) code, which are presented in other chapters of this book, the PBIST: – –

Prepared a compendium on the new Security Rules adopted by the United States in order to add additional security protections to US ports. Published a Report on the activities and actions taken by the EU and by the US to develop security measures and initiatives for detecting weapons of mass destruction and deterring terrorists from exploiting the vulnerabilities of containerized cargo (the so-called Container Security Initiative).

x The identification of the main problems related to the Protection of Seaports. This project is partly included in the general study on Critical Infrastructure Protection. However, the complexity of protecting a Seaport as a whole, including the interior road, rail and inland waterway terminals, must also be taken into account.

PBIST AND ITS PROJECTS, WITH FOCUS ON PORT SECURITY

21

5 Conclusion The protection of critical infrastructures involves a variety of actors and issues with a wide spectrum of different terminologies and concerns. What matters is the definition of common goals and values and a comprehensive approach to risks and vulnerabilities, as technological and social developments have created a complete interdependence between transport activities and other fields of activities apparently unrelated to them. Assessing and addressing the security threat, translating that threat into effective and efficient policy measures to mitigate the threat, while considering costs – who will pay for the measures, how these costs weigh against their benefits, and how tighter security can be reconciled with the safe and efficient transport of persons and goods throughout the economy – these are just some of the emerging transport security policy challenges to transport decision-makers in today’s transport security policy environment. Strategies to effectively deal with vulnerabilities in critical infrastructures, ranging from reforms of traditional instruments that enhance the protection of national assets to methodological approaches that emphasize analysis and consequence management, should be sought and any contribution to this effort, even a minor contribution, is important. The NATO Planning Board for Inland Surface Transport is ready to contribute in any possible and useful way to this undertaking.

IMPLEMENTATION OF THE ISPS CODE IN THE RUSSIAN FEDERATION: SHIPS AND PORTS

SERGEI NOVIKOV TRANSAS Eurasia, St. Petersburg, Russia

Abstract. Maritime security has always been given high priority in Russia. On April 22, 2000, the Government of the Russian Federation adopted Regulation 324, approving the federal statute of protection for the maritime industry from unlawful acts against the safety of maritime navigation. The paper describes measure taken and planned to ensure implementation of Chapter XI-2 and ISPS Code in the Russian Federation. Keywords: ISPS code, port facility, SOLAS, security assessment, security measures, ship security alert, resource allocation

1 Introduction Maritime security has always been given high priority in Russia. On April 22, 2000, the Government of the Russian Federation adopted Regulation 324, approving the federal statute of protection for the maritime industry from unlawful acts against the safety of maritime navigation. According to Regulation 324, the main task of the federal system is to fortify the security of vessels, their crews, passengers and cargo, navigation control units, communication and navigation, port facilities and personnel. The implementation of assigned tasks for the management and coordination of protective measures to be undertaken was entrusted to the Ministry of Transport of the Russian Federation and its departments. This work had been conducted in close cooperation with other federal authorities.

2 Chapter XI-2 and ISPS code implementation in the Russian Federation 2.1 MARITIME SECURITY SERVICE (MSS) In December 2002, the Ministry of Transport of the Russian Federation launched the implementation of the requirements listed in Chapter XI-2 of SOLAS and the E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

23

24

S. NOVIKOV

International Ship and Port Facility Security Code (ISPS Code), subsequent to their adoption by the International Maritime Organization (IMO). The Maritime Security Service (MSS) of the Ministry of Transport was appointed as the organization responsible for the implementation of Chapter XI-2 of SOLAS-74 and the ISPS code. The MSS fulfils the following functions: x Approval of the security plans of ships hoisting the Russian flag x Submission of the port facility security assessment and port facility security plan for approval x Verification of ships’ compliance with ISPS Code requirements and subsequent issuing of the International Ship Security Certificate x Verification and registration of port facilities in accordance with approved security plans x Systemized training of security officers x Coordination, organizational and methodical support of the activities of the maritime security service established in ports x Cooperation with IMO. Owing to the efforts of the MSS, a security assessment system and security measures for ships engaged in international voyages and for port facilities serving such ships has been established. Its approval by maritime authorities, the verification of ships and port facilities’ compliance with ISPS Code requirements was conducted according to the established scheme. As a result, all ships hoisting the Russian flag and engaged on international voyages are now in compliance with ISPS Code requirements. International Ship Security Certificates were issued to 1,544 vessels. The security assessment of port facilities and the approval of security plans were implemented in ports that handle international shipping for the Russian Federation. The security plans of 178 port facilities were approved, and they were entered into the IMO database. A training system for security officers of all levels was established. Fifteen training centres in maritime academies and colleges are involved in the training of maritime security specialists. More than 5,500 ship, company and port facility’ security officers have been trained there. 2.2 SHIP SECURITY ALERT SYSTEM (SSAS) In pursuance of Regulation 6 of SOLAS Chapter XI-2, plans were drawn for the development, engineering and deployment of a Ship Security Alert System (SSAS) on all vessels in compliance with SOLAS requirements. Russian ship owners had been eager to implement the security alert system to meet the new SOLAS requirements, which came into force on all ships of 5,000 gross tonnage and upwards engaged on international voyages after July 1, 2006. The Russian Maritime Register of Shipping is charged with carrying out control measures on board vessels to fulfil the technical requirements for the

IMPLEMENTATION OF THE ISPS CODE IN THE RUSSIAN FEDERATION

25

installation of the SSAS and executes procedures for the technical supervision of equipment. The installation of the equipment is carried out by enterprises approved by the Russian Maritime Register of Shipping according to a list compiled in cooperation with the MSS. The Maritime Rescue Coordination Centre provides around-the-clock reception and monitoring of alert messages. 2.3 WITHIN THE IMO’S GLOBAL PROGRAM ON MARITIME AND PORT SECURITY The first workshop “On Maritime and Port Security” was held in St. Petersburg on March 22–24, 2004 as part of the IMO’s Global Program with support of the Ministry of Transport of the Russian Federation. As a result of this seminar, a “Resolution on the implementation of maritime security measures for the Russian Federation was adopted.” On November 23–24, 2004, the First National Seminar on maritime security took place in St. Petersburg. Its topic – “ISPS Code implementation in the Russian Federation: Initial Results and Future Trends.” Such events are indicative of active measures taken by the Russian Federation to implement national and international requirements intended to ensure maritime security. Representatives of practically all state authorities, first-rate shipping enterprises, educational and scientific institutions of the maritime industry took part in the seminar. The purpose of the discussion was to summarize the results of work carried out under the new international standards of maritime security, to exchange the experience gained in this endeavour and inform participants of the IMO’s recent activities in the sphere of maritime security. The seminar called attention to the fact that the main task of the Contracting Governments implementing the ISPS Code is to prevent terrorist attacks against ships and port facilities, build new obstacles for piracy and armed robberies at sea and to restrict as much as possible the transportation of illegal drugs, cargo and passengers. The seminar was also meant to initiate the development of collective recommendations for optimizing ISPS Code implementation in Russia, whereupon necessary guidance was issued in the resolution of the seminar. At the conclusion of the discussion, participants came to the agreement that maritime transport entities (ships and port facilities) not involved in international sea trade must also be protected by additional security measures. In order to do this existing experience gained from ISPS Code’s initiation into the maritime community can be utilized. The successful implementation of this agreement would necessitate the adoption of a federal law “On transport security” amended by relevant national security requirements. This federal law is already under consideration by the Government of the Russian Federation.

26

S. NOVIKOV

The current task of Russia’s maritime authorities is to ensure that ships and port facilities are secured in accordance with the ISPS Code. Necessary spadework is already at a final stage. 2.4 RESOURCE ALLOCATION One of the main aims of ensuring transport security is the stable and safe functioning of port facilities and protection from acts of illegal interference, such as: x x x x

Unauthorized access Diversions and terrorist attacks Theft of goods, equipment and financial assets Sabotage and deliberate destruction or damage of equipment

Port facilities will be categorized and provided with engineering and technical tools for the integration and maintenance of security systems according to category. Categorization will be determined by the Federal Agency of Sea and River transport in accordance with procedure established by the Ministry of Transport in coordination with the Federal Security Service and the Ministry of Internal Affairs of the Russian Federation. Categorized entities will be entered into the national database, under the authority and supervision of the Ministry of Transport of the Russian Federation. Conditionally, all port facilities will be divided according to the extent of potential danger and possible threats into three categories: of high (1), medium (2) and low (3) danger, as per the specially designated “Methods of defining the category of a port facility.”

3 Conclusion The implementation of all actions planned for the near future will effectively increase the level of security in port facilities of the Russian Federation. The Russian maritime community contributes a great deal to implement ISPS requirements and to train ship, company and port facilities’ security officers. The strict compliance and fulfilment of the SOLAS-74 Convention and ISPS Code requirements is a vital necessity nowadays. But this is only one more brick in the wall that we must build together in order to protect our nations against the threat of global terrorism.

STEPS TO BETTER WATERSIDE PORT AND HARBOR SECURITY: THE DEVELOPMENT OF REGIONAL MARITIME SAFETY SYSTEMS IN THE RUSSIAN FEDERATION

ALEXANDER SOKOLOV TRANSAS Eurasia, Russia

Abstract. One of the most important components of port facility security is the complex of organizational and technological solutions intended for providing maximum navigation safety in coastal waters, on anchorage approaches, and in port harbors. The paper describes regional Maritime safety system developed by Transus. Keywords: Regional maritime safety system, coastal surveillance, CCTV camera, crisis management

1 Introduction One of the most important components of port facility security is the complex of organizational and technological solutions intended for providing maximum navigation safety in coastal waters, on anchorage approaches, and in port harbors. These security measures are necessary not only for implementing anti-collision measures and preventing potential, sometimes catastrophic, consequences for the environment but also averting destruction of the port infrastructure in the face of the sharply risen global terrorist threat, and most importantly, avoiding, human casualties in neighboring regions. The financial and human resources that can potentially be used by international terrorist organizations for attacking the most important transport centers of world economy (including large maritime ports, oil and chemical terminals) are really vast. The task of creating safety systems capable of countering existing and potential threat levels is literally a matter of the human race’s survival in the 21st century.

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

27

28

S. NOVIKOV

2 Regional maritime safety system Modern automated regional maritime safety systems should provide Seaside port facility security while ensuring compliance with constantly expanding requirements of the International Conventions defined by the International Maritime Organization and other international agreements on security and protection from unlawful acts. The reminder of this section describes the elements of the Regional maritime safety systems of the Russian Federation as well as a Regional maritime safety system operating in the eastern part of the Gulf of Finland built in 2001– 2004 by Transas. The Regional Maritime Safety System is a single soft and hardware complex with a high degree of component integration. This working system is built on the Data Fusion Technology principles as applied to maritime security. It includes: x x x x x x x x

Vessel traffic management systems GMDSS for sea area A1 and A2 A ramified network of microwave lines and multiplexers Coast UAIS stations A joint database Object power supply and life support subsystems Remote monitoring of equipment operation Meteorological stations and other subsystems

The System links eight main sites: Primorsk, Karavaldai, Kotlin Island, Petrodvorets, Vysotsk, Lomonosov, Ust-Luga and St. Petersburg. The main tasks performed by the RMSS are: x Collection of information from different sensors about on the navigational situation x Operational control over vessel traffic x Vessel traffic scheduling x Integration of data on targets and its presentation in tabular and graphic form with full multi-layered electronic charts x Analysis of the navigational situation and alarm message generation as per the operator’s set criteria x Provision of auxiliary navigational information and other data x Automatic identification in large part of a dangerous situation and actuation of appropriate alarm signals x Digital recording and storage of most data (targets motion, displayed messages, radar scanning of the area, exchange between operators, signals from video cameras) for a complete playback of situation dynamics within a chosen time segment Moreover, the system’s modular structure permits easy upgrading without the additional expenditure of already installed components. Based on this system

STEPS TO BETTER WATERSIDE PORT AND HARBOR SECURITY

29

modular structure, Transas has developed a Coastal Surveillance System called “Navi-Monitor”specialized in providing harbor and port facilities security. NaviMonitor is a high-tech surveillance and alert system derived from the popular Navi-Harbor Transas VTS system and specifically adapted to the needs of small and medium-size ports, guarded coastal objects, and offshore platforms. NaviMonitor is ideally suited for meeting the Port Facility Security requirements set forth in Chapter 11 of the SOLAS-74 and the ISPS Code and provides an efficient and flexible solution for oil and chemical terminals and other dangerous cargo processing coastal facilities (falling under the ISPS Code requirements), as well as civil facilities vulnerable to terrorist attacks from the waterside such as river or harbor bridges and other waterside constructions. The next section described Navi-Monitor in more details.

3 Navi-Monitor The Navi-Monitor performs the following functions: x Monitoring and identifying ships and other navigation objects in coastal waters x Generation of alarms and notification messages in accordance with preset criteria x Continuous recording of the data on navigational situation for further playback and analysis x Continuous and synchronous recording of radio communications and video from CCTV cameras for further playback and analysis x Integration the CCTV system with radar/AIS tracker to enable automatic CCTV focusing on a selected ship x Comprehensive presentation of navigational situation to the operator(s) x Providing a fully customized user-friendly operator interface x A complete set of sensors such as Radar(s), UAIS Base Station(s), controllable CCTV camera(s) and Thermo vision camera, VHF Radio, Direction Finder(s) x Remote monitoring of equipment status and environmental conditions x Sensor remote control x Flexible system configuration x Remote Internet access x The possibility to upgrade to the full-mission Navi-Harbor VTS system without dedicated hardware replacement The Navi-Monitor system may comprise several or all of the following subsystems: x x x x

Radar site Universal AIS Base Station CCTV camera system VHF radio station

30

x x x x

S. NOVIKOV

VHF Direction Finder EPIRB/MOB Direction Finder Navi-Monitor Operator Display Unit (ODU) User terminal to VTMIS Navi-Monitor has the following characteristics:

x x x x x x

Maximum processed range 32 nautical miles Range resolution rate 3.75 m Maximum target number 500 per radar site Maximum tracked target speed 70 knots Maximum acceleration 4 m/s Target tracking characteristics: -

Position accuracy better than 15 m Heading accuracy better than 2 degrees Speed accuracy better than 0.5 knots

The Navi-Monitor provides data with the UAIS base station via interfaces compliant with international standards. The Transas T201-01UAIS Base Station is available in a single or dual configuration. The UAIS Base Station detects AIS equipped targets within VHF radio coverage at a distance of up to 50 nautical miles. The Close Circuit Television (CCTV) camera can automatically track any target chosen by the operator. Its functional capabilities are: x x x x

One-touch camera focusing on the selected target One-touch camera focusing on an arbitrary chart object Capability to follow a selected target Graphic presentation of current CCTV camera position, camera azimuth and angle of view on the electronic chart x Manual control both of the camera’s zoom/pan and tilt angles x Control of service functions (wiper, washer, etc.) Such monitoring systems can be installed and used not only on the waterfront of harbors and for anchorage protection, but also to protect maritime fixed drilling platforms, which can be the possible object of terrorist attacks. The installation and use of this system restricts makes it somehow difficult to implement concealed monitoring of the guarded area or object.

4 CMS (Crisis Management System) This section addresses the high-level Data Fusion (DF) technologies for harbor protection. DF technologies allow the estimation and prioritization of the relative criticality of threats and the prediction and assessment of the situation. In cooperation with the Federal Center of the Emergency Response Institute of

STEPS TO BETTER WATERSIDE PORT AND HARBOR SECURITY

31

Russia, Transas has developed an automated system to monitor and analyze the situation, and provide decision support for emergencies of varying natures and scales. The system is called CMS (Crisis Management System) and incorporates tools for the visual presentation of information (including the real-time tracking of personnel and vehicle movements), planning, data storage, and the search for information and communication networks. The system’s components are designed to support damage estimation, planning of response actions and general control of counter-crisis operations. Main CMS functions are: x Timely notification of agencies and personnel concerned with critical situations x Setting up an effective communication system x Ensuring the safety of citizens and personnel involved in the elimination of an emergency x Control, monitoring, deployment of resources and tracking the positions and activities of resources and personnel x Manage resources for implementing strategy and tactics necessary to a prompt response while performing regular screening to allow rapid resource reallocation as required x Visual presentation of the current status of the accident and status of the resources involved x Providing accurate, consistent and timely information to the mass media and agencies concerned x Control over financial expenditures involved in the elimination of an emergency and in logistics System components are grouped into modules according to the nature of tasks to be accomplished. The user can therefore choose all the necessary components, and the modules can be added or removed as required. The Crisis Management System can be used in situations such as forest fire, oil spills, floods, accidents on potentially dangerous objects, search and rescue operations in sea, and acts of terrorism. According to signed contracts with the Admiral Makarov State Maritime Academy (AMSMA) and the Ministry of Civil Defense and Crisis Management, state-of-the-art simulators with 3D visualization capabilities are being developed for training specialists on various types of threats. The creation of such simulators intended for the advanced training of ship and port facilities security officers in accordance with ISPS requirements is also planned under the aegis of the Maritime Security Service of the Russian Federation’s Ministry of Transport.

32

S. NOVIKOV

5 Conclusions This paper briefly describes some elements and characteristics of a Coastal Surveillance System called N “ avi-Monitor” specialized in providing harbor and port facilities security as well as a Crisis Management System designed to provide monitoring and analysis of the situation, and decision support for emergencies of varying natures and scales.

Acknowledgements I would like to thank the Organizing Committee for the opportunity to participate in this seminar, which, as we believe will provide an additional incentive for further developments in the application of the principles of Data Fusion Technologies.

POLICE NATIONAL MARITIME SECURITY STRATEGY

JAMES DOUGLASS Deputy National Coordinator, Ports Policing, UK

Abstract. This paper describes a national strategy aimed at ensuring the delivery of a protective security regime that will ensure the seamless protection of portal areas and internal waterways from acts of aggression. Keywords: National Maritime Security, enforcement agency, ISPS Codes, threat and risk assessment, protective security

1 Introduction The Police National Maritime Security Strategy is a national strategy to protect public and UK interests in maritime regions by making them a harder target through better protective security. It is a joined-up partnership approach making effective use of current enforcement agency and partner resources. Most portal regions have a large number of facilities that are of national significance strategically and economically. Often in close proximity to centers of population, they are an attractive target for terrorists. Potential methods of attack by terrorist groups could include using vessels as weapons or targeting cruise ships & ferries to cause mass casualties. Additionally, there is a sub surface threat to security zones and key installations. We must utilize intelligence to make the best use of our resources, however we cannot rely on intelligence only to forewarn us of an attack. Learning from good practice at home and abroad, the strategy utilizes the resources of all agencies that have an interest in the portal business area. It includes the following elements: x x x x x x x x

Identify portal regions across the UK. Identify vulnerable sites and installations & develop domain awareness. ISPS codes. The maritime MATRA process. Regional maritime information & intelligence teams. Regional maritime response teams. Joint portal operational plans. Joint portal contingency plans.

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

33

34

J. DOUGLASS

x Joint operations command & control arrangements. x Supportive management structures. The remainder of the paper presents more detail description of these elements.

2 Elements of the police national maritime security strategy 2.1 PORTAL REGIONS ACROSS THE UK Portal regions often cut across police force/regional boundaries and partner agency business areas. Local agreements are necessary to identify the best individual arrangement for each portal region. The National Coordinators office will maintain a central overview. Some portal regions are served by more than one police force, which allows for the brigading of resources. Joint agency groups are being set up and the resources of these agencies are being examined to see where they can contribute to the security of the portal region. 2.2 IDENTIFICATION OF VULNERABLE SITES, INSTALLATIONS AND DEVELOPMENT OF DOMAIN AWARENESS Lord Carlisle mentioned the ability of police officers to spot a criminal in a crowd. He described it as ‘a coppers nose.’ This ability is developed by officers (and others) as they get to know their environment, enabling them to spot the unusual. There are a number of sources of information available to improve domain awareness in the maritime environment. These include Police, MDP, VTS Ops, QHM Ops, RNLI, Sea Rescue Associations, Pilots, Harbor Masters, Marina Staff, Boatyard Staff, HM Customs, Customs Cutters, UKIS, MCA, MRCC, MOD and the Fire Service. There are also a number of technological sources such as Seasearcher, Lloyds registry, CCTV, Radar, SONAR etc. Drawing this information together creates a knowledge base that will heighten awareness of the abnormal. 2.3 THE INTERNATIONAL SHIP & PORT FACILITY SECURITY (ISPS) CODES The ISPS codes deal with security at the ship/port interface. Although the new EU direction extends this to cover the whole facility, much more is needed to protect the whole portal region.

POLICE NATIONAL MARITIME SECURITY STRATEGY

35

2.4 THE MULTI AGENCY THREAT AND RISK ASSESSMENT (MATRA) PROCESS Post 9/11, Sir John Wheeler undertook a review of airport security. In his report, he recommended that a joint agency threat and risk assessment should be carried out at all NASP airports. This is a voluntary process that has proven very successful in the airport environment where joint agency teams examined the threat and risks to the airport, looked at the mitigation already in place and then agreed in joint agency plans to improve security. The Multi Agency Threat & Risk Assessment (MATRA) process is a more inclusive process that will engage all agencies and consider the wider portal area.

2.5 THE REGIONAL MARITIME INFORMATION & INTELLIGENCE TEAM (ReMIIT) Multi disciplinary Regional Maritime Information & Intelligence Teams are being developed to draw together all information and intelligence relevant to the maritime interface. They will draw upon information from sources already described and develop an analytical expertise. The work of the team would include ship profiling to identify risk vessels and the examination of crew lists etc. Ideally they should be co-located with access to various information systems to improve domain awareness and enable the proactive deployment of response teams. 2.6 REGIONAL MARITIME RESPONSE TEAMS (ReMReT) ReMReTs utilize the resources of all agencies/partners in a coordinated effort. A menu of options can be prepared that identifies the routine measures that will be undertaken by each organization to provide a joined-up capability for each threat level. A waterborne Tactics Manual has also been prepared. This assists police forces when developing their operational plans. 2.7 JOINT PORTAL OPERATIONAL PLANS Joint Portal Operational Plans document the joint agency activity necessary to detect, deter or disrupt waterborne and landside attacks within the portal region. These plans are comprehensive and they collectively provide a coherent security framework for the whole of the portal area. A Menu of Options is contained within this plan documenting the actions each agency has agreed to undertake. The plan would also include target hardening and other activity undertaken such as: exclusion zones, detection equipment, sub surface protection and random & non-random interdiction etc.

36

J. DOUGLASS

2.8 JOINT PORTAL CONTINGENCY PLANS The Civil Contingencies Act requires all agencies to collectively work towards Joint Contingency Plans. Such plans include the response to all incidents and possible acts of aggression whether committed landside, waterside or offshore. The plans of all agencies will be collectively examined and jointly rewritten to ensure they provide a seamless response. 2.9 JOINT OPERATIONS COMMAND & CONTROL CENTERS (JOCCC) The formation of Joint Operations Command & Control Centers (JOCCC) enables enforcement agencies, HMCE, UKIS, VTS, QHM, and the MCA to work together to ensure the most effective coordination of their resources.

2.9.1 Supportive management structures Portal regions are setting up appropriate Management Structures to ensure the delivery of the security strategy. A Strategic Group with senior managers from the agencies concerned sets the overall security strategy for the portal area. A Regional Management Group then determines the projects and work-streams necessary to achieve the strategy.

3 Conclusion The strategy described in this paper will ensure the delivery of a protective security regime that will help protect portal areas and internal waterways from acts of aggression. The police response to a terrorist incident involving ships at sea is a separate matter that is defined in the Manual of C/T Contingency Planning Guidance. This strategy builds upon good work already in place across the UK. Good practice identified in the Thames, Solent and other portal regions is being shared with colleagues. The Sea Port & International Rail terminals Group enables the sharing of good practice and will monitor the delivery of the strategy for the UK. Key figures within the police service support this work, including Assistant Commissioner Sir David Veness (who chaired the Thames Partnership Group and Chief Constable Barbara Wilding (South Wales) who provides the necessary strategic support to ensure the delivery of this strategy across the UK.

HARBOUR PROTECTION IN THE JORDANIAN PORT OF AQABA

BASSAM KAKISH Director General of Aqaba Port Authority, Jordan

Abstract. This paper reports information regarding the Jordanian Port of Aqaba, its capabilities, and all steps taken to ensure the security and safety of the port and to prevent any terrorist action from taking place. Keywords: Harbour protection, Jordan Maritime Authority, ISPS code, security assessment

1 Introduction This paper reports information regarding the Jordanian Port of Aqaba, situated on the tip of the Red Sea, its organization, and capabilities, and all steps taken by concerned parties in the field of harbour protection, in cooperation with parties involved in ISPS Code Implementation, to ensure the security and safety of the port and to prevent any terrorist action from taking place.

2 Jordan Maritime Authority The Jordan Maritime Authority (JMA), a specialized administration under the umbrella of the Ministry of Transport in Jordan, is responsible for the implementation of the SOLAS1 Convention provisions and the ISPS Code on Jordanian ships as well as on foreign ships crossing Jordanian Territorial Waters. In addition, the JMA is the designated authority for the port facilities security plan follow-up and approval. The JMA is responsible for supervising the comprehensive implementation of the maritime security regulations provisions (SOLAS), the International Ship and Port Facilities Security Code (ISPS), related national maritime security legislations in Jordan and all aspects of maritime transport security arrangements. The JMA assumes responsibility for executing follow-ups on Jordanian Flag ships including cargo ships, passenger ferries and cruise ships, so as to insure the 1

SOLAS: Safety of Life at Sea

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

37

38

B. KAKISH

fulfillment of the SOLAS Convention and ISPS Code requirements. The JMA also performs the security control of foreign ships under regulation 9 of the SOLAS XI-2 (control of ships in ports) within the Port Control System and in harmony with the MED MoU on PSC.2 The Aqaba Ports Corporation is a governmental institution, which supervises and carries out all port operations and activities; it is thus responsible for the security of port facilities. Recognized Security Organizations (RSOs) have been appointed in Jordan to carry out ship security assessments, prepare security plans and issue the International Ship Security Certificate (ISSC) by delegation of the JMA. An RSO has also been appointed for port security risk assessment and port security planning. All relevant security officer training courses for Port Facility Security Officers (PFSO), Company Security Officers (CSO) and Ship Security Officers (SSO) must be approved by the JMA. The first series of courses have already been approved and some security officers have been trained.

3 Legislation The new chapter XI-2 of SOLAS and the associated ISPS Code has been given direct effect in Jordanian Law. Secondary legislation under Jordanian Maritime Law will cover the amendments of SOLAS chapters V and XI-1. Another resolution has also been adopted by the JMA’s board of directors and has been published in the official gazette on 16/5/2004. This resolution includes a set of regulations to enhance ship and port facility security as well as instructions for the guidance of surveyors conducting ship security assessments, for preparing and approving security plans and for issuing the ISSC.

4 Ships Jordan has 24 ocean-going vessels, totalling 377.578 GRT,3 registered under the Jordanian Flag and covered by the new chapter XI-2 of SOLAS and the associated ISPS code. The JMA, as the maritime authority of the state, has taken all necessary measures to comply with the objectives of the code and the standards stipulated in the relevant chapters of the SOLAS convention. The JMA finalized Ship Security Plans (SSP) for Jordanian Ships: all SSPs were submitted to the JMA by June 25th 2004 so that they could be enforced at least 5 days before the final verification and consequent issuance of the ISSC. SSPs for ocean-going vessels have been reviewed and approved by the Det Norske Veritas (DNV), Lloyd’s Register (LR), Bureau Veritas (BV), Nippon Kaiji Kyokai

2 3

MED MoU on PSC: Mediterranean Memorandum of Understanding on Port State Control GRT: Gross Registered Tons

HARBOUR PROTECTION IN THE JORDANIAN PORT OF AQABA

39

(NKK) and Hellenic Register of Shipping (HRS) classification societies acting on behalf of the JMA.

5 Port facilities The Aqaba Port Corporation has worked with the recognized security organization (RSO) Global Village Strategies in co-ordination with the JMA to make necessary arrangements by June 24th 2004. An initial security assessment was prepared for Aqaba Port facilities in Jordan on 24/6/2004, which notified all port facilities that the security system should be applied. A security plan draft was prepared (covering all individual port facilities). An initial evaluation was completed by RSO (Global Village Strategies) and the final approval for individual port facility security plans was given by the JMA and signed on 30/6/2004. Procedure followed by the concerned authorities of Jordan to implement the ISPS code in the port of Aqaba: x Formation of a committee of authorized people from the Ministry of Transport, the Ports Corporation, and the Jordan Maritime Authority in accordance with the April 1st 2003 resolution of the Minister of Transport. x The committee had monthly meetings in Aqaba and studied each article of the ISPS code to determine which procedures should be implemented in order to prepare the required report and comply with all the demands of the ISPS code regarding both ship and port security. x The committee decided to summon a forum on ship and port security with the International Maritime Organization (IMO) in order to activate the ISPS code. x The Jordan Maritime Authority corresponded with the IMO as of November 2003 so as to hold this forum within the cooperation and assistance program presented to Jordan by the IMO. x The forum was held from January 24 to 28, 2004 by the Jordan Maritime Authority in Aqaba, and attended by IMO experts on technical assistance as well as by representatives of the Ministry of Transport, the Ports Corporation, the Jordan Maritime Authority, the Association of Maritime Agencies, Aqaba Customs, Ship Management Companies and the Security Authorities. x The committee started the actual preparations in February 2004 to specify the relevant actions to be taken by the Ports Corporation regarding port security and by the Jordan Maritime Authority regarding ships. Procedure followed by The Jordan Maritime Authority for the preparation of the security report: x Special instructions were prepared for the implementation of the SOLAS convention and the ISPS code for both ship and port security, and were published in the official paper.

40

B. KAKISH

x The Jordan Maritime Authority deputized seven international classification societies to inspect and perform security checks on ships and to issue security plans and certificates for ships hoisting Jordan’s flag. x All individuals in Jordan who are concerned with implementing the ISPS code on ship and port security where put through a training plan. x The Jordan Maritime Authority, under the coordination of the MED MoU on PSC Secretariat, implemented special marine security measures to enhance regional cooperation regarding the control of ships entering Jordan’s ports. x Identity forms for Jordanian mariners and ship crews were prepared. x Follow-up of ship-management companies operating under the Jordanian flag was executed to insure that they would commit to the marine security measures before 1st June, 2004. The Ports Corporation’s committee representatives assigned to prepare the report followed the procedures listed hereafter with the assistance of port security, shipping and marine engineering specialists.

6 Port facility security assessment As stated in article 15 of the code, Aqaba port was divided into the following areas for the purpose of security assessment and planning: main port, specialized bulk berth, passenger Terminal, oil terminal, container terminal, and the whole port of Aqaba. The assessment covered the code’s main requirements, summarized as follows: x Identification and evaluation of the main assets and infrastructures to be protected. x Identification of possible threats to the assets and infrastructures and of the likelihood of their occurrence to establish and prioritize security measures. x Identification, selection and prioritization of countermeasures, procedural changes and their level of effectiveness in reducing vulnerability. x Identification of weaknesses, including human factors, in the infrastructures, policies and procedures. x Preparation of a report summarizing how the assessment was conducted, including a description of countermeasures that could be used to address vulnerabilities. The port facility security plan report was prepared according to article (16) of the code and port facility assessment report on May 9, 2004. It is geared toward establishing measures to ensure the protection of port facilities, ships, individuals and goods against dangers arising from accidents threatening security. It also includes measures taken by the Port Corporation to reinforce security in the port facility and to protect against vulnerabilities mentioned in the assessment report. Measures are taken on the three security levels.

HARBOUR PROTECTION IN THE JORDANIAN PORT OF AQABA

41

An international recognized security organization (R.S.O.) was chosen to review the port security assessment and security plan to ensure the port’s compliance to the ISPS code. The assessment and security plan were approved and the information was sent to the IMO through The Jordan Maritime Authority as a representative of Jordan’s government. A statement of compliance was issued to all port facilities on June 30, 2004.

7 Conclusion Aqaba Port is considered a safe port due to its location at the tip of the red sea. All ships sailing to reach it via the Suez Canal are checked along the way and checked again at the moment of entry into the red sea (Gulf of Aqaba). All ships are checked upon arrival into Jordanian waters (ship, personnel and cargo). The Jordanian Coast guard is on patrol day and night in cooperation with port security. All measures of safety and security are taken on the ground in all port facilities, berths, entrances, exits, storage hangars, etc. All available measures are relentlessly being taken to ensure the security and safety of the port and to prevent any terrorist action from taking place, especially given that, and as long as, the port of Aqaba is one of the major ports receiving and delivering imports to Iraq.

FEASIBILITY STUDY ON SURVEILLANCE AND INTERDICTION TECHNOLOGIES FOR THE PORT OF HALIFAX*

FRANCINE DESHARNAIS1, DALE BLODGETT2, ROBERT BUSH2, ROBIN DURNFORD2 AND BILL YAKAMOVICH3 1 DRDC Atlantic, Canada 2 Lockheed Martin Canada, Canada 3 Unisys Canada, Canada

Abstract. This paper presents the preliminary results from the Feasibility Study on Surveillance and Interdiction Technologies for the Port of Halifax, lead by DRDC-Atlantic and performed by two independent teams of investigators. The two teams identified and specified the requirements for Detection, Deterrence and Response to water and land threats for a multimodal seaport, identified the technologies that are currently in place or will be in place in the very near future, then identified configurations of sensors, command and control elements, detection aids, and organizational infrastructure that will meet the requirements and the technologies or developments needed to deliver the recommended configurations. An Initial Operational Capability of 2011 was assumed. This feasibility study was a first step for DRDC to better understand the Port Security problem. The results from this study will be used to define the scope and structure of a larger port security project where we hope to bring all stakeholders together to reach coordinated solutions. Keywords: Marine security, multimodal seaport, surveillance, interdiction, detection, deterrence, response, port domain awareness, command and control, decision making, emergency response, domain awareness, recovery, preparedness

1 Introduction Over the past few years, the Canadian government has increased its focus on National Security. The National Security Policy has outlined several key areas that * Post-ARW research results preformed within the scope of the PSTP program presented by Dr. Maria Rey at the ARW. Reported also at NATO TTCP meeting in 2007. E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

43

44

F. DESHARNAIS ET AL.

need to be addressed, such as Marine Security. Following these priorities, Defence Research and Development Canada (DRDC) has been given the additional mandate of providing research and development support to the new department of Public Safety Canada (PS). In response to this new mandate DRDC is growing its activities to include Marine Security. DRDC is presently conducting a feasibility study on surveillance and interdiction technologies for a multimodal seaport. The study is first identifying requirements for Detection, Deterrence and Response to water and land threats for a multimodal seaport (without being limited by current technology) and second, it will identify what technologies are required to meet these requirements and suggest how they should be developed and demonstrated. These technologies should improve Port Domain Awareness, Command and Control, decision making quality and increased tempo in emergency response to a safety and/or security related threat, and the overall security of the harbour and port facilities. Halifax, Nova Scotia, is the test bed for this study. This report will outline the context of the study, and present the methodology followed by two parallel contracting teams supporting DRDC in this work. Preliminary results for the first phase of the study will be summarized.

2 Context In 2002, the Government of Canada launched the chemical, biological, and nuclear (CBRNE) Research and Technology Initiative (CRTI) to provide science solutions to CBRNE terrorist threats. The initiative has been very successful, and led to the launch of DRDC’s Center for Security Science, a joint endeavour between DRDC and Public Safety Canada to provide Science and Technology services and support to address public safety and security. The center is managing the CRTI portfolio, and incoming portfolios of the Public Security Technical Program (PSTP). PSTP has four mission areas, as listed in Table 1. Many of the programmatic activities of DRDC Atlantic have direct links with the Marine Security agenda, especially in the areas of Maritime Domain Awareness, and port or harbour security. In the area of Port Security, DRDC Atlantic is leading the Underwater Force Protection, Command and Control, and Vulnerability and Recoverability components of a Technology Demonstration Project (TDP) on Maritime Force Protection. The TDP is heavily biased towards the protection of military assets within a port setting, and maps to many civilian port security issues. The Marine Security and Port Security agendas have links with all four PSTP mission areas, however the related activities will be managed mainly through the Disruption and Interdiction portfolio. As a result, the feasibility study discussed in this paper is biased towards surveillance and response activities; CBRNE technologies, though important in the context of port security, will not be emphasized.

FEASIBILITY STUDY ON SURVEILLANCE AND INTERDICTION

45

TABLE 1. PSTP mission areas Area

Description

CBRNE

Capabilities to prevent, prepare for and respond to CBRNE threats to public security, whether derived from terrorist or criminal activity, natural causes or accidents.

Critical Infrastructure Protection (CIP)

The robustness, reliability and protection of physical and IT facilities, networks, services and assets, which if disrupted or destroyed would have a serious impact on the health, safety, security, economic wellbeing or effective functioning of the nation.

Disruption and Interdiction (DI)

The ability to identify and stop terrorists/criminals and their activities, including surveillance, monitoring, disruption and interdiction of their activities at pertaining to border and transportation security.

Emergency Management and Systems Integration (EMSI)

The performance, integration and interoperability of national and international public security and emergency management capabilities and supporting systems, including the enabling standards, and vulnerability and systems analyses.

The main objective of this study is to understand the current state of port security technologies and capabilities in Halifax harbour, which was selected as a starting point. This current state will be analyzed in terms of requirements, vulnerabilities, and technology maturity, in order to better direct short-term research and development efforts in the areas that need it most. Ultimately, the results of the study will be used to define the scope and structure of a port security demonstration or test bed that DRDC would like to introduce in the next 1–2 years. Halifax was chosen as the test bed for this study for several reasons. Not only is it the third largest Canadian port, it is also a major multi-modal transportation hub, where waterways, roadways, railways, bridges and airports are found within a very small radius (~25 km) of the main harbour/inner city. Military and civilian infrastructures are located at the waterfront, and all three levels of government (federal, provincial and municipal) are represented within the city. In most regards, Halifax paints as complex a picture as one can find within the Canadian context.

3 Methodology This study is an exploratory exercise with two phases, as outlined below. Phase 1: x To identify and specify the requirements for Detection, Deterrence and Response to water and land threats for a multimodal seaport. These technologies should improve Port Domain Awareness, Command and Control, decision-making quality, and increased tempo in emergency response to a safety and/or security related threat, and the overall security of the harbour/port facilities. x To identify the technologies that are currently in place or will be in place in the very near future, which support these requirements in Halifax Harbour. The

46

F. DESHARNAIS ET AL.

task will also examine equivalent systems in other ports, to determine the range and maturity of the technology. Phase 2 is based upon the requirements reported in Phase 1: x To identify configurations of sensors, command and control elements, detection aids, and organizational infrastructure that will meet the requirements of Phase 1. x To identify the technologies or developments needed to deliver the recommended configurations. An Initial Operational Capability of 2011 is assumed. The study initiated in January 2007. To have more people thinking about the problem and increase our technology coverage, two parallel contracts were awarded to two separate contracting teams: Lockheed Martin Canada (LMC) and a joint team from Unisys Canada, xwave and CarteNav Solutions. DRDC was responsible for facilitating the interaction between the contracting teams and the Partners/Stakeholders in the security of Halifax harbour. The scope of the study was much larger than the commercial port, and included the Halifax Port Authority, Canadian Forces Base Halifax (Base Operations), Royal Canadian Mounted Police (National Port Enforcement Team), Emergency Measures Organizations (both the provincial and municipal units), Halifax Regional Municipality Police Department, Canadian Coast Guard (Marine Communications and Traffic Services), Public Safety Canada (Regional Office), Canadian Border Services Agency, Transport Canada, and Canadian Security Intelligence Service. Representatives from each of these departments and agencies were identified to provide information regarding the main threats for the region of Halifax (based on their respective points of view), and their technology capabilities, gaps and vulnerabilities. 3.1 LOCKHEED MARTIN CANADA For this study, Lockheed Martin Canada (LMC) worked closely with DRDC Atlantic to arrange interviews with stakeholders for security in the port of Halifax, and to gain access to a large range of relevant documentation. In addition, LMC drew upon the experience of subject matter experts from within the corporation. Because of the tight schedule for Phase 1, particularly in regard to arranging and conducting the interviews with stakeholders, an iterative approach was used in the determination of requirements and technologies. As more information was obtained, the specification, classification and relationships among requirements and technologies continued to evolve. Before each interview, a specific agenda was prepared to elaborate on the unique role of that particular stakeholder, and to help clarify the overall picture of requirements and current technologies for port security. The information that was gathered for Phase 1 was organized into three separate matrices for port security requirements, threats, and current technologies (Table 2).

FEASIBILITY STUDY ON SURVEILLANCE AND INTERDICTION

47

TABLE 2. Requirement areas and definitions, LMC team Requirement area

Definition

Domain awareness

The ability to know what is happening on, below and above the waterside, and on or above the shore side of the port.

Deterrence

The ability to protect facilities, infrastructure, vessels, personnel, and visitors in the port, and prevent threats from affecting these.

Response

If a threat does arise, or a security related incident cannot be prevented, these are the actions taken to counter the threat or deal with the immediate impact of the incident.

Recovery

In the event a security related incident does occur, these are the actions taken to restore the port environment to the pre-incident status quo.

Preparedness

Including activities such as the development and implementation of common procedures, contingency plans, training, and technological systems for information exchange and control.

The threat matrix identified the threats to port security, and described them according to their associated capability (measurable properties and characteristics of a threat, e.g., lethality, range/area of impact, speed), intent (behaviour characteristics that indicate a threat is a danger, e.g., a vessel operating in a controlled access zone, not responding to communications, deviating from schedule), opportunity (likelihood, and when/where a threat must be to pose a risk), and vulnerability (how a threat can be stopped, e.g., what countermeasures apply). Also shown for each threat are related port security technologies that address one or more of the capability, intent, opportunity, or vulnerability of the threat. The current technologies matrix summarized the findings of the Phase 1 investigations of technologies that are either currently implemented, or that will be in place in the very near future. The LMC approach supports a systematic and complete assessment in Phase 2 of the deficiencies in current technologies to address the requirements of port security, and provides a means to demonstrate and prioritize the need for new technologies. By mapping entries from the current technologies matrix into the requirements and threat matrices, it will be clear where technology gaps exist for dealing with requirements and threats. Furthermore, even where there is a match, an additional assessment will be made about the level of maturity and appropriateness of the current technologies. These assessments will guide the search for relevant new technologies for port security, and form the basis for recommendations for the way ahead in developing and implementing new technologies in this domain, which will be part of Phase 2 of this study. 3.2 UNISYS CANADA, XWAVE AND CARTENAV Interviews were arranged with the same agencies as LMC, and with the Director of Cargo and Port Security Practice North America of Unisys Corporation. Before

48

F. DESHARNAIS ET AL.

each interview, the team carefully reviewed the aspects of security where the agency has a leading role, and prepared their questions to gather as much information as efficiently as possible. From these sources, the Unisys team gathered the background for the requirements and technologies picture. A security framework or requirements model was established to capture and categorize security requirements. While each requirement is important, some may be more urgent depending on the particular set of circumstances and context. The Unisys team identified terrorist activity as the primary threat, based on presentations heard at the Association of Canadian Port Authorities Conference (26 Feb 2007). Threat scenarios were broken down into five areas: terrorist threat objectives, terrorist targets, weapons, delivery and attack plans. The classification, categorization and organization of the captured requirements follows a model commonly used in the Information security domain and modified to accommodate the Port specific view that is being conducted under this study. The model used is based on ISO17799 with the ten elements listed in Table 3. TABLE 3. Requirement group classifications, Unisys team Requirement group

Definition

Threat specific

Requirements captured that are related to a specific direct threat.

Policy

Requirements related to management direction and support for port security in accordance with business requirements and relevant laws and regulations.

Asset management

Requirements related to the maintenance of appropriate protection of organizational assets (information assets, software assets, physical computing, communications, networking and surveillance assets, services, and general utilities).

Human resources security

Requirements to ensure that employees, contractors, visiting workers, and third party users understand their responsibilities, and are suitable for the roles they are considered for.

Physical and environmental security

Requirements that address unauthorized physical access, damage, and interference to the organization’s premises and information (landside, waterside, coastal, infrastructure).

Communications and operations

Requirements that address the correct and secure operation of information processing facilities, communications and operations.

Access control

Requirements related to the control and access to information and facilities.

Security incident management

Requirements related to security events, management, operations and control.

Business continuity

Requirements that deal with major interruptions to business activities and to the protection of critical business processes from the effects of major failure of systems or disasters.

Compliance

Requirements that address legal, statutory, regulatory or contractual obligations.

FEASIBILITY STUDY ON SURVEILLANCE AND INTERDICTION

49

Each security requirement was in turn mapped to a technology requirement, along with an annotation when an applicable commercial technology that satisfies the requirement exists. Technology requirements were separated into four main categories: (1) human resources security and access control, (2) physical and environmental security, (3) communications and operations, and (4) security incident management.

4 Preliminary results Port Security is very wide in scope, and is therefore difficult to cover adequately in its entirety. This is the main reason why two separate contracting teams were enlisted to support the study. On the first round, the results for both teams were very much centered on the commercial port. Since the intention is to consider the harbour as a whole, we had to increase the scope further to find multi-purpose technologies that will benefit the larger stakeholder community. Both contracting teams have used different categories to group requirements. For a first level analysis, we looked at the number of requirements listed under each category for both teams. The requirements from the LMC team are primarily under Deterrence (57) and Domain Awareness (31), while most requirements from the Unisys team are under Communications and Operations (25), Physical and Environmental Security (15), and Security Incident Management (12). In spite of the different categorization, there is a significant overlap between the technologies covered by the top headings of both teams. Further analysis will be required after the second phase of the study to determine the technologies that require further research and development effort, however, several common themes have emerged from the first phase. One of the top and yet basic requirements listed by the stakeholders relate to communication issues. The current means of exchanging classified information electronically between facilities/agencies is ineffective, or sometimes non-existent. To a lesser extent, this also applies to the exchange of secure voice communications between facilities/agencies, or even between a command center and its own field agents. Even at the unclassified level, the range of technologies used by the various agencies is not well coordinated, with technology choices driven either by cost or individual requirements. Deficiencies in the interoperability of equipment, communications, software, standards and even procedures between agencies/ facilities are a common occurrence. Specialized crisis and command centers partially resolve some of these issues, but even these centers are not completely free of them. Furthermore, not everything associated with Port Security is integrated into these centers, so the issues are still relevant. Halifax is linked to as many as 15 operations centers, for missions related to domain awareness and emergency management. This introduces many requirements for the individual ops centers, such as data collection and processing, data mining, data fusion, etc. Though the requirement for automation for many of

50

F. DESHARNAIS ET AL.

these tasks is on many agencies’ wish lists, it may not always be the best way to proceed. A stakeholder pointed out that automation leads to fewer operators, but these operators need to be more skilled to understand anomalies. In some instances, it may be more effective to have more unspecialized labour to monitor surveillance equipment, with fewer decision makers to handle anomalies. The benefits of automation should always be viewed within their operational and business contexts. The high number of operations centers can also lead to either real or perceived overlaps in roles, responsibilities and jurisdictions, and efficiency concerns. Each of these operations centers is involved in regular exercises to test its operational deficiencies and its interactions with outside agencies. Modeling and simulation capabilities could help reduce the exercise load and better manage operating procedures and overlaps, redundancies or general inefficiencies. Some concerns were raised about the level of backup and redundancy in the current security systems. Given the proximity of most of these operations centers to the port area, there may be concerns that some devastating occurrence in the port might also render them inoperable. Hence, alternative emplacements or portable units, with the ability to switch command and control operations over to these, may be needed to provide the backup and redundancy required for the command centers. Since many security measures relate to the commercial port, a recurrent theme is that security solutions need to be commercially viable. For example, if containers need to be moved out of their normal processing flow to pass through portals for detection and scanning (e.g., radiation detection, VACIS), the latencies introduced by the detection/scanning processes themselves can produce choke points in container processing that may introduce significant delays and inefficiencies. This introduces a need for technologies such as efficient scanning and detection, multi-purpose non-interfering scanning portals, and concept of operations studies to optimize container flow from pre-arrival of the container ship to exit of the port via rail or road. New deterrence measures are planned as part of ongoing facility upgrades at the commercial port (e.g., microwave intruder detection systems on the water approaches to the container facilities), but the effectiveness of such systems has yet to be demonstrated in this environment. Enhanced waterside presence and security measures are also greatly desired. In some cases, there are no robust current measures to counter noted vulnerabilities (e.g., some threats to large ships while docked coming from the waterside).

5 The way ahead This feasibility study is a first step for DRDC to better understand the Port Security problem. The results from this study will be used to define the scope and structure of a larger port security project where we hope to bring all stakeholders together to reach coordinated solutions. Industry has brought to the market many

FEASIBILITY STUDY ON SURVEILLANCE AND INTERDICTION

51

innovative technology solutions, but we need to bridge the inter-agency gap to get more impact from these technologies. Amongst the technologies of interest to both this study and the AG-9 group, we include: economical underwater detection systems (divers and pier/hull surveys), deterrence methods and systems and how to determine their effectiveness, nonlethal weapons, and command center operations (concept of operations, modeling and simulations).

Acknowledgments The authors would like to thank all of the Halifax port security stakeholders that have contributed to this study.

MARITIME SURVEILLANCE INFORMATION AVAILABILITY IN ESTONIA

REIN HAAVEL, MONIKA OIT AND AIVAR USK Cybernetica AS, Estonia

Abstract. This paper addresses the problem of the development of maritime surveillance information acquisition, analysis, and dissemination systems in Estonia. Coastal radar and radio (AIS, VHF) surveillance systems for air and marine vessel traffic monitoring have been set up or are currently in the process of being set up by Estonian national authorities. As a result, fragments of more or less recognized maritime surface situational picture of different track quality, reliability and extent of coverage are at the disposal of several governmental organizations. All relevant organizations would benefit from commonly contributing, correlating, fusing (where reasonable) and sharing this information with each other, but such mechanisms of common tactical depository and interorganizational secure surveillance information exchange do not yet exist. Keywords: AIS, data security, data sources, maritime surveillance

1 Introduction When Estonia reclaimed its independence in 1991, its legacy maritime surveillance systems were mainly outdated and provided insufficient coverage of the area of national responsibility. Therefore, authorities initiated the planning and procurement of modern national coastal surveillance and port radar systems, which became operational only in recent years. The availability of maritime surveillance information in Estonia in the early and mid-1990s can be described as follows: x Few data sources – outdated radars were employed. x Fragmentary sensor coverage of the area of responsibility, no redundancy. x Vessel tracking information was available only locally, while remote reporting capability was limited mainly to verbal communication (by phone or radio). x Recognized maritime picture – positions and movement data of identified air/sea platforms essential from a safety standpoint were not available for threat estimation.

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

53

54

R. HAAVEL, M. OIT AND A. USK

x No centers with a capability for the higher level analysis of the maritime situation. x VHF transceivers operated mainly locally, no recording/playback capabilities x No electronic databases of vessel movements or communications were kept. It is obvious that under such circumstances, both the maritime situation awareness and the effective intervention capability of organizations responsible for conducting maritime surveillance in law enforcement, vessel traffic management, as well as search and rescue, was limited.

2 Current maritime surveillance information availability Owing to timely decisions made by the national authorities, and to advances in information and communications technologies (ICT), the maritime situation awareness level of Estonian authorities has significantly improved after successful completion of the national coastal radar system, Vessel Traffic Management and Information System (VTMIS, VTS) and shore-based Automated Information System (AIS) procurements by the Board of Border Guard (BG) and the Estonian Maritime Administration (MA). It is important to note that at the time of identification of the requirements for most of the above systems, maritime security issues had not yet become a global imperative. By the end of year 2005, the following data sources (sensors) shall be continuously operated on the coastline of Estonia by governmental non-military organizations: x 20 surveillance radars (BG) x 2 VTS Tallinn radars (MA). x 12 networked shore-based AIS stations, 8 of these include VHF/DSC (MA). These sensors provide full coverage across the coast, with overlapping surveillance areas in most critical sectors of the Estonian maritime area of responsibility. Information from these data sources is digitized and routed to corresponding centers using IP-based communication links, and in some cases, tunnelling through the public Internet. Four regional centers are operated by the BG while two centers, VTS and GoFREP (see paragraph 2 below), are operated by the MA. Due to the deployment of these sensors, segments of more or less recognized maritime surface situation picture of different track quality, reliability, and extent of sensor coverage area are at the disposal of several governmental organizations. Military systems are excluded from the scope of this article. As a result, the availability of maritime surveillance information has improved: x Full radar and AIS sensor coverage of the Estonian maritime area of responsibility with redundancy in critical sectors exists. x Vessel tracking information is available at remote operating centers in digital formats supporting automated analysis and further dissemination.

MARITIME SURVEILLANCE INFORMATION AVAILABILITY IN ESTONIA

55

x Fragments of recognized maritime picture are becoming obtainable. x A shared remotely operated coastal marine VHF radio transceiver network with digital voice recording and playback capabilities is available. x Facilities for keeping electronic databases of vessel movements and VHF/DSC communications for an unlimited period of time are available. All relevant organizations would benefit from commonly contributing, correlating, fusing (where reasonable), and sharing this information with each other, but such mechanisms of universal common tactical depository and interorganizational secure surveillance information exchange do not yet exist. Discussions have been initiated to encourage the utilization of existing systems for effectively achieving common safety and security goals.

3 GoFREP, VHF for VTS and data security An innovative remotely operated integrated shore-based AIS surveillance and marine VHF radio communications network has been designed and fielded by Cybernetica in 2004, and contracted by the MA with the primary purpose of fulfilling international obligations of Estonia in cooperation with Finland and Russia within the framework of an IMO-approved Gulf of Finland vessel traffic separation and reporting scheme (GoFREP, [4]; see Figure 1). Cybernetica’s tasks in this project were the development of a software/hardware solution for the whole system, and the installation of shore-based AIS stations together with a computer controlled VHF transceiver network in Estonia, extending local marine radio traffic monitoring and communication system coverage over the entire GoFREP area. Also a ship reporting information system with a dedicated database facility was designed and deployed. Based on in-house research into cryptography and timestamping [2], Cybernetica had in the past successfully delivered a few projects of comparable content where secure centrally managed digital data transfer infrastructure and interface adoption issues were successfully solved. Besides a Virtual Private Network (VPN) solution called Privador, one of the latest highlights is a project named X-road [1] with the objective of interlinking all Estonian governmental databases. X-road offered a unified and Public Key Infrastructure (PKI) based secure method for exchanging information in a heterogeneous environment between a large number of individuals and organizations, covering all aspects of the communication: from security (availability, confidentiality, integrity, user authentication and access rights management) and presentation (formal description of messages) to a unified user interface that hides the variations of different information resources from end users. Same authentication and authorization mechanisms based on the use of strong cryptography and encrypted information interchange architecture with proprietary PKI were adapted to secure transfer of AIS and DSC (Digital Selective Calling) messages as well as digitized voice communications over IP in the GoFREP system.

56

R. HAAVEL, M. OIT AND A. USK

Figure 1. Gulf of Finland reporting system (GoFREP) monitoring area.

Cybernetica typically uses Linux as a software platform to guarantee sufficient availability and reliability parameters of applications. Due to the use of its own built-in VPN solution, the system can be securely operated even over the public Internet. Before deploying the full GoFREP solution, the same principles were applied on a smaller scale in a pilot project to equip the Tallinn VTS center with a remotely operated marine VHF radio communications system. This project concluded with the integration of Cybernetica’s VHF/AISnet software with the VTS software from the company HITT Traffic. The final VHF/AISnet system architecture is shown in Figure 2. In addition to local availability, AIS information is also forwarded to GoFREP partner countries as well as to the Helcom AIS database [3]. The architecture remains open allowing for the integration of new sensors and interfacing to third party systems. The hardware of the GoFREP and VHF for VTS systems is selected from the Commercial Off The Shelf (COTS) category as much as possible, including industrial computers for radio servers, AIS transponders, VHF transceivers, networking equipment, etc. Radio transceiver control is arranged using a standard SparcBus interface (Sailor). The only proprietary hardware configuration item is an interface adapter called MultiSparc, which is employed for interfacing the VHF radio transceivers to the radio server. A full remote system management capability is employed, including active process and equipment status monitoring by Built-In Test Equipment (BITE). While COTS sound cards are used for voice data conversion, suitable models are selected through rigorous testing for latency and stability prior to approval. Optimized software audio codecs are applied to guarantee the effective use of available transmission bandwidth. The remote basestation site equipment is housed inside a special rack designed for fully exposed installation, typically on the coastal radar masts at a height of about 50–60 m.

MARITIME SURVEILLANCE INFORMATION AVAILABILITY IN ESTONIA

57

PLASMA MONITOR1 PLASMA MONITOR2 GOFREP/VHF SERVER

ARAMIS FOR 2 plasma screens

ARAMIS SERVER

VTA LAN OTHER VTS EQUIPMENT

Bridge

Router

GOFREP ARAMIS1/1 GOFREP ARAMIS1/2

GOFREP ARAMIS2/1 GOFREP ARAMIS2/2

GOFREP /VHF 1

GOFREP/VHF 2

Bridge

Router

Public WAN

Link in Viimsi AIS Transponder

AIS Transponder

VHF transmitter 1 VHF transmitter 2

Router

VHF 1

2M

VHF transmitter 4 VHF transmitter 6

Radio server Valaste

ROUTER

ROUTER AIS Transponder ROUTER

ROUTER AIS Transponder

ROUTER AIS Transponder

AIS Transponder

ROUTER AIS Transponder

AIS Transponder

VHF 1

VHF 1

VHF 2 VHF 3

VHF 2 VHF 3 VHF 4

VHF 4

VHF 1 VHF 2 VHF 3 VHF 4

VHF 1

VHF 1

VHF 2 VHF 3

VHF 2 VHF 3

VHF 1 Radio server Juminda VHF 2 VHF 3

VHF 4

VHF 4 Radio server Tahkuna

Radio server Osmussaar

Link to VllMSI

VHF transmitter 5

Radioserver Viimsi

VHF 4

Radio server Kopu

b/s

VHF transmitter 3

VHF 2 VHF 3

Radio server Vaindloo

VHF 4 Radio server Pakri

Figure 2. GoFREP/VHF for VTS Tallinn system general architecture (partial).

The VHF/AISnet basestation rack with external AIS and VHF antennas can be considered as a remote sensor (Figure 3) in this system. The core component of the basestation is a radio server, which is responsible for communicating with the central server at the operations center, interfacing its data stream to site equipment, and returning the responses. This includes digital to analog conversion of the received voice messages before broadcasting over the air using connected radio transceivers, and digitizing the analog audio signal acquired from such radio transceivers. Operator workstations at the operations center communicate with the central server, monitoring the VHF/DSC traffic and initiating voice transmissions on available VHF channels as necessary. Information from all basestation subsystems is forwarded to the central server using the same IP network channel. Up to six VHF transceivers can be integrated within one basestation, providing monitoring and transmitting capability on six marine VHF channels at a time to remote operators. Modular architecture of the basestation allows for assembling different equipment configurations as required at a particular installation site and leaves the system open for the integration of additional sensors directly on the level of the local area network (LAN) of the basestation to utilize the existing data communications and system management infrastructure.

58

R. HAAVEL, M. OIT AND A. USK

Figure 3. VHF/AISnet system basestation block diagram.

4 Prospective developments in maritime surveillance and security While significant steps have been taken in Estonia within the last 10 years to ensure the availability of basic maritime surveillance information to organizations with the highest priority based on responsibilities, continuing efforts are still required to arrive at the recognized maritime picture and to utilize the acquired information for the purposes of shore facility security in full. It is encouraging to note that several positive examples of reciprocal utilization of networking resources and physical infrastructure objects between governmental organizations exist today, but common networking for automated data exchange between all relevant organizations, including communication of ISPS MARSEC levels and utilization of databases, has not yet been implemented. First, fusion of AIS information with radar tracks is desirable in the nearest future to effortlessly obtain track identification information for vessels equipped with an AIS transponder; this is achievable with the application of existing technical solutions.

MARITIME SURVEILLANCE INFORMATION AVAILABILITY IN ESTONIA

59

Second, fusion of information from other relevant sensors and systems with basic radar and AIS surveillance information is necessary; this would require a more sophisticated approach, including data source reliability analysis, etc. Based on existing information, at least two analysis mechanisms should be considered for implementation: x Artificial Intelligence based “maritime picture assessors” to assist human operators with exposing suspicious behaviour of vessels x Automated monitoring and analysis of VHF and DSC communications Third, sharing of the resulting information pool with all relevant authorities, including ports that can also contribute to inbound and outbound vessel identification, to improve the situational awareness level. Organizations such as ports could improve their situational awareness and thereby reduce vulnerability by utilizing available near-real-time radar and AIS surveillance information for threat assessment. Surface vessel tracking information paired with underwater sensor data, if available, can further improve the effectiveness of threat identification. Many existing data sources employ proprietary protocols, often lacking any open interfaces for effortless integration into larger systems. At the same time, vessel tracking and surveillance data items need to be transferred between stakeholders securely with minimum delays, and with acquisition time (timestamp) attached. It is obvious that a solution cannot be achieved by simply cross-linking all existing data sources and display systems, but a (central) information acquisition, processing, and dissemination system is needed to avoid chaos. In addition to radar and AIS tracks, other data sources like sonar, CCTV, digital visible light and infrared camera imagery, possibly also satellite imagery, radio direction finders, monitoring of VHF radio communications, and even human observer/intelligence input should be considered. A data dissemination management function must support content filtering by client location, subscribed information type (“need to know”), and other requirements. Protocols and interfaces need to be selected or developed for each data source and client. One of the as yet unexploited potentially available sensor platforms is the visual aids to navigation (AtoN) outstation network consisting of buoys, lighthouses, and range lights, totalling over 180 sites. Remote monitoring and control of this equipment is the responsibility of the MA, while technical systems have been supplied and are supported by Cybernetica. This network is currently using public GSM cellular telephone services for CSD data communications, with a migration trend of new outstations to the GPRS service. Being a mature technology with at least 10 years of expected usable lifetime, GPRS is cost efficient for low bandwidth data communications, offering an excellent quality of service in Estonian coastal regions. While the visual AtoN network has spare communications capacity and the capability to handle additional functionality, a main obstacle in case of buoy systems is their limited capacity to supply power to additional equipment.

60

R. HAAVEL, M. OIT AND A. USK

5 Conclusions Realizing that further development of maritime surveillance information acquisition, analysis, and dissemination systems in Estonia to understand maritime situational picture is a necessity for improving both navigational safety and shore facility security, we can conclude as follows: x The integration of AIS, VHF and radar surveillance information and communication capabilities (TETRA) available from national systems with relevant port authorities improve their threat preemption capability. x Secure and timely automated data communications are essential to guaranteeing port mission success and reduction of loss of life and damages to property as consequences of an incident. x The utilization of COTS system components allows for designing and fielding of reasonably priced solutions within a short timeframe.

References 1.

2. 3.

4.

Ansper A, Buldas A, Freudenthal M, Villemson J. 2003. Scalable and Efficient PKI for Inter-Organizational Communication. 19th Annual Computer Security Applications Conference ACSAC. Las Vegas, USA. pp. 308–318. . Cybernetica AS Research & Development Information. . HELCOM launches a common Baltic maritime traffic monitoring system. 2005. Helsinki Commission press release of July 01, 2005. . Navigation safety on the Gulf of Finland has increased essentially. 2004. Cybernetica AS news release of September 01, 2004. .

CASE STUDY: THE REALITY OF DF TECHNOLOGY TRANSITION FOR MARITIME DOMAIN AWARENESS WITH A FOCUS ON CONTAINER SECURITY

MICHAEL MOSKAL Calspan-UB Research Center, Inc., USA

Abstract. The identification and tracking of shipping container based threats is a particularly complex challenge due to the multi-jurisdictional nature of international maritime shipping, the distributed nature of the shipping container supply chain that comprises an ever-changing constellation of many hundreds of thousands of manufacturers, distributors and other parties, and, the resulting disparate data sources that must be integrated to draw actionable conclusions. In this paper, we will describe the challenges facing both Government and Industry with regard to tracking containers in a cost effective manner. We will also discuss a number of ongoing initiatives designed to overcome some of these challenges. Keywords: Data fusion, Container shipping, vulnerability analysis

1 Introduction More than eighty percent of goods traded worldwide are transported by sea, much of them in containers involving thousands of container vessels. Each year there are more than 250 million container movements representing approximately 90% of the world’s cargo moves [5]. Approximately 16 million of these cargo containers enter the U.S. carried aboard by thousands of container vessels entering U.S. ports. It is generally recognized in the United States and in a majority of countries throughout the World that the movements of cargo containers are vulnerable to some form of terrorist or criminal action. Container based threats facing Governments and industry are: the loss of the container and its contents by hijack, the use of containers to transport illegal immigrants, contraband, weapons, drugs or other dangerous goods, and, theft or pilferage of the container contents somewhere along the transportation supply chain. In the worst case, a single container could serve as a “Trojan horse” used to transport and deploy a Weapon of Mass Destruction (WMD) with the intent of causing significant loss of life and creating a number of secondary effects.

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

61

62

M. MOSKAL

The consequences of a terrorist incident involving a container based WMD at a seaport or other inland location are severe. In addition to killing people and causing physical damage, the event will most certainly have serious economic consequences. Recent studies suggest that a successful attack utilizing a nuclear device concealed within a shipping container would create disruption of U.S. trade valued at $100– 200 billion, property damage of $50–500 billion, and 50,000 to 1,000,000 lives could be lost. Global and long-term effects, including the economic impacts of the pervasive national and international responses to the nuclear attack, though not calculated, are believed to be substantially greater [1]. The cost of a bioterrorism event would create disruption of U.S. trade valued at $20–200 billion, property damage of $1–10 billion and 30,000 to 3,000,000 lives could be lost [2].

2 Responding to container based threats 2.1 INTERNATIONAL INITIATIVES The vulnerability of cargo containers has been a focus of international policy since the terrorist attacks on the United States on September 11, 2001. In response, the International Maritime Organization and its members adopted the International Ship and Port Facility Security (ISPS) Code [4]. The objectives of the code are to establish an international framework that promotes cooperation, establishes respective roles and responsibilities, and ensures the early and efficient collection and exchange of security related data between Governments, Government agencies, local administrations and the shipping and port industries to strengthen the security of this portion of the supply chain. The code also provides a standard methodology for security vulnerability assessments to be used by port facilities and ships that enforce ISPS mandated security plans and processes. Adoption of the ISPS code is mandatory for the current 165 member states with a target date of July 1, 2004, for implementation of the code by port facilities and ships. 2.2 UNITED STATES INITIATIVES The U.S. Government (USG) currently employs multiple layered security initiatives to reduce the threat to the United States posed by terrorists acting through international shipping containers. Primary programs addressing this problem are the U.S. Customs and Border Protection’s (CBP) Container Security Initiative (CSI), CBP’s 24-Hour Advance Manifest Rule, and CBP’s Customs-Trade Partnership Against Terrorism (C-TPAT).

2.2.1 Container Security Initiative (CSI) The Container Security Initiative, currently in place in over 30 ports outside the United States, is a cooperative effort between United States Customs and Border

THE REALITY OF DF TECHNOLOGY TRANSITION

63

Protection and its participating partners to target and identify maritime containers that pose a risk for terrorism. CSI consists of four core elements: 1. Using intelligence and automated information processing to identify and target containers that pose a risk for terrorism. 2. Pre-screening those containers that pose a risk at the port of departure before they arrive at the U.S. ports. 3. Using detection technology to quickly pre-screen containers that pose a risk after they arrive at the U.S. ports. 4. Using smarter, tamper-evident containers. As part of its layered approach, US Customs and Border Protection employs its Automated Targeting System (ATS) computer model at both US and CSI ports to review documentation on all containers arriving at or leaving the U.S. Bound containers and help select or “target” containers for additional documentary review and/or physical inspection. The ATS was originally designed to help identify illegal narcotics in cargo containers. ATS automatically matches its targeting rules against the manifest and other available data for every arriving container, and assigns a level of risk (i.e., low, medium, high) to each container. At the port level, inspectors use ATS, as well as other data (e.g., intelligence reports), to determine whether to inspect a particular container. In addition, CBP has a program, called the Supply Chain Stratified Examination, which supplements the ATS by randomly selecting additional containers to be physically examined.

2.2.2 The 24-hour advanced manifest rule The 24-hour Advanced Manifest Rule requires sea carriers and NVOCCs (NonVessel Operating Common Carriers) to provide U.S. Customs with detailed descriptions of the contents of sea containers bound for the United States 24 hours before a container is loaded on board a vessel. The rule allows U.S. Customs officers to analyze container content information and identify potential terrorist threats before the U.S.-bound container is loaded at the foreign seaport, and not after it arrives in a U.S. port. The key data source of this rule is the required use of the Automated Manifest System (AMS). The AMS is a multi-modular cargo inventory control and release notification system that interfaces directly with U.S. Customs allowing faster identification and release of low risk shipments. Data obtained from the AMS system is a key element of the CSI Automated Targeting System.

2.2.3 Customs-Trade Partnership Against Terrorism (C-TPAT) The U.S. Customs and Border Protection Customs–Trade Partnership Against Terrorism (C-TPAT) program is a voluntary government/private sector partnership program similar to the IMO ISPS code. One major difference is that C-TPAT does not limit itself to the maritime portion of the supply chain. C-TPAT extends its reach

64

M. MOSKAL

to all of the participants within a supply chain, from raw material provider to final distribution channels. The current security guidelines for C-TPAT program members address a broad range of topics including: personnel, physical and procedural security; access controls, education, training and awareness; manifest procedures; conveyance security; threat awareness; documentation processing. Companies that apply to CTPAT must sign an agreement with CBP that commits their organization to the program’s security guidelines. These guidelines offer a performance based customized solution for members while providing a clear minimum standard that approved companies must meet. As of November 2004, over 7,400 supply chain members have either completed or are in the process of complying with C-TPAT requirements [6]. Most of the current program participants are companies operating within the United States.

3 Tracking of containers along the entire supply chain With the exception of C-TPAT, which has not gained wide acceptance outside the United States, neither the ISPS Code nor the Container Security Initiative address the security of the entire supply chain. This is particularly true for the inland transportation part of the chain where containers are transported by truck or rail. Due to the multiple actors involved, there is a lack of comprehensive security measures at various points along the supply chain, and because the movements of containers by truck are relatively uncontrolled, the land transport parts of the supply chain are particularly vulnerable to exploitation by terrorist organizations or other criminal elements. In response to this, the United States Congress appropriated funds to assess the security of multiple international supply chains that were representative of the typical flow of goods from foreign manufacturing sites to their final destinations in the United States. The project, named Operation Safe Commerce, was designed to assess the vulnerabilities present within each individual component of the supply chain, and evaluate process challenges as well as the application of various technologies that were intended to increase overall security of the supply chain. Eighteen supply chains from Europe, Asia, and South America were evaluated and used to exhibit the various processes and technologies assessed during the course of the project. This project revealed various challenges in general and information security systems and processes, personnel identification and credentialing, installation and use of devices to increase physical security of the containers and conveyances, installation and use of RFID based and other container tracking systems, technical monitoring and physical inspection of containers at critical waypoints along the supply chain, and technical monitoring and inspection of containers to detect the presence of WMDs or other nonhazardous materials.

THE REALITY OF DF TECHNOLOGY TRANSITION

65

3.1 USE OF TRANSACTIONAL DATA TO TRACK CONTAINER MOVEMENTS Figure 1 presents a process flow chart that, at a high level, describes the interactions between all of the actors within the supply chain for a notional shipment of goods from Europe to the United States. This basic process flow can be applied to nearly every container shipment, regardless of commodity, point of origin or destination. At each node of the process, considerable amount of data regarding the transaction is generated and, with the exception of the creation of the customer order shipment request, each node has several sub-processes, each producing additional transactional data that could be used to track the container as it progresses through the supply chain. For the most part, the data are neither shared between actors nor disclosed. Because of the complexity of the supply chain, the multiple actors and the diversity of the data generated within the supply chain, the information necessary to identify container-based threats must be captured across a wide range of commercial and government (both domestic and international) sources. Many of the data sources include foreign language content so translation of large volumes of data is another component of the challenge. Ensuring the integrity of a container or other goods once they are accepted into the transportation system is somewhat more straightforward because the number of participants is more finite and they are generally subject to some degree of regulation. In order to meet data collection and subsequent information fusion challenges, significant cooperation is required to enable data and information sharing between Governments,

Figure 1. Typical supply chain shipping process.

66

M. MOSKAL

international corporations, and other supply chain participants who may be reluctant to share security sensitive or proprietary information with other Governments, international regulators or third parties. To meet some of these challenges, the Operation Safe Commerce team established a sophisticated Data Clearinghouse (DCH) that provides a critical and unique foundation for the integrated approach by capturing and fusing data from multiple disparate sources pivotal to trade logistics and the supply chain [3]. The DCH receives and stores data from many different sources including webbased tools, electronic data interchange (EDI) messages, spreadsheets, sensor data and database feeds. The DCH uses extraction and transformation techniques to discern information relevant to the DCH from these structured and unstructured data sources. As a result, commercial and government organizations gain real-time access to pertinent information through a unified view of data sources which enables queries, accelerates analytical tasks, produces actionable intelligence, facilitates decision-support, enhances customer service, and enhances efficient utilization of labour and equipment. During the Operation Safe Commerce testing, the Data Clearing House was able to capture 100% of all data flows from external interfaces and these data were successfully consolidated in the DCH and associated to the appropriate container. Moreover, the DCH successfully executed 100% of all planned rule set notifications indicating an increased level of risk for a specific container. This resulted in the ability to track each container as it passed through the system and also to detect and notify the appropriate authorities when anomalous events were deliberately introduced into the test. 3.2 TECHNICAL MONITORING OF CONTAINERS During the Operation Safe Commerce demonstration and testing a number of technologies were used to assess the physical security of the containers during transit (intrusion detection monitoring) and assist in the tracking of containers as they progressed through the supply chain. Figures 2 and 3 below show a small number of these devices and their associated interrogation and reading equipment. Readers were placed at strategic waypoints along the transportation route to monitor a container’s progress and collect data from the various devices. Data collected from these sensors were transmitted back to the DCH through a variety of means including landline telephone, Internet and satellite uplink. Once housed in the DCH, the technical monitoring data were fused with the transactional data to provide the complete picture of a container’s status as it moved through the system. This combination of transactional data with technical monitoring data proved to be a critical element in effectively and accurately tracking individual container shipments along the supply chain.

THE REALITY OF DF TECHNOLOGY TRANSITION

67

Figure 2. (a) Typical supply chain shipping process; (b) anti-tamper seal, E-seal and conventional seal applied to container.

Figure 3. (a) RFID container tag; (b) RFID and E-eal readers installed at port terminal.

4 Conclusions During the course of the Operation Safe Commerce project a number of challenges were encountered that, if not cooperatively addressed by Governments and the shipping community, will hinder efforts to increase the security of worldwide supply chains. These challenges included: x Political differences over the scope and management of security efforts. x Cultural differences related to the need and implementation of increased security measures, particularly those related to personnel credentialing. x Resistance from supply chain participants to implementation of increased security due to the associated increased costs. x The lack of international technical standards, particularly for the RFID communications infrastructure, and the lack of standard transactional data formats. In some cases, no transactional data were available. x Economic issues related to the release and sharing of proprietary transactional data to assist container targeting and tracking. x Economic issues related to the additional cost of the various physical security and container tracking devices. This includes increased infrastructure costs

68

M. MOSKAL

related to the installation and maintenance of RFID readers and data transmission and storage systems. In order to overcome some of these challenges, there must be international agreement and cooperation standards developed, which go beyond the ISPS code and touch each participant in the supply chain. Securing our ports and sea conveyances only solves part of the problem; security must start at the point of product manufacture or container stuffing in order to be completely effective. The security measures must not significantly increase cost and must be applied fairly across the supply chain so as not to create a competitive advantage among transporters. Making data available for re-use to enhance container targeting and tracking is critical. This will require increased interoperability between data systems, the adoption of technical data standards, and making a clear business case for a community that creates increased security, value and efficiency through data sharing.

References 1.

2.

3.

4. 5.

6.

Abt. Clark C. 2003. The Economic Impacts of Nuclear Terrorist Attacks on Freight Transport Systems in an Age of Seaport Vulnerability. Final Report USDOT Contract DTRS57-03-P-80130, 30 April 2003. Abt. Clark C, Rhodes W, Casagrande R, Gaumer G. 2003. The Economic Impacts of Bioterrorist Attacks on Freight Transport Systems in an Age of Seaport Vulnerability. Report USDOT Contract DTRS57-03-P-80130, 9 May 2003. Kothman D, Brosi S, Quartel R. 2003. Sharing the Gains of Globalization in the New Security Environment, the Challenges to Trade Facilitation. Chapters 3.8, 3.9, 3.10 of the United Nations Economic Commission for Europe. United Nations Publications, ISBN 92-1-116889-9. International Maritime Organisation. 2003. International Ship & Port Security Code and SOLAS Amendments. . United States General Accounting Office. 2003. GAO Report: GAO-04-325T. Preliminary Observations on Efforts to Target Security Inspections of Cargo Containers. UNCLASSIFIED, 16 December 2003. United States Customs and Border Protection. 2005. C-TPAT Strategic Plan. <www.cbp.gov>.

CERBERUS

ANDREW B. WEBB1 AND WILLIAM J. BORROWDALE 2 1 2

QinetiQ, UK QinetiQ Consutant, UK

Abstract. This article lists some of the products and technologies Qinetiq provides for maritime and harbour protection. Keywords: Underwater system, naval platforms, swimmer detection sonar, submarine sensors

1 Sea Division overview The Sea Division has decades of experience in developing brilliant solutions to important Naval operational problems. The Division operates three principal business streams: x Underwater Systems x Above Water Systems x Naval Platforms The Division has a business turnover exceeding £80 million and over 750 scientists and engineers. Our concepts form the basis of many systems presently in service in the Royal Navy. The most recent example of this is the introduction of sonar 2193, the new wideband mine hunting sonar for the Hunt Class Mine Countermeasures Vessel (MCMV). QinetiQ developed the wideband concept and led the design, manufacture and demonstration of prototypes fitted to the Hunt Class MCMV and to experimental remotely operated and autonomous underwater vehicles. QinetiQ is a world leader in 1:3 composite wideband array technology and it was this knowledge that was applied to the development of the Cerberus swimmer detection sonar. Sea Division has been one of the leading areas in the strategy to develop products. Recent examples of this are: x A contract to supply the Loki Countermeasure to the Norwegian Navy x A contract from four oil majors to demonstrate four component seismic optic fibre sensing systems, and x A partnership with GeoAcoustics to develop and supply synthetic aperture sonar systems E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

69

70

B. BORROWDALE

Sea Division has a wealth of knowledge and experience in many other areas of maritime operations. These include: x Hullforms and hydrodynamics. QinetiQ are world leaders in the design and development of ship and submarine hull forms and quiet propulsors. x Marine power system communications. QinetiQ is able to provide a control system whereby ship propulsion and generation systems can be run from anywhere onboard via a laptop computer. x Marine structures. QinetiQ has a whole life approach to marine structures from initial concept through design and structural assessment to final design specification. x Innovative marine concepts. QinetiQ’s philosophy is to ‘never stand still.’ Development is constantly taking place to improve: replenishment at sea, catapult efficiency for Navy aircraft carriers, the amphibious deployment of war-fighting assets and to above and below water sensor technology and application. x Submarine sensors, processing and communications. More and more operations are taking place on and around the coast and the shore – often in environments that are noisy, highly reverberant and highly variable. Current research is focusing on flank arrays, towed arrays, wideband active sonar, transducers and hydrophones and open system architecture. x Submarine to surface communications. QinetiQ are working with other partners to design and develop a device that allows a deeply submerged submarine to conduct covert two-way, communications at a high data-rate. x Submarine concepts and designs. Because submarines present some of the most complex engineering tasks, QinetiQ places great emphasis on concepts and design before deciding on any structural details. x Unmanned underwater vehicles. QinetiQ is at the forefront of technology in the use of UUV’s for intelligence, surveillance and reconnaissance (ISR).

EMERGING SENSOR TECHNOLOGIES AND METRICS FOR HARBOUR SECURITY

RUTH M. WOODWARD HT Consultants Ltd., Magdelen Centre, Oxford Science Park, Oxford, UK

Abstract. Since the terrorist attacks of September 11th, extensive security measures have been implemented for improved harbour protection, the most global standard of which has perhaps been the introduction of the International Ship and Port Facility Security (ISPS) code and regulations in July 2004. The absence of appropriate standards and metrics in port and harbour security and uncertainty in customer requirements has in some instances resulted in confusion and misplacement of technologies. This paper discusses standards and sensor metric initiatives underway by HT Consultants Ltd. aimed at facilitating suppliers of security solutions and end users in identifying and implementing emerging technologies with existing systems to improve the safe and economic passage of trade internationally. Keywords: ISPS Code, standards, metrics, guidelines, biometrics, sensing

1 Introduction Since the terrorist attacks of September 11th, extensive security measures have been implemented for improved harbour protection, the most global standard of which has perhaps been the introduction of the ISPS code and regulations in July 2004. Nevertheless, sabotage, piracy and smuggling are still a common occurrence, with an increase in reported piracy incidents of 20% between 2002 and 2003, according to the International Chamber of Commerce (ICC) International Maritime Bureau. Numerous challenges in harbour protection exist, including the tracking of cargo from the point of packing to receipt by the end customer and the detection of threats in fog with a resolution high enough to be able to identify a small boat or person, particularly close to the surface of the water. The demand for new technologies capable of addressing existing deficiencies in the security infrastructure has driven new developments of sensors and systems, particularly in the optics and photonics industry. The solutions offered however are sometimes not as effective as promised, incompatible with end user needs, E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

71

72

R.M. WOODWARD

incomplete, or too expensive. The absence of appropriate standards and metrics in port and harbour security and uncertainty in end-user requirements has in some instances resulted in confusion and misplacement of technologies. This has been highlighted of late through recent reports by government agencies and port authorities regarding the implementation and associated time frame for the development of biometric systems. The problem of setting appropriate standards and metrics for harbour security that are suited to both the end-user needs and the technology is vast and often considered too complex to resolve. This development of metrics is thus frequently avoided, or the follow-up of implementing the defined standard, including the financial support required by government agencies, is ignored. The inappropriate implementation of a technology or methodology can however lead to disruption and delays, and may reduce rather than improve operational effectiveness. The potential long term consequences include inducing a new security threat. This paper discusses recent initiatives underway internationally concerning the identification and implementation of standards in homeland security and the issues associated therewith. The future forward steps required by both government and industry in order to improve the economic passage of trade through the implementation of standards and metrics in harbour security will be presented.

2 Security standards – why bother? There currently exist no internationally approved standards, metrics or guidelines to facilitate the implementation of emerging technologies into the security sector, which may demonstrate significant advantage over existing systems for enhancing security whilst ensuring the economic passage of trade. Although many studies have been conducted highlighting the importance of standards and the subject is continuously commented on at international maritime and security conferences, the ultimate questions remain: x x x x

Are port and harbour security standards and metrics necessary? Who defines the standards? Who pays for defining the standards? Who is going to take responsibility for the implementation of standards in port and harbour security at both a national and international level? x How do you introduce a standard which can be implemented to a particular problem when the type of sensors and systems used vary widely between harbour terminals both nationally and internationally? x How will the implementation of standards and guidelines affect operational effectiveness? x Who pays for implementing the standards? Following a recent standards study released in June 2005 by the UK Department of Trade and Industry in conjunction with the British Institute of

EMERGING SENSOR TECHNOLOGIES AND METRICS

73

Standards it was revealed that standards account for an annual contribution of £2.5 billion per annum to the UK economy, resulting in a 13% increase in labour productivity, enabling innovation and facilitating technological change [2]. The role of standards is at the heart of the drive for excellence in many businesses ranging from small enterprises to large transnational corporations. Despite this extensive and comprehensive study highlighting the fundamental importance of standards to industry, there is a greater need for government and industry to address more rapidly the development and implementation of viable, realistic standards initiatives for port and harbour security. 2.1 INTERNATIONAL SECURITY STANDARDS INITIATIVES Since the launch of the American National Standards Initiative (ANSI) Homeland Security Standards Panel (HSSP) in February 2003 in direct response to the needs expressed for standards in this area by the US government, HSSP has identified key players internationally working towards developing security standards. In 2004, the International Standards Organization (ISO) Advisory Group on Security (AGS) was tasked to review ISO and other international standards work capable of supporting security standardization, with ANSI taking an active role therewith [1]. The ANSI-HSSP seeks to promote a positive, cooperative partnership between the public and private sectors in order to meet the homeland security needs of the nation, with the mission of identifying consensus standards, or if none exist, assisting the US Department of Homeland Security (DHS) to accelerate the development and adoption of necessary standards. A recent report following a workshop organized by ANSI on 17 May 2005 on “Standardization for Perimeter Security” highlighted several fundamental questions and comments regarding perimeter security: x x x x

What is the definition of a perimeter? What is the ultimate goal of perimeter security? What happens if a violation occurs? There are integrated solutions for perimeter security but no integrated standards.

Following a Chemical and Biological Sensor Standards Study by the Defence Advanced Research Project Agency (DARPA) published in April 2004 [3], the following key metrics were identified for the best sensor characterization: x x x x

Sensitivity Probability of correct detection False positive rate Response time

These metrics are now being used and developed further by the DHS and implemented into the development of new chemical and biological sensor systems.

74

R.M. WOODWARD

At a Global Homeland Security Standards Subcommittee meeting chaired by HT Consultants Ltd. at the Defence and Security Symposium in Orlando, 2005, the need for standards specifically relating to airport security for next generation travel was highlighted by the DHS. Generic system metrics suited for the security application rather than the technology area were considered essential. HT Consultants Ltd. is now liaising with the DHS, companies and other government agencies to facilitate the identification and implementation of emerging technologies for enhanced operational effectiveness and economic international air travel.

3 System standards and metrics The terms regarding standards, sensors and metrics are defined in Table 1. TABLE 1. Definitions as applicable to standards, sensors, systems and metrics Parameter

Definition

System

A product, methodology or technique. Thus this could be a metal detector, a method of physically checking an item or a process used to identify suspects such as questioning.

Sensor

A system aimed at identifying a target of interest.

System metric

A parameter used to define the quality of a particular system.

Guideline

A procedure used in order to determine a particular system metric.

Standard

The threshold level of the system metric which must be reached by the system or collection of systems for identifying a particular problem.

Baseline

The minimum standard necessary for a system to be considered for a particular marketplace problem.

Stand-off detection

Detection whereby the system used to analyze the target is not in contact with the target.

Point detection

Detection whereby the system used to analyze the target is in contact with the target.

Remote detection

Detection whereby the system and target are not co-located with the operator performing the analysis.

Trigger

Event which results in a low disruption action.

Alarm

Event which results in a high disruption action.

Low disruption action

No significant interruption in normal activity occurs.

High disruption action

Significant interruption in normal activity occurs.

SOP

Standard operating procedure – a set method to carry out a particular procedure.

Thus it is possible to have a remote point detection system or a remote stand-off detection system. Before even designing a system suited for any homeland security application the following system metrics and questions need to be reviewed:

EMERGING SENSOR TECHNOLOGIES AND METRICS

x x x x x x x x x x x

75

Stand off or point detection – what is the target material? Remote or co-located – where should the system be positioned? Sensitivity – suspicious/not suspicious: what are the user requirements? Specificity – e.g. talc or drug: how important is this parameter? False alarm rate – unnecessary security control: what impact will this have on the reaction of the operator? Response time – how does this vary depending on the above settings? Source, detector, algorithms – what is required and how robust are they to environmental variations? Passenger delays – is the system rapid enough to enable high passenger throughput? Complacency – does the false alarm rate induce complacency in the operator leading to additional errors? Operator training – how much is required and what are the costs? Labour and system cost – what is the maintenance required and are easy replacements possible?

4 Integrating emerging systems for enhanced harbour security When developing any security solution, the question to the end user is ultimately “Will it improve existing operational effectiveness whilst remaining economical to implement?” The process for integrating an emerging system into an existing solution is demonstrated in Figure 1. Rather than building the system to fit a requirement, the target the user is interested in is first identified. The user defines the requirements, which are set using system metrics. There may already be a system or combination of systems which produce an associate metric value, activating an event – the trigger or alarm. In Figure 1 this event is a trigger activated by the system metric threshold levels of the combined systems A + C. If the threshold level is exceeded, a low disruption action occurs which activates system B. This system may have a completely different set of system metric TARGET REQUIREMENTS

SYSTEM

ACTION

OPERATIONAL EFFECTIVENESS

METRICS

A

B

C

TRIGGER

ALARM

ENHANCED

REDUCED

Figure 1. User metric schematic showing the implementation of systems A, B and C for enhanced operational effectiveness.

76

R.M. WOODWARD

values and thresholds, such as a higher sensitivity and lower false positive rate. If the threshold level exceeds the system metrics defined for system B, the alarm is activated, resulting in a high disruption action. Each trigger and alarm system will have an associate set of system metrics, the threshold levels of which will be set by the user to activate the desired event. For this system to work efficiently and effectively, system metrics must firstly be defined, measured and understood for varying environments, targets and associate security levels desired. These should be known by the supplier of the system to facilitate the end user. If these are poorly understood or the threshold levels incorrectly set, high disruption reactions and unnecessary delays may occur. In this above example, the system is completely technology independent. System A for example could be a metal detector – a technology solution, system B a trained dog for sniffing explosives – a methodology, and system C the intuition of the operator through the observation of the suspect person after a double security questionnaire after portal screening – a technique. Without the appropriate system metrics and guidelines, choosing the appropriate system solution for the end user is extremely difficult. In addition, the absence of system metrics and guidelines for emerging technologies and alternate system solutions presents a challenging task. This is because without such metrics generic to the problem, convincing the end user that operational effectiveness can be enhanced by implementing a new system and to what extent for particular targets of interest is more difficult.

5 Discussion Although many studies and conferences have discussed the importance of standards in homeland security, the implementation of appropriate, realistic standards and metrics which are both suited to industry and the end user is still poorly addressed. Since the introduction of the ISPS code in 2004, the economic impact on trade and associate rates of piracy and other security issues, which may be enhanced or reduced, remains to be reviewed. Despite the repeated system failures in port and harbour security since 2001, there still exist no definitive standards, metrics and guidelines for facilitating end users and suppliers identifying appropriate security solutions, including the implementation of new security systems. Standards and guidelines for purchasing common items such as mobile phones are now used effectively by the entire supply chain, from research and development to the supplier and end user. These standards ultimately facilitate the identification of the appropriate phone or combination of phone and additional electronic system or subsystems to suit the needs of end user and supplier. The concept of developing and implementing standards and guidelines for a marketplace problem is not novel, however the complexity of security solutions and associate systems implemented for harbour security makes this a challenging task. The implementation of port and harbour security standards and metrics is

EMERGING SENSOR TECHNOLOGIES AND METRICS

77

essential to ensure the efficient and economic passage of trade, whilst providing a cost effective security solution to the supply chain and end user. This is particularly important for international trade. The responsibility for the implementation of standards in port and harbour security at both a national and international level lies jointly with government, end user and supply chain, as the entire process and associate economy is entwined. It is thus essential that these sectors work together in a cohesive, efficient and effective manner to define viable standards and metrics which can be easily understood, implemented and monitored by all. The implementation of a metric and standard to a particular problem when the type of sensors and systems used vary widely between harbour terminals both nationally and internationally requires a multi-layered system and standards approach. This in turn requires international collaboration and a thorough understanding of needs and problems across the entire supply chain and levels of response to the threat. Standards need to be trialed and modeled prior to implementation to ensure that operational effectiveness is enhanced rather than reduced and the associate standard developed does not introduce an additional security threat. The cost of implementing the standards should be distributed between government and consumer, and should not be the sole responsibility of the maritime authority or transportation agency.

6 Summary This paper has discussed the needs for security standards, metrics and guidelines to facilitate the identification and implementation of emerging technologies into the harbour security sector and the associated forward steps required. It is essential that governments, corporations and agencies internationally take a proactive, supportive, cooperative and collaborative approach. This is necessary to facilitate the appropriate identification, development and implementation of security standards capable of facilitating the economic and safe passage of trade. Current international initiatives are underway and HT Consultants Ltd. is coordinating with government agencies and companies, which aim to address this important issue. Government support and funding is essential to enable this to happen efficiently and effectively.

References 1. 2. 3.

Deane M. 2005. Accomplishments of the Panel at Its Two-Year Anniversary. American National Standards Institute Homeland Security Standards Panel. The Empirical Economics of Standards. 2005. DTI Economics Paper; 2005 June. Chemical and Biological Sensor Standards Study. 2004. Defence Advanced Research Project Agency; 2004 April.

LOW POWER MODULAR SONAR SYSTEMS

MART RÜÜTEL R-Süsteemid OÜ, Estonia

Abstract. Sonars developed in Tallinn Technical University use complex structure-probing signals that allow effective measurements to be made with significantly lower power emissions and consumption. In current implementations, feature detection is possible up to a distance of 170 m using a transmitter with emissions of 5 W, and up to a distance of 300 m with emissions of 30 W. Lower power emissions, aside from saving energy, reduce environmental damage, and can remain practically undetected. This sonar is designed as a device/module that allows the construction of sonar systems varying in size and purpose – it has been used for bathymetry surveys. In addition to several multichannel and multi-beam sonar systems, experimental sidescan and subbottom sonar systems have also been developed. Keywords: Sonar systems, multichannel, multibeam, sidescan, subbottom, DSP, multiple target recognition

1 Introduction The development of sonar systems in Estonia began in the mid-1990s in the Tallinn Technical University Department of Radio and Communication Engineering. Within ten years, a variety of sonar systems and sonar applications had been developed. All sonar systems are based on the same working principle, a probing signal with a complex structure [1–3] and full Digital Signal Processing (DSP). This approach imparts special qualities and features to the sonars. They are listed as follows: low power, ecologically safe [4], high precision, modular and flexible, multiple target recognition. Various systems have already been implemented along these principles, some are still under development, and many other potential applications remain.

2 Current usage Sonar systems currently under development are mainly used by the Hydrographic Department of the Estonian Maritime Administration. Several multichannel, E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

79

80

M. RÜÜTEL

multibeam and hybrid [5] sonar systems were built for bathymetric surveys to measure water depths [6]. The sidescan sonar system was developed for bottom feature detection [7, 8]. The geological sonar system (subbottom profiler) was developed for exploring geological sediment layers and features below the sea bottom surface. Full-scale hydrographic software was developed for sonar data acquisition, processing and presentation. Special algorithms and automatic tools were created to exploit the sonar’s potential to detect multiple alternate echoes/targets within a single measurement.

3 Working principle The base module of a sonar system is the sonar device. The sonar device is implemented as a PCI-board for a PC. The sonar device has different models with a varying number of channels and output interfaces. Depending on the model, the sonar device has one, two or four commutated channels for transducers. The available data output interfaces are PCI and RS232. In any case, the main working principle remains the same and corresponds to the structure shown on Figure 1.

Figure 1. Sonar device structure.

Sonar device components: x x x x x x x

DSP: digital signal processor INT: interface (PCI or RS) GEN: clock signal generator (40 MHz) DAC: digital-analogue converter ADC: analogue-digital converter MUX: multiplexer CONTROL LOGIC: automatic receiver amplification, active channel switching

The sonar starts each measurement by generating the sounding signal with the DSP. Multicoded sounding signals with a complex structure and spread spectrum are used [1–3]. Afterwards, the generated signal is converted to an analogue signal by the DAC, then it is amplified, and finally it is transmitted to the chosen

LOW POWER MODULAR SONAR SYSTEMS

81

transducer. Once the transmission is over, the transducer is switched to reception mode in order to receive the scattered signal returning from the scanned environment. As soon as the signal is received, the gain of the analogue amplifier is automatically increased from 10 to 100 dB so as to adapt the received signal to the dynamic range of the ADC. This signal is digitalized by the ADC and then processed by the DSP. The DSP filters out signal parts that do not match the transmitted signal pattern. This process reduces noise up to a hundred times more than the probing signals currently being used. In multichannel and multibeam mode, the DSP extracts 1 to 16 (the number is configurable) potential targets from the returned signal. Each target is represented with delay time and intensity. In sidescan and subbottom mode the entire sampled signal is available. The results are then sent to data acquisition and presentation software via a PCI or RS interface. Output data is equipped with GPS UTC time stamps, which increase positioning precision.

4 Implementation Sonar devices are implemented as PCI-boards that can be inserted into a PCs available PCI sockets. For small-scale portable systems, a standalone lightweight waterproof unit with additional controller and RS232 interface is also available. SHARC© processors are used for digital signal processing. The number of commutated channels for transducers varies between 1, 2 and 4. A sonar board with a PCI interface and 4 commutated channels is shown in Figure 2. The technical specifications of main models are shown in Table 1. Combining proper sonar devices, transducer placement and a program code for DSP, it is possible to construct modular sonar systems of different sizes and purposes. The smallest system consists of a standalone sonar RS unit and a laptop computer for data acquisition. The intermediate system consists of a desktop computer with one or a couple of sonar PCI boards and data acquisition software running on the same computer. The largest multibeam and multichannel-multibeam hybrid systems [5] consist of a network of industrial computers, each hosting up to 14 PCI boards and data acquisition running on a separate workstation.

Figure 2. Sonar device type A4-P5-PCI with PCI-interface and four commutated channels for transducers.

82

M. RÜÜTEL TABLE 1. Technical specifications of sonar devices

Model

A4-P5-PCI

A4-P5-RS

A2-P30-PCI

A1-P100-PCI

Measurement range [m]

0.7–170

0.7–170

0.7–300

1–500

Frequency [kHz]

180–400

180–400

180–400

12

Measurement rate [nr per s]

1–16

1–16

1–32

Regulated

4

4

2

1

0.31; 1.25; 5

0.31; 1.25; 5

5; 10; 20; 30

100

Power consumption [W]

8

16

8

8

Power supply [V DC]

5

18

5

5

341 × 107

360 × 270 × 90

341 × 107

341 × 107

Interface

PCI

RS232

PCI

PCI

Radial resolution [cm]

3; 6

3; 6

3; 6

65

Multichannel, multibeam, sidescan

Multichannel, sidescan

Multichannel, multibeam, sidescan

Subbottom

Number of channels Emitted power of transmitter [W]

Dimensions [mm]

Applications

5 Key qualities The sonar systems described in this paper have specific key qualities that distinguish their capabilities from the usual performance of sonars to detect objects – sea bottom, surface, or certain features. These qualities include: 5.1 MODULARITY AND FLEXIBILITY By combining the same sonar devices with different transducer layouts and signal processing algorithms, it is possible to build a wide range of systems varying in scale and function. 5.2 LOW POWER The output power of the transmitter is from 5 to 30 W on a distance of 0.7–300 m. For example, the average consumed power of the 48-channel multibeam is about 100 W, and the total impulse power of transmitters in active mode does not exceed

LOW POWER MODULAR SONAR SYSTEMS

83

60 W. This is achieved by using long-term low-power impulses instead of short but powerful impulses. Due to a minimized influence on sea wildlife [4], sonars with low power emission are ecologically safe. 5.3 SIGNIFICANTLY LOW SIGNAL TO NOISE RATIO The sounding signal could be about 100 times weaker than noise, but the use of a specific signal structure [1–3] still makes it possible to catch the returning signal describing precisely the objects it reflected from. This also means that the actively emitting sonar is practically undetectable by anyone who does not know the exact signal pattern. An example of raw and extracted sounding signals is shown on Figure 3.

Figure 3. The noisy raw input signal (above) and the sounding signal (below) extracted from it. The scattered sounding signal clearly shows a single strong reflection from a feature not detectable on the raw input signal without knowing the emitted signal structure.

5.4 SMALL DIMENSIONS A 1 to 4-channel sonar device takes up a single PCI slot in a PC. For larger systems, industrial 19"-rack computers are used to host up to 14 sonar devices. For small-scale systems, the standalone compact sonar unit with RS interface is available. With commutated channels, more transducers can be used with a lesser number of DSP-s, thus reducing system cost and size. The trade-off is that commutated transducers can only be used one at a time.

84

M. RÜÜTEL

5.5 HIGH PRECISION AND RESOLUTION The measurement is precise to the nearest 5 mm, and the target resolution of the radial direction is about 3–6 cm at frequencies of 180–400 kHz. At the low frequency of 12 kHz for bottom penetration the resolution is 65 cm. 5.6 VERY LOW RELATIVE INTENSITY OF SIDELOBES The intensity of sidelobes is about 1/140 of the main lobe. 5.7 REDUCED SENSIBILITY TO DOPPLER-SHIFTS OF FREQUENCY Reduced sensibility to Doppler-shifts becomes important when the distance between the sonar and the targeted object fluctuates rapidly. In current systems, it can improve slant angle beam performance when a survey ship is rocking, but will also help when targeting moving objects. 5.8 MULTIPLE TARGET RECOGNITION In multichannel and multibeam working modes, within each measurement not just the single strongest reflection is selected as a range to a sea bottom, but a set of alternate values. The maximum number of alternate echoes is programmable from 1 to 16. In bathymetric surveys, 2 to 4 echoes are currently used, and scattered signals can thus be received simultaneously from bottom features (rocks) and the bottom itself, providing additional information about bottom structure.

6 Applications Currently developed applications of low power modular sonars are concerned with on-ship hydrographic surveys. Constructed sonar systems and the application software are designed mainly for measuring the sea bottom profile and searching for objects potentially dangerous to navigation. The sonar data is combined with data from other sensors – RTK GPS receiver, gyrocompass, and motion sensor – to provide precise positioning. Different sonar data presentations are available in 2D and 3D views (see sidescan example in Figure 4), also in combination with digital navigation charts. Besides conventional bathymetric, sidescan and subbottom sonar systems, other specialized applications could also be developed on the existing basis.

LOW POWER MODULAR SONAR SYSTEMS

85

Figure 4. Sidescan image obtained from four-channel sonar using transmitter with 30 W power emission.

Some not yet implemented but potential uses of low power digital sonars: x Practically undetectable active underwater surveillance due to low power complex signal emission. x Exact positioning and a high level of detail for relatively small objects like divers and underwater vehicles due to high precision and resolution capabilities. x Better detection and tracking of moving targets due to a reduced sensibility to Doppler-shifts. x Harmless assistance to manned underwater work due to low power emission, x Survey and analysis of the inner structure of underwater facilities, such as quays, due to low power emission combined with low frequency usage. Maybe even take a peek inside ships? x Digital radar systems could be developed on a similar basis.

References 1. 2.

3. 4. 5.

I. Müürsepp, Complex Amplitude-Phase Probe Signals, Baltic Electronics Conference 2004, Tallinn. A. Duhnik, V. Kozevnikov, Formation and Processing of the Orthogonal Sounding Signals in Multichannel Sonars, Proceedings BEC 2002, Tallinn, TTU Press, 2002, 191–194. J. Derkats, Multi-beam Sonar and Sounding Signals, Baltic Electronics Conference 2004, Tallinn. I. Arro, J. Derkats, V. Kozhevnikov, Ecologically Safe Multichannel Sonar System, ISPC 2003 Conference Proceedings, April 1–3 2003, Dallas, TX, 1–5. I. Arro, J. Derkats, A. Duhnik, Multichannel Multi-Beam Sonar System, Undersea Defence Technology Conference Proceedings CD, La Spezia, Italy, UDT Europe, 2002, 1–5.

86 6.

7. 8.

M. RÜÜTEL M. Rüütel, V. Kozevnikov, P. Väling, Use of Multichannel Sonar A1-P0-PCI in Bathymetric Surveys, Proceedings of the 8th Biennial Baltic Electronics Conference 2002, Tallinn, TTU Press, 2002, 149–152. I. Arro, A. Duhnik, V. Kozevnikov, Sidescan Sonar with High Resolution, UDT Europe 2003, Conference Proceedings, 23–25 June 2003, Malmö, Sweden, 1–9. A. Gavrijaseva, Sidescan Sonar with High Resolution, Baltic Electronics Conference 2004, Tallinn.

AN UNCONSTRAINED VIEW OF THE EMPLOYMENT AND EXPLOITATION OF DEFENCE DATA FUSION TECHNOLOGIES FOR HARBOUR PROTECTION

JAMES LLINAS State University of New York, USA

Abstract. This chapter provides a view of the implications of apparent requirements for Harbour Protection (as seen through the eyes of a novice in the field of Maritime Security) on the design of a Data Fusion process that would aid harbour security staff in an improved understanding of the harbour-related “maritime picture”. To examine the implications of other related Maritime Security problems to the design of Data Fusion processes, the paper also examines process design implications of the broad problem of Maritime Domain Awareness and the narrower problem of Container Shipping security. A summary opinion is offered that: (1) these problems have much in common in terms of system-level design issues but that (2) the solution of specific sub-problems in each domain could benefit significantly from leveraging existing Defence Data Fusion knowledge and capabilities. Keywords: Data Fusion design, problem context, harbour protection, Maritime Domain Awareness, Container Shipping

1 Introduction Terrorist events around the world have spawned a series of extensive selfexaminations among and within nations of the vulnerabilities of various national operations and activities. These sobering self-examinations have often and typically revealed a glaring realization that many national operations are and have been extraordinarily vulnerable to a host of possible attacks, ranging from singleperpetrator, limited-scale attacks to attacks from organized terrorist teams and to organized nation-state forces. Among the scariest of these realizations are the vulnerabilities associated with the broad area of Maritime Security. Even the simplest examinations, for example, of the statistics of container shipments by sea, are daunting. “More than 95 percent of our non-North American foreign trade arrives by ship. In 2001, approximately 5400 ships carrying multinational crews

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

87

88

J. LLINAS

and cargoes from around the globe made more than 60000 US port calls. More than 6 million containers enter the country annually” [6]. What makes Harbour Protection difficult is the extent and complexity of a harbour. Harbours are not “just about the water,” they are full three-dimensional environments comprising the above-water domain to include air, near-shore land masses, the water-surface environment, and also the below-water environment. Harbours attract industry, recreation, warehousing, marinas, etc., resulting in a very complex multi-dimensional entity. Developing a fusion-based situational picture for this environment that satisfies the informational needs of the wide range of multi-jurisdictional decision-makers in the harbour is as complex a task as developing a fusion-based situational picture for military decision-makers in, say, the urban warfare environment, which is equally three-dimensional and complex. In this chapter we take an “unconstrained” view of the nature of these maritime-related problems and the implications they impute onto the design of a Data Fusion (DF) process that may be helpful to relevant situational picture in the environment. By “unconstrained” we mean that the views offered herein are unconstrained – one could say uninformed – by extensive domain knowledge; this type of unfettered view of an area has the benefits of unbiasedness but also the danger of naiveté, so the views proffered here must be taken, carefully, in this context.

2 Maritime security in the large The maritime meta-context for any sizeable nation in the world with a measurable coastline along large bodies of water and in particular along the oceans is enormous, in that connectivity to the oceans in effect means connectivity to the world at large. These coastlines are continuous portals that afford entry into and exit from a country for all types of entities ranging from people to cargoes to fish. Control of these “transactions” requires first developing policy then methods for control. For most nations, setting of policy and the invocation of methods of control involves a broad range of governmental agencies spanning coast guards, police, drug control, fire protection, and environmental agencies, among others; an inherent complexity of this control problem is this extensive multi-jurisdictional aspect. As has been experienced in the United States in its formation of the Department of Homeland Security, significant organizational restructuring to synthesize an integrated organization to deal with such issues as maritime security in the large takes a very long time, and is a necessary precursor to the employment of advanced technologies in control methods. If any nation has not addressed such issues, the likely result of its control policies and methods will be a series of “stovepipes” as have existed in so many real-world cases. Opportunities for implementation and exploitation of Data Fusion technologies for maritime security go hand-in-hand with these organizational and policy issues; the more is done in one the more can be done in the other, and vice-versa.

AN UNCONSTRAINED VIEW OF THE EMPLOYMENT AND EXPLOITATION 89

3 Harbour protection One of the “sub-problems” of maritime security is harbour protection. In beginning to address the protection of harbours, an initial question is a definitional or ontological one – what is a harbour? A dictionary-level definition [7] is given as “a sheltered part of a body of water deep enough to provide anchorage for ships.” But this definition seems too weak for our purposes here. From a limited survey of maritime definitions, we see that the term “port” may be more appropriate for this work. In [8] we see that “Harbours and ports are often confused. A port is a man-made coastal or riverine facility where boats and ships can load and unload. It may consist of quays, wharfs, jetties, piers and slips with cranes or ramps. A port may have magazine buildings or warehouses for the storage of goods and a transport system, such as railway, road transport or pipeline transport facilities for relaying goods inland.” To our naïve view, it seems that the best way to think of our problem domain is as a combination of a harbour and a port, one the rather natural aspect of a body of water whose configuration is hospitable to sheltering, and one which is man-made, dealing with all of the aspects of commerce surrounding the harbour. We return to this ontological question later. 3.1 THE HARBOUR PROTECTION PROBLEM CONTEXT The context of the harbour protection problem is quite complex, especially when viewed as a “combined harbour and port” as discussed above (henceforth we just use “harbour” to denote this concept). This environment has the following characteristics (an incomplete list): x Full 3D environment: Surface, subsurface, airborne, landborne x Broad range of entities: Ships – but also small craft; people – crew/passengers; containers/cargo; high value targets in port vicinity x “Terrorist-type” and coordinated force-level threats: Force, platform, subplatform threats; CBRN-to-cyber level x Foreign language aspects x Multi-agency responsibilities; including international x Possibly no integrated Maritime Security Policy/Strategy If we take the view that a DF process, among other things, should produce a harbour protection situational picture, then there are surface, subsurface, underwater, and landborne pictures, each partitioned with aggregated entities of people, ships, cargo, etc., and associated layers of events and behaviours; the combinatory of forming such pictures are clear. A better strategy or framework for thinking about DF for harbour protection may be to structure the picture-formation process about threat classes rather than having a geospatial foundation. A threatclass view would integrate various components of the geospatial framework into

90

J. LLINAS

an integrated structure tuned to decision-making and action-taking, which seems to be the most pragmatic view. Two things are needed to do this – threat ontology, and the rules of engagement (ROE) for harbour protection. 3.2 FRAMING THE DATA FUSION SYSTEM ARCHITECTURE The threat ontology and the ROE provide, in a sense, the desired fusion-based informational products needed to support decision-making and action-taking. An immediate next question is in regard to the nature and extent of the evidence upon which such fused products shall be formed. In turn, we can ask: whose sensor or observational/evidence-gathering resources will be part of the DF system? It can be seen that this goes right back to the multi-organizational/jurisdictional issue. Assuming the system will not be stovepiped, there will be a federation of sensor and observational/data-gathering resources connected by some type of communication backbone (itself not a trivial matter). This sets the stage for the design of a distributed or decentralized DF (DDF) system and its attendant complexities. Because of this, decisions will have to be made regarding whether certain major components of the DDF are specified or given, or whether they are part of the overall architectural design trade-space for the DDF. Such considerations will also involve contemplation of the issue regarding any standards that may apply or that will be needed to be factored into the design approach. For example, in the US there is a current great inertia toward “service-oriented architectures (SOA)” that would likely need to be considered in the design of a DDF for harbour protection. New system designs also have an interesting limitation in that, since there is no system of the type being designed in existence, there is no way to (easily) address the human engineering side of the system, i.e. the manning, training, cognitive systems engineering-type issues which, together, result in a concept of employment. The concept of employment can only be notionally specified a priori but will have to be tuned subsequent to system development. An initial design question is of course what the system boundaries are; a harbour protection system would necessarily be linked to a layered, overall maritime security system architecture, e.g. tied into immediate offshore/littoral surveillance systems that would provide initial input into the harbour protection DDF system. 3.3 CHARACTERISTICS OF THE HARBOUR PROTECTION DATA FUSION PROCESS Once an overarching architectural design begins to unfold, the DDF processing details need to be designed. Various characteristics of such a DDF process can be envisioned – although it must be remarked that the “real” characteristics are those that derive from the ontology and ROE information mentioned above. Thus, a preliminary or notional list of characteristics might be:

AN UNCONSTRAINED VIEW OF THE EMPLOYMENT AND EXPLOITATION 91

x x x x x

Anomaly-oriented/Change-detection approaches. Data mining for subtle statistical patterns. Counter-deception techniques need to be embedded. Negative information should be included. Focus on tracking/ID of high risk vessels including tankers carrying gas, oil, chemicals, LNG, etc. x Focus on tracking/ID of high risk vessels with concentrations of passengers, e.g. cruise ships/ferries. x Low false alarm rate requirements imply possible use of Detection Fusion and/or high-reliability Target ID. These characteristics are of course in addition to the “traditional” DF process requirements involving entity tracking and ID, entity aggregation, etc., across the Level 1 to Level 3 DF functions. Broadly speaking, it seems that much of the maritime security problem would be involved with anomaly detection and “unusual” events, or entities, or behaviours, etc., due in large part to the high volumes of “normal” data and behaviours. One core issue is that the “quanta” of possible threats are large enough in one sense – e.g. a single container – to cause significant worry about the potential consequences of not detecting that entity, while at the same time this quanta is actually small in the context of the surrounding (multi-thousand container) environment. Another aspect of the system is its inherent distributed/decentralized nature. This will require that all of the current design knowledge in the DF community regarding DDF be applied, e.g. the knowledge about how to design and build distributed tracking algorithms and processes that accounts for such factors as redundant and correlated data control. However, the community is lacking a strong foundation of design knowledge for distributed ID fusion, and certainly lacks the knowledge base for design of distributed L2, L3 capabilities. In the nearterm, the opinion here is that a mixed-initiative type DDF system design will be required, so that automated and human capabilities can be made synergistic and cooperative. It would seem that in the spirit of a divide-and-conquer approach to design, that the various sub-problems related to DDF and harbour protection should be partitioned and specified. This would allow a coordinated design approach where the boundaries and synergies among and between sub-problem areas would be well defined. Operating policies, rules of engagement, and ontological specifications for each sub-problem area are among the early tasks that need to be carried out. At the meta-system level, an early task would involve analyzing and specifying (a) the nature and details of inter-process operations between the subsystems, and the related task of (b) defining the overall approach to systemlevel quality control, i.e. control of the quality of the fused outputs.

92

J. LLINAS

3.4 AN EXAMPLE SUBSYSTEM IN THE US: VESSEL TRAFFIC SYSTEM In the US, there is a system that would be an example of a natural part of any harbour protection DDF system; this is the Vessel Traffic System (VTS; aka Vessel Traffic Services), whose purpose is to provide active monitoring and navigational advice for vessels in particularly confined and busy waterways. There are two main types of VTS, surveilled and non-surveilled.1 Surveilled systems consist of one or more land-based sensors (i.e. radar, AIS and closed circuit television sites), which output their signals to a central location where operators monitor and manage vessel traffic movement. Non-surveilled systems consist of one or more reporting points at which ships are required to report their identity, course, speed, and other data to the monitoring authority. The ROE for VTS [3] allow port state control officers to take appropriate measures, including delay, restriction of operations, and denial of entry or expulsion from port when vessel operations are outside normal procedures. Exact specification of the (fused) information needed to implement these actions is an example of the type of fusion system specification needed for this case. In our research on the harbour protection problem, we were able to find several references to Data Fusion related research for VTS applications, all from the US Naval Postgraduate School (NPS); see [1–5]. These documents, all Master’s-level academic theses, are good examples of the application of DF technology to a harbour protection subsystem, such as a VTS. The theses address both fundamental issues such as data association issues but also extend to threat and intent estimation techniques.

4 Container Shipping and Maritime Domain Awareness Our presentation associated with the NATO Data Fusion Technologies for Harbour Protection included some remarks on the Container Shipping and Maritime Domain Awareness problems that are both related to the Harbour Protection problem. The Container Shipping (CS) problem is another example of what could conceivably be considered as another sub-problem of the Harbour Protection problem, which would be addressed with a DDF-based CS decision support sub-system. The CS problem brings into the DDF design domain at least two new challenges: (1) automated analysis and fusion of the possibly-large database of all the documentation associated with transshipment of goods, and (2) the alignment, association, and state estimation techniques needed for modern-day container content assessment based on nuclear, chem./bio, and explosives sensors. The Maritime Domain Awareness (MDA) problem brings into the DDF design domain several challenges associated with the extensive boundary or region of concern for MDA, spanning from foreign points of origin to near-offshore 1 In the maritime environment, there seems to be a history of such “surveilled and non-surveilled (or voluntary)” type systems and/or operating procedures; the idea of a non-surveilled aspect of a possibly-dangerous environment of course has to be considered in light of the possible threat risks associated with non-cooperative platforms and entities.

AN UNCONSTRAINED VIEW OF THE EMPLOYMENT AND EXPLOITATION 93

situational analysis; this is the domain of very large-scale DDF systems involving for example a broad range in the quality and availability of surveillance data, in part relying on international-level cooperation, the use of national (satellite-based) surveillance over the open oceans, and ranging to high-quality/availability national organic systems.

5 Summary The various surveillance and decision-support requirements for maritime security applications span a broad range of complexity but bear considerable similarity to many modern-day Defence applications. It is clear that the relatively mature bodies of knowledge and existing operational capabilities for Level 1 Data Fusion systems can be cost-effectively extended to address some of the maritime security issues, to include certain aspects of the Harbour Protection problem. Since the Defence community is just now seriously addressing the design challenges of large-scale DDF systems, it would be wise for the Coast Guards and related governmental agencies concerned with technology insertion to maritime security problems to partner with the Defence DDF community in jointly addressing common design and also algorithmic concerns.

References 1. 2.

3. 4. 5. 6.

7. 8.

Anzano ES. 1999. Fuzzy Clustering Means algorithm for track fusion in U.S. Coast Guard Vessel Traffic Service systems [dissertation]. US Naval Postgraduate School. Glenn IN. 1995. Multilevel data association for the Vessel Traffic Services system and the Joint Maritime Command Information System [dissertation]. US Naval Postgraduate School. Koh LPL. 1995. A neural network approach to multisensor data fusion for vessel traffic service [dissertation]. US Naval Postgraduate School. Kok ST. 2005. A multi-agent system for tracking the intent of surface contacts in ports and waterways [dissertation]. US Naval Postgraduate School. Ruthenberg TM. 1995. Data fusion algorithm for the Vessel Traffic Service system: a fuzzy associative system approach [dissertation]. US Naval Postgraduate School. United States General Accounting Office. GAO-03-297T. 2002. Container Security: Current Efforts to Detect Nuclear Materials, New Initiatives, and Challenges. Unclassified; 2002 Nov. p 2. See http://www.answers.com/topic/harbour See http://encyclopedia.laborlawtalk.com/Harbour

AN APPROACH TO THREAT ASSESSMENT

ALAN N. STEINBERG CUBRC, Inc., USA

Abstract. A concept for characterizing, predicting and recognizing threat situations is developed. The goal is to establish a systematic approach to automating some of these functions. The proposed approach addresses the fundamental problems of (a) sparse and ambiguous indicators of potential or actualized threat activity buried in massive background data; and (b) uncertainty in threat capabilities, intent and opportunities. Attack hypotheses are adaptively generated, evaluated and refined as the understanding of the situation evolves. This effort builds upon advances in Situation, Ontology and Estimation theory. Specific features of the approach include (a) fuzzy definition of situations and relationships; (b) integration of diverse inference bases: logical/semantic, causal, conventional, etc.; and (c) context-conditioned reasoning with uncertain evidence formulated in terms of “probabilistic infons.” Keywords: Threat assessment, JDL data fusion model, functional model, situation assessment, knowledge representation, ontology, situation theory, context, combinatory logic, belief propagation

1 Objectives The goal of this paper is to present new ideas in estimating and predicting threat relationships and situations, given uncertain evidence and uncertain models of such relationships and situations. This is part of a line of research into methods (a) for representing, recognizing, discovering and predicting situations and (b) for inferencing – whether by people, machines or some combination thereof – across information levels; from features, individuals, relationships, situations and impacts (e.g. outcome costs).

2 Situation and threat assessment Data fusion in general involves the use of multiple data – often from multiple sources – to estimate or predict the state of some aspect of reality. Among data fusion problems are those concerned with the estimation/prediction of the state of

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

95

96

A.N. STEINBERG

one or more individuals; i.e. of entities whose states are treated as if they were independent of the states of any other entity. Many formulations of target recognition and target tracking problems make this assumption. These are the provinces of “Level 1” in the well-known JDL Data Fusion Model [9, 10, 12]. Other data fusion problems involve the use of context to infer entity states. Such contexts can include relationships among entities of interest and other entities and of the type of context, or situation, itself. The ability to exploit such relational and situational contexts, of course, presupposes an ability to characterize and recognize relationships and situations. A situation is a partial world state, in some sense of ‘partial’. Devlin provides an attractive working definition for ‘situation’ as ‘a structured part of Reality that is discriminated by some agent’ [4, p. 31, paraphrased]. Such structure can be characterized in terms of the states of entities and of relationships among them. Determining just which entities, which state elements and relationships are parts of a situation or relevant to a situation is an important part of the problems of Situation Semantics and of Situation Assessment (level 2 data fusion). The problem of Situation Semantics is that of defining situations in the abstract; e.g. specifying the characteristics that define a situation as a political situation or as a scary situation or, more specifically, as a tennis match or an epidemic. In the terminology of Peircean semeiotics, Situation Semantics involves abductive processes that determine general characteristics about types of situations and inductive processes that generalize these characteristics to establish an ontology of situations [7]. In contrast, the problem of Situation Assessment is that of recognizing and predicting situations; i.e. of classifying concrete situations (whether actual, realworld situations or hypothetical ones) as to situation type. Situation Assessment – like most data fusion – involves primarily deductive processes. Situation Assessment has only recently received serious attention in the military data fusion community, which has hitherto found it both easier to build and to market target identification and tracking technologies. The newfound interest in Situation Assessment reflects changes both in the marketplace and the technology base. The former change is largely motivated by the transition from a focus on traditional military problems to asymmetrical threat problems. The latter change involves developments in the theories of Generalized Belief Propagation [8, 13], in Situation Logic and in Machine Learning [5]. Per the JDL Data Fusion Model, Threat Assessment is a level 3 data fusion process. Indeed, the original model version [12] used ‘Threat Assessment’ as the general name for level 3, indicative of the importance of that topic. In a subsequent revision [9], the concept of level 3 has been broadened to that of Impact Assessment. Threat Assessment involves assessing situations to determine whether adversarial events are either occurring or expected [11]. Generally speaking, Threat Assessment includes the following functions:

AN APPROACH TO THREAT ASSESSMENT

97

x Threat Event Prediction: Determining likely threat events (“attacks”): who, what, where, when, why, how x Indications and Warning: Recognition that an attack is imminent or under way x Threat Entity Detection & Characterization: Determining the identities, attributes, composition, location/track, activity capability, intent of entities involved in a potential or current attack x Attack (or Threat Event) Assessment: – – – – –

Responsible parties (country, organization, individuals, etc.) and attack roles Intended target(s) Intended effect (e.g. physical, political, economic, psychological effects) Threat capability (e.g. weapon and delivery system characteristics) Force composition, coordination & tactics (goal and plan decomposition)

x Consequence Assessment: Estimation and prediction of event outcome states and their cost/utility to the responsible parties, to affected parties or to the system user. These can include both intended and unintended consequences. The relative difficulty of the higher-level Situation and Impact/Threat Assessment problems can largely be attributed to the following three factors: Weak spatio-temporal constraints on relevant evidence: Evidence relevant to a level 1 estimation problem (e.g. target recognition or tracking) can be assumed to be contained within a small spatio-temporal volume, generally limited by kinematic or thermodynamic constraints. In contrast, many situation and threat assessment problems can involve evidence that is wide-spread in space and time, with no easily defined constraints; Weak ontological constraints on relevant evidence: The types of evidence relevant to threat assessment problems can be very diverse and can contribute to inferences in unexpected ways. This is why much of intelligence analysis – like detective work – is opportunistic, ad hoc and difficult to codify in a systematic methodology. Rather, the methodology in threat assessment is second-order: not to discover instantiations of pre-scripted threat scenarios, but (i) to discover patterns of activity that are consistent with unanticipated threat scenarios and (ii) to nominate searches for data that could either confirm or refute such possibilities. Weakly-modeled causality: Threat assessment often involves inference of human intent and human behaviour. Such inference is basic not only to predicting future events (e.g. attack indications and warning) but also in understanding current or past activity. Needless to say, our models of human intent, planning and execution are far less complete and far more fragile than the physical models used in target recognition or target tracking. As an example, the problem of recognizing and predicting terrorist attacks is much different from that of recognizing or predicting the disposition and activities of battlefield equipment. Often the key indicators of potential, imminent or current

98

A.N. STEINBERG

threat situations are in the relationships among people and equipment that are not in themselves distinguishable from common, non-threatening entities. Situation Assessment – whether implemented by people, an automatic process or some combination thereof – requires the capability to make inferences of the following types: x Inferring the presence and the states of entities on the basis of relations in which they participate x Inferring relationships on the basis of other relationships x Recognizing and characterizing extant situations x Predicting undetected (e.g. future) situations Broadly speaking, relationships to be inferred and exploited in situation assessment can include: x Logical/semantic relationships (e.g. definitional, analytic, taxonomic, mereologic relationships) x Physical relationships (e.g. spatio-temporal or causal relationships) x Functional relationships (e.g. structural or organizational role) x Conventional relationships (e.g. ownerships, legal and other societal conventions) x Cognitive relationships (e.g. sensing, perceiving, believing, fearing) To be sure, relationships of most of these types are generally not directly observable, but rather must be inferred from the observed attributes of entities, their context and from other relationships.1 Indirect inference of this sort is not unique to Situation Assessment, of course. Inference of relationships and of entity states given relationships is also essential to model-based target recognition, in which the spatial and spectral relationships among components are inferred and these, in turn, used to infer the state of the constituted entity.

3 Threat assessment system architecture Given the above definition, threats are modeled in terms of potential and actualized relationships between threatening entities (which may or may not be people or human agencies) and threatened entities, or targets (which often include people or their possessions) [11]. A threat situation is one in which there is some likelihood of certain types of potential events (e.g. attacks) by some agent (which may or may not be a person or human agency) against threatened entities (which often include people or their possessions). Threat events may be intentional or unintentional (e.g. potential natural disasters or human error). 1

We define a relationship as a function mapping from an n-tuple of entities to a relational state, R(m)(x1,…,xn) = y, m > n; e.g. a four-element relationship, R(4)(w,x,y,z) = w trades x to y in exchange for z.

AN APPROACH TO THREAT ASSESSMENT

99

In the present discussion, we are not directly concerned with unintentional threats such as from natural disasters. Rather, we are most interested in characterizing, predicting and recognizing situations in which a wilful agent intends to do harm to something or somebody. This, of course, does not assume that intended threat actions necessarily occur as intended or at all, or that they yield the intended consequences. Furthermore, we need to recognize threat assessment as part of the more general problems of assessing intentionality and of characterizing, recognizing and predicting intentional acts in general. Based on the ontology of Little and Rogova [6], indicators of Threat Situations relate to the capability, opportunity and (where purposeful agents are involved) intent of agents to carry out such actions against various targets. The threat ontology decomposes these three sub-elements of the threat situation into subelements. Figure 1 shows the elements of a representative threat situation hypothesis and some important inference paths. The three sub-elements of the threat situation – an entity’s capability, opportunity and intent to affect one or more entities – are each decomposed into sub-elements. The Threat Assessment process (a) generates, evaluates and selects hypotheses concerning entities’ capability, intent and opportunity to carry out an attack and (b) provides indications, warnings and characterizations of attacks that occur. An entity’s capability to carry out a specific type of threat activity depends on its capability to design, develop and deploy or deliver the resources (e.g. weapon system) used in that activity. The hypothesis generation process searches for indicators of such capability and generates a list of feasible threat types. Capability • Design: Concept, Theory, Technology • Development: Materials, Facilities, Skills • Deployment/Delivery

Indicators

Threat Types e.g. CBN, cyber, econ,… Threat Delivery Constraints

Intent (as applicable) • High-Level Objectives: e.g. Security, wealth, prestige, revenge • Means Decomposition: e.g. Force, trade, coercion, trickery, thirdparty pressure, internal pressure • Desirable Targets & Effects: e.g. Capture, Destruction, Demoralization

Feasible Threat Types

Desirable Targets and Effects

Attack Outcome (perceived by threat agent) Attack Likelihood

x x x x

x x x x

x x x x x x x x

x x x x x x x x

x x x x x x x x x x xx x x x x x xx x x x x x x x x Opportunity Target Access Target Vulnerabilities Opportunity Assessment: Assessed target vulnerabilities & access, Outcome assessment: Predicted goal satisfaction, Predicted Secondary Effects: e.g. retaliation, international political or economic reaction x x

x

x

Attack Opportunity

• • •

• x

x

Figure 1. Elements of a threat situation hypothesis.

100

A.N. STEINBERG

Intent is inferred by means of goal decomposition, based on decision models for individual agents and for agent conclaves. The postulated threat type constrains a threat entity’s opportunity to carry out an attack against particular targets (e.g. to deploy or deliver a type of weapon as needed to be effective against the target). Other constraints are determined by the target’s accessibility and vulnerability, and by the threat entity’s assessment of opportunities and outcomes. Capability, opportunity and intent all figure in inferring attack-target pairing. The Threat Assessment Process evaluates threat situation hypothesis on the basis of likelihood, in terms of the threat entity’s expected perceived net pay-off. Be it noted that this is an estimation, not of the actual outcome of a postulated attack, but an estimation of the threat entity’s estimation of the outcome. The system’s ontology provides a basis for inferencing; capturing useful relationships among entities that can be recognized and refined by the intelligence analysis process. An ontology can capture a diversity of inference bases, to include logical, semantic, physical, and conventional contingencies (e.g. societal legal or customary contingencies). The represented relationships can be conditional and/or probabilistic. This ontology will permit an inferencing engine to generate, evaluate and select hypotheses at various fusion levels. In data fusion level 0, 1 and 2, these are, respectively, hypotheses concerning signals (or features), individuals and relationships. A level 3 (threat or impact assessment) hypothesis concerns potential interactions among entities and their outcomes. In Intentional Threat Assessment we are, of course, concerned with interactions, in which adverse outcomes are intended. The Threat Assessment process will need to generate, evaluate and select hypotheses concerning threat situations – i.e. of situations, in which threat events (“attacks”) are likely – in terms of entities’ capability, intent and opportunity to carry out various actions, to include attacks of various kinds. It will do so by reference to an ontology that decomposes capability, opportunity and intent into indicators that can be recognized in constructing threat hypotheses. By evaluating and selecting such hypotheses, the Threat Assessment system will provide indications, warnings and characterizations of possible, imminent or occurring attacks. Because of the focus on human intentional threats and the three difficulties (A, B, C) listed above, the threat assessment process will need to be adaptive and opportunistic. That is to say, it cannot expect to have a pre-defined set of threat scenarios available for a simple pattern-matching recognition algorithm (e.g. using graph-matching). Rather, situation and event hypotheses will need to be generated, evaluated and refined as the understanding of the situation evolves. A notional Threat Assessment architecture is shown in Figure 2. As illustrated, the process is adaptive, by which hypotheses concerning threat situations and threatened events are successively generated, evaluated and selected for response or for further refinement. A secondary feed-back loop refines the models whereby

AN APPROACH TO THREAT ASSESSMENT

101

threat situations and events are recognized or predicted. As noted above, this involves the abductive and inductive processes of Situation Semantics. The proposed threat assessment architecture involves the following processes: Data Collection: sensor management and data mining to obtain reports of realworld entities, relationships and events. Sensor reports can be considered to have the form of probabilistic infons (r,x1,…,xn,h,k,p), for attributes/relations r, entities xi, locations h, times k and probabilities p. x Hypothesis Generation: building candidate threat hypotheses consistent with the available data. Such hypotheses – which also can be stated in the form of probabilistic infons – will be instantiations of situations and events per the threat ontology. x Hypothesis Evaluation: ascribing a likelihood score to each candidate hypothesis and requesting additional data expected to either support or refute the hypothesis. x Hypothesis Selection: selecting among candidate hypotheses on the basis of global likelihood. x Alerting: providing indications of current and predicted threat situations and threat events for Event Prediction, Indications & Warning, Threat Characterization, Attack Assessment and Consequence Assessment. x Model Management: building and refining threat models. Model management is typically performed as an off-line task, involving abductive and inductive processes for pattern explanation and generalization. Advanced learning methods can be employed to develop and validate predictive models of threat activity [5].

Abstract Threat Models*

Hypothesis Generation • Build cases consistent with data

* Threat Models • Entity Capability models • Opportunity models

Hypothesized Situations

Model Mgmt* • New Model Generation • Model Refinement

• Entity Intent models • Attack Planning & Preparation models • Attack & Consequence models

Sensor Source Data

Data Collection • Data Mining • Sensor Management

Hypothesis Evaluation • Test Existing Hypotheses

Hypothesis Selection • Nominate Response

External Events**

Alerting • Who, What, Where, Why, When, How • Probabilities

** External Events: (r,x1,…,xn,h,k), with attributes/relations r, entities xi, locations h, times k)

Figure 2. Threat assessment functional flow.

102

A.N. STEINBERG

Objective World Situation

Entity x State

Entity y State

Threat Situation (posed by x to y)

Physical • Physical Resources • Security, etc.

Capability

Physical • Physical Resources • Security, etc.

Opportunity Informational

Informational

• Data bases • Data sources

Intent

• Data bases • Data sources

Perceptual

Perceptual

• Beliefs concerning world current & future physical state • Beliefs concerning reliability of available information • Beliefs concerning physical, informational & perceptual states of other agents • Goals, constraints & threats • Plans & predicted outcomes

Threat Event (e.g. x attacks y) Direction Execution Outcomes

• Beliefs concerning world current & future physical state • Beliefs concerning reliability of available information • Beliefs concerning physical, informational & perceptual states of other agents • Goals, constraints & threats • Plans & predicted outcomes

Figure 3. Generic threat model.

In general, Hypothesis Generation is expected to be the most complex and difficult of the processes in the assessment of intentional threats, due to the abovelisted factors: x Weak spatio-temporal constraints on relevant evidence x Weak ontological constraints on relevant evidence x Dominance of relatively poorly modeled causal processes (specifically, human and group behaviour versus simple physical processes) This is in contrast to level 1 data fusion, in which Hypothesis Generation is relatively straightforward. It is expected that Threat Assessment in general – and particularly threat hypothesis generation – will remain an intensely manual process for some time. Adaptive data collection – seeking evidence to support or refute threat hypothesis – can also be expected to evolve to greater levels of automation, largely driven by anticipated advances in data mining and collection management. Figure 3 shows the high-level Intentional Threat model. Threat hypotheses include: x Inference of threat situations; i.e. of the capability, opportunities and intent of one entity x to (adversely) affect other entities y.2 x Prediction of threat events (“attacks”); i.e. interactions whereby entities intend to adversely affect others. Threat situations and threat events are inferred on the basis of the attributes and relationships of the entities involved.

2

For simplicity of exposition, we will ignore cases of self-threat, where x = y.

AN APPROACH TO THREAT ASSESSMENT

103

As indicated by green arrows in the figure, estimates of physical, informational and perceptual states of such entities are used to infer both actual and potential relationships among entities; and specifically to infer threat situations, in terms of one entity’s capability, opportunity and intent to affect (other) entities. The red arrows indicate information used to characterize a predicted or occurring threat event, in terms of its direction (i.e. planning, command and control), execution and outcomes.

4 Formal foundations – entities, situations and inferences We find it convenient to explicitly include attributes and relationships into our ontology. Doing so allows us to reason about such abstractions without the definitional baggage or extensional issues in reductionist formulations; e.g. in which uncertainties in the truth of a proposition involving multiple entities are represented as distributions of multi-target states X = {x1,…,xn,h}. A multi-target state in Situation or Threat Assessment cannot in general be inferred from a set of single-target states X = {x1,…, xn}. E.g. from P = ‘Menelaus is healthy, wealthy and wise’, we can’t be expected to infer anything like Q = ‘Menelaus is married to Helen’ (however, in such cases we can sometimes infer Q Ÿ ™P ). It does appear feasible, however, to restrict our ontology of relationships to those of finite and determinate order; i.e. any given relation maps from a space of specific finite dimensionality R:X(n) tY. To achieve the additional expressive power needed to reason about relationships and situations under uncertainty, we adapt a Situation Logic model to the problems of estimating and predicating states of entities and of situations. Situation Logic, as developed by Barwise and Perry [1], Devlin [4], et al., employs a second-order predicate calculus, related to the Combinatorial Logic of Currey and Feys [3]. The latter represents first order propositions R(x1,…,xn), involving an m-place predicate R, m > n, as second-order propositions, Applies(r,x1,…,xn,), employing a single second-order predicate “Applies”. This abstraction amounts to the reification (i.e. admission to the ontology) of relations r corresponding intentionally to predicates R (the concept of relation is extended to include attributes, i.e. 1-placed relations). There being but one second-order predicate, we can often abbreviate ‘Applies(r,x1,…,xn,),’ as ‘(r,x1,…,xn,),’. This is the basis of Devlin’s notion of an Infon, construed as a “piece of information”. Under this formulation [4], an infon has the form ı = (r,x1,…,xn,h,k,p);

(1)

for m-place relations r, m > n, entities xi, locations h, times k and polarities (i.e. truth values) p. Hypotheses can also be represented as sets of infons. We may read

104

A.N. STEINBERG

‘(r,x1,…,xn,h,k,1)’ as “relation r applies to the n-tuple of entities <x1,…,xn> at location h and time k”. Similarly, ‘(r,x1,…,xn,h,k,p.0)’ can be read “attribute or relation r doesn’t apply …”. Barwise and Perry [1], Devlin [4], et al., broaden the expressive power of the second-order predicate calculus by a further admission into the working ontology. The new entities are our friends, situations. A new operator ‘|=’ expresses the notion of contextual applicability, so that ‘s |= ı’ can be read as “situation s supports ı” or “ı is true in situation s”. This extension allows consistent representation of factual, conditional, hypothetical and estimated information.3 Some situations can be crisply defined; e.g., a chess game, of which the constituent entities and their relevant attributes and relationships are explicitly bounded. On the other hand, we should allow that many situations may have fuzzy boundaries. Fuzziness is present both in abstract situations (e.g. the concepts of economic recession or naval battle) and of concrete situations that occur in the real world (e.g. in the 1930s, the Battle of Trafalgar). Both, it would seem, can naturally be characterized via fuzzy membership functions. In this way, we may represent the relevance of individual facts to characterizing a situation or to recognizing an instance of an abstract situation type.4 One further expansion of situation logic is necessary to make it a useful tool in data fusion. To use infons as vehicles for reasoning under uncertainty, we replace the bipolar truth-value model of [4] with a probabilistic model, i.e. we wish to treat the term p in expression (1) as a continuous density function. This extension allows us to infer hypotheses – stated in terms of sets of probabilistic infons – across uncertain real-world state estimates together with uncertain inference models (e.g. expressed as conditionals among infons) and conditional and hypothetical state estimates (e.g. in counter-factual historical analysis or analysis of fiction). In summary, we propose that situations – to include abstract situations (or situation types) and concrete situations (or situation tokens) – can be represented as sets of infons. Sensor measurement reports, track reports and fused situation estimates alike can be stated in terms of probabilistic infons. This formulation allows inferencing from one perceived entity in a situation to another and from estimated entities, their attributes and relationships to situations. Attributes of individuals can be treated as1-place relationships. Real-world situations can be classified in the same way as we classify realworld individual entities. Indeed, an entity is – by definition – a kind of situation. A situation-type or entity-type s can be defined in terms of an equivalence class R(s) of relations: 3 The term ‘infon’ was coined as an analogy to ‘electron’, ‘meson’, ‘photon’, suggesting a discrete “particle” – though not necessarily an elementary, indivisible particle – of information. Note that an infon is not the same as a proposition, which is a claim about the world. An infonic proposition is one in which a set of information is claimed concerning the world or another situation. Such a statement has the form s |= I, where s is a situation and I is a finite set of infons [4, pp. 62f]. 4 If we can completely specify both abstract and concrete situations by the use of fuzzy membership functions, then there is no need to include situations as ontological primitives.

AN APPROACH TO THREAT ASSESSMENT

105

‘x is of type s’ can be defined as r(m)  R(s) Ÿ y1,...,ym1[s | (r, x, y1,...,ym1, h, k,1)].

(2)

The Bowman model for a data fusion node [2, et al.] applies to level 2 and 3 processes as well as to level 1 processes. In this model, data association involves generating, evaluating and selecting hypotheses that a set of information (e.g. a set of sensor measurement reports) is the maximal set of information to be used in inference: x A level 1 hypothesis is a set of measurements, hypothesized to be the maximal set of measurements of a perceived entity. x A level 2 or 3 hypothesis is a set of perceived entities (whether from individual sensors or from a multi-sensor data fusion process) hypothesized to be the maximal set of entities in a perceived relationship or situation. Level 1 state estimation (e.g. concerning target identification, location or tracking) typically involves determining infons involving 1-place relationships (or, possibly, n-place relations with n-1 bound or parameterized variables). Level 2 and 3 fusion hypotheses generally concern multi-place relationships. A situation can imply and can be implied by the states and relationships of constituent entities. As in Level 1 inferencing (i.e. with 1-place relations), we can write production rules based on logical, semantic, causal, customary or material (etc.) relationships among predicates of any length. Characteristic inference patterns include the following (in which the infon notation is replaced with more concise predicate calculus notation):

x

Situational Inference pattern

P(n) ( X (n) ) Ÿ S

x

(3)

Level 1 Inference pattern

f ( P (1) ( x) | Q (1) ( x), S )

(4)

(e.g. single target likelihood functions or Markov transition densities)

x

Level 2 and 3 Inference patterns

f ( P ( 2 ) ( x, y ) | Q (1) ( x), R (1) ( y ), S ) (Level 1o2 deduction)

(5)

106

A.N. STEINBERG

f (y[ P ( 2 ) ( x, y )] | Q (1) ( y ), S )

f ( P (1) ( x ) | Q ( 2 ) ( x, y ), S )

(Level 1o2 induction)

(6)

(Level 2/3 o1 deduction)

(7)

(Level 2 o2, 3o3, 2 o3 or 2 o3 deduction).5 (8)

f ( P ( 2 ) ( x, y ) | Q ( 2 ) ( x, y ), S )

Such inferences are amenable to the machinery of Generalized Belief Propagation, whereby entity states are inferred conditioned on other entity states (i.e. on other states of the same entities or on the states of other entities) [13]. In such a formulation, belief in a state xj of an entity X is modeled as kI X ( x j )

bX ( x j )

–m

W ,X

( x j );

(9)

W N ( X )

in terms of “local” evidence MX(xj) and evidence passed as “messages” from other nodes W in the immediate neighbourhood N(X) in the graph that represents the set of relationships in the relevant situation:

mW , X ( x j )

¦I

W

( wi )\ W , X ( wi , x j )

–m

Y ,W

( wi ).

Y N (W ) \ X

wi

(10)

This can be restated in terms of a density function on attributive and relational states among X and its neighbours: mW , X ( x j )

¦I

W

wi

( wi ) ³ f ( wi , x j | r (W , X )) f (r (W , X )) r

–m

Y ,W

( wi ).

Y N (W ) \ X

(11)

Joint belief, then, is given as k\ W , X ( wi , x j )IW ( wi )I X ( x j )

bW , X ( wi , x j )

–m

Y ,W

Y N (W ) \ X

( wi )

–m

Z ,X

( x ).

Z N ( X ) \W

(12)

Here, as above, k is a normalizing constant such that beliefs sum to 1.

5 These types of deduction can include, for example, estimating multi-target likelihood functions or multi-target Markov transition densities.

AN APPROACH TO THREAT ASSESSMENT

107

An important feature of this formulation for the present purpose is the fact that infons and situations can be elements of other infons. For example, consider an infon of the form 1= (Believes,x1,Ș2,h,k,p); to be interpreted as “with probability p, at place and time h, k, x1 believes that Ș2”, x1 being another infon. Similar nested infons can be created with such predicates R = Perceives, Hypothesizes, Suspects, Doubts that, Wonders whether, etc., or such compounds as “with probability p, at place & time h,k, x1 asked x2 whether x3 reported to x4 that x5 believes that Ș2”. In this way, the Probabilistic Situation Logic representational scheme can be used to characterize and reason about counterfactual and hypothetical situations, their relations to one another, and their relations to reality. Probabilities embedded in recursively nested infons allow these pieces of information to be combined with sensor reports and information of other types. The second-order formulation also permits inferencing concerning propositions other than assertions. For example, a first-order interrogative “is it true that ı?” can be restated as the assertion “I ask whether ı” or, in infon notation, “(ask whether, ı,h2,k2,p)”. Similarly for other modalities, replacing the relation ‘ask whether’ with ‘demand that’, ‘believe that’, ‘fear that’, ‘pretend that’, etc.6 Infons can be nested recursively: “x perceives that y perceives that ı”. Situations, too, may be nested recursively to characterize, for example, my beliefs about your beliefs or my data fusion system’s estimate of the product of your data fusion system. One can use such constructs to reason about such perceptual states as one’s adversary’s state of knowledge and his belief about present and future world state; e.g. our estimate of the enemy’s estimate of the outcome of his available courses of action. In this way, the scheme supports an Operational Net Assessment analysis framework that can reason simultaneously from multiple perspectives.

References 1.

Barwise J, Perry J. 1983. Situations and Attitudes. Cambridge, MA: Bradford Books, MIT Press. Bowman CL, Steinberg AN. 2001. A systems engineering approach for implementing data fusion systems. In: Handbook of Multisensor Data Fusion, Hall DL, Llinas J, editors. London: CRC Press. Curry H, Feys R. 1974. Combinatory Logic, Volume 1. Amsterdam: North-Holland. Devlin K. 1991. Logic and Information. Cambridge: Press Syndicate of the University of Cambridge. Khalil A. 2005. Computational Learning and Data-Driven Modeling for Water Resource Management and Hydrology [dissertation]. Utah State University.

2.

3. 4. 5.

6

In our formulation, instances of such modalities (e.g. sensing, perceiving, believing, fearing) are cognitive relationships r and therefore can appear either as the initial argument of an infon or as one of the succeeding n-tuple of arguments.

108 6. 7.

8. 9.

10.

11.

12.

13.

A.N. STEINBERG Little E, Rogova G, Boury-Brisset AC. 2005. Theoretical Foundations of Threat Ontology for Data Fusion Applications, DRDC, Valcatier TR-2005-269. Nozawa ET. 2000. Peircean Semeiotic: a new engineering paradigm for automatic and adaptive intelligent systems. In: Proceedings of the FUSION’2003-3th Conference on Multisource-Multisensor Information Fusion, Paris. Pearl J. 1988. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Francisco, CA: Morgan Kaufmann. Steinberg AN, Bowman CL, White FE. 1998. Revisions to the JDL Model. In: Joint NATO/IRIS Conference Proceedings; 1998 Oct.; Quebec. Reprinted in: Sensor Fusion: Architectures, Algorithms and Applications. Proceedings of the SPIE, Volume 3719; 1999; Quebec. Steinberg AN, Bowman CL. 2004. Rethinking the JDL data fusion model. In: Proceedings of the 2004 MSS National Symposium on Sensor and Data Fusion, Volume 1; 2004 June; Columbia. Steinberg AN. 2005. Threat assessment technology development. In: Proceedings of the Fifth International and Interdisciplinary Conference on Modeling and Using Context (CONTEXT’05); 2005 July; Paris. White FE. 1988. A Model for Data Fusion. In: Proceedings of the First National Symposium on Sensor Fusion, Volume 2. GACIAC, IIT Research Institute; Chicago. p 143–158. Yedidia JS, Freeman WT, Weiss Y. 2002. Understanding belief propagation and its generalizations. In: Exploring Artificial Intelligence in the New Millennium, Lakemeyer G, Nebel B, editors. San Francisco, CA: Morgan Kaufmann.

HARBOUR PROTECTION AND HIGHER LEVEL FUSION: ISSUES AND APPROACHES

GALINA L. ROGOVA Encompass Consulting, USA

Abstract. Harbour security is a strategic area of concern for the international community since ports and approaches to them are very important for the national and world economy. Reducing port vulnerabilities requires increased situation awareness, which can be achieved by utilizing higher-level fusion processes (situation assessment and impact prediction). The result of higher-level fusion is a coherent composite picture of the current and projected situations, which provides decision makers with essential information to help them understand and control the situation, and act effectively to mitigate its impact. Higher-level fusion processes utilize fused sensor data, intelligence information, databases, maps, information on prior similar situations, preliminary assessment of the vulnerability of the area under consideration, and intent and capability of a possible adversary. The purpose of this paper is to discuss major challenges, specific requirements, and approaches to designing higher level fusion processes as applied to the problem of harbour security. Keywords: Harbour protection, higher-level fusion, ontology, reasoning

1 Introduction Harbour security is a strategic area of importance for the international community since “a terrorist incident against a marine transportation system would have a disastrous impact on global shipping, international trade, and the world economy in addition to the strategic military value of many ports and waterways” [1]. One of the key goals of strengthening harbour security is to “increase maritime domain awareness – a combination of intelligence, surveillance, and operational information to build as complete a picture as possible to assess the threats and vulnerabilities in the maritime realm” [2]. Maintaining coherent situation awareness is essential for making informed decisions aimed at detecting and deferring threat and assessing the impact of those decisions more effectively. Situation awareness requires a contextual understanding and interpretation of the events and behaviours of interest, which can be achieved by utilizing E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

109

110

G. L. ROGOVA

higher-level fusion processes. Higher-level fusion processes (situation and threat assessment) employ information about objects, human and electronic intelligence, historical databases, and domain knowledge to understand the relations between these objects and the dynamics of these relations within a specific context in order to assess and monitor the current situation, and predict threat. Building higher level fusion processes for harbour protection is complicated by specific characteristics of the maritime environment. Assessing situations and threat requires monitoring not only critical harbour facilities but also vast coastal areas, sea surface, underwater, the air, approaching objects, ship personnel, cargo, cargo handlers, etc. All this involves gathering and fusing a large amount of uncertain heterogeneous information of varying significance coming from geographically distributed sources, often of low fidelity, contradictory, and redundant, to gain knowledge of the entire domain. Information to be processed and made sense of includes but is not limited to data obtained from Automatic Identification System (AIS) for commercial vessels larger than 65 feet, sensor data (underwater sensors, infrared imagers, radars, etc.), surveillance reports, human intelligence reports, and information obtained from open sources. The lack of a priori statistical information about this information as well as its reliability and relevance further complicates the problem. Successful processing of this information also demands information sharing and dissemination, and action cooperation of multiple maritime stakeholders such as different nations, navies, cost guards, lawenforcement and regulatory agencies, and commercial shipping companies. Such a complex environment calls for designing a higher-level fusion system capable of processing all of this information to support multiple decision makers in monitoring the environment, and detecting and deferring threat. This system is not to replace analysts but to serve as a decision aid and to help decision makers to better understand current situations and to see their possible impact. Specific characteristics of the problem of situation awareness for harbour protection define specific requirements for the architecture and processes of a higher level fusion system. Such a system should allow for complex distributed modeling reflecting the distributed nature of the harbour environment, information characterizing this environment as well as the distributed nature of decisionmaking and the scalability issue. The system has to be able to process an enormous amount of information coming in different forms, i.e. numerical sensor readings and linguistic information obtained from human intelligence reports and open source information mining. It has to deal with uncertainty, poor reliability, and contradiction inherent to this information as well as the lack of statistical information and knowledge of all the wealth of possible threat scenarios. The system has to provide for inter- and intra-agency cooperation, and information sharing while satisfying significant privacy and security requirements. The needs for cooperation and information sharing between different stakeholders require also a capability of reconciling the differences between different information formats accepted by different agencies and call for creating a common vocabulary uniquely understandable by everybody involved An important issue in designing

HARBOUR PROTECTION AND HIGHER LEVEL FUSION

111

such systems is a capability of modeling the utility of various actions in response to predicted or recognized threat. This paper discusses a possible approach to designing higher-level fusion processing as applied to the problem of harbour protection.

2 General approach The purpose of situation and threat assessment (STA) is to develop probable explanations of the situation based on prior knowledge and incoming transient information about individual entities obtained at lower level fusion and to predict possible consequences based on the constructed situational picture. Several threat stages are identified in the literature (see, e.g., [3, 4]): x x x x

Potential threat including Strategic Warning Imminent threat Attack assessment and response Consequence management, including secondary threats detection and management

Although every stage has its specific characteristics, threat assessment during each of these stages calls for assessing and monitoring the states of the environment, detecting, associating, and characterizing single objects and events of interest, and reasoning about relations between them, their aggregates at different levels of granularity, and their behaviour within a specific context. It is a difficult problem, especially when dealing with terrorist or asymmetric threat since characteristics of asymmetric threat “may be considered to be the “different,” “unknown,” “unconventional,” or “unimaginable.” With many of these threats, we know only that there is something lurking out there, yet we are not certain how, when, or where to look for them” [5]. Therefore, important components of STA are the identification of this “different,” “unknown,” “unconventional,” or “unimaginable” and the discovery of underlying causes of observed situational items and their behaviour. Figure 1 presents a possible notional architecture for STA for harbour protection. There are several essential components of STA processing required for building current and predicted situational pictures, and detecting and managing threat: x Formally structured and computationally tractable domain representation capturing the basic structures of relevant objective reality and users’ goals, functions, hypotheses, chain of command, information requirements, and domain knowledge. x Dynamic reasoning about situational entities of interest, attributes, aggregates, relations and their behaviour over time within a specific context for making sense of current situation and predicting possible outcomes. x Inter- and intra-fusion level quality and consistency control procedures.

112

G. L. ROGOVA

x Domain specific models used to support reasoning processes. Domain specific models for STA for harbour protection have to include models of typical situations and normal behaviour of situational items along with models of possible threat scenarios. It is necessary to note that all of the system components described above have to be designed to support highly distributed fusion processing. The remainder of this paper comprises a description of the processes presented in Figure 1 in greater detail. Reasoning About Situations and Threat Quality Check of Level 1 Estimations Objects Attributes Credibility Quality

Delay vs. Sensor Mng

Dynamic reasoning about objects, attributes, aggregates, relations and their behavior over time within the context of maritime security

Inconsistency , Unusual behavior

?

Belief change via abduction (Discovery of possible causes of detected events) Quality ? Dynamic Situational Picture formation Fused information about items of interest

Level 1 Fusion processes

Impact (threat) assessment

Formally structured and computationall y tractable domain representation

Observations: •Sensor data •Intelligence reports •AIS information •Open world information

Real World

Quality Check of STA Quality ?

Delay

Current and predicted situational pictures

•Goals •functions •Information needs (EEI) •Hypotheses •Actions •Domain Knowledge

•Domain specific models: - Models of usual behavior - Simulated threat scenarios - etc

Environmental parameters: •Time of the day •Weather •Geospatial parameters • etc

Figure 1. Notional architecture of STA processing.

3 Domain representation STA deals with context-dependent information about dynamic reality to infer and approximate critical events and characteristics of the environment in relation to particular goals, capabilities and policies of decision makers. Designing STA processes requires a formally structured representation of reality capable of

HARBOUR PROTECTION AND HIGHER LEVEL FUSION

113

capturing the situational and threat entities at different levels of granularity, and the wealth of relations (e.g., spatio-temporal, intentional, causal, and dependence relations) between them. At the same time, STA requires an understanding of all of the goals, objectives, function, and organizational structures of decision makers, their knowledge about problem solving, and essential elements of information required to achieve these goals and solve the problems. A combination of formal ontology of situation and threat with “ontology of problem solving” [6] provides a framework for reasoning about situations and threat [7, 8]. “Ontology of problem solving” for STA in the maritime domain has to provide a decomposition of the decision maker’s chains of command, goals, functions, information requirements, along with procedures and regulations related to ships, cargo, and people movements, to support a domain-specific generation of hypotheses about the environment and high-level reasoning about these situational hypotheses. Decision makers’ goals, functions, their relations, organizations and policies along with their information needs may be identified by means of Cognitive Work Analysis (CWA) [8, 9]. CWA is a systems-based approach providing an understanding of what exactly various decision makers are expecting from a situation picture and what information should be represented. In such an “unknown,” “unconventional,” or “unimaginable” environment as harbour protection, information provided as a result of CWA has to be augmented by knowledge obtained from the results of simulations of possible threat scenarios. Formal ontology of situations and threat comprises two types of items: spatial (situational items) and temporal (processes), together with the relations between and among them. Spatial items, elements of the embedded snap ontology, and relations between them are defined by a set of spatial and mereological attributes. The values of these attributes define the state of these items and a corresponding state of the environment. Temporal items, i.e. processes, are elements of the related span ontology, which describe the temporal behaviour of the situational items and dynamics of attributes and relations. Important characteristics of processes are events representing a transition between states of the environment defining situations. Each relation characterizing a situation falls into one of two basic categories: inter-class relations and intra-class relations. Intra-relations (i.e., internal relations) are spatial, temporal, or functional relations that exist within a given set of ontologically similar items while inter-class relations (e.g., external relations) exist between various items. It is necessary to mention that information utilized in STA is highly uncertain, noisy, contradictory, redundant, of varying significance, and of low fidelity. This makes it necessary to incorporate uncertainty, reliability, and imprecision into characterization of qualitative mereotopological relations to provide for reasoning about situations and threat [10]. Ontologically, threat is defined as an integrated whole of intent, opportunity, and capability [11, 12]. An ontological analysis of threat elements, inter-relations between them and intra-relations between their components allows for a formal description of various threat conditions such as potential and viable threat as well as different types of threat such as dispersed and unitary potential threats.

114

G. L. ROGOVA

Understanding the distinctions between the various formal configurations and relations of threat elements allows for an understanding of what kinds of entities and relations need to be present to constitute each threat condition or when a threat item changes its ontological status from that of a potential threat to a viable one. Since intent, capabilities, and opportunities and their components are connected to one another via relations of dependence, the disruption or dissolution of any one of these elements results in a disruption or dissolution of the threat as a whole. For all of the reasons mentioned above, it is important to understand the role of each element of a threat condition by providing an appropriate ontological decomposition of it as well as of those relations that tie it to other threat items. A more detailed description of a formal ontology of situation and threat is presented in [10–12].

4 Reasoning about situations and threat Building and predicting situational pictures require a contextual understanding of the states of the environment critical to decision makers. Harbour security involves multiple hierarchically organized and geographically distributed decision makers with various goals, functions, and information requirements. The organizational structure of decision makers and relations between them call for distributed reasoning about situations at various levels of granularity. The STA processes comprise the dynamic generation of hypotheses about possible current and future states of the environment and an assessment of their plausibility. Due to reasoning about situational items at different levels of granularity, an assessment of the plausibility of more complex hypotheses may require hierarchical processing, which includes not only reasoning about situational items and relations between them but also relations between hypotheses and assessments of plausibility of lower-level hypotheses. Dynamic reasoning about plausible states of the environment is based on reasoning about characteristics and on the behaviour of situational items and their relations within a specific context. It requires one to match relations and behaviour inferred from current uncertain observations that represent concrete situations (whether actual, real-world situations or hypothetical ones) to situation type [3]. These relations and behaviours are constantly updated by newly processed observations and evaluated for quality, such as reliability, relevancy and consistency with current knowledge about the situations and their “normal behaviour.” The discovery of characteristics, relations, and behaviour inconsistent with knowledge about their “normal behaviour” corresponding to the assessed current situation invokes causal inference aimed at possible explanations of observed inconsistency. In the asymmetric threat environment, observed inconsistency can be caused either by poor quality of the incoming data, or insufficient knowledge about “normal behaviour,” or underlying threat conditions.

HARBOUR PROTECTION AND HIGHER LEVEL FUSION

115

Discovery of the underlying causes of observed situations is the result of abductive reasoning or “inference for best explanations.” This abductive process of reasoning from effect to cause requires [13, 14]: x Constructing or postulating possible hypotheses and the states of the world explaining observations x Computing the plausibility of these hypotheses x Selecting the most plausible hypothesis Automatic hypothesis generation is the most difficult process to implement within STA. In general, hypothesis generation may be performed via case based reasoning by utilizing historical real-life as well as simulated situation and threat scenarios and/or by human experts. A reasoning framework for evaluating plausibility of hypotheses has to allow for combining expert knowledge and observations, and provide an efficient uncertainty representation. The process of hypothesis evaluation has to take into account the following considerations [14]: x To what degree the hypothesis to be selected is better than alternatives. x How good the hypothesis is in and of itself independently of alternatives considered, i.e. one should be cautious about accepting a hypothesis, even if it is clearly the best one we have, if it is not sufficiently plausible in itself. x Reliability of incoming data. Since, in the asymmetric threat environment, knowledge about possible explanations is incomplete and the unimaginable can happen, the set of hypotheses considered may not be exhaustive and an open world assumption has to be included in the process of plausibility computation and hypotheses evaluation. A possible approach to reasoning about situations and threat may be constructed in the framework of generalization of the Probabilistic Argumentation System (PAS) (see, e.g. [15]). PAS has many important properties, which make it appropriate for reasoning about situations in the harbour protection environment. It exhibits the non-monotonicity required for reasoning under uncertainty, provides a mechanism for “cost-bounded approximations” (anytime argumentative reasoning), and supports inferring from a given knowledge to a possible explanation for an observed or reported fact. Following [15], PAS can be described as an approach to non-monotonic reasoning under uncertainty, combining symbolic logic with probability theory for judging hypotheses about the unknown or future world by utilizing given knowledge. Logic is used to find arguments in favour and against a hypothesis about possible causes or consequences of the current state. An argument is a defeasible proof built on uncertain assumptions, that is, a chain of deductions based on assumptions that make the hypothesis true, or false. Every assumption is linked to an a priori probability that this assumption is true. The probabilities that the arguments are valid are used to compute the credibility of the hypothesis,

116

G. L. ROGOVA

which can then be measured by the total probability that it is supported by the totality of supportive and refuting arguments. The resulting degree of support corresponds to belief in the theory of evidence, and used to make a decision as to whether there is enough information to select one of the hypotheses and which hypothesis is to be selected. Situation and threat assessment for harbour protection is characterized by high uncertainty and lack of domain knowledge and therefore a priori probabilities that the assumptions are true are rarely available. In such an environment, PAS has to be generalized to “Evidential Argumentation System” to utilize sub-additive subjective belief measures. These subjective beliefs can be expressed in a linguistic form, e.g., very high, high, low, very low with subsequent quantization of these linguistic values. The belief measures can be also represented numerically as a function of the values assigned to attributes and relations characterizing the state of environment [16].

5 Quality control The success of dynamic STA greatly depends on the quality (e.g. uncertainty, vagueness, reliability, and relevance) of individual and integrated in level 1 fusion data as well as information resulting from all steps of higher-level fusion processing. The information quality considerations play an important role in transferring information within fusion levels as well as in between levels [17]. The strategies for quality control within as well as between fusion levels can include: x x x x

Eliminating information of low quality from consideration. Incorporating information quality into fusion processing. Utilizing process refinement by sensor management. Delaying the transferral of results to the next processing level or to decision makers until information of better quality can be obtained as a result of additional observations and/or computations (anytime processing).

Quality control is highly context specific since the notion of “good,” “poor,” or “satisfactory” quality greatly depends on context. Incorporation of information quality into STA processing is a difficult task. There is a relatively small body of literature in the data fusion community that addresses this topic. It is also not clear how to measure the quality of the result of different processes and how different dimensions defining information quality are interrelated. Utilization of anytime processing has to take into account the fact that dealing with threat requires timely decisions and swift actions. Waiting may result in unacceptable decision latency, leading to significant damage and casualties. At the same time, the cost of false alarms can be very high, since it can result in the disruption of port operations and in the waste of valuable resources. Therefore, the cost of waiting for additional information or the cost of additional computation delay to produce information of better quality and reduce false alarm has to be justified by the benefits of obtaining results of better quality. This can be achieved

HARBOUR PROTECTION AND HIGHER LEVEL FUSION

117

by either implicitly modeling the expected utility of making a decision at a certain moment or by comparing the quality of information achieved at a certain time with a time varying threshold [18].

6 Conclusions This paper presents a discussion on major challenges, specific requirements, and approaches to designing higher-level fusion processes as applied to the problem of harbour security. It is clear that the utilization of higher-level fusion can contribute to successful identification, prevention, and management of terrorist attacks. It also appears that a belief based argumentation framework can be used as a foundation for building reasoning processes for STA. Designing an effective higher-level fusion system for harbour protection requires special attention to the problems of designing: x Sophisticated methods to support information sharing among various countries and agencies to promote interoperability while ensuring security and access control. x A comprehensive and computationally tractable maritime domain representation, which includes a combination of “ontology of problem solving” providing a decomposition of all of the goals, functions, knowledge about problem solving, and information requirements of decision makers, with a formal ontology of situations and threat for the maritime domain recognizing the uncertainty of the environment. x A realizable and scalable methodology of reasoning under uncertainty for recognizing complex situations and threat. x A library of simulated possible threat scenarios and models of “unusual behaviour” to support reasoning processes for an assessment of the causes of such behaviour and the recognition of potential threat. x Sophisticated techniques for the characterization of the quality of all sources of information as well as of the quality of interim and final results of all processes. x A decentralized fusion architecture addressing multi-jurisdictional, multinational cooperation. x Methods of evaluation of the effectiveness of utilizing STA.

References 1.

2.

Admiral T. H. Collins, R. C. Bonner, Admiral D. M. Stone. 2004. Statement on maritime security status before the committee on commerce, science, and transportation, U.S. Senate, March 24, 2004, can be found on http://www.globalsecurity.org/security/library/ congress/2004_ h/040324-collins.pdf. Admiral T. H. Collins. 2003. Statement on transportation security before the committee on commerce, science, and transportation, U.S. Senate, September 9, 2003, can be found on http://www.globalsecurity.org/security/library/congress/2003_ h/030909-collins.pdf.

118 3.

4.

5. 6.

7.

8.

9. 10.

11. 12.

13. 14.

15.

16.

17.

18.

G. L. ROGOVA A. Steinberg, “An Approach to Threat Assessment,” in Proceedings of the FUSION2005 – 8th International Conference on Multisource Information Fusion, Philadelphia, 2005. E. Shahbazian, M. DeWeert, G. Rogova, Findings of the NATO Workshop on Data Fusion Technologies for Harbour Protection, Defence & Security Symposium, SPIE, Orlando, Florida, 2006. S. Lambakis, J. Kiras, K. Kolet, Understanding “Asymmetric” Threats to the United States, National Institute for Public Policy, Fairfax, 2002. B. Chandrasekaran, J. R. Josephson, V. R. Benjamins, Ontology of Tasks and Methods, in Proceedings of the 11th Knowledge Acquisition Modeling and Management Workshop, KAW’98, Canada, 1998. P. Scott, G. Rogova, Crisis Management in a Data Fusion Synthetic Task Environment, in Proceedings of the FUSION’2004 – 7th Conference on Multisource Information Fusion, Sweden, 2004. A. Bisantz, G. Rogova, E. Little, On the Integration of Cognitive Work Analysis Within Information Fusion Development Methodology, in Proceedings of the Human Factors and Ergonomics Society Annual Meeting, New Orleans, 2004. A. Rasmussen, J. Pejtersen, L. Goodstein, Cognitive Systems Engineering. Wiley, New York, 1994. E. Little, G. Rogova, Ontology Meta-Model for Building a Situational Picture of Catastrophic Events, in Proceedings of the FUSION’2005 – 8th Conference on Multisource Information Fusion, Philadelphia, 2005. E. Little, G., Rogova, A. C. Boury-Brisset, “Theoretical Foundations of Threat Ontology (ThrO) for Data Fusion Applications”, DRDC, Valcatier TR-2005 -269, 2005. E. Little, G, Rogova, An Ontological Analysis of Threat and Vulnerability, in Proceedings of the FUSION’2006 – 9th Conference on Multisource Information Fusion, Florence, Italy, 2006. G. Harman, The Inference to the Best Explanation, Philosophical Review 64, 88–95, 1965 P. Thagard, C. P. Shelley, Abductive reasoning: logic, visual thinking, and coherence. In M.-L. Dalla Chiara et al. (Eds.), Logic and Scientific Methods. Dordrecht: Kluwer, 413–427. HTML, 1997. R. Haenni, J. Kohlas, N. Lehmann, Probabilistic argumentation systems. In J. Kohlas, S. Moral (Eds.), Handbook of Defeasible Reasoning and Uncertainty Management Systems, Volume 5: Algorithms for Uncertainty and Defeasible Reasoning. Dordrecht: Kluwer, 2001. G. Rogova, P. Scott, C. Lollett, R. Mudiyanur, Reasoning about situations in the early post-disaster response environment, in Proceedings of the FUSION’2006 – 9th Conference on Multisource Information Fusion, Florence, Italy, 2006. J. Llinas, C. Bowman, G. Rogova, A. Steinberg, E. Waltz, F. White, Revisions to the JDL Data Fusion Model II, in Proceedings of the FUSION’2004 – 7th Conference on Multisource Information Fusion, Sweden, 2004. G. L. Rogova, P. Scott, C. Lollett, Distributed Fusion: Learning in multi-agent systems for time critical decision making. In E. Shahbazian, G. Rogova, P. Valen (Eds.), Data Fusion for Situation Monitoring, Incident Detection, Alert and Response Management. FOI Press, 123–152, 2006.

PERCEPTUAL REASONING MANAGED SITUATION ASSESSMENT FOR HARBOUR PROTECTION

IVAN KADAR Interlink Systems Sciences, Inc., USA

Abstract. A central problem in Harbour Protection situation assessment is the rapid acquisition and interpretation of information from multiple potentially disparate sources. A perceptual reasoning system paradigm framework is presented as a candidate toward the solution of the problem. The perceptual reasoning system is shown to be a natural governing mechanism for adaptive interacting control of information fusion, the use and update of prior knowledge bases, the generation of anticipation/prediction hypotheses, reinforcement learning and the timely management of uncertainty in situation assessment. The constructs and interaction among elements of the perceptual reasoning system are described. A conceptual data fusion/situation assessment system architecture based on the perceptual reasoning system paradigm is illustrated for harbour protection situation awareness using information from distributed C4ISR and local sensor systems, contextual, a priori and anticipatory information for enhanced target detection, localization, tracking and identification. Concepts are discussed for applying the candidate system to facets of the harbour protection problem as a means to enhance timely maritime domain awareness. Keywords: Harbour protection, C4ISR, situation assessment, perceptual reasoning, maritime awareness

1 Introduction A central problem in Harbour Protection situation assessment is the rapid acquisition and interpretation of information from multiple potentially disparate sources. In order to comply with the influx of information in establishing situation awareness, humans require decision support technologies such as Data/Information Fusion (DF) of which Situation Assessment (SA) is denoted as Level 2 of a five level taxonomy [5]. We will first define the Harbour Protection problem setting, along with available information sources, identify needs and then introduce SA methodology paralleling human perceptual reasoning – referred to as the Perceptual Reasoning Machine (PRM) [2, 3] – to rapidly gather, assess, interpret, combine information, develop

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

119

120

I. KADAR

situation awareness and identify potential intent. Subsequently, a conceptual data fusion/situation assessment system architecture based on the perceptual reasoning system paradigm is illustrated for harbour protection situation awareness, using information from distributed C4ISR and local sensor systems, contextual, a priori and anticipatory information for enhanced target detection, localization, tracking, identification and intent.

2 The harbour protection problem The envisioned conceptual problem setting is divided into local harbour/port/ perimeter protection surveillance systems [4, 10] coupled with distributed regional/coastal C4ISR systems, e.g., “Deepwater” [7]. It is postulated that suspicious vessels are identified and located by distributed systems while in transit towards ports, and information might be exchanged between the “local” and “distributed” systems, which we define as “coupled” systems, to represent mutual aid. Information sources and sensor suites for both systems may include: x An Automatic Identification System (AIS) for commercial vessels larger than 65 feet and merchant ships over 300 GRT, providing, within LOS range, unique ship ID, position and kinematic states, destination and ETA, cargo and other relevant information via transponder [6]. x Human observer reports for vessels in transit and near shores and ports. x For “local” harbour protection and, specifically, perimeter protection systems, potentially acoustic (sonar) underwater sensors augmented with unmanned underwater vehicles, surface night vision technologies such as infrared imagers and dual use (surface and under water) low-light level cameras, multispectral airborne imaging, radar and seismic sensors [4, 10]. x For “distributed” regional/coastal C4ISR systems, spatially and geographically diverse multi-platforms-based-sensors may include SAR, ISAR, GMTI, EO/FLIR [7]. These sensor suites in turn provide “target states” (kinematic and identification/classification) information collected and fused at a command center for target vessels within operational range of the platforms. x Standardized International Ship and Port Facility Security (ISPS) codes have been instituted world-wide [11] for vessels (and ports receiving them) over 500 GRT, at several security and warning levels which can serve as input (cues) to combined Situation and Threat Assessment (SA/TA) [2, 3] components of the surveillance systems. The above sources serve as input to the DF Levels, illustrated in Figure 1.

PERCEPTUAL REASONING MANAGED SITUATION ASSESSMENT

121

C4ISR C4ISR Distributed/ Distributed/ Regional Regional Coastal Coastal ISPS ISPSCode Code AIS AIS SAR SAR ISAR ISAR GMTI GMTI IRIR EO/FLIR EO/FLIR Harbor/Port Harbor/Port Local Local Visual Visual Sonar Sonar LL-EO LL-EO Seismic Seismic Radar Radar IR IR ISPS ISPSCode Code AIS AIS ...... Data Data bases bases

SOURCES

Level Level00 Source SourcePrePreProcessing Processing

DATA FUSION DOMAIN

Level Level Level22 Level33 Processing Processing Processing Processing SITUATION IMPACT SUB-OBJ SITUATION IMPACT SUB-OBJDATA DATA REFINEMENT REFINEMENT REFINEMENT REFINEMENT ASSESSMENT ASSESSMENT Level Level11 Processing Processing

Level Level44 Processing Processing PROCESS PROCESS REFINEMENT REFINEMENT

Human/ Human/ Computer Computer Interface Interface

Database DatabaseManagement Management System System Support Database

Fusion Database

Figure 1. Input to the data fusion processes.

It should be noted that distributed C4ISR system data might not provide adequate discrimination of vessels and underwater targets at ports for perimeter protection. This is why the concept of coupled systems was introduced. Furthermore, the lack of expected AIS and/or ISPS information is envisioned as an important characteristic feature data to the DF/SA/TA system. How to assure compliance with the ISPS code in different countries and how to adhere to standard terms is a question for the establishing organizations. If potential conflicts were to occur, an automated integrated DF/SA/TA system using all available data could serve as a decision aide. In addition, commercial vessels less than 65 feet, merchant ships less than 300 GRT and vessels less than 500 GRT are not expected to use either the AIS or the ISPS codes. An issue arises: how does one assess information about vessels that do not fall into either reporting class? It is conjectured that Data Fusion methodologies may resolve this issue.

3 Definitions of Level 2 fusion-situation assessment Level 2 fusion, referred to as SA or “situation refinement” has had several definitions over the years [5, 8, 9]: “estimation and prediction of relations among entities, to include force structure and cross force relations, communications and perceptual influences, physical context” [5, 8]; “an iterative process of fusing the spatial and temporal relationships between entities to group them together and form an abstracted interpretation of the patterns in the order of battle data” [9].

122

I. KADAR

Issues with the above definitions and some subsequent models based on these definitions are: x They do not incorporate human thought processes, human perceptual reasoning, the ability to control sensing, to gather information, and the essence of response time. x They imply the use of limited or no a priori information. x They only imply a potential for new knowledge capture. That is, an ability to control Levels 1–4 of DF [5] processes (refer to Figure 1) for knowledge capture in SA is necessary. In addition, if SA is to establish relationships (not necessarily hierarchical) and associations among entities, it should anticipate using a priori knowledge in order to rapidly gather, assess, interpret and predict what these relationships might be; it should plan, predict, anticipate again with updated knowledge, adaptively learn, and control the fusion processes for optimum knowledge capture and decision making. These features are similar to the characteristics of human perceptual reasoning [1–3]. Therefore it is conjectured that the “optimum” SA system should emulate human thinking as much as possible. For Harbour Protection specifically, one needs to develop audit trail of data from “distributed” C4ISR and “local” sensor systems, associate, correlate and optimally combine (fuse) human observer, ISPS, AIS and multisensor information in order to rapidly assess the situation and provide effective response strategies. In the following sections, several aspects of the above stated and related issues, as well as constructs derived thereof are discussed and illustrated.

4 The human perceptual system and perceptual reasoning Perception will be defined here as the response of a “Perceptual System” (defined as a feedback system) where the said perceptual system refers to relations between output and input, both assumed observable [1]. The role of the user, given the knowledge of the variables that enter into the relations or the adaptive feedback system itself, is to make inferences or guesses about relations. That is, in order to perceive, one must: (1) sense and deliver stimuli to the “system”, and (2) the “system” when “properly stimulated” can deliver feedback (“reinforcement”) to the system’s input in order to modify the system’s output [1]. We note here, that one can consider the change in the system’s output by reinforcement feedback as a learning process [3]. Hence, by definition, perception can be considered both an adaptive and a learning process. The generalization of a perceptual system and its adaptive feedback control process is termed the PRM. How the system is expected to respond is dependent on a priori learned information, inferred by “proper stimulation” providing fast response [3].

PERCEPTUAL REASONING MANAGED SITUATION ASSESSMENT

123

5 Generic Information Fusion Process Model System (PMS) The generic information fusion process model system (PMS) is depicted in Figure 2. This system utilizes an adaptive embedded control mechanism, known as the PRM [2, 3] illustrated separately in Figure 3. In this section, we will establish a definition of process modeling followed by a description of the operation of the PMS/PRM systems. Process modeling is defined as a set of procedures and algorithms that capture the functional (temporal and spatial) dependency relationships of tasks and/or processes being modeled. These models may be represented in either a relational database or expert system framework, which can be structured to execute data dependent opportunistic tasks. Thus, the process modeller serves as a meta-level

World State

Information Collection

Preprocessing

Resource Assignment Current Information (Knowledge) Sources

Prior Domain Knowledge

Learning Algorithms & Memory

Process

Anticipate/Predict (AP)

Model

Reinforcement

Systems

Interface to Decision Maker

Estimation, Evidence Queries Accumulation and Control Algorithms Gather/Assess (GA) &Beliefs Hypotheses Reinforcement

Figure 2. Generic information fusion process model system (PMS).

Prior Domain Knowledge

Learning Algorithms & Memory Anticipate/Predict (AP)

Current Information (Knowledge) Sources

Interface to Decision Maker

Estimation, Evidence Queries Accumulation and Control Algorithms Gather/Assess (GA) &Beliefs Hypotheses

Reinforcement (Perceptual Reasoning Cycle) Figure 3. Perceptual reasoning machine.

124

I. KADAR

controller of procedures and algorithms. The PMS can request data at various levels of abstraction and processing, select, cue and manage sensors (via resource assignment) under control of the embedded PRM, gather/assess (GA) and anticipate/predict (AP) adaptive learning control-loop mechanism. The PMS represents the time dependent functional applications domain states and utilizes a priori information. The application of the PMS/PRM constructs will be illustrated in the description of a conceptual DF system architecture for Harbour Protection in the following section. Details of PRM can be found in [2, 3].

6 A data fusion/situation/threat assessment system concept Figure 4 illustrates a conceptual DF architecture for a net-centric surveillance system. A key function of this system is the acquisition and interpretation of timely information. By narrowing the field of search, and by providing evidence about observations and mission constraints, the Situation Assessment/Threat Assessment (SA/TA) systems will be able to answer questions such as: What is it? Who is it? Whose is it? Which one did it? Which is which? Where is it? Where is which? Is it or isn’t it? Whence did it originate? Where is it going and what is its intent? What resource to use? Verify or Negate Most Likely Hypothesis (Reinforcement)

Evidential Reasoning & Identification Fusion (GA) Local Pre& Processing Distributed Alignment NetAssociation Centric Tracking Sensors/ Sources Levels 0 & 1

Situation & Threat Assessment Process Models

Level1

Human Computer Interface

Level 2,3

Feature Extraction & Classification Level0

Kinematic Fusion Level1

Learned knowledge base

Anticipate/ Predict (AP) Level1,2,3 4

Resource Mgmt & Sensors Mgmt Level4

Anticipation knowledge base

Process knowledge base

Prior Knowledge

Figure 4. Conceptual net-centric data fusion/situation/threat assessment architecture.

6.1 TRACING THE PMS/PRM FUNCTIONS The bold lines in Figure 4 provide a guide for tracing PMS/PRM functions [2, 3]. The active and passive data from “distributed” C4ISR and “local” sensors are controlled by the SRM depending on the scenario (e.g., target detections trigger information pull or push (cue sensors) from portal regions followed by information from “local” sensors in harbours and at ports). Requests for SRM are

PERCEPTUAL REASONING MANAGED SITUATION ASSESSMENT

125

derived from the process model systems (imbedded in the PMS/PRM constructs). A multi-target multi-sensor association/tracking/fusion system resolves track ambiguities. An identification system makes inferences based on declarations from target identification information (e.g., sensor data, AIS and ISPC codes and human reports). An evidential reasoning scheme resolves conflicts and manages/checks the convergence of hypotheses among anticipated and observed events thereby shortening response time. The SA/TA system (part of the PMS under PRM control) assesses the situation and prioritizes potential threats [2, 3].

7 Conclusions In this chapter, conceptual architectures providing effective situational awareness for Harbour Protection were presented. The concept of “a perceptual system” based on human perceptual reasoning was shown to be analogous to the PRM for the adaptive learning and control of the DF processes. The application of a PMS/PRM construct was illustrated as the integral component of a conceptual net-centric data fusion and SA/TA system for Harbour Protection. The PRM paradigm was shown to control gathering/assessment and anticipation/prediction of information from coupled “local” and “distributed” C4ISR sensor systems for SA/TA. It is suggested that these system concepts may provide an improvement in achieving data fusion system objectives and enhance timely maritime situational awareness.

References 1. 2.

3.

4. 5.

6. 7. 8.

Dember WN. 1965. The Psychology of Perception. New York: Holt, Reinhart & Winston. Kadar I. 2001. Perceptual Reasoning Managed Situation Assessment and Adaptive Fusion Processing. In: Kadar I, editor. Signal Processing, Sensor Fusion and Target Recognition X. Proc. SPIE Volume 4380; 2001 Apr. Kadar I. 2002. Perceptual Reasoning in Adaptive Fusion Processing. In: Kadar I, editor. Signal Processing, Sensor Fusion and Target Recognition X. Proc. SPIE Volume 4729; 2002 Apr. Sato T, Guthmuller H, Deweert M. 2005. Ports of Call. In: SPIE EO Magazine; 2005 Apr.; p 15–17. Steinberg A, Bowman C, White F. 1999. Revisions of the JDL Fusion Levels. In: Proc. of the SPIE Sensor Fusion Architectures, Algorithms and Applications III Conference; 1999 Apr.; p 430–441. Automatic Identification System (AIS) [Internet]. Available from: http://www.cruising. org.uk/CYCC/AIS.htm Coast Guard Integrated Systems [Internet]. Deepwater Objectives. Available from: www.icgsdeepwater.com/objectives/C4ISR/C4ISR_overview.php Data Fusion Lexicon. U.S. Department of Defence, Data Fusion Subpanel of the Joint Directors of Laboratories, Technical Panel for C3 [Internet]. Available from: http://www.data-fusion.org/article/php?sid=70

126 9.

I. KADAR

DSTO (Defence Science and Technology Organization) Data Fusion Special Interest Group, Data fusion lexicon. Department of Defence, Australia, 7 p., 21 September 1994. Available from: http://www.data-fusion.org/article/php?sid=70 10. SAIC Maritime Security [Internet]. Available from: www.saic.com/maritime/ security.html 11. What is the ISPS Code? Lloyd’s Register [Internet]. Available from: http://www.lr.org/ market_sector/maritime-security/what_is_ISPS_code.htm

UNDERSTANDING MILITARY INFORMATION PROCESSING – AN APPROACH TO SUPPORT INTELLIGENCE IN DEFENCE AND SECURITY

JOACHIM BIERMANN FGAN – FKIE, Germany

Abstract. Decision makers in defence and security require timely and accurate understanding of the situation in their respective area of responsibility as well as a prediction of the likely intentions and capabilities of supposed or potential adversaries. To achieve this, intelligence cells have to process large volumes of information and data from all kinds of sources to deduce situation awareness. Particularly in the area of non-conventional conflicts, e.g. in the fight against terrorism, heterogeneous and complex non-military information factors are influencing the production of intelligence. To be able to support automatically the production of intelligence, a sound understanding of the principles of reasoning and the cognitive processes of the operators in the human dominated area of heuristic information processing and fusion has to be developed. The main information processing steps in intelligence are described and recommendations for automatic support are given. Semantically based flexible information structuring methods are a necessary precondition to provide the user with automated support for the exploitation and fusion of information from unstructured text. A second challenging aspect in the automation of information fusion is induced by the heuristic nature of the real human processing of imperfect information. The human method of default reasoning, based on knowledge about the behaviour or structure of adversary factions, can be used as an template based approach to support the Collation, Analysis and Integration of information in intelligence. Keywords: Intelligence processing, semantic information structuring, behaviour model, default reasoning, closed world representation, interactive fusion system

1 Military view on information processing in intelligence 1.1 THE STRUCTURE OF OPERATIONAL INTELLIGENCE PROCESSING Intelligence processing is an important part of Command and Control (C2) in defence and security because the production of a most accurate situational E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

127

128

J. BIERMANN

awareness and understanding is essential prior to all decisions and activities. It is a basic requirement which is independent from the ever-changing variety of conflicts and threats. Within network centric operations and the global information environment of asymmetric threats, the growing challenge is to rapidly manage large volumes of data and information and to portray the results in a timely and appropriate manner. This task includes determining the most likely location, strength, and activities of all engaged faction forces and the deduction of their capabilities and likely intentions. A wide variety of information produced by the full spectrum of sensors and human sources has to be collected, filtered, processed and disseminated. This is commonly done in a structured and systematic series of operations which is called the “Intelligence Cycle” (IC). The representation of the military intelligence process in Figure 1 shows the interrelation between the Command and Control Cycle and the Intelligence Cycle [2]. The decision process of the OODA Loop interfaces with the Direction phase of the IC. OODA Loop Observation

Action

Perceived Situation

Orientation

Decision

Intelligence Cycle Direction

CCIR+PIR Assessment

Dissemination

Collection

Targeting Processing

Figure 1. Intelligence cycle interfacing with OODA loop.

The IC includes four stages which are defined as follows [5]: x Direction: “Determination of intelligence requirements, planning the collection effort, issuance of orders and requests to collection agencies and maintenance of a continuous check on the productivity of such agencies.” x Collection: “The exploitation of sources by collection agencies and the delivery of the information obtained to the appropriate processing unit for use in the production of intelligence.” x Processing: “The production of intelligence through collation, evaluation, analysis, integration and interpretation of information and/or other intelligence.”

UNDERSTANDING MILITARY INFORMATION PROCESSING

129

x Dissemination: “The timely conveyance of intelligence, in an appropriate form and by any suitable means, to those who need it.” These four discrete stages are conducted culminating in the distribution of the finished intelligence product. The Processing phase within the Intelligence Cycle is a structural series of actions where the information, which has been collected in response to the directions of the commander, is converted into intelligence products. It is here that the intelligence branch needs automation to be more effective in their work. A simplified representation of this phase of the IC is shown in Figure 2. The military understanding of these processing steps is kept in Allied Joint Intelligence Counter Intelligence and Security Doctrine (AJP 2.0) [6] and in NATO Glossary of Terms and Definitions (AAP6) [5]: Beha viour models are descriptive and qualitat ive

Level 2, 3 Behaviour Models & Doctrines

Documents Symbolic Inform ation

Input proces sing

Meaning of inform ation is context depende nt

Im pe rfec t Information

“Raw Data”

Hum an level 2, 3 methods are

Qualitative subjective and qua litative Methods

Match Fit

What does it mea n? What is it?

» Situation awareness » Impact ass essment

Output “Fused Information” » Classific ation

Numerical Data

» Identification » Tra ck

Sensor Data Level 0,1

Physical Models

Mathematical Methods

Models and methods a re normative, qua ntitative, and obje ctive

Figure 2. Integrated data and information fusion – low and high level fusion aspects.

x Collation: The registering and logging of the incoming information followed by its decomposition into individual information items. These individual information items are subject to categorizing according to either predefined categories or to new identified categories adapted to the mission. The categorized information items are finally cross-referenced, each one with the others. x Evaluation: The constituting appraisal of an item of information with respect to the reliability of the source and the credibility of the information. x Analysis: A step in which information is subjected to review in order to identify significant facts for subsequent interpretation. It consists of a number of interacting sub-processes to bring the analyst questions like: “Who is it?”

130

J. BIERMANN

“What is it?” “What does it means?” “Why is it happening?” etc., in order to recognize indicators and warnings. x Integration: A step whereby analyzed information or intelligence is selected and combined into a pattern in the course of the production of further intelligence. The process of building pictures of the current and of predictive situations from all gathered and analyzed information. x Interpretation: A step in the processing phase of the intelligence cycle in which the significance of the analyzed and integrated information is judged in relation to the current body of knowledge. Thus far, in its essence, the processing of information and the production of intelligence has been a pure human, cognitive process applying heuristic procedures based on personnel skill and experience. The appraisal of each piece of input information and of the deduced results is not done by numerical procedures based on pure technical measurements but by the subjective individual evaluation and assessment of textual information by a human operator. The analysis, integration and interpretation of information are performed to support the decision making of military leaders. They take responsibility for their decisions. If high level fusion is to be automated it must be taken into account that this involves substituting a human operation which is different e.g. to tracking in sensor data fusion, a task which in defence was hardly performed by man himself but rather was always performed automatically. So every methodical approach to information fusion has not only to be scientifically sound but also psychologically convincing to be accepted by the user. One way to do so is to carefully analyze the defined human procedures, to understand the view of the experts and their cognitive processes and to adapt human principles and methods for automated fusion concepts and procedures.

2 Characteristics of high-level fusion Regarded from a sufficient level of abstraction there is a structural similarity between sensor data fusion and high level information fusion. In both cases, we have imperfect input data about the real situation, coming from several sources, which is matched against a model of the world. The resulting output is taken to update the so far known picture of the real situation. An important difference between these two similar structured data fusion tasks lies in the following two aspects: x The kind of data to process x The kind of model to match against In sensor data fusion, numerical data from technical sensors is processed to acquire insight into the physical aspects of the battlespace elements, their type (object classification and identification) and their space-time behaviour (tracking). It answers the question: “What kind of object is it and how is it acting actually?”

UNDERSTANDING MILITARY INFORMATION PROCESSING

131

The relevant attributes and features are determined by the physical nature and characteristics of the objects; their dynamic behaviour can be modeled appropriately by the rules of physics. Accordingly, the matching of the input data against the physical models is done by mathematical methods (probability, statistics, optimization, artificial neural nets). These models and methods are of normative and quantitative nature and they are regarded to be objective. In heuristic intelligence processing, rather than numerical data, symbolic information, mainly documents with text, has to be handled as input. The aim is not to discover and quantify single objects but to understand the behaviour and intentions of the adversary. A most reliable picture of the battlefield has to be integrated from imperfect information and the impact on the perceived development of the military situation has to be deduced. Information fusion answers the question: “What does it mean?” In order to do this, intelligence officers practice by default a method of heuristic reasoning, which relies on their knowledge about the standards and rules of the adversary as well as the assumption that opposing forces will act according to these regulations and constraints for the benefit of their mission. It is common military experience and expert knowledge that the production of intelligence can be achieved by integrating current information based on the assumption of default behaviour. Behaviour modeling, such as doctrinal templating, is a descriptive, qualitative method of knowledge representation. Complex battlespace elements are described in a general manner as patterns or templates. The descriptions include, if available, relations, dependencies, and dynamic behaviour as well as the operational intensions and concepts, ethnic and or religious position, their moral concepts and political involvements. For an information fusion system supporting intelligence, two different kinds of models are needed [3]: x Behaviour models describing tactics of potential adversary factions and all necessary pre-conditions for the constituting activities x Models defining the “normal” situation, the common and unsuspicious behaviour In the first case, information fusion is the task of detecting indicators for a pattern of hostile activity defined by the behaviour model. This approach is used e.g. in low and high intensity conflicts of military or paramilitary type. In the second case, the task is to detect changes in the pattern of “normal” behaviour. These models have to be used in combating terrorism and asymmetric threat to be able to define indicators of suspicious activities.

3 Challenges and difficulties for information fusion in intelligence For an information fusion system in intelligence it is necessary to automatically obtain access to the semantic content of the input and background information. As high level fusion is mainly based on document input, it is very important to have automated extraction of the content of the single input document. The semantic

132

J. BIERMANN

meaning of a document must be accessible for subsequent automated context based information analysis and integration. It is necessary to have methods and procedures for automated text extraction and text analysis. Efficient linguistic methods are needed to understand the semantic meaning of the document input, tools which are able to cope with the information workload in network-centric intelligence and other C2 activities. Basically this is not a fusion problem in itself, but it is strongly connected to the domain modeling problem, e.g. by its ontological approach to semantic text analysis. Knowledge based information fusion is focusing on the heuristic human evaluation process [1]. Military information processing can be modeled as a context dependent and template-based heuristic reasoning process and by this the real information domain and workflow is modeled by a “Closed World” representation. This approach to information fusion in intelligence is based on the above mentioned common expert assumption that forces are organized in a structured manner and that they operate in a military reasonable and typical way. This has been shown to be true not only for classical war where doctrines are the basis for operations and training, but also for terrorist activities, such as in Northern Ireland or in Spain [7]. Even the preparation and planning activities of groups like al-Qaida exhibit some system and structure, which could have been recognized. Respective rules, doctrines, or modus operandi can be used as a basis for matching templates for default reasoning. By this, the analysis and fusion of incomplete and imperfect information of military reports and background knowledge can be supported substantially in an automated system. Such models can be developed by a thorough analysis of the real processes in intelligence and command and control [3 4]. Contrary to the finiteness of such a “Closed World” representation, the information context, which is relevant for a sufficient understanding of the situation, is not fixed, but it is open. Especially in new missions with low background intelligence and little knowledge about the behaviour of the adversary side, the initially available knowledge will not be sufficient and the models will have to be modified or new ones will have to be created. Presently, ontological models are very much used in knowledge representation, but they also define a “Closed World” representation of an application area. Any concept which is not covered by an ontology cannot be taken into account during semantic analysis and information processing. As a consequence, a fusion system based on such a module will not be able to consider the not defined information context; there is a blind spot for the fusion system. A solution to the obvious learning problem might be the concept of an interactive high level fusion system. It supports the human operator by providing formally correct fusion hypotheses using all information which is available to the system but leaves knowledge maintenance and final decisions to him. Such interactive systems will be adaptable to new and evolving situations. All these challenges define at least the following basic functional requirements for automated support of information fusion in intelligence:

UNDERSTANDING MILITARY INFORMATION PROCESSING

133

x Information extraction from unstructured text documents x Domain modeling: representation of the application area, knowledge, and information context x Information structuring methods for extracted input information x Matching procedures for current information into the representation of a perceived situation According to the descriptive and qualitative nature of the behaviour models which are represented by non-numerical methods and the fact that high level information fusion has to deduce from incomplete information, it is likely to use qualitative non-numeric methods for the matching of imperfect information with qualitative models. Among these are methods of non-monotonic logic like default logic and the close world assumption, which both correlate very much with human cognitive processing. High level fusion will probably benefit from quantitative mathematical methods like fuzzy logic, evidential reasoning, and probability if a suitable abstraction of the problem is reached in order to be able to define detail problems which reasonably can be formulated as quantitative questions.

4 Approach to an automated support of collation and analysis Table 1 shows three promising steps from the Processing phase of the IC, which are identified as candidates where functionality of information processing and fusion can be supported automatically [9]: TABLE 1. Candidates for automated information processing support Step

Semantic information processing functionality

Collation

Text extraction, categorization, cross-reference

Analysis

Correlation/link analysis

Integration

Pattern matching/aggregation/fusion

4.1 COLLATION Incoming information is firstly logged and stored into an appropriate repository. This technical information management can be fully automated, but there is no aspect of semantic processing or information fusion. To answer the commander’s pending “Requests for Information” (RFIs), topical information has to be retrieved from the data stores and decomposed into individual items, which are categorized and cross-referenced with previously categorized information items. A structured information set is thus established and is the main bases used to produce intelligence.

134

J. BIERMANN

To automatically support this information structuring process, starting with the decomposition of unstructured text into individual items of interest, there has to be a semantic processing to exploit the content of the documents. At present, text extraction and comprehensive semantic representation cannot be done in depth completely automatically. Some tools for text extraction exist (like GATE from Sheffield University) but they remain on a syntactical level of linguistic analysis. The necessary semantic exploitation of the content of a document is far beyond the ability of those tools. But they can be used within more complex application tools to support a human operator building semantic nets to structure the information. This has been achieved within the experimental work of the NATO Research Task Group on Information Fusion Demonstration (IST-038/RTG-016). Two nationally developed interactive tools (Idéliance (FR) and PARANOID (NL)) are used to support the structuring of information from unstructured input text. The tools use different linguistic approaches to support a human operator to establish interactively a semantically meaningful and automatically questionable representation of the content of incoming messages [8]. The main advantage of the implemented structured representation of the information content is its flexibility and openness to the varying requirements of the user. This flexibility is due to the semantic bases and interactive establishment of the representation. 4.2 CATEGORIZATION Categorization is a function of classification, by which a reported item or event is associated to a class of the underlying domain representation. Basically it is a semantic matching procedure. It answers the question: “Who or what is it?” The appropriateness and quality of the domain representation model mainly influences the quality of the results. If the key classes, which are used during the Categorization (e.g. people, places, events, organization, and equipment) are high level classes of the domain representation model then we can interpret Categorization as an initial step of Identification. The number and definition of the categories depends on the type of the mission, which is conducted. These categories are deduced from the commander’s Critical Information Requirements (CCIR) and RFIs. They define the main aspects of the evolving situation and reflect the relevant information context, which has to be covered to be able to give a description of the situation that fully reflects the abilities and intentions of the opposite parts and to make an assessment. The concrete association of the content of a report with a certain category of information is an experience guided correlation. Based on a semantic information representation, Categorization can be supported, as the defined categories will be part of the domain model.

UNDERSTANDING MILITARY INFORMATION PROCESSING

135

4.3 CROSS-REFERENCING Cross-referencing associates different pieces of categorized information because they share certain attributes, e.g. same time, same location, same person mentioned. No further interpretation of the content of a single piece of input is performed at this step, or any fusion of information (objects). The establishment of such links can easily be supported or even done automatically, if a structured representation of the evolving situation and its elements is available. 4.4 ANALYSIS Link analysis follows Categorization. It is the function by which information from different sources about the same item (event, activity, person, organization) is correlated to improve the knowledge about the single item. Different information items are associated to each other because they share certain attribute values. The richer the set of attributes of the observed or deduced situation elements the better this fusion process can be. This does not take into account that the different objects may match attributes of higher level classes of the domain representation model, classes of higher complexity different from the classes of all constituting objects. No “Part-of” or any “alike” relationship is tested because that is the functionality of the “Integration” step. If link analysis results in evidence about the identity of the related objects, all respective information is fused into one object which is kept for further processing, reducing by this the number of situation elements. Cross-referencing during the collation step may be interpreted as an initial link analysis step. The attribute or object fusion process within link analysis also can be supported by a semantic information representation. Fusion hypotheses can be developed automatically but the assessment will normally be left to the operator as part of an interactive assessment and decision process. 4.5 INTEGRATION Regarding the RFIs, the information items processed thus far have been transformed into situation elements of special interest. Finally, they have to be aggregated and fused with respect to underlying and hitherto confirmed templates about relevant aspects for situation awareness and understanding. Different lower level domain objects will be fused in order to deduce new and more valuable information, if they all satisfy the pattern of a more complex situation element according to basic intelligence information and expert knowledge about the relevant information context (doctrines, templates, patterns, etc.) of the situation. The following sub-functions of the Integration step are regarded as candidates for automated user support with knowledge based methods:

136

J. BIERMANN

x To carry out pattern analysis in order to recognize activity or behaviour patterns and thereby to deduce more complex activities and manoeuvres. x To integrate information items into the known generic structure of organizations and thereby verify or deduce the actual hierarchical structure of the organization. x To trigger an alarm function when significant parts of an activity pattern, taken as evidence of a certain type of threat, have been recognized. This approach to support Integration by knowledge based methods of default reasoning (template based information fusion) has been proved to be feasible for situation awareness in classical war situations [4].

5 Conclusion The operational requirements in asymmetric conflicts (urban war, harbour protection), with its more complex and varying situations and the growing amount of information from network centric warfare, makes it evident that the traditional way to process information is no longer sufficient and must be supported by new methods adapted to the respective situational context and the flood of unstructured information. The necessary development of domain and process models and associated information fusion methods for intelligence has to be based on a thorough analysis of the human cognitive processes and the actual information models used in operation. Tools which support the user in information extraction from unstructured text documents and to establish highly structured representation of all available information on the situation, give promising results for generating machine readable intelligence data, close to the operational means of understanding and processing information. Processing methods can be based on the human heuristic procedures of default reasoning and semantic domain representation. As long as automatic learning has not developed to a mature state, the disadvantages of “Closed World” concepts of the suggested default reasoning approach could be met by interactive control of automated information fusion systems. This approach would allow additional human knowledge to be included, a huge advantage in the context of rapidly evolving urban and harbour situations and would give responsibility to the operator. The average operational staff will probably be able to cope with and manage supporting systems using structured information (semantic nets) and default reasoning, as these two approaches perfectly correlate to human mental processing. Thus, it clearly appears that for the production of intelligence, especially for Collation, Analysis, and Integration, semantic information structuring and knowledge based methods can give substantial support to the human operator.

UNDERSTANDING MILITARY INFORMATION PROCESSING

137

References 1. 2.

3. 4.

5. 6. 7.

8.

9.

Anderson JR. 1996. Kognitive Psychologie: Eine Einführung. Spektrum, Akademischer Verlag. Biermann J et al. 2004. From Unstructured to Structured Information in Military Intelligence: Some Steps to Improve Information Fusion. SCI Panel Symposium, 2004 Oct. 25–27. London. Biermann J. 2004. Challenges in High Level Information Fusion. In: The 7th International Conference on Information Fusion; 2004 June. Stockholm, Sweden. Biermann J. 2003. A Knowledge-Based Approach to Information Fusion for the Support of Military Intelligence. NATO RTO Symposium on Military Data and Information Fusion IST-040/RSY-012; 2003 Oct. 22. Prague, Czech Republic. NATO Standardization Agency (NSA). 2003. AAP-6(2002) NATO Glossary of Terms and Definitions, http://www.nato.int/docu/stanag/aap006/aap6.htm NATO, AJP 2.0. 2002. Allied Joint Intelligence Counter Intelligence and Security Doctrine, NATO/PfP Unclassified, Ratification Draft 2. Joint Inquiry into Intelligence Community Activities Before and After the Terrorist Attacks of September 11, 2001. Report of the U.S. Senate Select Committee on Intelligence and U.S. House Permanent Select Committee on Intelligence. December, 2002, S. REPT. NO. 107–351 H. REPT. NO. 107–792, http://www.gpoaccess.gov/ serialset/creports/911.html RTGonIFD, Final Activity Report 2003–2005. RTO Task Group Report, NATO Research Task Group On Information Fusion Demonstration IST-038/RTG-016, to appear in 2006. TGonIF, Final Activity Report 2000–2002. RTO Task Group Report, NATO Task Group on Information Fusion, IST-015/TG-004, Dec. 2002.

APPLYING DECISION SUPPORT AND DATA/INFORMATION FUSION/MANAGEMENT CONCEPTS FOR EMERGENCY RESPONSE IN THE CONTEXT OF HARBOUR PROTECTION

ÉLOI BOSSE AND JEAN ROY Defence R&D Canada, Valcartier, Canada

Abstract. The events of September 11, 2001 moved the issues of anti- and counter-terrorism, national/public security and collective emergency response (both crisis and consequence management) to the forefront of many nations’ concerns. Harbour protection against terrorist attacks is an example of a kind of concern typical of the post 9/11 security paradigm. In the event of a large-scale terrorist emergency situation that would necessitate the ability to coordinate multiagency and multi-national operations, advanced decision support, knowledge exploitation, information fusion and management tools can significantly improve the ability to respond to such emergency. This paper presents a brief review of Canada’s national security policy and discusses the roles of National Defence. Key concepts and characteristics of emergency management are outlined with associated information management requirements for the collective response to an emergency. A collaborative workspace aiding military experts to work together effectively with other civil authorities at all levels and international allies is proposed. The collaborative environment supports the fusion of various perspectives to better interpret the situation and the problem, identify candidate actions, formulate evaluation criteria, decide on what to do, and synchronize a diverse set of plans and actions. Keywords: Decision support, emergency response, knowledge exploitation, collaborative environment

1 Introduction The events of September 11, 2001 moved the issues of anti- and counter-terrorism, national/public security, and the collective emergency response (both crisis and consequence management) to the forefront of many nations’ concerns. Harbour protection against terrorist attack is an example of a kind of complex situation typical of the post 9/11 new security paradigm. In the event of a large-scale terrorist emergency situation that would necessitate the ability to coordinate E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

139

140

É. BOSSE AND J. ROY

multi-agency and multi-national operations, advanced decision support, knowledge exploitation, information fusion and management tools can significantly improve the ability to respond to such emergency. The military standpoint alone is not sufficient to meet the diverse and unpredictable increase in terrorist threat; such a threat requires a consideration of collective security that expands to include cooperation with multiple non-military organizations (e.g. harbour protection). Working effectively in actual terrorist emergency situations requires the ability to communicate and to coordinate multinational and multi-agency operations in a seamless environment. There are vast quantities of data and information requiring weeding, sorting, and analysis. Clearly, advanced information management technology, for example, is required to enable emergency response communities to timely and securely access data, information, services, etc., relevant to their roles and responsibilities, regardless of what agency operates the facilities where the critical data and services reside. This paper presents a brief review of Canada’s national security policy and the roles of National Defence. Key concepts and characteristics of emergency management are outlined with the information management requirements for the collective response to an emergency. A collaborative workspace aiding military experts to work together effectively with other civil authorities at all levels and international allies is presented. The collaborative environment eases the fusion of various perspectives to better interpret the situation and the problem, identify candidate actions, formulate evaluation criteria, decide on what to do, and synchronize a diverse set of plans and actions.

2 Canada’s national security policy and DND responsibility Canada’s first-ever comprehensive statement of national security policy is presented in [11]. The document outlines the integrated security system that the Government of Canada (GoC) will build. Addressing many of the threats requires a more integrated, coordinated approach to national security, integrated inside the GoC and with key partners (i.e., provinces, territories, communities, the private sector and allies). The focus of the policy is on events and circumstances that generally require a national response as they are beyond the capacity of individuals, communities or provinces to address alone. The policy adopts an integrated approach to security issues across governments. It contains several measures to help build a more integrated security system in a way that is consistent with the policy’s goals. 2.1 SOME RESPONSIBILITIES OF THE DEPARTMENT OF NATIONAL DEFENCE (DND) The military viewpoint alone is not sufficient to meet the increase in terrorist threat that is diverse and unpredictable; such a threat requires a consideration of

DECISION SUPPORT & DATA/INFORMATION FUSION/MANAGEMENT

141

collective security that expands to include cooperation with multiple non-military organizations. The asymmetric threat study [10] examined existing and potential future responsibilities and roles, both international and domestic, of the DND in dealing with the asymmetric threat. The currently accepted DND responsibilities were determined by examining the various acts, orders-in-council and government policy directives that apply to DND and the responsibilities of other federal government departments and the provinces. The responsibilities outlined below are unlikely to change significantly in the foreseeable future [10]. These responsibilities are mandated by federal law and long-term government policies, are deeply engrained in our federal democratic system of government and its civilmilitary relationship, and have existed for some time. 2.2 INTERNATIONAL RESPONSIBILITIES DND may be required to provide military response to direct and indirect threats to Canadian interests, and will deal with threats to DND operations, including the provision of force protection measures during collective defence and peace support operations. The Department of Foreign Affairs and International Trade (DFAIT) will retain primary responsibility for the security of Canadian citizens and interests abroad and, when requested by DFAIT, DND will provide armed or unarmed assistance for the protection and evacuation of Canadians from areas threatened by imminent conflict. 2.3 DOMESTIC RESPONSIBILITIES Domestically, the provinces and other federal government departments and agencies are responsible for preventing, deterring, crisis managing and consequence managing an asymmetric attack on Canada. If called upon, DND will need to respond to requests for assistance from these organizations and will function in a supporting role. Regardless of which government department has the lead, the inherent flexibility of military units makes DND a potential source of assistance in all domestic emergencies and civil disturbances. However, the DND response is bound by legal considerations. Typically, DND is the Canadian government’s instrument of last resort. This being said, DND may be required to respond to a domestic attack designed to delay or hinder the deployment of Canadian forces overseas. DND will be required to provide military response to direct and indirect threats to Canadian sovereignty. It will deal with threats to DND assets, installations and operations and will provide communications security support to all government departments. The Solicitor General will retain primary responsibility for terrorist incidents in Canada and DND will provide armed or unarmed assistance when requested. Provincial and territorial governments are responsible for law enforcement and public safety within their jurisdiction, and DND will be required

142

É. BOSSE AND J. ROY

to respond to requests from provinces for Aid of the Civil Power to assist law enforcement agencies during riots or disturbances of the peace. Other federal government departments (OGD) and provinces retain responsibility for domestic economic threats and for broader health and criminal issues, and DND will provide armed or unarmed assistance when requested. In many cases this assistance will be discretionary. The provinces and territories assume primary responsibility for emergency preparedness and consequence management during a domestic emergency, and the Minister responsible for emergency preparedness will be required to coordinate federal assistance.

3 Emergency management This section briefly outlines the key concepts of emergency management with some emphasis given to the management of large-scale terrorist events. Reference [12] defines an emergency as any incident(s), human-caused or natural, that require(s) responsive action to protect life or property. It also defines an incident as an occurrence or event, natural or human-caused, requiring an emergency response to protect life or property. Incidents can, for example, include major disasters, emergencies, terrorist attacks, terrorist threats, wild land and urban fires, floods, hazardous materials spills, nuclear accidents, aircraft accidents, earthquakes, hurricanes, tornadoes, tropical storms, war-related disasters, public health and medical emergencies, and other occurrences requiring an emergency response. 3.1 CRISIS AND CONSEQUENCE MANAGEMENT Crisis management, also referred to as incident management, is defined as the government-led coordination of efforts to contain, alleviate or terminate an apprehended terrorist incident, to identify and make accountable the terrorist agents, and to gather information and preserve evidence to that end, primarily through the intervention and resources of law enforcement and related security agencies. Consequence management is defined as measures that mitigate the damage, loss, hardship, and suffering caused by acts of terrorism. It also includes measures to restore essential government services, protect public health and safety, and provide emergency relief to affected governments, businesses and populations. Coordinated crisis and consequence management is a modern approach recognizing that crisis and consequence management often need to be addressed concurrently rather than sequentially. The approach blends crisis and consequence management to create a more coordinated and effective response.

DECISION SUPPORT & DATA/INFORMATION FUSION/MANAGEMENT

143

3.2 EMERGENCY MANAGEMENT CYCLE Figure 1, adapted from [9], shows the emergency management (EM) cycle. Reduce Hazard / Risk / Vulnerability

Update Hazard / Risk Analysis

Mostly Back to “Normal”

Vulnerability Reduced

mMonitoringo

M Ro itig bu at st ion ne ss y er ov c Re

“Healing”, Repair, Remediation, and Reconstruction

R&D, Policies Plans, SOPs, Training and Exercises

/ n io at ion r a t ep ec Pr rot P Re sp o

Detection of Threat / Disaster / Emergency

ns e

mRisk Managemento Situation Stabilized No Longer Direct Threat to Life / Safety and Property and Critical Infrastructure

Figure 1. Emergency management cycle. (Adapted from [9].)

Aspects of this cycle that are relevant to this paper are described next, using a combination of the definitions provided in [6] and [7].

3.2.1 Response The response phase refers to measures and activities undertaken immediately after an emergency has occurred, and for a limited period thereafter, that address the short-term, direct effects of an incident. Response includes immediate action taken primarily to save human lives, treat the injured (e.g., contaminated and overexposed persons), protect property, meet basic human needs, and prevent and minimize further injury and other forms of loss, impacts and unfavourable outcomes. These actions may necessitate the activation and execution of emergency operations plans, the opening and staffing of emergency operations centers, mobilization of resources, issuance of warnings, advisories and directions, provision of aid, and may include the declaration of states of emergency. As indicated by the situation, response activities may also include applying intelligence and other information to lessen the effects or consequences of an incident, increased security operations, continuing investigations into the nature and source of the threat, ongoing public health and agricultural surveillance and testing processes, immunizations, isolation, or quarantine, and specific law enforcement operations aimed at pre-empting, interdicting, or disrupting illegal activity, and apprehending actual perpetrators and bringing them to justice. This phase may last from a few hours to several weeks after the commencement of the emergency and is followed by a recovery phase, as

144

É. BOSSE AND J. ROY

necessary. The time frame for recovery begins as soon as a reduction in critical response activities permits the re-allocation of some resources to longer-term recovery activities. Recovery measures can begin within the initial response phase and may last up to several years after the emergency. 3.3 COLLECTIVE RESPONSE TO LARGE-SCALE EMERGENCIES This section briefly outlines the collective typical response to large-scale emergencies in Canada. This can be generally applied to its main allies as well. The description focuses on large-scale Chemical-Biological-Radiological-Nuclear (CBRN) emergencies, as these represent a major response challenge involving many key players at all levels. A number of assumptions can be made about such emergencies: x No single federal, provincial-territorial (P/T) or local government agency has the capability or requisite authority to respond independently in handling all crisis and consequence management aspects of a CBRN terrorist incident. x Given the wide-ranging implications of a CBRN terrorist incident, municipal governments, P/Ts and the federal government will almost certainly all be involved in the management of the event. x Municipal and P/T response capabilities vary across the country and within different regions of each province/territory. x A terrorist attack may occur at any time, with little or no warning, may involve more than one geographic area and may result in mass casualties. x The time and place of a biological attack may be impossible to determine. The subsequent spread of contamination is likely to progress exponentially. The lack of temporal and spatial boundaries of a biological attack may dictate an imposed, multi-community quarantine based on appropriate legal authorities. x While the consequences of a chemical incident could likely be managed at the local or provincial level, a biological incident such as a smallpox outbreak would require a swift national and international response from the onset. x Communicating between GoC departments, with other levels of government, international partners, the media, private sector and public to prevent and respond to CBRN incidents is a significant challenge.

3.3.1 Primary responsibilities in crisis and consequence management The federal government assumes primary responsibility for the policy and operational response (crisis management) to criminal aspects of a terrorist incident, but recognizes that local and P/T authorities will likely be the first to respond. The federal government is also responsible for managing CBRN incidents that occur on federal property (e.g., federal building or aboriginal reserve) or related to federal jurisdiction (e.g., maritime approaches). The federal government will work closely with P/T jurisdictions in developing its response.

DECISION SUPPORT & DATA/INFORMATION FUSION/MANAGEMENT

145

Public Safety and Emergency Preparedness (PSEPC) is the designated federal lead responsible for coordinating Canada’s preparedness for, and in response to, terrorist incidents occurring within Canada. The operational response to domestic terrorist incidents is the primary responsibility of the Royal Canadian Mounted Police (RCMP). P/T governments, and their respective police services, are responsible for law enforcement and public safety within their jurisdiction. Consequence management differs from crisis management in that the province or territory in which a CBRN terrorist event occurs has primary, overall responsibility for managing its consequences, assisted, as necessary and when requested, and to the full extent possible, by the federal government. Provinces and territories are responsible for determining what type(s) of assistance they might require. The federal government assumes primary responsibility for coordinating the international aspects of a CBRN incident affecting Canada, but will work with provinces and territories to ensure that interests and issues are properly represented when dealing with the international community. Clearly, effective preparedness and response to a large-scale CBRN terrorist event requires rapid, accurate and channelled communications between all levels of government.

3.3.2 Multiple interacting jurisdictions

s Municipalities Mun icip litie ipa alit nic ies Mu Ontario Qu ba o t e i bec n Ma an

Br N un e sww ic k

RCMP

EC AAFC

fou nd lan

DFO

d

un ic

t vu na Nu

CF

s Municipalities Mun i ci p litie i pa alit Prince i es Ne Edward M w Island

NRCAN

PCO

RCMP CF

TC DFAIT

RCMP DND CFIA GOC

va Nootia Sc

HC

ni c Mu

CF PSEPC

s ie

ew

it al ip ic un M ip al it ie s

s Municipalities Mun icip litie ipa alit nic i es British Mu M s e ts Columbia Alb s w un e h e i e i rt a it ortritor ic l N ip er Sa pa T i al c sk i iti at es un R C c MP M h

Reference [12] defines a jurisdiction as a range or sphere of authority. Public agencies have jurisdiction on an incident related to their legal responsibilities and authority. Jurisdictional authority on an incident can be political or geographical (e.g., city, county, tribal, State, or Federal boundary lines) or functional (e.g., law enforcement, public health). As illustrated in Figure 2, the collective response to a large-scale terrorist emergency typically involves numerous jurisdictions.

T e Yu rr ko ito n ry

Figure 2. The collective response typically involves numerous jurisdictions.

146

É. BOSSE AND J. ROY

4 Emergency management domain characteristics A number of characteristics can be identified for emergency management (EM), especially as concerns the collective response to large-scale emergencies caused by terrorism: x x x x x x x x x x x x x x x x

Diversity in the nature of operations. Geographically dispersed operations. Abundance and diversity of data, information and knowledge types and sources. Abundance and diversity of services, applications, tools and products types and sources. Staff resources to perform EM functions have remained basically stable over the years. EM workers possess an increasing level of information technology literacy. The EM environment continually adjusts to expanding information technology capabilities. Operating in the EM environment, decision makers at all levels (e.g., incident commanders) and their staff need. To rapidly develop situational awareness (e.g., understand how a situation has developed and is expected to develop). To rapidly develop shared understandings of the operational environment. To plan operations. To monitor the situation and the execution of plans. To ensure that each individual worker is productive in and concentrated on assigned roles and tasks. To deal with complex crises. To deal with multiple, simultaneous crises (e.g., multiple operations monitoring). To perform routine office tasks.

Initial discussions with members of the emergency management operational communities reveal that there is a need to address some limitations very similar to those identified in [4], in a different context: x The required data, information and knowledge and services, applications, tools and products originate from various stovepipe systems. x Information is not provided or accessed in a timely fashion because of the high operational tempo. x There is a significant information overload (e.g., a huge quantity of messages at a high rate of message reception). x There are limitations in timely fusing information of different natures. x There are information management constraints due to different security domains. x There is a lack of tools for understanding how a situation has developed and is expected to develop. x There are limited visualization capabilities.

DECISION SUPPORT & DATA/INFORMATION FUSION/MANAGEMENT

147

x There are limited decision-aid/planning tools. x Decisions are being made on incomplete/unreliable information.

5 Situation awareness Situation awareness (SAW) has emerged as an important concept in dynamic human decision-making, especially in complex military and national security environments. Endsley [1] proposed a general definition of situation awareness that was found to be applicable across a wide variety of domains. She describes SAW as the perception of elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future. The perception of cues is fundamental. Without a basic perception of important information, the odds of forming an incorrect assessment of the situation increase dramatically [2]. Endsley also states that SAW, as a construct, goes beyond mere perception. It also encompasses how people combine, interpret, store, and retain information. Thus, it includes more than just perceiving or attending to information; it includes the integration of multiple pieces of information and a determination of their relevance to the person’s goals. Such comprehension of a situation demands that the problem of meaning be tackled head-on. A person with situation comprehension is able to derive operationally relevant meaning and significance from the data perceived. At the highest level of SAW, the ability to forecast future situation events and dynamics marks decision makers who have the highest level of understanding of the situation. This ability to project from current events and dynamics in order to anticipate future events (and their implications) allows for timely decision making. 5.1 DECISION SUPPORT SYSTEMS Operational trends in warfare and the collective response to large-scale terrorist emergency events put the command and control (C2) process under pressure. The technological evolution constantly increases the scope of the operational theatre and the tempo of the response. Moreover, a huge load of uncertain data and information is generated about the environment. Evidently, all these data and information may exceed human information processing capabilities. Yet, the military community typically maintains that the dominant requirement to counter the threat and ensure the survivability of people, or of a critical infrastructure, is the ability to perform C2 activities quicker and better than the adversary.

148

É. BOSSE AND J. ROY

Internal and External Communications

Situation Analysis and Awareness Support Capabilities

Peripheral Systems Interface

Decision Making Support Capabilities

A Priori Knowledge Management and Exploitation Server

Visualization and Human Computer Interactions

Core System Architecture Response Planning Support Capabilities

Response Control and Monitoring Support Capabilities

Figure 3. A generic decision support and knowledge exploitation system.

Information technology support is thus typically required to cope with human limitations in such complex environments. This emphasizes the need for real-time, computer-based Decision Support Systems (DSSs) to bridge the gap between the cognitive demands inherent to the accomplishment of the C2 process and human limitations. A DSS is a computerized system intended to interact with and complement a human decision maker [3]. Whatever the nature of the DSS, the objective is to develop DSS features that intuitively fit the perceptual and cognitive processes of the human user. The ideal DSS is one that provides the information needed by the human decision maker as opposed to raw data, can be controlled effortlessly by the human, complements the cognitive power of the human mind, and supports a wide variety of problem solving strategies. Effective DSSs are the ones that “make the problem transparent to the user.” Figure 3 illustrates a generic decision support and knowledge exploitation system. On the right-hand side are capabilities supporting C2 activities per se. On the left-hand side are necessary ancillary capabilities. All of these heterogeneous capabilities are glued together through the core system architecture. 5.2 DEFINITION OF DATA AND INFORMATION FUSION Information fusion is a key enabler to meeting the demanding requirements of decision support systems, in complex military and national security environments, mainly as regards situation awareness. It can be defined as the process of utilizing one or more information sources over time to assemble a representation of aspects of interest in an environment [5]. Clearly, data and information fusion has a critical role in achieving situational awareness in complex military and national security environments. Indeed, it is often asserted that mental information fusion is situation awareness. Fusion allows

DECISION SUPPORT & DATA/INFORMATION FUSION/MANAGEMENT

149

the decision maker to cope with the complexity and tempo of operations in the modern dynamic battlespace, and has an important role in asymmetric conflict. Fusion techniques allow the management of large volumes of information, and the correlation of seemingly unrelated, overlooked, or deceptive information to present a coherent picture of an evolving situation to a decision maker.

5.2.1 The JDL data fusion model The data fusion model maintained by the Joint Directors of Laboratories’ Data and Information Fusion Group (JDL DIFG) is the most widely used method for categorizing data fusion-related functions [8]. Figure 4 shows the revised model from [8]. The JDL distinction among fusion “levels” provides a valuable means of differentiating between data fusion processes that relate to the refinement of “objects,” “situations,” “threats” and “processes.” The definitions are as follows. Level 0  Sub-Object Data Assessment: Estimation and prediction of signal/object observable states on the basis of pixel/signal level data association and characterization. Level 0 assignment involves hypothesizing the presence of a signal (i.e. detection) and estimating its state. Level 1  Object Assessment: Level 1 data fusion combines data from single or multiple sensors and sources to provide the best estimate of objects and events in the battlespace in terms of their position, kinematics (e.g. tracks), identity, or identification features. Level 1 involves tracking and identification that includes reliable location, tracking, combat ID, and targeting information. Such Level 1 information may be of high quality particularly when it leverages multiple sensors to provide robustness to spoofing or deception, reliability in the face of sensor malfunctions, and extended space-time coverage due to the diversity of observations. Level 3

Impact Assessment Situations/Plans Plans

Level 4

Process Refinement (Resource Management)

Situations Situations

Plans

Level 2

Situation Assessment

Resources Situations

Objects

Level 1

Object Assessment Signal/Features

Objects

Level 0

Sub-Object Assessment Measurements

Signal/Features

Figure 4. A Revised data fusion model [8].

150

É. BOSSE AND J. ROY

Level 2  Situation Assessment: Level 2 data fusion focuses on situation assessment. This requires recognition of objects/entities in regions of interest, as well as the activities of these objects and their relationships. Issues that must be addressed include: x Automated target/object recognition x Automated activity recognition from multiple sensors and reports collected and stored for historical analyses x Inferring relationships of objects in the scene based on their identities or coordinated behaviours and historical analyses x The automated system’s capability to estimate certainties about object identities and activities and to request human assistance or additional information from sensors or databases in order to resolve ambiguities. Level 3  Impact Assessment: Level 3 fusion estimates the “impact” of an assessed situation, i.e., inferring intent of objects/entities, or groups of objects, in regions of interest and/or the outcome of various plans as they interact with one another and with the environment. The impact estimate can include likelihood and cost/utility measures associated with potential outcomes of a player’s planned actions. Issues that need to be addressed include: x Methods for constructing and learning a wide variety of models of threat behaviour x Methods for reasoning with uncertain and incomplete information for assessing threats from object activities x Methods for efficient data-mining of databases Level 4  Process Refinement: Adaptive data acquisition and processing to support mission objectives. Level 4 processing involves planning and control, not estimation. Level 4 assignment involves assigning resources to tasks. 5.3 DATA, INFORMATION, KNOWLEDGE, EXPERTISE, SERVICES, APPLICATIONS, TOOLS, PRODUCTS, ETC. Multitudes of vital data, information, knowledge, expertise, services, applications, tools and/or products for disaster management exist and are maintained up-to-date by the different individual organizations that are responsible for them. As information production and maintenance are costly and require particular expertise, it is natural to designate several organizations responsible for creating and maintaining critical information and providing mechanisms to efficiently distribute the information. The tool sets include many kinds of data, information, knowledge, expertise, services, applications, tools and/or products that help in many ways during emergency response situations. Examples of data, information, knowledge and expertise include:

DECISION SUPPORT & DATA/INFORMATION FUSION/MANAGEMENT

x x x x x x x x x x x x

151

Documents E-mails, queries Working-track history Lessons learned Personal comments Maps Geospatial imagery Socio-political/cultural context (historical evolution of boundaries and political contexts, religious and cultural factions, organizational links, personality profiles) Resources, infrastructures, terrain 3D urban and building models Weather (e.g., past and present meteorological observations, and weather/ atmospheric information available for a particular area that could be relevant for a crisis response) Intelligence

These data, information, knowledge and expertise are provided from a wide variety of sources (including open sources such as internet, commercial, TV channels, human). There are dynamic sources, providing real-time data and information, and sources of a priori data, information, and knowledge. Examples of services, applications, tools and products include: x x x x x x x x x x x x x x x

Electronic messaging Chatting Conferencing Message processing Document management Federated/contextual search and retrieval, knowledge discovery User assistance/assistant wizard Briefing production Lessons learned Alerting/triggering Situation analysis/assessment Information fusion Decision-aid, development of course of actions Numerical weather prediction Optimal path computation (e.g., vehicle path optimization for VIP evacuation purposes as infrastructures are being disrupted), crowd phenomena simulation, etc.

Of particular importance are leading edge numerical models, algorithms and technology that are able to simulate (in a quasi-real time mode) the past/present/future 3D dispersion plumes of toxic matter. Precisely knowing the contaminated region helps save time and lives. With the application of numerical

152

É. BOSSE AND J. ROY

models to meteorological data, it is also possible to accurately trace the source of a contaminant by doing a backward simulation in time. 5.4 INFORMATION MANAGEMENT REQUIREMENTS In view of the above discussions, working effectively in terrorist emergency situations requires the ability to communicate and coordinate multi-national and multi-agency operations in a seamless environment. There are vast quantities of data and information requiring weeding, sorting and analysis. Clearly, advanced information management technology is required to enable the emergency response communities to timely and securely access data, information, services, etc. relevant to their roles and responsibilities, within and across jurisdiction boundaries, regardless of what agency operates the facilities where these critical assets reside. Unfortunately, the current situation limits the integration of assets from different sources, thereby limiting their impact on efficient emergency interventions. Broad integration has yet to be achieved in Canada, or within other nations, and would represent a breakthrough in information management. Interoperability between systems and data is also a major problem in many information technology (IT) projects. It would be beneficial to better exploit the existing legacy IT systems through synergistic integration without the need for extensive re-engineering or major alteration to the business process of each individual participant. This is a key problem currently faced by governments and industries. In this regard, one objective of the CRIS project is to implement the necessary message and data exchange mechanisms for the ever-increasing amount of digital information that needs to be exchanged within and across jurisdiction boundaries. Clearly, this must be achieved with low overhead, and with member agencies retaining autonomy of business practice and technology.

6 High-level requirements of a decision support system The aim of this support system is to enable the members of the operational communities at all levels (from first responders through upper-level management) to work together effectively in large-scale terrorist emergency situations, when engaged in crisis response and consequence management. The numerous objectives of the project can be summarized as follows: 1. Enhance collective decision-making capabilities: (a)

Support users in achieving information superiority: (i) Improve situation analysis support to enhance the situation awareness of responders. (ii) Provide an emergency situation scenario tailored to the user’s needs. (iii) Provide users with the specific information required to perform their functional responsibilities during crisis or conflict.

DECISION SUPPORT & DATA/INFORMATION FUSION/MANAGEMENT

(iv) (v)

153

Support the development, maintenance and sharing of the collective “battlespace knowledge.” Exchange information within and across jurisdiction boundaries.

(b) Improve decision support to enhance the critical decision-making of the responders regarding scarce response resources. (c) Support rapid incident assessment and management (immediate reaction and near-term consequence management). 2. Improve the effectiveness and efficiency of inter-agency co-operation, coordination, interoperability and decision-making: (a) Coordinate multi-national and multi-agency operations in a seamless environment. (b) Provide a solid collaborative solution to the many participating organizations and individuals across multiple jurisdictions. (c) Link jurisdictions in a trusted federation (with low overhead, and with member agencies retaining autonomy of business practice and technology). (d) Facilitate secure interaction between jurisdictions while leaving access control decisions under each jurisdictional authority. (e) Better exploit critical data, information, knowledge, services, applications, tools and products offered by the legacy systems of the operational community. (f) Improve the integration of existing operational information systems of the emergency response community (the CRIS will not replace these systems). (g) At multiple scales regarding time, space, computer capacity, visualization and human-computer interface capacity, bandwidth, security levels, training levels, financial resources, etc. (h) Support scenario rehearsal, exercises, training, readiness assessment, postevent reconstruction (i.e., other segments of the emergency management cycle) to enhance collective readiness. 3. Support the different needs of various actors. 4. Improve the efficiency of emergency response workers: (a) As individual workers, as communities and as groups. (b) Independently of their computer literacy level. (c) Without revolutionizing organization values, business rules and work layouts. 5. Reduce response time. 6. Enable more efficient and effective collaboration among cluster members of the CBRN Research and Technology Initiative (CRTI), and enhanced cluster participation in the response to CBRN events (laboratory cluster management and operations):

154

É. BOSSE AND J. ROY

(a) Enable the capture and sharing of knowledge and expertise with first responders and operational communities. (b) Enable the exchange of secure data and information before and during an operation to support cluster operations. The Decision Support System (DSS) is an example of an innovative information management and integration system (providing the information management technological glue) developed to meet requirements such as those listed below: 7. Allow a timely and secure integrated/federated access: (a) To multiple types of data, information, knowledge and expertise. (b) To multiple types of services, applications, tools and products. (c) From a wide variety of heterogeneous, distributed sources and disparate providers. (d) To data, services, etc., maintained by their owners. (e) From any media. (f) From multiple formats. (g) From varying levels of abstraction. (h) Within and across jurisdiction boundaries, regardless of what agency operates the facilities where the critical data, information, knowledge, expertise, services, applications, tools and/or products reside. 8. Allow arbitrary navigation, from a single workstation, on the sources of data, information, knowledge, expertise, services, tools and products. 9. Find and provide users with timely, right, trustable data, information, knowledge and expertise: (a) Allow semantic connections on sources of data, information, knowledge, expertise, services, applications, tools and products. (b) Allow searching and extracting information that is truly relevant to the work and to what is needed at the time the search is performed. (c) Integrate/aggregate/fuse data, information and knowledge: (i) (ii)

Allow various forms of aggregation/fusion depending on user preference and/or the true value of results. Create new products on the fly by combining different datasets.

(d) Provide data, information, knowledge and expertise relevant to various user roles and responsibilities in tailorable views, with adaptable and flexible tools. (e) In context of the work: (i)

Take into account individual user interests and group constraints within a dynamic and evolving task context (in a changing environment).

DECISION SUPPORT & DATA/INFORMATION FUSION/MANAGEMENT

155

10. Manage data, information, knowledge, expertise, services, applications, tools and products. 11. Distribute/exchange/share data, information, knowledge, expertise, services, applications, tools and products: (a) (b) (c) (d) (e)

Rapid dissemination toward the intervention site. In the appropriate form and level of detail. To users at all echelons. Allow easy contribution to the collective “battle space knowledge.” Allow easy contribution to the collective response.

12. Provide services for synchronous and asynchronous collaboration among a variety of responders: (a) Interact with other people. (b) Expertise grouping. 13. Make relevant task support tools readily/easily available to users, according to the task being performed, and according to the preferred ways of working, skills and community of practice. 14. Provide personalization capabilities in terms of data, information and knowledge content and expertise, services, applications, tools and products: (a) Provide users with designated working spaces, called portfolios. (b) Display, manage or produce documents related to user assignment and responsibility. (c) Tap into task-specific tools (templates, wizards, lessons learned, etc.). (d) Access specific specialized tools such as systems or applications (e.g., a planning application). 15. Allow smart navigation from any elements of the portfolio universe. For instance, to jump from a particular paragraph in a working document into the appropriate section of an external system source corresponding to the meaning of that paragraph. 16. Allow the activation or deactivation of the contextual assistant in certain fuzzy scoped areas. 17. Allow a continuous formulation of the contextual meaning of the work as actions are performed. 18. Keep track of all documents and actions taken when assuming the duty. 19. Tackle many critical issues regarding system and data security, privacy and confidentiality, and authentication: (a) Provide fine grain user management and access control. (b) Allow a security strategy that matches the nodal network and information pull approach.

156

É. BOSSE AND J. ROY

6.1 A PROPOSED SYSTEM ARCHITECTURE Figure 5 from [7] presents the Horseshoe Architecture Concept (HAC). The HAC shows, at a conceptual level, the interaction between the end-users, represented by “roles,” and the DSS (decision support system). The proposed architecture is based on a web portal solution that can be simply accessed by an Internet client by means of standard communication protocols such as http or https. An end-user, such as a first responder or an incident decision maker, will be able to operate using a simple web browser engine from its client station. The system will provide a wide variety of end-users with a transparent and highly powerful operational service point for a variety of services ranging from portal core services, such as a “Data Organizer” and a “Data Viewer,” to more specialized CBRN emergency data and processing services. The system will be characterized by its facility to exploit the CBRN data, information and knowledge by the emergency staff before, during and after a CBRN incident. The HAC highlights the fact that four main thrusts will drive the development and demonstration capability of the system: the “operation,” the “exploitation,” the “technology” and the “security” thrusts. A proper balance between these four thrusts during the evolution of the project will ensure that the “horseshoe concept” will be preserved. As can be expected, the operation thrust has to do with the various roles of the operational community. As for the technology thrust, it can simply be deduced that a large number of technology solutions/products will be used in order to render the system portal and available services. The exploitation thrust is related to the resources offered to end-users to exploit the capabilities of the system. For instance, having a knowledge database with a large relevant CBRN emergency content but poor operating mechanisms to access this valuable information is an example of weak exploitation thrust. Finally, in the context of the proposed system, in which a large number of organizations of different types and from various jurisdictions participate, the security issue shall also be considered as a major driving thrust. So why the horseshoe? Having such a system deployed on the Internet means that an almost infinite number of resources can collaborate to the capability of the system. Thus, with this sole consideration in mind, an image like an infinite straight line would have been more appropriate than the horseshoe. However, some access control mechanisms must be applied to protect the valuable resources of the main collaborators involved in this project without limiting access to the large potential of the Internet. Having this extra consideration in mind, the reader can better appreciate the horseshoe architecture.

DECISION SUPPORT & DATA/INFORMATION FUSION/MANAGEMENT

SECURITY

CBRN EMERGENCY DATA SOURCES

CBRN EMERGENCY PROCESSING SERVICES

OPERATION

WEB INTERFACES

ROLES NON WEB DATA

WEB DATA

Prop. Open Sources Sources

Prop. Open Sources Sources

CBRN Data Provider

157

WEB INTERFACES

CBRN Sit. Analyst

PROC. APPS. WEB PROC. SERVICES

Prop. Sources

Open Sources

First Incident Responder Commander Internet http/https/ftp

TECHNOLOGY

EXPLOITATION DSS PORTAL

SERVICE REGISTRY

USER COLLABORATION Event Free Form Formal Logger Exchange Exchange

Decision Support

PORTAL CORE Data Portal Data Organizer Viewer Management

Conference

Portal DB

DATA / KNOWLEDGE EXPLOITATION Search Capability

Portfolio Portlets

Data Classification

Exploitation DB

Figure 5. A Horseshoe Architecture Concept (HAC).

6.2 ALL INFORMATION? THE RIGHT INFORMATION? Situation Awareness (SAW) quality can be related to the amount of information available to an individual. Clearly, circumstances where no information is available should result in poor SAW, leading with high probability to a very low decision quality. In such a case, a natural reaction would be to provide mechanisms to increase the amount of information available to decision makers in order to improve SAW quality. One could even claim that a good approach to reach optimal SAW and decision-making would be to provide as much information as possible. However, this does not necessarily represent the best solution [5], as more information does not automatically mean better SAW in ensuring improved human performance. First, all of this information may exceed human information processing capabilities, causing cognitive overload. Second, not all of the data and information available in the environment is relevant and useful for reaching an optimal decision. In fact, in some situations, most data can be considered as distracters and noise for the decision maker and may thus reduce his/her level of SAW. The decision maker must detect and use only a specific fraction of this information to enhance his/her SAW and decision-making processes. Such considerations lead to the concept of “the right information, at the right place, at the right time,” as opposed to “all information, everywhere, at all time.”

158

É. BOSSE AND J. ROY

Clearly, however, research and technological advancements toward providing “all information, everywhere, at all time” is necessary, as such progresses ensure that “the right information, at the right place, at the right time” will actually be available to the decision-makers. Figure 6 presents a different perspective of a generic DSS that takes into account these issues. In Figure 6, support capabilities are represented as a set of independent system tools and services supporting situation analysis, decision making, knowledge exploitation, etc. Part of the necessary interactions between these tools/services is enabled by the system integration and interoperability layer. However, the system also requires appropriate mechanisms based on technological enablers such as information fusion and knowledge management to provide "the right information, to the right person, at the right time. People

A Variety of Heterogeneous Sources of Information System Tools / Services for Situation Analysis Decision Making Knowledge Exploitation HCI

..

..

The Right Information To The Right Person At The Right Time

Info Source #1

Info Source #2

All Information Everywhere At All Time

..

..

..

..

Info. Fusion, Knowledge Management, Advanced Visual., Contextual Portfolios Task Oriented Services Info. Centric Workspace

Info Source #N

Tool / Service #1

Tool / Service #2

Tool / Service #N

System Integration & Interoperability, Middleware, Net-Centric Enterprise Services

Figure 6. Exploiting information sources and tools/services.

7 Conclusion This paper discussed decision support and information fusion/management concepts and requirements applied to complex military and national security environments. A brief review of Canada’s national security policy was presented. The key concepts of emergency management, with some emphasis given to the collective management of large-scale terrorist events, were outlined, along with a coarse analysis of the characteristics of the emergency management domain, and of the high-level information management requirements associated with the collective response to emergencies. Highlights of a vision developed to structure a high-level knowledge environment framework capable of laying the foundations of a situational awareness knowledge portal were presented. The proposed decision support system could link to a broad range of terrorist events (including

DECISION SUPPORT & DATA/INFORMATION FUSION/MANAGEMENT

159

harbour protection) and will enhance collective readiness and the safety of people and critical infrastructures.

References 1.

Endsley MR. 1995. Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors Journal, 37(1), pages 32–64. 2. Endsley MR, Garland DJ. 2000. Situation Awareness Analysis and Measurement, Lawrence Erlbaum, Mahawah, NJ. 3. Elm WC, Potter SS, Gualtieri JW, Roth EM, Easter JR. 2002. Applied Cognitive Work Analysis: A Pragmatic Methodology for Designing Revolutionary Cognitive Affordances. 4. Gouin D, Gauvin M, Woodliffe E. 2003. COP 21 TD – Towards a Situational Awareness Knowledge Portal. Proceedings of SPIE 2003 - Aerosense/Defence Sensing, Simulation and Controls, Orlando, 21–25 April 2003 Vol. 5101 - Battlespace Digitization and Network-Centric Systems III. 5. Lambert DA. 2003. An Exegesis of Data Fusion, in: L. Reznik and V. Kreinovich (Eds.) (2003), Soft Computing in Measurement and Information Acquisition, Studies in Fuzziness and Soft Computing No. 127, Chapter 6, Springer Verlag. 6. Roy J, Dessureault D, Létourneau F. 2004. Crisis Response Interoperability System: Enabling Multi-National and Multi-Agency Defence Against Terrorism, Proceedings of the NATO Systems Concepts & Integration (SCI) Panel Symposium (NATO SCI158 Symposium) on “Systems, Concepts and Integration (SCI) Methods and Technologies for Defence Against Terrorism,” MoD, London, 25 – 27 October 2004, 27 pages. 7. Roy J. 2004. Multi-Environment Decision Support and Knowledge Exploitation in Terrorist Emergency Responses. 9th International Command and Control Research and Technology Symposium (ICCRTS) – Coalition Transformation: An Evolution of People, Processes and Technology to Enhance Interoperability, Copenhagen, Denmark, September 14–16, 2004, 22 pages. 8. Steinberg AN, Bowman CL, White FE. 1998. Revision to the JDL data fusion model. Joint NATO/IRIS Conference. Quebec City. 9. Tucker C. 2002. Canada – Presentation at the Canada – United States CIP R&D Roundtable. Ottawa, Canada. 10. Ajilon Canada. 2001. The Asymmetric Threat, Report Number 2001-1, Prepared for DND/DCDS. 11. PCO. 2004. Securing an Open Society: Canada’s National Security Policy. <www.pco-bcp.gc.ca>. 12. U.S. Department of Homeland Security. 2004. National Incident Management System.

ANALYSES OF THE CONCEPT OF TRUST IN INFORMATION FUSION AND SITUATION ASSESSMENT

HASMIK ATOYAN1 AND ELISA SHAHBAZIAN2 1

OODA Technologies Inc. Canada

2

MSc Student, École Polytechnique of the University of Montreal, Canada

Abstract. The ultimate goal of any new technology is to be helpful for the end user. However, the level of usefulness of this technology for the end user can be impacted by the level of trust a human operator has in the system. Maintaining trust is critical, since it is potentially harder to gain, easier to loose and even more difficult to recover when lost. Hence, it is essential to foster human trust in the system. At the same time this trust should tuned to the appropriate level, because neither mistrust nor complacency is desirable. The paper discusses various factors affecting human trust in automation identified in the current literature and presents proposed guidelines for tuning appropriate trust in new automated systems. Keywords: Appropriate trust, automated systems, human computer interface

1 Introduction The ultimate objective of any new technology is to provide an effective support to the human operator. One of the factors that impact the usefulness of new technology is the level of trust the decision-maker has in the system [1]. If the system is not trusted, it is unlikely to be used; if it is not used, the operator may have limited information regarding the capabilities of the system [26]. Trust has several meanings and can be defined simply as the confidence or degree of belief in the strength, ability, truth or reliability of a person or information. In the context of complex human-machine systems, Madson and Gregor [20] have defined trust as follows: “Trust is the extent to which a user is confident in, and willing to act on the basis of the recommendations, actions, and the decisions of computer-based tool or decision aid.” Trust is becoming an important issue, since it is potentially harder to gain, easier to loose and even more difficult to recover when lost. Hence, the human computer interface should support the operator to develop trust in the system. But E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

161

162

H. ATOYAN AND E. SHAHBAZIAN

this trust tuning should be at an appropriate level, i.e. there should be a correlation between the user’s trust in the system and the system’s capabilities. Supporting appropriate trust is critical in order to avoid the misuse (i.e. overly rely) or the disuse (i.e. underutilize) of the system [5].

2 General factors influencing trust More and more automated systems are being incorporated into new technologies; significant attention should therefore be paid to the operator’s trust in automation. Lee and Moray [15] defined the general factors influencing trust in automation: performance, process and purpose. Performance refers to the current and historical operation of automation, its ability to achieve the operator’s goal; it describes ‘what the automation does.’ Process is the degree to which the automation algorithms are appropriate for the situation to achieve the operator’s goal. It describes ‘how the automation operates.’ Purpose is the degree to which the automation is being used within the realm of the designer’s intent. It describes ‘why the automation was developed.’ The operator will tend to trust the system if it performs in a manner that reliably achieves the operator’s goals, if its algorithms can be understood and seem capable to achieve the operator’s goals in a current situation and if it is designed to achieve the goals. In early relationships with automation, trust can depend on purpose and not performance, since there may be little history of performance but clear statement regarding the purpose of automation [11]. As experience increases, operators may develop a feeling for the automation’s performance and process.

3 Cognitive processes underlying trust tuning Designing an interface and training to provide operators with information regarding the purpose and performance of automation could enhance the appropriateness of trust. However, the availability of this information is not sufficient to ensure appropriate trust. The information should be presented in a manner, which is consistent with cognitive processes underlying the development of trust [17]. Humans develop and tune their trust in three qualitatively different ways: based on analytic processes, analogical processes, and affective, emotional processes. Each of these processes demands a different degree of cognitive processing. During the analytic trust tuning process, the information is processed and plans are formulated and evaluated by utilizing the user’s knowledge, experience, and mental model of the system. Here, decision-making is made on the basis of what is

ANALYSES INFORMATION FUSION AND SITUATION ASSESSMENT

163

known about the motivations, interests, capabilities and behaviour of the other party. So far, there is no explicit explanation of how trust develops during analytical processes. The analytic process of trust tuning demands the highest cognitive sources of the human operator. This process is similar to knowledge-based performance described by [33]. In the analogical process the trust is developed according to analogical judgments. This process is based on rules and procedures, and it demands fewer cognitive resources than the analytic process. When rules are consistent with trustworthy behaviour, they can increase people’s expectation of satisfactory performance by pairing situations with rulebased expectations. This is true during times of normal operation, but rules can have negative effects when situations diverge from normal operations. If trust is primarily based on rules that characterize performance during normal situations, it can be very fragile and could collapse during abnormal situations. Analogical trust can also be based on factors such as reputation and gossip [22]. In this case, trust will be developed without any direct contact with the system [4]. The affective process is based on affective aspects of trust representing the core influence of trust on behaviour [14]. Emotional aspects are critical, because people not only think about trust, but they also feel it [10]. Nickerson and Reily [27] propose that the more positive the effect a machine can generate in a human, the more it will be trusted. Emotions can guide behaviour when rules fail to apply and when cognitive resources are not available to support a calculated rational choice. The affective process demands less cognitive sources than the analytic and analogical trust tuning processes. There is a temporal element in trust building since it takes time to develop trust, which can be gained via self-experience, training, hearsay or the experience of others. Miller [22] argues that the affective and analogical processes will be more important in human interaction with the system than the analytic process especially while interacting with a novel system, While experiencing a novel system for the first time, with no background knowledge about the agent’s motivations and behaviour, the only information the person may have about whether or not to trust the system, is the affective information. If the affective information is negative enough to prompt a strong “do not trust” response, then no further information will be gathered. But when there is a moderate level of affective trust tuning, the user will continue to use the system and will develop the trust via the analogical process. Hence, if the system does not provide appropriate cues to achieve at least moderate levels of affective and analogical trust, there may never be a chance to build analytic trust [22]. The experimental results [22, 28] also revealed that analogical and affective methods will play a greater role in trust tuning than the analytic method with its added memory, knowledge, and experience.

164

H. ATOYAN AND E. SHAHBAZIAN

4 Trust and reliance, context, and information display In developing appropriate trust, three main elements should be taken into account: the dynamics of trust and reliance, the importance of context, and the role of the information display. Trust is strongly affected by system reliability [21, 23]. Unreliability lowers the operator’s trust and can therefore undermine the potential system performance benefits of the automation [32]. Relying on automation provides the operator with the opportunity to observe how the automation works and, thus, to develop greater trust. Parasuraman et al. [31] found that reliance strongly depends on the type of automation. Automation can differ in type and complexity, from simply organizing the information sources, to integrating them in some summary fashion, to suggesting decision options, or even carrying out the necessary action. In their taxonomy, Parasuraman et al. [31] provide 4 levels (stages) of automation and these levels can vary depending on situational demands during operational use: x Information acquisition, i.e. collection and filtering of information x Information analyses, i.e. integration of information in a form of inference or diagnoses x Decision and action selection, i.e. selection among decision alternatives x Action implementation, i.e. execution of chosen action It is possible for the operator to observe the behaviour of information acquisition, even if they are not relying on it [42]. At this stage, the raw data is still available, so the operators can balance their attention between raw data and automation [40]. In contrast, it is not possible for operators to observe action implementation unless they are relying on it [16]. Trust and reliability are strongly affected by faults in the automation [8, 18, 37]. According to Dzindolet et al. [8], Madhavan and Wiegman [19]. There are subtle differences in how humans perceive and react to automated systems compared to their reaction to human team mates. Humans expect an automated system’s performance to be more or less perfect, whereas they have expectations of “imperfections” in their fellow team mates. These expectations trigger a rapid decline in trust when an automated system makes errors [1, 8, 19]. Moreover, there will always be a set of conditions (e.g. the uncertain nature of information sources, sensor imprecision, conditions in which the algorithms used by automation are inappropriate, system malfunctions) under which the automation may reveal incorrect results. Hence, high reliability cannot be always guaranteed. However, people rely on faulty automation when they are aware of automation imperfections [35]. If the user understands why and under what conditions an automation tool can make an error, he or she can develop a control strategy [7, 9, 36]. However, a discrepancy between the operator’s expectations and the automation’s behaviour can undermine trust even when the automation performs well [34].

ANALYSES INFORMATION FUSION AND SITUATION ASSESSMENT

165

The analysis of 20 different studies revealed that “diagnostic automation with reliability greater than 0.70 is a “crossover point” below which unreliable automation is worse than no automation at all” [39]. Trust also depends on context. In order to reach an appropriate level of trust, one should understand how context affects the capability of the automation. Besides, it’s important to see the distinction between the two psychological states of “trust” and “reliance,” and then to consider the interaction of context with trust and reliance. One can rely upon automation even when it is not trusted: the user may know that the automation can fail, but use it nevertheless if he/she is overloaded or there is no possibility to carry out the task without the automation [41]. Reliance on automation depends on other system and human performance constraints, such as multitask demands, time constraints, varying levels of risk for consequences, familiar or unfamiliar situations, self-confidence levels, stress, individual differences, stereotypes, training, etc. A particularly important variable that interacts with trust to influence reliance is self-confidence. When the operator self-confidence is high and trust in the system is low, they are more inclined to rely on manual control. The opposite is also true: low self-confidence is related to a greater inclination to rely on the automatic controller [16]. Thus, biases in self-confidence can have a substantial effect on the appropriate reliance on automation. Unfamiliar situations also affect trust. In unfamiliar situations, people place more trust in automation [12]. Another factor that affects the level of trust is the risk-level and the magnitude of the consequences of an automation error [21]. The user weighs each case differently, depending on the risk involved [25]. An automated system with a high level of risk calls for a large amount of trust. Riley [35] found that it takes longer to recover after automation failure in high-risk situations than in low-risk situations. The multitasking demands or high workload of a situation can also interact with trust to influence reliance. A situation when the operator has a highly reliable automation combined with a responsibility to perform multiple tasks in addition to monitoring the automation, can lead to excessive trust in automation and undermining the detection of automation failures [29]. Experiments carried out by Mosier et al. [24] also showed that in a multitask environment, a highly reliable system can reinforce the perception that the task can be completely carried out by the automation and that there is no need to pay attention to other cues. They call this phenomenon automation bias. Ultimately, over-reliance or excessive trust, or in ergonomics terminology, “complacency,” leads to less vigilant monitoring. Reducing the over trust in automation and increasing the detection of failures can be achieve through adaptable automation, which shifts between manual and automatic control according to the capabilities of the person and the situation [30]. Since in adaptable automation the operator can define when to use automation and instructs the system as to what behaviours to exhibit, it can produce better trust tuning. This approach can also be helpful to avoid skill degradation and to

166

H. ATOYAN AND E. SHAHBAZIAN

increase the situation awareness of the operator in case the system fails and human intervention is required. Stress and trust are interrelated as well. The more stress there is, the worse decision making becomes, the more individual differences and cognitive processes such as stereotyping will surface, which may lead to an inappropriate level of trust [27]. The environmental context also influences trust and reliance. Automation may perform well in certain circumstances and not in others. For this reason, appropriateness of trust often depends on how sensitive people are to the level of influence of the environment on the performance of automation. In this regard, Bisantz and Seong [2] found that trust can be affected differently depending on the source of automation failure. Their findings show that trust is less degraded if the source of failure is an abnormality outside of the automation system itself (different environmental situations, such as a lost power supply, intentional sabotage of hardware in the military domain, etc.) than if it is a failure within the automation (e.g. software bugs). Individual, organizational and cultural context [17] should also be acknowledged. Individual context includes the individual propensity to trust. Individuals who are “trustworthy” tend to have more trust in others and are more willing to delegate tasks to automated systems. The organizational context reflects the interaction between people. This includes reputation and gossip, which can affect the individual’s willingness to trust. The cultural context influences trust through social norms and expectations. Thus, we can summarize by stating that precisely resolving differences in context can lead to the development of more appropriate trust [6, 40]. The third critical factor in developing appropriate trust is the effect of content and the format of the display. Perception of automation-related information is usually mediated by the display. Organizing information on the display in a way that supports analytic-, analogical-, and affect-based assimilation of this information may be an important means of guiding appropriate expectations regarding the automation. If the information is not available in the display or if it is formatted improperly, trust may not develop appropriately. Results described in [17] show that trust tends to increase when the information is displayed in a way that provides concrete details that are consistent and clearly organized. Bisantz et al. [1] conclude that a decision maker will trust the decision aid systems more if the information is provided in a format that conveys important aspects of that information (e.g., its uncertain nature). In many cases, trust and credibility depend on surface features of the interface that have no obvious link to the true capabilities of the system [3, 38]. For example, Kim and Moon [13] found that certain interface features such as cool colours and a balanced layout can enhance the user’s perception of the interface’s trustworthiness.

ANALYSES INFORMATION FUSION AND SITUATION ASSESSMENT

167

5 Implications While designing new automated systems, in order to support the development of an appropriate amount of trust in the system, the following guideline should be considered: x The system should be designed for appropriate trust, neither too much, nor too little. x Involve the real user in all phases of system development. x The operators should be adequately trained for appropriate trust, and not for “blind trust.” Note that training should not be used to compensate for poor design. x Show the rationale behind automation and design so as to relate to the user’s goals. x Make the algorithms of automation simpler or more understandable. Automation will be trusted to the extent that operators understand the algorithms underlying automation use. x The user should be aware of whatever problems may arise. Provide means to indicate to the user that data are missing, incomplete, unreliable or invalid. Distrust is more resistant to change than trust. x During initial introductory periods with new systems, prepare both the system (hardware and software) and the user for the eventuality of some form of system failure. x Provide the user with the source of the automation failure. x The system’s potential to “miss” should be as low as possible. x Data (particularly raw data) needed by the user should be easily accessible. x The more positive the effect a machine can generate in a human, the more it will be trusted. x Context should be revealed in such a way as to support the assessment of situations relative to the capabilities of automation. x Provide the user with an adaptable automated system, i.e. a system where the flexibility of the information and automation behaviour are controlled by the user. x Present the interface in a simple, well organized manner, with a high degree of consistency and compatibility (with regard to the task and to the user’s expectations) with effective feedback. x A reliability of 0.70 is a “crossover point,” below which, unreliable automation is worst than no automation at all.

6 Conclusion In order to support the utilization of new automated systems by human operators, it is essential to develop trust in that system. The trust tuning process should be

168

H. ATOYAN AND E. SHAHBAZIAN

maintained at an appropriate level, neither too much, nor too little. To support this process, the organization of human computer interface should be consistent with cognitive processes underlying the development of trust, and the operator should be adequately trained. Based on the literature review, the paper has proposed a guideline for the development of an appropriate level of trust in new automated systems.

References 1.

2.

3.

4.

5.

6.

7.

8. 9.

10. 11. 12. 13.

14.

Bisantz AM, Finger R, Seong Y, Llinas J. 1999. Human performance and data fusion based decision aids. Proc. of the 2nd International Conference on Information Fusion – Fusion ’99. Vol. 2, Sunnyvale, CA, 918–925. Bisantz AM, Seong Y. 2001. Assessment of operator trust in and utilization of automated decision-aids under different framing conditions. International Journal of Industrial Ergonomics, 28(2), 85–97. Briggs P, Burford B, Dracup C. 1998. Modeling self-confidence in users of a computer-based system showing unrepresentative design. International Journal of Human-Computer Studies, 49, 717–742. Burt RS, Knez M. 1996. Trust and third party gossip. In Editors: Kramer RM and Tyler TR. Trust in organizations: frontiers of theory and research. Thousand Oaks, CA: Sage. pp. 68–89. Cohen MS. Parasuraman R, Freeman JT. 1999. Trust in decision aids: a model and its training implications. (Technical Report USAATCOM TR 97-D-4). Cognitive Technologies. Arlington, VA. Cohen, MS. 2000. Training for trust in decision aids, with applications to the rotorcraft’s pilot’s associate. Presented at the International Conference on Human Performance, Situation Awareness, and Automation, Savannah, GA. Dzindolet M, Pierce LG, Beck HP, Dawe L. 1999. Misuse and disuse of automated aids. Proc. of the Human Factors an Ergonomics Society 43rd Annual Meeting, Santa Monica, CA, 339–343. Dzindolet MT, Pierce LG, Beck HP, Dawe LA. 2002. The perceived utility of human and automated aids in a visual detection task. Human Factors, 44, 79–94. Dzindolet MT, Pierce LG, Dawe LA, Peterson S, Beck HP. 2000. Building trust in automation. Proc. of the Human Performance, Situational Awareness and Automation: User-Centered Design for the New Millennium: The Fourth Annual Technology and Human Performance Conference and the Third Conference on Situation Awareness in Complex Systems. Savannah, GA, HPSAA. Fine GA, Holyfield L. 1996. Secrecy, trust, and dangerous leisure: generating group cohesion in voluntary organizations. Social Psychology Quarterly, 59, 22–38. Hoc JM. 2000. From human-machine interaction to human-machine cooperation. Ergonomics, 43, 833–843. Kantowitz BH, Hanowski RJ, Kantowitz SC. 1997. Driver acceptance of unreliable traffic information in familiar and unfamiliar settings. Human Factors, 39, 164–176. Kim J, Moon JY. 1998. Designing towards emotional usability in customer interface – Trustworthiness of cyber – banking system interfaces. Interaction with Computers, 10, 1–29. Kramer RM. 1999. Trust and distrust in organizations: emerging perspectives, enduring questions. Annual Review of Psychology, 50, 569–598.

ANALYSES INFORMATION FUSION AND SITUATION ASSESSMENT

169

15. Lee JD, Moray N. 1992. Trust, control strategies and allocation of function in humanmachine systems. Ergonomics, 35, 1243–1270. 16. Lee JD, Moray N. 1994. Trust, self-confidence, and operators’ adaption to automation. International Journal of Human-Computer Studies, 40, 153–184. 17. Lee JD, See KA. 2004. Trust in automation: designing for appropriate reliance. Human Factors, 46, 50–80. 18. Lewandowsky S, Mundy M, Tan G. 2000. The dynamics of trust: comparing humans to automation. Journal of Experimental Psychology: Applied, 6(2), 104–123. 19. Madhavan P, Wiegmann DA, Lacson FC. 2003. Automation failures on tasks easily performed by operators undermines trust in automated aids. Proc. of the 47th Annual Meeting of the Human Factors and Ergonomics Society. 20. Madsen, M. Gregor, S. 2000. Measuring human-computer trust. Proc. of Eleventh Australasian Conference on Information Systems. Brisbane, 6–8 December. 21. Masalonis AJ, and Parasuraman R. 1998. Trust as a construct for evaluation of automated aids: past and present theory and research. Proc. Human Factors and Ergonomics Society 43rd Annual Meeting, Santa Monica, CA, 184–188. 22. Miller CA. 2004. Trust in adaptive automation: the role of etiquette in tuning trust via analogic and affective method. Proc. of the 1st International Conference on Augmented Cognition, Las Vegas, NV, July 22–27. 23. Moray N, Inagaki T, Itoh M. 2000. Adaptive automation, trust and self-confidence in fault management of time-critical tasks; Journal of Experimental Psychology: Applied, 6, 44–58. 24. Mosier KL, Skitka LJ, Heers S, Burdick M. 1998. Automation bias: decision making and performance in high-tech cockpits. International Journal of Aviation Psychology, 8, 47–63. 25. Muir BM. 1994. Trust in automation: Part 1. Theoretical issues in study of trust in the study of trust and human intervention in automated systems. Ergonomics, 37(11), 1905–1922. 26. Muir BM, Moray N. 1996. Trust in automation, Part II. Experimental studies of trust and human intervention in a process control simulation. Ergonomics, 39, 429–460. 27. Nickerson VJ, Reilly RR. 2004. A model for investigating the effects of machine autonomy on human behaviour. Proc. of the 37th Hawaii International Conference on System Sciences, Big Island, HI. 28. Parasuraman R, Miller CA. 2004. Trust and etiquette in high-criticality automate systems. Communications of the ACM, 47(4), 51–55. 29. Parasuraman R, Molloy R, Singh I. 1993. Performance consequences of automationinduced “complacency”. International Journal of Aviation Psychology, 3, 1–23. 30. Parasuraman R, Mouloua M, Molloy R. 1996. Affects of adaptive task allocation on monitoring of automated systems. Human Factors, 38, 665–679. 31. Parasuraman R, Sheridan TB, Wickens CD. 2000. A model for types and levels of human interaction with automation. IEEE Transactions on Systems Man and Cybernetics – Part A: Systems and Humans, 30, 426–432. 32. Parasuraman R, Riley V. 1997. Humans and Automation: use, misuse, disuse, abuse. Human Factors, 39, 230–253. 33. Rasmussen J. 1983. Skills, rules, and knowledge: signals, signs, and symbols, and other distinctions in human performance models. IEEE Transactions on Systems, Man and Cybenetics, SMC-13, 257–266. 34. Rasmussen J, Pejterson AM, Goodstein LP. 1994. Cognitive systems engineering. New York: Wiley. 35. Riley V. 1994. Human use of automation. Unpublished doctoral dissertation, University of Minnesota.

170

H. ATOYAN AND E. SHAHBAZIAN

36. St. John M, Manes DI. 2002. Making unreliable automation useful. Proc. of the Human Factors and Ergonomics Society 46th Annual Meeting (pp. 332–336). Santa Monica, CA: Human Factors and Ergonomics Society. 37. Tan G, Lewandowsky S. 1996. A comparison of operator trust in humans versus machines. Proc. of CybErg 1996: The 1st International Cyberspace Conference on Ergonomics. International Ergonomics, Perth, Australia, Association Press. 38. Tseng S, Fogg BJ. 1999. Credibility and computing technology. Communications of the ACM, 42(5), 39–44. 39. Wickens CD, Dixon S. 2005. Is There a Magic Number 7 (to the Minus 1)?: The Benefits of Imperfect Diagnostic Automation: A Synthesis of the Literature. University of Illinois Human Factors Division Technical Report AHFD-0501/MAAD-05-1 40. Wickens CD, Gempler K, Morphew ME. 2000. Workload and reliability of predictor displays in aircraft traffic avoidance. Transportation Human Factors, 2, 99–126. 41. Wickens CD, Xu X. 2002. Automation Trust, Reliability and Attention HMI 02-03, University of Illinois Human Factors Division Technical Report AHFD-0214/MAAD-02-2. 42. Yeh M, Wickens CD. 2001. Display signalling in augmented reality: effects of cue reliability and image realism on attention allocation and trust calibration. Human Factors, 43, 355–365.

ISSUES WITH DEVELOPING SITUATION AND THREAT ASSESSMENT CAPABILITIES

JEAN COUTURE AND ERIC MENARD Lockheed Martin Canada, Canada

Abstract. Lockheed Martin Canada has recently developed a Situation and Threat Assessment (STA) and Resource Management (RM) application using some recent technologies and concepts that have emerged from level 2 and 3 data fusion research. The current paper focuses on the issues pertaining to the design and implementation of threat opportunity, capability, and intent in threat assessment computation. The discussion about intent is particularly relevant for harbour protection since it is the key factor in identifying asymmetric threats. Keywords: Threat assessment, asymmetric threat, situation assessment

1 Introduction In recent years, Lockheed Martin Canada (LMC) has been very active in data fusion research, specifically towards evaluating methods, algorithms, techniques and architectures. This research does not only include theory and modelization investigation but also involves significant effort in experimentation by implementing and testing the most promising findings of the research. Experimentation allows access to problems and issues that are not obtainable by a pure top-down theoretical approach. One of the main products of this experimentation is the Situation and Threat Assessment (STA) and Resource Management (RM) application (hereafter STA/RM) that has been developed over the last five years. This application is meant to fulfill the capabilities of a modern high-level fusion system, using the most recent technologies and concepts that have emerged from level 2 and 3 data fusion research. STA/RM was originally designed to be used in blue water environment but is now expanding towards more complex environments, e.g. the littoral environment. The current paper will focus on the experimentation performed on threat assessment. Threat assessment consists of estimating and predicting effects on situations of planned or estimated actions by the participants [6] and comprises three main components: opportunity, capability and intent. The current paper will E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

171

172

J. COUTURE AND E. MENARD

discuss issues pertaining to the design and implementation of each of those components. The discussion about intent is particularly relevant for harbour protection since it is the key factor for identifying asymmetric threats (e.g. small boats, swimmers, unconventional air threats, etc… ).

2 Development of a situation and threat assessment application Higher fusion levels, namely Situation Assessment and Threat (or Impact) Assessment, have not yet reached the maturity of level 1 data fusion. Significant research on several domains pertaining to higher fusion levels such as human learning process, ontology or artificial intelligence systems is still required. Therefore, the development of a STA/RM application must allow for a large potential of growth to take advantage of new trends in methodologies, modelization and technologies that will emerge in the near future. LM Canada (LMC) has developed Cortex, a Knowledge-Based System (KBS) using a Blackboard (BB) architecture that eases the development of applications requiring a large amount of: x Parallelization x Growth (expandability) x Speed For these reasons, Cortex has been used as the backbone framework for STA/RM [1, 7]. Cortex stores the dynamic data in the blackboard and the processing capabilities are encapsulated in the so-called agents. This is depicted in Figure 1. It is therefore very easy to modify, remove or add a capability without perturbing the other capabilities of the application. In addition, any new capability that is implemented can use the whole blackboard as its data playground. This is very useful when a lot of experimentation is required to develop a capability.

Figure 1. Blackboard architecture.

In order to optimize the growth potential of STA/RM, the following incremental approach has been followed: x Develop an open and scalable software design (i.e. defining the data model, the agents and their relation) based on what we know about STA and RM and what is expected from level 2–3 new trends/methodologies/technologies.

DEVELOPING SITUATION AND THREAT ASSESSMENT CAPABILITIES

173

x 1st iteration: implement minimum Threat Evaluation (TE) and Weapon Assignment (WA) capabilities in order to perform the basic tasks of a command and control system (from entity detection up to threat engagement and kill assessment). x Subsequent iterations: introduce new sets of functionalities/concepts or refine existing capabilities. After each implementation iteration, the new or modified STA/RM capabilities are evaluated using a closed-loop test bed comprising a target generator, sensor models and a Human Computer Interface (HCI). The main functional components of the current STA/RM application are: x Threat Assessment: Evaluation of threat values and threat ranking x Target Weapon Analysis: Computation of engageability; target-weapon pairing; engagement recommendations x Engagement planning: Reactive and deliberative planning x Situation Assessment: Identity refinement; corridor/lane correlation; clustering; manoeuvre detection; and more… The rest of this paper will describe how the architecture and development approach presented above have been used to mitigate the issues that inevitably occur during the implementation of such a complex application as STA/RM. For sake of brevity, the discussion will mainly focus on issues related to threat assessment.

3 Threat assessment implementation The main tasks of a “real life” threat assessment system are the evaluation of the threat level of non-friendly tracks and the ranking of those threat levels to build a prioritized threat list. This list is in turn used to establish engagement planning and reserve and assign resources to the most threatening entities. Threat level evaluation usually addresses three different aspects of a threat: its opportunity to do damage, its capability to do damage and its intent. Recent design and implementation of threat assessment capabilities have brought to light several issues that are discussed further in this section. 3.1 OPPORTUNITY ASSESSMENT The problem of opportunity assessment has been addressed during the first implementation phase of STA/RM. The opportunity is defined relative to a specific location or a valuable asset, like the ownship, and represents a time and space measure of

174

J. COUTURE AND E. MENARD

how close the threat is to its assumed target (i.e. the location or asset). Its computation requires the following threat information: x x x x

Speed Heading Closest Point of Approach (CPA) Time to reach CPA

As an example, Figure 2 illustrates the information at play for the opportunity computation of three threats. Many different algorithms exist to perform the computation, each one having a particular method of weighting the different pieces of information. In any case, using only the opportunity to deduce threat values can be misleading for the following reasons: x Threat lethality is not taken in account: –

–

Slow entities (e.g. ships, submarines) are usually assigned low threat values in spite of the possibility that they may have very threatening long range weapons. Projection of entities trajectory may not lead directly towards the asset to be protected but they still can launch weapons directly on the asset.

x Threats may not have the intent to attack. It is therefore necessary to include threat’s capability and intent in the threat evaluation process.

Figure 2. Information involved in threat’s opportunity computation.

3.2 CAPABILITY ASSESSMENT Capability assessment has just been implemented in the latest STA/RM implementation phase. In order to include threat’s capability in the overall threat evaluation process, the two following information sources are required:

DEVELOPING SITUATION AND THREAT ASSESSMENT CAPABILITIES

175

x Threat’s identity provider x Characteristics and capabilities of the identified entities The first source is generally fulfilled by a data fusion (i.e. level 1 fusion) module. For instance, LMC has developed a Multi-Source Data Fusion (MSDF) engine that uses Dempster-Shafer evidential theory to compute entities’ identity. The second source has been addressed more recently in collaboration with and sponsored by DRDC Valcartier [2, 8] and consists of a vast database containing a priori information about: x x x x

Aircraft and ships (more than 2,000) of all types and their characteristics Sensors (more than 1,500) and their characteristics Weapons (more than 500) of all types and their characteristics, and Ground infrastructures, maps, pictures, and documents

Figure 3 displays diverse types of information contained in the a priori database: characteristics of an F16, its weapons and its sensors are presented, as well as additional information on the owner’s country (the Netherlands). With those two sources of information in place (i.e. MSDF and a priori database), we have investigated two different methods that include threat’s capability in the threat evaluation [5]: the Constant Velocity Time Earliest Weapon Release (CVTEWR) and the Manoeuvre Time Earliest Weapon Release (MTEWR). Those two methods are explained in Figure 4. The former uses the estimated time the identified threat would take to launch its most threatening weapon (e.g. the one having the longest range) if the threat keeps its current velocity. The second method is similar except that the threat is assumed to break its trajectory instantaneously in order to launch its most threatening weapon in a minimum of time. We have finally implemented the MTEWR method, which constitutes a “worst case scenario” compared to the other method. This method requires the following a priori information: x x x x x x

Precise threat identity “Most threatening” weapon identity Weapon velocity Weapon maximum range Weapon type (Missile, CIWS, Gun, Cannon, Torpedo, Mortar) Weapon Utility (air/surface/subsurface)

176

J. COUTURE AND E. MENARD

Figure 3. Database browser showing various information from the a priori database.

Figure 4. Definition of CVTEWR and MTEWR computational methods.

The implementation of the MTEWR method has been extended to handle threats that have an incomplete identity (due to lack of information or conflicts) or no weapons (e.g. a missile) [4]. Some refinements and additional issues are being investigated, such as: x Improving the handling of threats with incomplete identity x Refining the concept of “Most Threatening” weapon by including threat sensors, jamming and softkill capabilities x Introducing Measures of Performance (MOPs) based on probability of killing targets and probability of survival The inclusion of threat capability in the threat evaluation may still not provide satisfactory results when dealing with asymmetric threats, since they do not

DEVELOPING SITUATION AND THREAT ASSESSMENT CAPABILITIES

177

necessarily behave accordingly to their type. For instance, the presence of explosives or bombs in a small boat cannot be revealed by looking at the small boat’s capability as stored in the a priori database. Therefore, a third aspect of threat evaluation must be included: the threat’s intent. 3.3 INTENT DETECTION Threat’s intent is being designed and has not yet been implemented in STA/RM. The design is inspired by the study of Liebhaber and Smith [3] who extracted behaviour profiles from a team of experimented US Navy officers through voice recording during realistic simulation exercises. Profiles are decomposed into several attributes such as: x x x x x

Whether or not an aircraft emits on a commercial frequency Whether an aircraft follows an air corridor or deviates from it Low vs. high altitude Constant vs. changing velocity IFF mode

Each attribute is characterized by a weigh representing the importance of the attribute in the profile. In other words, a profile represents the “normality” model of a given entity that performs a given function (such as commercial aircraft, fighter preparing an attack, surveillance UAV, fishing boat, etc… ). Profiles may be used for different purposes: x Assessment of threat’s intent x Identification of an unknown entity from its profile analysis x Detection of an asymmetric threat (an entity deviates from its normal behaviour) Figure 5 shows a preliminary design for implementing intent detection in STA/RM. Assuming that a profile of interest has been selected, a number of single-purpose Situation Assessment (SA) agents assess whether a track fulfils specific profile attributes or not. After each SA agent has reported its result in a Track-Profile Correlation Table, the agent “Evaluate Threat Intent” estimates at which degree the track fulfils the overall profile, and then decides if the threat value must be changed and if a warning has to be sent to the operator. The same agent could also evaluate if the track represents an asymmetric threat. Several SA agents have already been implemented (commercial corridor correlation, manoeuvre detection, fast inbound threat detection, track splitting, and formation/cluster detection). Additional agents will need to be developed according to the profiles that will be defined. Obtaining implementable profiles is probably the most critical step and would require the use of modeling (e.g. cognitive models), knowledge acquisition techniques (e.g. neural network) and Subject Matter Experts (SMEs).

178

J. COUTURE AND E. MENARD

PROFILE

PROFILE ATTRIBUTE #1

Situation Assessment Agent #1

PROFILE ATTRIBUTE #2

Situation Assessment Agent #2

PROFILE ATTRIBUTE #3

Situation Assessment Agent #3

THREAT THREAT THREAT LIST LIST LIST

Evaluate Threat Intent Agent

TRACK

TRACK-PROFILE CORRELATION TABLE

Figure 5. Design to implement intent detection in STA/RM.

4 Conclusion In this paper, we demonstrated how the incremental approach used to develop STA/RM was successful in iteratively improving the computation of threat assessment. This is largely due to the expandability and modularity allowed by Cortex and to the implementation of a very versatile and generic architecture on top of the Cortex blackboard environment. The next step will consist of implementing the concepts related to threat’s intent. This exercise will provide the opportunity to expand the applicability of the LM Canada STA/RM application to other contexts or domains, such as asymmetric threats and harbour protection.

References 1.

2. 3.

4.

5.

Bergeron P, Couture J, Duquet JR, Macieszczak M, Mayrand M. 1998. A New Knowledge-Based System for the Study of Situation and Threat Assessment in the Context of Naval Warfare, in FUSION 98 Las Vegas, 6–9 July 1998, Vol. II, pp. 926– 933. Couture J, Duquet JR, Allard Y. 2002. MSDF/STARM Libraries Study – Final Report, Doc. No. 6520014004, Lockheed Martin Canada. Liebhaber MJ, Smith CAP. 2000. Naval Air Defence Threat Assessment: Cognitive Factors and Model, In Proceedings of the 2000 Command and Control Research and Technology Symposium, Monterey, CA: Naval Postgraduate School. Menard E, Couture J. 2005. Application of Improved Threat Evaluation for Threat Assessment, Multisensor Data and Information Processing for Rapid and Robust Situation and Threat Assessment. NATO Advanced Study Institute, Albena, Bulgaria, 16–27 May 2005. Oxenham MG. 2003. Enhancing Situation Awareness for Air Defence via Automated Threat Analysis, In Proceedings of the Sixth International Conference on Information

DEVELOPING SITUATION AND THREAT ASSESSMENT CAPABILITIES

6.

7. 8.

179

Fusion (FUSION 2003), Cairns, Australia, 8–11 July 2003. International Society of Information Fusion. pp. 1086–1093. Shahbazian E, Duquet JR, Valin P. 1998. A Blackboard Architecture for Incremental Implementation of Data Fusion Applications, in FUSION 98 Las Vegas, 6–9 July 1998, Vol. I, pp. 455–461. Steinberg AN, Bowman CL, White FE. 1998. Revisions to the JDL Data Fusion Model, in Joint NATO/IRIS Conference, Quebec City, Quebec, 19–29 October, 1998. Truchon JF, Couture J. 2002. MSDF/STARM Libraries Study – Data Representation for the Information Libraries, Doc. No. 6520014004, Lockheed Martin Canada.

DISTRIBUTED DATA FUSION AND MARITIME DOMAIN AWARENESS FOR HARBOUR PROTECTION

SIMON J. JULIER1 AND RANJEEV MITTU2 1 ITT Advanced Engineering Systems/Naval Research Laboratory, Washington, DC 2 Naval Research Laboratory, Washington, DC

Abstract. Protecting a harbour against intentional and accidental threats is extremely difficult. Harbours are not closed systems. Rather, they are focal points for the movement of people and cargo, both in land and on water. As such, threats can arise from many sources that range from the smuggling of illegal and dangerous goods to the placement of mines to damage or destroy shipping. To detect the many different types of threats, a harbour must be monitored by multiple sensing systems with different sensing modalities. To be practical, such a large sensing system must be cost effective to install, can be readily upgraded, and should be robust to sensor and communication failures. In this chapter we discuss the role that distributed data fusion can play in harbour protection. We define and discuss distributed data fusion algorithms and illustrate how they could be used in port surveillance and Maritime Domain Awareness applications. Keywords: Distributed data fusion, harbour security, maritime domain awareness, distributed operations

1 Introduction The need for Harbour Protection is extremely important. Harbours are critical for commercial activities. However, the sheer volume of people and material moving through them means that they are not closed systems. Rather, threats can arise in many different ways from many different sources. These include the use of commercial vessels for contraband smuggling and trafficking (people and/or weapons), the potential use of commercial vessels to support other illegal activities that could lead to terrorist activities, and threats that directly impact the harbour itself (such as mining the waterways). These difficulties are exacerbated by the fact that some types of threats – such as the hijacking of commercial ships – means that an effective terrorist attack can be initiated even while the ship is far from port and there is substantial time before the threat manifests itself [8]. E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

181

182

S.J. JULIER AND R. MITTU

Therefore, threats should be detected as far away and as early as possible before they have an opportunity to reach their destination. In the best case, detection is achieved before the threat departs from port headed towards the destination. However, detection may occur while vessels are in transit but at a sufficient distance from the destination. Therefore, the ability to recognize, monitor, track and intercept suspect maritime vessels on a global scale is being seen as a major capability that will enable the United States and its allies to stop future global terrorist activities. This capability, known as Maritime Domain Awareness (MDA) is being pursued by many agencies in the Department of Defence (DoD). Whatever the source of the threat, one means of identifying and responding to it is to start with an accurate Maritime Common Operational Picture (MCOP). The MCOP is formed by integrating multi-source intelligence information obtained through a worldwide network. The information may contain raw measurements that are fused with other raw measurements (Level 1 fusion) to enable the estimation of objects including their identity and kinematics. Level 1 fusion is a necessary precursor that enables Level 2 fusion, which is concerned with situation assessment and the ability to recognize activities and their relationships. Level 3 fusion concerns itself with threat assessment and ability to reason about entity intent. Generally, systems that provide the MCOP are concerned with Level 1 fusion. However, as the DoD moves towards the vision of realizing networkcentric warfare operations, it is reasonable to expect that services supporting Level 2/3 fusion will be available. The MCOP can be formed using the centralized architecture shown in Figure 1a: all the raw sensor data is sent to a central fusion site where it is fused together. However, an alternative is to use the distributed data fusion (DDF) system illustrated in Figure 1b. Such systems can be flexible, robust and tiered. They replace the notion that the network consists of sensors and a fusion center by a set of processing nodes connected to one another through communication links [2].

Sensor 1

Sensor 2

Sensor 2

Node 2

Node 3

Node 1

Fusion Node

Sensor 6

Sensor 5

a

Sensor 4

Node 6

Node 5

Node 4

b

Figure 1. Centralized and distributed fusion architectures. In a centralized architecture, all the sensor data is routed to a central fusion site. In a distributed architecture, sensor data is fused throughout the network in processing nodes.

DISTRIBUTED DATA FUSION AND MARITIME DOMAIN AWARENESS

183

Each processing node can have zero or more sensing devices attached to it. There is no single central fusion center (the system state can be extracted by a “system monitor” which can be attached to any node in the network); there is no common communication facility (all communication is managed on a node-to-node basis); there is no need for global knowledge of network topology (nodes need only know the other local nodes they communicate directly with). Given the potential benefits of distributed data fusion, the purpose of this chapter is to describe its basic principle of operation, discuss different types of architectures, and discuss how it can be applied to Harbour Protection. A description of DDF and the different network topologies is provided in Section 2. An application to Harbour Protection is given in Section 3. Conclusions and a summary are drawn in Section 4.

2 Distributed data fusion 2.1 COMPONENTS OF A DDF SYSTEM As explained above, a distributed data fusion (DDF) system consists of a set of processing nodes connected together through communication links. Each node possesses zero or more sensing devices. Nodes fuse data from two sources: data collected from local sensors (if available) and data distributed to it from other nodes. The communication between nodes is entirely local: a single node only knows the list of other nodes it communicates with; no single node need know the entire topology of the network. There is no direct one-to-one correspondence between processing nodes and platforms. Figure 2 illustrates a possible configuration of processing nodes on a single platform such as an Unmanned Aerial Vehicle (UAV). The UAV possesses two types of sensors such as Forward Looking Infra-Red (FLIR), Laser Radar (LADAR) and a database. These are configured in two separate nodes: one for handling the low-level data, the other for handling the database. There are many attractive properties to DDF systems including [2]: x Reliability: the loss of a subset of nodes and/or links does not necessarily prevent the rest of the system from functioning. In a centralized system, however, the failure of a common communication manager or a centralized controller can result in an immediate catastrophic failure of the system. x Flexibility: nodes can be added or deleted by making only local changes to the network. For example, the addition of a node simply involves the establishment of links to one or more nodes in the network. In a centralized system, however, the addition of a new node can change the topology in such a way as to require massive changes to the overall control and communications structure.

184

S.J. JULIER AND R. MITTU

x Bandwidth: nodes do not need to distribute raw sensor data. Rather, by propagating fused sensor products significant bandwidth savings can be achieved. For example, a processing node with a camera could use computer vision algorithms to process the image and identify and track targets. Therefore only the fused projects (e.g., trackID, pixel coordinates, and pixel velocity) need be distributed. In a centralized system, however, all the raw sensor data (video) would have to be transmitted to the central node to be processed. As the example in Figure 2 shows, the capabilities of all nodes need not be the same and can vary in at least five different ways [5]: 1. Local sensing capability: there are many sources of intelligence information [12] (e.g., Signals Intelligence and Electronic Intelligence, to name a few) and a priori data (databases and other offline sources of information). However, some nodes might possess no sensors at all. Rather, they can perform the role of aggregating, forwarding and disseminating information. 2. Signal processing: a small unattended ground sensor, for example, might perform simple low pass filtering and use crude localization algorithms to localize a target within a fixed detection region. At its most complicated, a node might perform target class recognition using a variety of pattern recognition algorithms, constraining the results using a set of geospatial and other databases. In the most extreme case, a node might be a fusion center consisting of many analysts utilizing many types of data. It should be noted that although most DDF algorithms have been applied to Level 1 Fusion there is no difficulty, in principle, with applying these methods to Level 2 and Level 3 Fusion as well. 3. Available bandwidth: different types of nodes have access to different network resources. This can depend on both the capability of the node and its current activity. As a result, the bandwidth available on different communication links can vary and signal compression schemes must be used [7]. 4. State information maintained: each node maintains a subset of the MCOP depending on its sensing capabilities, purpose and security level. 5. Roles assigned to nodes: depending on available hardware, different nodes can be assigned different roles. For example, some nodes can be assigned information collection roles (significant sensing capacity; little onboard fusion), others specialize in fusion (few sensing capabilities; significant onboard fusion) and some can handle dissemination and monitoring capabilities. Furthermore, some can act as master or slave nodes. DDF networks can be configured in a number of different network topologies, each with their own advantages or disadvantages. We now outline these.

DISTRIBUTED DATA FUSION AND MARITIME DOMAIN AWARENESS

N1 FLIR

FLIR Picture

LIDAR Picture

DB Picture

185

N2 LIDAR

DB

Platform

Figure 2. Configuration of processing nodes in a UAV.

2.2 NETWORK TOPOLOGIES The different types of DDF network topologies that have been developed are illustrated in Figure 3: x Fully-connected: all nodes share all their sensor information with all other nodes in a timely manner. This architecture has been deployed in the Cooperative Engagement Capability [11]. Each node has the same information and is provably optimal. However, a large number of communication links are required and the topology is brittle: if any communication link fails, the assumption that all nodes have the same state is no longer true. x Tree-connected: in these networks, nodes need only communicate locally with one another and a single path exists between the nodes. Fusion algorithms have been developed which are provably optimal [3]. Furthermore, the network topology can change. However, this topology is brittle. Because there is a single path between any two nodes there is no redundancy. Adding multiple links to form redundancy leads to double-counting, as discussed below. However, the tree can reconfigure itself: if a root node is compromised, the network can be reconfigured and a new node replaced. x Hierarchical: in these networks, different nodes are assigned different roles. There is a central node and all data flows there through a set of intermediate nodes [9]. The intermediate nodes can, in effect, be considered a type of signal compression (for example, raw imagery data is processed to give a track of a target). However, this is a tree connected topology (hence, there still exists a central point of failure). Furthermore, the specification of roles means that there can exist a single point of failure if, for example, the master node is compromised. x Ad hoc: these have no special global topology. It is possible for loops and cycles to exist, leading to flexible communication architectures and redundancy. However, with general topologies no optimal local fusion algorithm can be developed. A significant problem with distributed data fusion is the potential risk for double counting.

186

S.J. JULIER AND R. MITTU

Node 2

Node 3 Node 2

Node 1

Node 8

Node 6

Node 5

Node 4

Node 2

Node 5

Node 7

Node 4

b Node 2

Node 3

Node 3

Node 1

Node 1

Node 5

Master Node

Node 6

a

Node 6

Node 3

Node 1

Node 6 Node 4

c

Node 5

Node 4

d

Figure 3. Various network topologies that can be used in distributed data fusion networks: (a) fully-connected; (b) tree-connected; (c) hierarchical; (d) ad hoc.

2.3 DOUBLE COUNTING AND SOLUTIONS One of the most serious problems which may arise in a DDF network is the effect of redundant information. Specifically, pieces of information from multiple sources cannot be combined within most filtering frameworks unless they are independent or have a known degree of correlation (i.e., known cross covariances). The effect of redundant information can be seen in the following scenario, sometimes referred to as “rumor propagation” or the “whispering in the hall” problem: x A node incorporating a sonar sensor detects a weak track that might be caused by an underwater threat. A hypothesis is generated that a threat exists and is propagated into the network. This information can be synopsized, augmented, or otherwise transformed as it is relayed through a sequence of nodes. x A threat database node receives this information and notes that a threat might be present. There are many possible interpretations of this data, but the possibility of a threat (e.g., diver) is deemed to be of such tactical importance that it warrants the transmission of a low confidence hypothesis. Again, the information can be transformed as it is relayed through a sequence of nodes. x The sonar sensing node receives the low confidence hypothesis that a diver threat exists. A check of available sensor data shows a feature that is consistent with the hypothesis. Because the node is unaware that the hypothesis was based on exactly the same sensor evidence, it assumes that the feature that it observes is an independent confirmation of the hypothesis. The node then transmits high confidence information that the feature represents the threat.

DISTRIBUTED DATA FUSION AND MARITIME DOMAIN AWARENESS

187

x The threat database node receives information from the sonar sensing node that a diver threat has been identified with high confidence. The threat database node regards this as confirmation of its early hypothesis and calls for an aggressive response to the situation. This problem cannot be solved using optimal data fusion algorithms [10]. However several classes of suboptimal data fusion algorithms have been developed which can be used to overcome these difficulties [4]. We now illustrate how distributed DDF can be applied to MDA for Harbour Security.

3 Examples Harbour surveillance and protection is critical in the Defence of shore based assets. Therefore, threats should be detected as far away and as early as possible before they have an opportunity to reach their destination [8]. There may be many factors that help in detecting possible threats while they are in transit, and one example may be an unusual change in the vessel’s normal path or in its emissions pattern. Detecting the change in vessel behaviour is part of the remit of MDA. MDA is a complex problem which spans numerous dimensions, including the sharing and analysis of complex and sometimes incomplete data to identify vessels of interest and inferring vessel intent based on that analysis. It is a process that requires the coordination and sharing of data/information between all the entities (i.e., nodes), both mobile and fixed, that take part in the global war on terror. The process must be resilient to errors, and must be carried out as efficiently as possible to mitigate threats in a timely manner worldwide, but particularly in ports and harbours which represent the last stages of Defence. As explained above, the concept of a node can be an extremely broad one, and can span from low-level sensing devices to entire fusion centers. We now provide two examples which illustrate the flexibility of the concept. We also discuss how DDF can be used to develop multi-tiered systems. 3.1 DISTRIBUTED, MOBILE SENSOR NODES The Automated Identification System (AIS) can be used to track a vessel. However, it is a signal that can be impacted by the effects of the environment such as weather phenomena. The same applies to emissions. Because effects such as these are dynamic over time, stationary nodes may not be able to receive and process such signals all the time, leading to possible gaps in collection. One means of overcoming this difficulty is to augment fixed nodes by a set of mobile nodes that can dynamically adjust and reallocate themselves to specific areas of interest. Furthermore, they are appealing due to their lower cost of operation. For example,

188

S.J. JULIER AND R. MITTU

the use of UAVs can play a significant role to support MDA due to their flexibility in reallocation. UAVs can be used to perform many tasks (such as information gathering and target search) in many types of environments (dynamic, uncertain, dense) with both known and unknown targets and threats. Because of the dynamic nature of the environment, fixed guaranteed communication topologies between UAVs might not exist. The use of DDF for tracking and fusion with UAVs has already been demonstrated for picture complication and distributed tracking both in simulation and in real hardware implementations [6]. It has already been shown to be a robust and scalable solution and is highly likely to convey similar benefits for MDA. 3.2 SHORE-BASED DATA FUSION AND ANALYSIS CENTERS The generalization of a node can extend to the scale of shore-based data fusion and analysis centers. These centers collect and fuse multi-source intelligence data and coordinate the distribution of the resulting MCOP to other centers. There are numerous collection and analysis centers worldwide responsible for collecting, analyzing and disseminating information on potential maritime threats to other regional centers or to tactical military centers. The tactical military centers also manage tactical systems and data links to produce an MCOP for dissemination to operational units and command elements throughout their areas of responsibility. These centers process data from many sources. One source of information is from other shore-based analysis centers. Because track and other data are shared frequently between systems, and because the intervening signal processing and transformation steps are extremely complicated, it is impossible to calculate the degree of common information between the two or more sites. Robust decentralized data fusion techniques, however, provides a formal mechanism for describing how the interaction between the different centers can occur. 3.3 TIERED SYSTEMS The previous two examples have shown how DDF can be used to support fusion between piers – whether they are UAVs or entire fusion centers. However, a key enabler of a sustainable military force is the notion of a tiered system [1], and is likely one of the more complex decentralized data collection, information processing and sharing systems. Conceptually, a tiered system is an integrated, multi-tier intelligence system encompassing space and air-based sensors linked to close-in and intrusive lower tiers (Figure 4). The lower tiers are not only the critical source of intelligence; they can also serve as a key cueing device for other sensors. Given the diversity of the assets, and the fact that information will need to be shared across the horizontal and vertical planes, and the environments in which

DISTRIBUTED DATA FUSION AND MARITIME DOMAIN AWARENESS

189

From : Report of the DSB on Future Strategic Strike Force (Feb 2004)

Multi/Bistatic Target Location Error

l

Space (Tier 4)

ta on

iz or s H ation s e gr ml ea Inte

National SBR/SBIRS Hig

S

h Altitude (Tier 3) UAVs, Theater ISR e-IOA like Futrue EP-3

ltitude close-in (Tier 2) wA Lo

UAVs, Tactical ISR

Sensing (Tier 1 rusive ) Vnt

UGS, Tags, Cyberspy, “Critters”

rs

so

en

es tiv ac le e for b or M Lo res fuela e r t e N u r t o ss PE ria er e Le FO prop y ap ranc - p lth du - A tea en - S ong -L

Better Altitude (standoff)

Increasing

{Blair} Figure 3-1 : Future tiered ISR systems.

Figure 4. Tiered systems concept.

the components of a tiered system will likely operate, it is not practical to envision a single data fusion architecture. Given the diversity of the information and processing nodes that may be participating in the network, it is more reasonable to expect that a given “system of systems” should be flexible, and employ a mix of architectures most suited for the operational environment, while maintaining interoperability across the spectrum. Future maritime scenarios may involve a collection of components from a tiered-system hierarchy, with both internal communications between the tieredsystem components, as well as external communications with shore-based data fusion and analysis centers. Due to the different types of information flowing within the networks ranging from raw measurements to partial estimates, coupled with decentralized system control due to the size and complexity of such an aggregate system, it is likely that a decentralized approach will provide a robust, scalable and flexible approach to data fusion.

4 Summary and conclusions Protecting a harbour against intentional and accidental threats is extremely difficult. Because they are focal points for the movement of people and cargo both in land and on water, they are open systems. One means of identifying threats is to develop an accurate MCOP. However, to develop an accurate and consistent MCOP requires the integration of data/information from multiple sensing systems with different sensing modalities. To be practical, such a large sensing system must be cost effective to install, can be readily upgraded, and should be robust to sensor and communication failures. In this chapter we have argued that distributed data fusion networks exhibit many of these properties. We have described the properties of these networks and we have argued that algorithms to overcome double counting – one of the most important problems of such networks – can be

190

S.J. JULIER AND R. MITTU

resolved using suboptimal fusion algorithms. We have briefly described examples of how DDF can be used to coordinate teams of UAVs, fuse data between command centers, and may offer capabilities to achieve the vision of tiered systems. Given these advantages, we believe that DDF is a valuable fusion paradigm which can be used in harbour protection and MDA.

Acknowledgements This work was supported in part by the Office of Naval Research.

References Dalburg J et al. 2007. Developing a Viable Approach for Effective Tiered Systems. NRL Memorandum Report 1001-07-9024. 2. Durrant-Whyte HF, Stevens M. 2001. Data Fusion in Decentralised Networks. University of Sydney, Australia. 3. Grime S, Durrant-Whyte HF. 1994. Data Fusion in Decentralized Sensor Fusion Networks. Control Engineering Practice, Volume 5, no. 2. p 849–863. 4. Julier SJ, Uhlmann JK. 2001. General Decentralized Data Fusion with Covariance Intersection (CI). In: Hall DA, Llinas J, editors. Handbook of Multisensor Data Fusion. CRC Press LLC, Boca Raton, London, New York, Washington, DC. p 12-1–12-18. 5. Julier SJ, Uhlmann KJ, Walters J, Mittu R, Palaniappan K. 2006. The Challenge of Scalable and Distributed Fusion of Disparate Sources of Information. In: Proceedings of the Multisensor, Multisource Information Fusion: Architectures, Algorithms, and Applications Conference. SPIE Defence and Security Symposium, Volume 6242. Orlando, FL. 6. Nettleton EW, Durrant-Whyte HF, Gibbens PW, Goktogan AH. 2000. Multiple Platform Localisation and Map Building. In: Proceedings of Sensor Fusion and Decentralized Control in Robotic Systems III Conference, SPIE Photonics East, Volume 4196. Boston, MA. 7. Nicholson D and Leung V. 2004. Managing a Distributed Data Fusion Network. In: Proceedings of the Signal Processing, Sensor Fusion and Target Recognition, XIII Conference, SPIE Defence and Security Symposium, Volume 5429. Orlando, FL. 8. Shahbazian E, DeWeert J, Rogova G. 2006. Findings of the NATO Workshop on Data Fusion Technologies for Harbour Protection. In: Proceedings of the Photonics for Port and Harbour Security II Conference, SPIE Defence and Security Symposium, Volume 6204. Orlando, FL. 9. Pao LY. 1994. Distributed Multisensor Fusion. In: Proceedings of the AIAA Guidance, Navigation and Control Conference. Scottsdale, AZ. p 82–91. 10. Utete SW. 1995. Network Management in Decentralised Sensing Systems [dissertation]. Robotics Research Group, Oxford. 11. The Cooperative Engagement Capability. 1995. Johns Hopkins APL Technical Digest, Volume 16, no. 4. p 337–396. 12. http://www.fbi.gov/intelligence/di_ints.htm [Internet]. 1.

INTELLIGENT GEO-INFORMATION SYSTEMS AND HARBOUR PROTECTION

RUSLAN P. SOROKIN St. Petersburg Institute for Informatics and Automation of the Russian Academy of Science, Russia

Abstract. We propose an approach to simulation of complex spatial processes within the scope of harbour protection. Different concepts of the simulation of spatial processes are formalized into scenario ontology. Integrated visual scenario development and environment replay using free open source software products are described. Keywords: Geo-information system, artificial intelligence, ontology, expert system, rulebased system, scenario simulation

1 Introduction Harbour protection combines many challenges of the contemporary world, such as terrorist organizations, refugee and immigrant flows, drug trafficing, ecological disasters, epidemics and cyberterrorism. How can we analyze and predict the consequences of such phenomena? How can we teach people to behave correctly during disasters? How can we teach administrators to make good decisions? How can we drill personnel to act fast and effectively? The answer is in disaster modeling and simulation. Modeling is the abstraction of the complexity of a phenomenon and the representation of one of its many inherent aspects, while simulation is the representation of all aspects of a phenomenon as naturally and correctly as possible. The root of many problems of simulation is the problem of software development. Conventional software development techniques require knowledge of an exact algorithm before programming. Complex processes imply complex algorithms. Programming, debugging and testing complex algorithms is an arduous, time consuming effort, and as a result, it is expensive. Moreover, it is difficult, even impossible, to know the entire algorithm of a complex phenomenon beforehand because it consists of several parallel complex processes interacting with each other during execution. It is only possible to know the exact output of this interaction after its execution. The approach to simulation proposed in this article is based on artificial intelligence techniques such as a rule-based systems and ontology for knowledge representation.

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

191

192

R.P. SOROKIN

2 Intelligent geo-information system The spatial aspect of the abovementioned phenomenon must be emphasized in the domain of harbour protection. A proper study would thus require spatial simulation. The purely mathematical simulation of spatial processes without visual representation is referred to as “blind” simulation, and essentially decreases the possibility that experts will discover the patterns regulating these phenomena. Geo-information systems (GIS) are especially suited for the visual representation of a spatial simulation on the earth-scale. GIS rapidly evolved from simple browsers depicting electronic digital maps to complex software applications capable of producing visual simulations of spatial processes in the real world. As the power of GIS applications increases and their inner logic becomes more complex, the use of conventional programming languages in their development becomes a restraining factor. GIS applications are in urgent need of intelligent support, such as rule-based systems, ontology, data mining, multi-agent systems and other artificial intelligence techniques. We define intelligent GIS as a complex software product including the GIS and a host of artificial intelligence components for solving complex tasks including spatial simulation and modeling [1]. At present, we emphasize the following mandatory components of the intelligent GIS. x An ontology-based subsystem for knowledge representation x A rule-based subsystem for knowledge processing In addition, we bear in mind optional components, such as: x A multi-agent subsystem for creating distributed simulation environments supporting knowledge exchange x A data mining subsystem for knowledge discovery, and x A learning subsystem

3 Principles of research prototyping Modern commercial GIS are very expensive software products. Before commencing commercial product design, it is desirable to develop a research prototype and comprehensively investigate its properties and features. The following principles are a useful guide to obtaining the research prototype quickly and cheaply: x Use ready software where appropriate. x Use free open source. x Use the Internet (solid academic sources). Well-debugged and tested algorithms are very useful because programming is a lengthy, tedious, error-prone process, and is consequently expensive. If you had bought very expensive software for your research prototyping and it turned out to be

INTELLIGENT GIS AND HARBOUR PROTECTION

193

unsuitable it would be difficult for you to abandon it. An expensive system is a hitch, while free software that turns out to be unsuitable can easily be discarded for something else. Open source code grants the user full command and the possibility to make alterations as needed. Despite widespread opinion, open source systems are very reliable because, in the process of their development, they undergo intensive testing by volunteer programmers. Open source systems are rapidly developing based on the responses of numerous users. Their authors deeply respect user opinions and readily include new functionalities upon user request. Free open source software development is quickly becoming a mainstream alternative in modern computer industry. Commercial software companies are now more than ever before publishing free open source products to test their ideas, and naturally enough, using open source in advanced research projects. On the internet, you can find software for absolutely any purpose. It is rather more difficult to invent something that does not yet exist on the internet. Though not all internet sources provide good quality published software, we believe that established sources affiliated with academic institution and companies can be trusted. Based on the above principles, we searched the internet and found the following software to aid us in the development of our intelligent GIS research prototype: 1. Protégé ontology editor 2. OpenMap GIS library 3. JESS – Java expert system shell All three are unified by a single programming language, JAVA, simplifying the creation of a new software application into which they can be integrated.

4 Scenario ontology Ontology can be defined as a formal explicit description of concepts in a domain of discourse (classes, which are sometimes called concepts), properties of each concept describing various features and attributes of the concept (slots, which are sometimes called roles or properties), and restrictions on slots (facets, which are sometimes called role restrictions). Ontology, together with a set of individual instances of classes, constitutes a knowledge base [2]. Domain of discourse in our case is a simulation of complex spatial processes. We concluded that this domain can be better described by using a concept of scenario of the process, and specific concepts related to it. Furthermore, we decided that the concept of scenario is a generalization of the concept of the algorithm. We therefore defined scenario as a sequence of phases and decisions. A phase is a set of actions that are executed step by step or simultaneously. A decision is a point in which a scenario can change its flow depending on some condition and proceed along one or another branch. A decision is thus a set of branches. Actions are the elementary building blocks of a scenario. They represent specific activities or operations and can be implemented differently. An action can have its own sub scenarios and we can represent a complex scenario

194

R.P. SOROKIN

as a hierarchy of embedded simple scenarios. A fragment of the scenario ontology in the Protégé user interface is represented in Figure 1 as a class tree. When the ontology is defined, we can represent specific scenarios as instances of the class Scenario, its phases as instances of the class Phase, and so on. It is very useful to represent scenarios visually with 2-dimentional diagrams or flow-charts. The diagram of a somewhat simplified scenario of rescue operation at sea is shown on Figure 2. Dark grey rectangles represent scenario phases, Light grey diamonds represent decisions. Also, similar diagrams (tree diagrams) exist for phases that show how elementary actions would be executed.

Figure 1. Fragment of scenario ontology.

FORCE RECUIREMENTS next step FULFILLED? variants

FORCE DEPLOYMENT

next-step HAVE CONTACT? variants

SEARCH

next-step variants CONTACT RECOVERED? FORCE GUIDANCE

variants

variants

next-step variants TAKE ABOARD next-step FORCE RETURN

REPORT

Figure 2. Diagram of rescue operation scenario.

INTELLIGENT GIS AND HARBOUR PROTECTION

195

5 Scenario implementation As mentioned in the introduction, conventional programming requires knowledge of exact algorithms. Complex processes yield complex algorithms. The entire algorithm of the complex spatial process consisting of other processes from different domains interacting together is complex in its own right and, in all probability, unknown beforehand. For this reason we rejected a conventional programming approach to scenario implementation and adopted a rule-based approach. Using this approach, we formulate comparatively simple rules governing distinct parts of entire processes and leave the organization of parallel execution and the interaction between different processes to the rule engine. Modern rule engines, such as Rete [3], are intended for such tasks. We devised three sets of rules: (1) ontology rules, a stable set of rules governing the common flow of scenario execution independently of specific elementary actions and the specificity of decisions; (2) generated rules, created prior to scenario execution from descriptions of specific elementary actions and decisions. The description of an elementary action consists of three parts, which we call the Begin rule, Repeat rule and End rule. The Begin rule is a short program (or script) in rulebased language describing what to do prior to action execution. The Repeat rule describes what to do during action execution. The End rule describes what to do when action execution is stopped. The theoretical basis is: 1. Every real action is generally continuous. 2. Computer simulation is a discrete process corresponding to a repetition of cycles with time interval ¨t during, which calculations are accomplished, conditions are checked and events are modeled. 3. Hence, in the general case, action execution consists of three phases (in the informal sense): initial state, changing state and examining the conditions of finalization, final state. 4. Any phase can be absent in some specific case. The description of a decision is unipartite: the decision rule contains a rulebased language script for selecting a given branch to continue the scenario depending on the fulfillment of certain conditions. There exists, also, a stable set of creation rules to help create generated rules before scenario execution. Ontology rules and creation rules are comparatively small sets, while ‘generated rules’ is a large set representing the majority of rules. An example of the Repeat rule can be found in Figure 3.

196

R.P. SOROKIN

Figure 3. Example of rule (moving object along the route).

6 Scenario development and use in harbour protection Visual integrated development environments (IDE) have recently become prevalent in software development. They can remarkably accelerate the development process. With ready-to-use components and drag-and-drop operation, it becomes virtually impossible to make an error in code because the code is generated automatically by the IDE. We accepted this approach to scenario development, especially since Protégé has the necessary features to implement it. In our case, lists of components make up phases and decisions in scenario diagrams, and actions in phase diagrams. Such lists can differ according to application domain. In fact, they are part of the corresponding ontology. See in Figure 1 the components of subclasses of the Action class. While creating an ontology of the harbour protection application domain, it would be very useful to design specific actions beforehand, such as: x x x x x

Creation of a navigating object (ship, boat, plane, helicopter, etc.) Movement of the object along a route Oil spill Fire Explosion, etc.

When the specific actions and decisions suitable for the application domain are designed and implemented (and corresponding Begin rules, Repeat rules and End rules are written) we can design different scenarios visually by dragging

INTELLIGENT GIS AND HARBOUR PROTECTION

197

components from the special palettes and dropping them on the scenario or phase diagram form. For the harbour protection application, scenarios can include: x x x x

The entrance and mooring of a cruise liner in a harbour Terrorist attack of a small fast boat on the liner Rescue operation Oil spill harbour protection operation, etc.

When all scenarios needed for the simulation of a complex spatial process or phenomenon have been developed, the corresponding rules are generated for these scenarios. They can be played and replayed for any purpose on different time scales by starting and running the rule engine. On Figure 4, the scenario of a collision and the resulting oil spill prior to harbour entrance is shown during execution.

7 Conclusion x Intelligent GIS can simulate complex spatial processes in the harbour protection domain. x Spatial processes can be described visually using scenarios. x Visual scenarios can be translated to rules automatically. x Rule engine can replay scenarios. x We integrated visual scenario development and environment replay, see .

Figure 4. Scenario of oil spill in harbour during execution.

198

R.P. SOROKIN

References 1. 2.

3.

Noy NF, Deborah L. 2001. McGuinness Ontology Development 101: A Guide to Creating Your First Ontology. Stanford University, Stanford, CA. Sorokin RP, Shaida SS, Smirnov AV. 2003. Intelligent geo-information systems for modeling and simulation – Harbour, Maritime and Multimodal Logistics Modeling & Simulation, Riga, Latvia, September 18–20, 2003. pp. 395–398. Forgy CL. 1982. RETE: A fast algorithm for the many pattern/many object pattern match problem. Artificial Intelligence, 19(1). pp. 17–37.

USING MODELING AND SIMULATION AS A FRAMEWORK FOR TESTING NEW SOLUTIONS DEVOTED TO SECURING GLOBAL FLOWS OF GOODS AND PEOPLE

AGOSTINO BRUZZONE1 AND CHIARA BRIANO2 1 MISS/DIPTEM, Italy 2 DIP Consortium, Italy

Abstract. This paper proposes an innovative initiative in the EU aim is to develop an interoperable simulation federation for analyzing security issues in Ports and Logistics Nodes. The goal of this initiative – GLOWS (Securing GLobal FLOWS of Goods and People) – is the development of a modeling and simulation (M&S) framework for resolving security issues in physical and ICT data infrastructures for managing people & goods flows, especially in Ports. GLOWS interdisciplinary models are designed to provide strong support for ports with the design and reorganization of people transportation and logistics. Indeed, the aim of GLOWS is to develop a simulation framework for assessing the impact of policies, data communication systems, innovative technologies, investments and reorganizations for increasing security and efficiency in global flows of people & goods in an international context. GLOWS integrates advanced research on policy, bioinformatics, Cyber Attacks on Transportation, Data Fusion Architecture, and M&S in Port Logistics. At the core of GLOWS is a new innovative simulation federation based on up-to-date technology (High Level Architecture), which has benefited from the input of Europe’s top experts. The GLOWS simulator is dedicated to the review, testing and certification of Ports Security Plans, Risk Assessments and Gaps Identification. The GLOWS federation supports the design and re-engineering of logistics procedures/infrastructures, and a definition of Standing Operation Planning for the preventive design of containment measures and partial or full recovery and continuity of operation in critical scenarios. GLOWS ideally provides very promising support for the design and evaluation of innovative data fusion architectures aimed at improving the security of logistics processes within ports. The GLOWS synthetic environment is expected to support training as per new security regulations (with dynamic links to real equipment and exercises). The GLOWS simulator is extensively tested in real Port case studies provided by partners (i.e. APV, IFF). The GLOWS initiative benefits from the support of a similar initiative, I-GERT [1], a USA NSF project involving several GLOWS partners that has established a research network on security for policy analysis, port operation M&S, data communication standards, and sensor & IT technologies. E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

199

200

A. BRUZZONE AND C. BRIANO

Keywords: Simulation, security

1 Introduction Security is becoming a fully integrated component of all logistics procedures. It poses a real challenge, especially in ports where different realities, such as passengers, cargo, town, etc., coexist. Commercial ports are extremely integrated systems, involving a large and complex number of entities dynamically interacting in an open environment 24 hours/day 365 days/year. Security issues within ports must be analyzed with the knowledge that all operative necessities and many critical aspects (i.e. recovery operations, integration with other bordering systems, etc.) are still under analysis. The September 11 terrorist attacks directed much attention to this problem, whereupon several regulations and standards promptly evolved, requiring the port network, as well as the ship operator, to introduce security controls and equipments into their processes. While the evolution of port security regulations and organizations is dynamically ongoing, there is a current demand for enabling technologies and innovative techniques that significantly improve infrastructure security without affecting the overall performance of logistics systems. This paper presents an approach to developing a simulation framework for identifying security requirements and the effectiveness and costs/benefits analysis of different solutions. The authors are currently establishing a consortium for GLOWS (Security for GLobal FLOWS of Goods and People) project development in the European Union, and are concurrently actively involved in I-GERT (Interdisciplinary Education and Research in Policy, Science and Technology for Securing Global Flows of People and Cargo) in the USA. Considering the expectations and goals of the European community from this point of view, it is possible to identify important strategic objectives such as: x The development of novel modeling/simulation techniques and synthetic environments for critical infrastructure protection in understanding ICT-related interdependencies, for the prevention and limitation of threats and vulnerabilities propagation, and for recovery and continuity in critical scenarios (FP6-IST-2004-2.4.3, Towards a Global Dependability and Security Framework, Strategic Objective No. 2). x The development of integrated interdisciplinary frameworks and related technologies for the provision of resilience, dependability and security in complex interconnected and heterogeneous communication networks and information infrastructures that underpin our economy and society (FP6-IST2004-2.4.3, Towards a Global Dependability and Security Framework, Strategic Objective No. 1).

USING MODELING AND SIMULATION AS A FRAMEWORK

201

2 Scientific and technological objectives of the GLOWS project and the state of the art The GLOWS initiative has established a project partnership among a number of academic, government institutions and companies in the EU that are already involved in equivalent initiatives with international and especially USA institutions. The GLOWS project is focused principally on ports, and its primary scientific and technological purpose is the development of modeling and simulation techniques for building an integrated environment for the management of transportation networks and information infrastructures in order to improve the security of global flows of people and cargo while minimizing the negative impact on the production industry’s performance. GLOWS expects to focus on the following goals. 2.1 DEVELOPMENT OF A MODELING AND SIMULATION INFRASTRUCTURE The use of M&S in considering the impact of security components on logistics/industrial facilities is more than justified by the need to analyze complex interactions between numerous factors. Simulation techniques could be employed to evaluate the impact of security requirements on people and cargo flow. Security equipment and procedures of logistics facilities can usually be grouped into two major classes: x Internal: security control of logistics flow x External: security control of external components The first class refers to all additional operations that must be carried out to ensure that goods are not dangerous (i.e. container scanning, gate controls, custom checks, etc.), while the second class refers to the fact that external entities (i.e. terrorists) might be introduced at certain points in the facility’s logistics flow (i.e. contamination of food along the supply chain) or may affect the efficiency of the entire infrastructure (i.e. attacks on a port/airport). It is interesting to note the fact that intensive flows of entities (i.e. people, materials, etc.) in logistics facilities have mutual influence and entity flows are a potential threat to other entity flows (consider port logistics as a pertinent example). Regarding security activity, intrusion models must also be used to control the potential interference between cargo and people entities in various areas of ports. In this area of application, M&S is already being used to study processes, performance levels and costs in regular operating conditions. These simulation models can be used to form an assessment of the impact that safety and security procedures may have on logistics throughput, and an evaluation of the role that human resources play in ensuring reliable surveillance and control during continuous and discrete monitoring (i.e. monitoring and personnel inspections for facility border surveillance). Such models enable managers

202

A. BRUZZONE AND C. BRIANO

and decision-makers to make a quantitative evaluation of the effectiveness of manmachine interactions during standard operations and emergencies (i.e. high and very high intrusion alert). Using these models for risk analysis purposes helps to identify best practices for emergencies related to relevant targets. These simulators can also be used to complete tests related to the effectiveness of different organizational models within critical structures, providing experimental results on the impact of the consequences of interfacing with public institutions (e.g. police patrols). In operative terms, the general experience obtained from simulation models helps to improve integration of automated control systems due to the opportunity of designing an architecture that combines information coming from different sources, and evaluating the impact resulting from the use of all possible alternative systems. Simulation tools can be used to carry out a comprehensive assessment of the system’s security weaknesses by modeling its constitutive parts and providing a means for verifying the performance level for the system as a whole during the simulation run. Considering that maritime port infrastructure simulators have to be developed and many different models need to be integrated, GLOWS expect to create an HLA (High Level Architecture: USA Department of Defence Regulation, IEEE Standard) federation of simulators. The GLOWS federation includes an extensive synthetic environment devoted to supporting effectiveness in Design of Facility, Security Plan Certification, Security Training, etc. GLOWS simulator execution and VV&A can provide a measurable objective for assessing the success of this aim. 2.2 SECURITY POLICY RESEARCH The current situation and trends of Communication Networks for People and Goods Flows, Logistics Infrastructures, Port Policies need to be analyzed by GLOWS simulation in order to aid decision makers to quantify the trade-off between achieving security goals and their attendant costs, as well as discovering improved procedures. Modeling and Simulation techniques can be used to develop a framework to evaluate and optimize the performance of security-related systems. These studies would support capacity planning of processing facilities and gauge the performance impact of various transit and security technologies, by estimating operating metrics (e.g., queue sizes, waiting times) resulting from a given traffic volume. Research should focus on biometric passports, advanced electronic submission of passenger name record (PNR) data and passenger manifests, and automated watch lists. This work would include an analysis of the necessary border infrastructure, gaps in the data being entered, limitations on interoperability of such data, potential theft of RF transmitted biographical and security data from passports, and security of digitized biometrics in government databases, as well as the development of passenger screening and baggage screening performance measures, along with an associated impact analysis of the impact on demand for air travel, when achieved and not achieved. For marine

USING MODELING AND SIMULATION AS A FRAMEWORK

203

ports, emphasis must be placed on security operations as applied to passengers and freights (especially containerized cargo). In fact, the GLOWS simulator also supports security system design/analysis in relation to different kinds of “external threats” and their interaction/impact on all port activities and processes, including material handling, data communication, human procedures, business processes, etc. Research focuses in detail on the container inspection procedure, risk factors for containerized cargo, standardized data submission, sensor technology and economics of security. Detailed reports of the project are expected to be submitted to the end user and designers (i.e. AVP, IFF) for the validation of the quality of achieved results. 2.3 BIOMETRICS ANALYSIS AND SURVEILLANCE RESEARCH AND MODELING When biometrics data (facial profiling, fingerprints, iris scans, etc.) for passengers is required, some difficulties may arise: x Passenger data must be compared with data already contained in a database. x Biometrics data have many false negatives and false positives, fingerprints and facial profiles change and can be forged or distorted. x Legal issues and privacy protection for guaranteeing citizens. x Biometric databases are not likely to contain the fingerprints and irises of terrorists. In addition, there are open issues in the data fusion of data sensors as well as behaviour identification. Nowadays, Computer Vision can work with motion analysis to extract semantic information from movements in human behaviour. This information could be automatically analyzed and translated into to a natural language. Based on the description, it would be possible to react in real time to anomalous actions or warn operators who are unable to control all the cameras installed in a port. Moreover, it is possible to synthesize a virtual sequence to simulate the flow of people or events in a situation of crowd panic. The creation of models to be included in the GLOWS simulation is the clear and measurable deliverable related to this aim. 2.4 CYBER ATTACKS ON TRANSPORTATION Information is essential to move both people and goods. Some information systems are often poorly secured against cyber attacks that could allow terrorists to disrupt systems directly or manipulate information that could create vulnerabilities or facilitate attacks. Information systems in the transportation sector are usually highly interconnected with other IT infrastructures, exposing them to additional vulnerabilities. Due to differences in cyber-security approaches among the hemisphere's developed and less developed countries (namely Latin

204

A. BRUZZONE AND C. BRIANO

America and the Caribbean, in which there is less frequent e-commerce owing to lack of development) there is no capability to accurately track attacks. This project includes a survey of IT infrastructure, a determination of where vulnerabilities lie, and the establishment of a cost effective program to address them. The creation of models to be included in the GLOWS simulation is the clear and measurable deliverable related to this aim.

3 Modeling and Simulation as a Support for the Development of a Security Framework Logistics, or flows of material are focal points in the global just-in-time economic environment, and security is usually considered just an additional cost for the industry. In GLOWS, security operations efficiency is improved by the modeling and simulation of transportation infrastructures, even capsizing the situation and introducing savings and novel efficiencies through the design and introduction of new technologies and procedures (i.e. by introducing safe data communications, effective management and new technologies, the material handling costs and crossing time within a logistics node, improving security). This approach guarantees the minimization of costs and increases competitiveness. Moreover, the flow of people, one of the bases of modern development, is positively affected by an increased security level with respect to personal rights and privacy regulations. GLOWS, owing to its cooperation with I-GERT, can contribute to establish standards in security procedures. I-GERT (Securing Global Flows of People and Cargo: Interdisciplinary Education and Research in Policy, Science and Technology) is a U.S. governmental research program (by NSF) involved in the security of FLOWS of people and goods. IGERT extensively cooperates with GLOWS in standardization activities on different aspects, such as, for instance, data submission, in order to implement pre-screening cargo procedures. Prescreening cargo is one of the most promising solutions for increasing security while reducing the time required for inspection procedures. It is thus possible to optimize inspection procedures. Considering main ports worldwide, this procedure requires high security and standardization. Different scenario are modeled and simulated in GLOWS to contribute to the standard definition. The aim of GLOWS is to develop novel modeling & simulation (M&S) techniques and a synthetic environment capable of supporting the mentioned applications. This approach enables the protection of critical infrastructures such as ports, which are in fact one of the most relevant nodes in the communication network to be protected today. The location of these infrastructures could have a strong impact in terms of vulnerabilities propagation (i.e. the port of Venice is one of the most important world cultural sites and top tourist destinations worldwide). In this complex framework, which encompasses many different components and interactions, GLOWS has the capability to evaluate possible scenarios, to establish best practices and operative procedures and plans.

USING MODELING AND SIMULATION AS A FRAMEWORK

205

In addition, GLOWS can be used to understand ICT-related interdependency. Today, security is strictly connected to information and communication technologies in particular, biometrics techniques for the identification of people, mobile wireless GPS-based devices that are exposed to attacks by cyber menaces. In order to understand interdependency, pre-screening cargo procedures should be considered as strictly inter-connected. Additional challenges are related to scenario assessment and improving the behavioural modeling capabilities. Moreover, GLOWS addresses the issue of guaranteeing recovery and continuity of operation in critical scenarios. In ports, this is traditionally a very complex issue, because it is impossible to keep the system down for long periods of time. Owing to developed solutions, it is possible to suggest rapid responses to different situations and eventually establish procedures to recover parts of the infrastructure while others remain on a high level of security alert. GLOWS incorporates the following research activities for the identification of needs and gaps in the following areas: x Biometrics research, namely computational biomedicine, computer vision, computer graphics and deformable algorithm x Cyber attacks on transportation x Wireless non-human entities as elements of cyberspace Similarly, GLOWS tackles important facets of freight handling, with special attention to passenger and container inspection procedures: x x x x

Queuing models and inventory control analytical and simulation methodologies Risk factors for containerized cargo Data mining and analysis on people and material flows – Bayesian methods Standardized data submission

The GLOWS approach takes into account all relevant stakeholders in the security of cargo and people. Researchers and companies involved in GLOWS contribute different points of view on performance and security issues for port material, data and people flows. The GLOWS simulator was designed as a federation of interoperable distributable models integrating legacy systems, new components, and real systems. The simulator can be used for understanding ICT-related interdependencies, prevention and limitation of threats and vulnerabilities propagation in the complex context of ports in relation to passengers and material flows. The international dimension of GLOWS is guaranteed by its cooperation with IGERT, a similar project funded by US government, which is currently designing an international network for the interoperable modeling of logistics networks. GLOWS activity is integrated in existing NoE (Network of Excellence) and IP (Integrated Project) in the field of security. To date, there are different security related NoE and IP, such as TERANOVA, EJUSTICE, PRIME, MAGNET, INSPIRED, OASIS, BIOSECURE, but none specifically addressed to flows of goods and people and their interdependency with the social economic

206

A. BRUZZONE AND C. BRIANO

infrastructure. GLOWS interacts with such initiatives in order to add its contribution to the research community.

4 Contributions to standards There are standards to which GLOWS can effectively contribute based on its capability to investigate and research needs on the area (a priori analysis) and, at the same time, to experiment in the synthetic environment with different possible solutions (a posteriori analysis). Firstly, GLOWS has a strong impact on the standardization of data transmission in logistics. For instance, there is currently a set of open issues in pre-screening procedures for guaranteeing homogeneous procedures and international standards in data submission, custom control and material handling among European member states, the USA, as well as internationally. In much the same way, these standards are a critical concern in bioinformatics, in which legal issues, regulation, fidelity and credibility, cost impact and real/data infrastructure solution compatibility are still very open issues. GLOWS simulation enables a detailed investigation of the impact and effectiveness of different approaches both in terms of risk assessment and overall cost performances. Some of the working prototypes produced by the project consortium for experiments and investigations may be released as open source software in order to guarantee the spread of proposed technologies to wide community of potential users. This is usually one of the first steps in the generation of de-facto standards from project related deliverables. Furthermore, the project incorporates a number of existing standards. Consequently, contributions, adaptations and ideas related to the producing standard bodies are foreseen. Since its introduction in 1996 as an open, interactive process for developing and revising the HLA (standard and technology specifications for distributed simulation), the simulation community has fostered the evolution of the High Level Architecture. In Europe, this technology was introduced especially by DIPTEM – Genoa University, Magdeburg University. Areas where the consortium may contribute are: x x x x

Federation Object Models in goods and people flows. Web-enabling of the HLA Interface Specification. Mixed HLA and Web-services based on service-oriented architectures (SOA). The set of APIs for inter-organizational messaging can extend the construction of secure SOAP message exchanges.

Although several proposed standards such as BPEL4WS, XLANG, WSFL, and XPDL exist for business process definition and interoperability, their development is currently driven mostly by concrete applications or commercial interests. These process modeling techniques combine expressiveness, simplicity, and formal

USING MODELING AND SIMULATION AS A FRAMEWORK

207

semantics, and their use in process definition and interoperability could be a major force in the development of a unified standard. The results of this project can be injected into relevant standardization boards (like W3C or IEEE), thus making it available to the public.

5 Conclusions It is evident that M&S provides a very fundamental support to Port Security in different application areas: x x x x x

Architecture/Infrastructure/Procedure Design Training Testing New Solutions Standing Operation Planning Operative Support & Emergency Management

In order to achieve results in these areas, GLOWS approaches the problem by customizing the solutions in to comply with different existing needs: in fact, it is very unlikely that one single system will be able to deal with all of these different aspects. It is evident today that simulation allows security issues to be transformed into opportunities to redesign logistics operations by testing innovative techniques and enabling technologies that are able to support this process. Alternatively, ports are faced with the necessity to competitively improve infrastructure design, control system architecture and operation management in considering security requirements. In this context, M&S provides quantitative support both to logistics operators and to the public community in order to guarantee solutions for reducing the impact of new threats within reasonable cost and without losing operative efficiency. Considering port evolution and competition in the market, it is reasonable to expect that, in the future, the availability of effective models will present a strategic advantage by enabling the transformation and reduction of the costs/impact of new regulations, as well as improvements in overall security: this will guarantee that final destinations will receive goods/people through a “safe” security system at competitive costs, consequently providing additional motivation to promote these ports.

Acknowledgments The authors thank their colleagues from the I-GERT project, and especially Tayfur Altiok, for a fruitful cooperation and sharing of initiative to support security in ports through innovative technologies.

208

A. BRUZZONE AND C. BRIANO

References 1. 2. 3.

4.

5.

6.

7.

8.

9.

10.

11.

12. 13. 14.

Altiok T. 2005. I-GERT. NSF Report/Rugters University, Piscataway, NJ. Bruzzone AG. 2005. GLOWS. DIPTEM Press Genoa, Italy. Bruzzone AG, Reverberi A, Rocca A, Brandolini M and Massei M. 2005. Security Management Systems in Logistics: An Innovative Approach in Solution Design. Proceedings of ASTC2005. Bruzzone AG, Brandolini M, Briano C and Petrova P. 2004. Poly-functional Intelligent Agents for Computer Generated Forces. Proceedings of Wintersim2004, Washington, DC, December. Bruzzone AG, Briano C, Bocca E and Simeoni S. 2004. Advanced Systems for Logistics Network Design Based on Interdisciplinary Approaches. Proceedings of I3M2004, Bergeggi, Italy, October. Bruzzone AG, Brandolini M and Viazzo S. 2004. Massive Training Based on Virtual Reality Equipment Applied to Logistics and Heavy Haul Tracking. Proceedings of SCSC2004, San Jose, CA, July. Bruzzone AG, Orsoni A and Giribone P. 2003. Fuzzy and Simulation Based Techniques for Industrial Safety and Risk Assessment. Proceedings of ICPR, Blacksburg, VA, August 3–7. Bruzzone AG, Brandolini M, Briano C and Procacci V. 2001. FLODAF: Fuzzy Logic Applied to a Multi-Sensor Data Fusion Model. Proceedings of FLODAF2001, Montreal, August. Bruzzone AG, Mosca R, Revetria R and Rapallo S. 2000. Risk Analysis in Harbour Environments Using Simulation. International Journal of Safety Science, Vol. 35, pp.75–86. Bruzzone AG, Cotta G and Cerruto M. 1997. Simulation & Virtual Reality to Support the Design of Safety Procedures in Harbour Environments. Proceedings of ITEC97, Lausanne, April 22–25. Merkuryev Y, Bruzzone AG, Merkuryeva G, Novitsky L and Williams E. 2003. Harbour Maritime and Multimodal Logistics Modeling & Simulation 2003. DIP Press, Riga. Marine Log. 2004. Jitters as ISPS/MTSA deadline nears. Simmons-Boardman, Omaha, NE. Marine Log. 2004. MTSA and ISPS: final rules issued. Simmons-Boardman, Omaha. Mosca R, Bruzzone AG and Costa S. 1996. Simulation as a Support for Training Personnel in Security Procedures. Proceedings of Military Government and Aerospace Simulation. New Orleans, LA, April 8–11.

FUSION OF INFORMATION FROM DISPARATE ELECTROOPTICAL IMAGERS FOR MARITIME SURVEILLANCE

MICHAEL J. DeWEERT BAE Systems Spectral Solutions LLC, Hawaii

Abstract. This work presents an overview of a portion of the sensor-fusion work being done at BAE Spectral Solutions LLC. The emphasis is on imaging technologies with near-term applications for maritime homeland security. General categories of sensor fusion are outlined, followed by a more in-depth discussion of work being done at two BAE North America facilities on systems for maritime security. Keywords: Homeland security, maritime security, sensor fusion, electro-optics, infrared imaging, automatic target detection

1 Introduction BAE Systems Incorporated has grown to be one of the top ten suppliers to the U.S. Department of Defence – dedicated to solving our customer’s needs with both highly innovative and leading-edge solutions across the Defence electronics, systems, information technology and services arenas. Employing 40,000 people, BAE Systems Incorporated operates in 30 States, the District of Columbia, and the UK, generating annual sales of more than $11 billion. BAE Systems designs, develops, integrates, manufactures, and supports a wide range of advanced aerospace products and intelligent electronic systems for government and commercial customers. This paper summarizes a small portion of the BAE Systems Incorporated work on maritime security, with emphasis on visible and infrared optics and the components required for developing systems for information exploitation. This paper is structured as follows: a brief discussion of fusion levels is presented, followed by summaries of the work of two of BAE Systems’ subsidiaries, BAE Systems Advanced Information Technologies and BAE Systems Spectral Solutions. The full range of BAE Systems’ work in this is beyond the scope of this paper – only a manageable fraction of BAE North America activity will be

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

209

210

M.J. DeWEERT

discussed. The interested reader is encouraged to contact the author or other BAE Systems Incorporated personnel for more information.

2 Fusion levels As will be noted from reading the papers by other authors in this volume, the JDL (Joint Directors of Laboratories) have developed a categorization scheme for data fusion. While the JDL categorization is still evolving, its main elements are summarized in six (0–5) fusion levels. 2.1 ASSOCIATION AND ESTIMATION Level 0 – Organize: This is the initial processing accomplished at or near the sensor that organizes the collected data into a usable form for the receiving system or person. 2.2 REFINE OBJECT Level 1 – Identify/correlate: Takes new input and normalizes its data; correlates it into an existing entity database, and updates that database. This level tells you what is there and can result in actionable information. 2.3 REFINE SITUATION Level 2 – Aggregate/resolve: Aggregates individual entities or elements, analyzes those aggregations, and resolves conflicts. This level captures or derives events or actions from the information and interprets them in context with other information. This level tells you how entities are working together and what they are doing. 2.4 REFINE THREAT Level 3 – Interpret/determine/predict: Interprets enemy events and actions, determines enemy objectives and how enemy elements operate, and predicts future enemy actions and their effects on friendly forces. This is a threat refinement process that projects current situation (friendly and enemy) into the future. Level 3 Fusion tells you what it means and how it affects your plans.

FUSION OF INFORMATION FROM DISPARATE ELECTRO-OPTICAL

211

2.5 REFINE AND MANAGE PROCESS Level 4 – Assess: This level consists of assessing the entire process and related activities to improve the timeliness, relevance and accuracy of information and/or intelligence. It reviews the performance of sensors and collectors, as well as analysts, information management systems and staffs involved in the fusion process. This process tells you what you need to do to improve the products from fusion level 0–3. 2.6 VISUALIZATION Level 5 – Visualize: This process connects the user to the rest of the fusion process so that the user can visualize the fusion products and generate feedback/control to enhance/improve these products. The work presented in this paper addresses levels 0, 1, and 5. These levels can be efficiently automated or computerized. Some of the automatable tasks accomplished in a maritime surveillance system are: x Multispectral Imaging Frame Co-registration. This requires precise coregistration of disparate bands that may or may not be collected from the same focal plane. For example, LWIR (long-wave infrared) and VNIR (Visible through Near-Infrared) are typically collected with separate cameras. Once the co-registration has been completed, detection of objects of interest is performed, via spatial-spectral matched filtering or other methods, followed by classification, localization, and tracking. x Temporal Co-Registration. This task is performed when images are collected over time, with co-registration being used to reject clutter. Examples of clutter rejected this way include stationary objects when targets are known to be moving, or objects which change shape over time, such as ocean whitecaps. The temporal registration steps are followed by classification and tracking functions analogous to those discussed earlier. x Post-Detection Fusion. In many cases, the sensors are so disparate or are operated in such a way that rather than “fuse, then detect,” the system must “detect, then fuse and confirm.” Detection occurs in each sensor mode independently, and then detections are correlated. This kind of processing might apply, for example to the fusion of acoustic-and optical detection modes. The means for accomplishing this are numerous and heavily application dependent, including, for example, N of M processing, Bayesian classification and Dempster-Shafer techniques.

212

M.J. DeWEERT

3 Research at BAE Systems AIT BAE Systems AIT (Advanced Information Technologies), headquartered in Burlington, Massachusetts has key programs in Wide-Area Automated Surveillance (WAAS), Intrusion Detection in the Littoral Zone, and harbour surveillance (SeeCoast). The WAAS system, developed in collaboration with a major regional airport, implements multi-camera, wide-area continuous tracking in high traffic environments, such as the airport shown in Figures 1 and 2. A key capability of the software is the ability to learn what constitutes normal traffic patterns and automatically flag deviations from normal.

Figure 1. Computerized tracking of persons and baggage in an indoor pedestrian area.

Vehicle traffic Pedestrian traffic

Figure 2. Automated tracking of pedestrian and vehicle traffic outdoors.

FUSION OF INFORMATION FROM DISPARATE ELECTRO-OPTICAL

213

BAE Systems AIT’s project on Intrusion Detection in the Littoral Zone fuses VNIR, LWIR and MWIR (mid-wave infrared) sensors, with the following program objectives: x x x x x

Small target detection sensor Fused VNIR, MWIR, LWIR imagery Target learning, search & cue Extended operating conditions Cue active SWIR (short-wave infrared) imager for operator

Key technologies used to accomplish the above goals include: neural models of colour vision, in-field target learning & search, multi-hypothesis target tracking, dual-band MWIR/LWIR focal-plane arrays, and active SWIR imagers for covert surveillance. Application of the technologies entails fused sensor-processor systems for littoral surveillance & search, in port, periscope & ship-mounted configurations. Some of the technologies are illustrated in Figure 3 and 4, which describes how we can apply multi-sensor image fusion and target learning & detection methods to the littoral zone. The imagery shown in the upper panels are from a multisensor data collection of small watercraft (kayaks) collected around sunset using daytime VNIR, MWIR, and LWIR sensors. Imagery from the three complementary sensors is combined using our opponent-colour image fusion techniques, creating the colour image shown below. In this image, the important information from each sensor is retained and displayed to the user as a single colour image. The three kayaks are easily seen near a buoy (triangular object in scene), due to their strong brightness and colour contrasts compared to the background. Using our on-line target learning and search methods, a user can teach the system to detect the small boats by selecting examples of the boats in the context of the background. The detections highlighted in the lower-left show all three kayaks being detected with no false alarms (although the image is cropped there were no false alarms throughout the rest of the image). These on-line learning methods are robust to changing operating conditions. These detections can be used to alert an operator or to feed a multi-hypothesis tracker. Relevant applications include port monitoring, littoral surveillance, and intrusion detection. The system could be mounted on land at a port, on a ship, or in a periscope.

214

M.J. DeWEERT

VNIR

MWIR

LWIR

Fused Kayaks

Detections

Figure 3. Multisensor image fusion and target learning in the littoral zone.

Figure 4. Images from the SeeCoast user interface. SeeCoast generates an all-source track picture and provides automated alerting.

FUSION OF INFORMATION FROM DISPARATE ELECTRO-OPTICAL

215

The third key program of BAE Systems AIT Key Program is SeeCoast, developed for the USCG (United States Coast Guard) with technologies sponsored by DARPA and HSARPA. The SeeCoast prototype system enhances USCG Project Hawkeye (radar and AIS tracking system). SeeCoast provides multi-sensor data fusion and video recognition for improved situational awareness and security in ports, including detection of anomalous watercraft events and automated sensor cueing. The SeeCoast program is building port situational awareness capability. It supports an understanding of the coastal and port security situation by: x Using multiple sensors, including video, radar, and AIS sensors x Tracking, fusion, behaviour analysis x Scripted & learned indications & warnings An example of the system and software flow is shown in Figure 5. Other Data Radars

AIS Data

Radar feed

Radar Tracking

Radar Tracks

Video feed

Video Tracking

Video Tracks

Cameras

Camera Tasks

Camera Mgr

Alert Generation

Operator Alerts

All source tracks

Display

Track Fusion

Target Locations

Figure 5. Data and processing flows in SeeCoast.

4 Work at BAE Systems Spectral Solutions LLC BAE Systems SSL (Spectral Solutions LLC), headquartered in Honolulu, Hawaii, develops systems and solutions for a variety of ISR (intelligence, surveillance and reconnaissance) tasks, including maritime surveillance and search and rescue. The difficulty of finding and classifying objects amid marine clutter is shown in Figure 6. Waves on the sea surface create glints and whitecaps that can produce a confusing, dynamic environment. The difficulty of detection increases rapidly as wave height increases and target size decreases. In the case of search and rescue, 10% of USCG searches utilize 80% of time and cost, due to difficult visual search condition and small targets, such as a PIW (person in the water) with no signalling device.

216

M.J. DeWEERT

Figure 6. Illustration of clutter in a marine scene. Even in clear weather with good visibility, the glint and whitecaps create difficulties for maritime imaging tasks. As the altitude increases (in this case from 500 to 1,000 feet), the search rate increases, but detection probability decreases.

Figure 7. Test objects for the BAE Systems SSL ESAR demonstration.

FUSION OF INFORMATION FROM DISPARATE ELECTRO-OPTICAL

217

The goal of the BAE-SSL ESAR (Enhanced Search and Rescue) demonstration is to reduce the false alarm rate while maintaining a high detection probability for small targets in seas with wave heights over 3 feet (1 m). Reducing false-alarm rates reduces search time for small targets, cost and operator fatigue. The experiments employed HSI (hyperspectral imaging) and MSI (multispectral imaging) sensors operating in VNIR bands, as well as a bolometric LWIR camera. A variety of targets, shown in Figure 7, were imaged from altitudes ranging from 500 to 1,500 feet. The experiments were conducted in waters near Hawaii, which provided a wide variety of wave heights under similar atmospheric and thermal conditions.

5 Hyperspectral imaging for enhanced search and rescue The utility of hyperspectral imaging derives from the high degree of correlation of clutter spectra over wide areas of the ocean. Glints, whitecaps, and the average “non-glinty” background spectra are consistent and predictable, allowing automated rejection of clutter. In addition, objects of interest to search and rescue operators have spectra which are readily distinguishable by HSI and MSI instruments, even though they may not be distinct to unaided human vision, which has three highly-overlapping spectral bands, as is illustrated in Figure 8. 0.04

4

0.03 Relative Intensity 3 0.02

5 2 Background

1

Whitecaps Glint Raft Boat

0.01

0 400

450

500

550

600

650

700

750

Wavelength (nm)

Figure 8. Spectra of various target and clutter entities for the ESAR experiments. Even the boat, a white object with a blue canopy, is spectrally distinct from the ocean background.

218

M.J. DeWEERT

Demeaned Data (V) Matrix Product

Input Data Inverse Covariance Matrix ( 6 )1

Threshold and Cluster Anomalies

Figure 9. Detection method using HSI (Hyperspectral imaging).

The hyperspectral data were analyzed in 150-line blocks, which yielded a hypercube datacube having two spatial dimensions, plus a dimension for each spectral band. For each location on the sea surface, we simply calculated

A

V T u 6 1 u V

(1)

Where 6 is the spectral covariance matrix of the marine background, and V is the target-background signal excess. And set a threshold on A to have high PD (Probability of detection). The detections were clustered detections based upon probable target size, and size thresholds applied. The result is illustrated in Figure 10, which shows detection ellipses drawn on an operator’s display. The ellipses include far fewer false alarms than an unaided eyes-only search would, reducing the number of events requiring human-operator inspection to a few. Because the consequences of missed detections are severe, the detection thresholds were set to insure high PD.

Figure 10. Hyperspectral data hyper-cubes rendered in false colour. Left: image with autodetection ellipses. Right: magnified images of target detections.

FUSION OF INFORMATION FROM DISPARATE ELECTRO-OPTICAL

219

6 Wave and glint removal via spectro-temporal fusion of framing MSI BAE Systems Spectral Solutions has also developed techniques for passive (nonlidar) bathymetry and submerged-target classification using HSI and MSI systems. The sea surface not only causes obscuring glints and reflections, but refraction through surface waves causes distortions in the images of subsurface objects. In addition, breaking waves create foam, which also obscures subsurface objects. These three effects: glints, distortion, and foam, are all dynamic, evolving on time scales ranging from milliseconds to several seconds. By acquiring multiple looks at the sea surface, we can capitalize on the dynamics to obtain clear views of subsurface objects, as is shown in Figure 11. Combining the spatio-temporal fusion processing that removes wave clutter with multi-spectral processing to remove glints and classify objects provides a robust capability for subsurface imaging. Before Glint Removal After Glint Removal

(a) Glint Patterns observed on Digital Video Camera

(b) Multi-Spectral Still Image

(c)

Figure 11. (a) Glint patterns observed from airborne digital video camera; (b) image synthesized from multi-spectral image of the same area; (c) the area sown in (b), after operation of spectrotemporal glint-removal algorithm.

7 Infrared imaging in maritime searches While spectral processing can remove glints from VNIR mages of the ocean, an alternate strategy is to image the sea surface in infrared bands, as shown in Figure 12. Because infrared radiation doesn’t penetrate seawater, only surface objects or reflections of above-surface objects, appear in maritime IR images.

220

M.J. DeWEERT

Figure 12. Simultaneously collected RGB digital camera (left) and IR (right) images. Targets deployed (from top right to bottom left) were (1) 7-m boat, (2) a partially inflated life raft, and (3) 3-m paddleboard.

Figure 13. LWIR data collected over surfers in breaking waves. The SNR for these subjects range from 8.8 (farthest left) to 18.5 (bottom right) dB. With improved processing and fusion with VNIR data, unambiguous automated detection is possible.

Furthermore, the strong absorption of certain IR bands by atmospheric moisture greatly reduces the solar glint and cloud reflections in those bands, so that surface objects with emissivities or temperatures that differ from the ocean stand out clearly, as shown in Figure 12. Even individual swimmers can be easily distinguished from the sea surface, as shown in Figure 13. BAE Systems Spectral Solutions has demonstrated that enhanced search and rescue (SAR) systems using multispectral discrimination, multi-sensor fusion, and computer-aided image processing, can increase target SNR and improve clutter mitigation. The implications of this for SAR over rough water include decreased search times and increased search rates, with improved detection probabilities. This translates directly into more lives saved by the intelligent application of information-fusion technologies.

FUSION OF INFORMATION FROM DISPARATE ELECTRO-OPTICAL

221

8 Conclusions BAE Systems is building on experience in imaging hardware, algorithms, and systems integration. The fusion of visible and infrared and visible-band multispectral and hyperspectral data is an example of the work we are doing with maritime-surveillance and homeland-security applications. This paper has presented work by BAE Systems AIT and BAE Systems SSL, just two of the many BAE Systems units in North America engaged in developing technologies and solutions for maritime security and ISR. We look forward to further progress in providing leading-edge technologies to serve our customer’s needs.

AUTOMATIC VIDEO SURVEILLANCE OF HARBOUR STRUCTURES

LAURO SNIDARO, GIAN LUCA FORESTI AND CLAUDIO PICIARELLI Department of Mathematics and Computer Science, University of Udine, Italy

Abstract. Since 9/11 naval structures and facilities have been recognized as potential critical targets of terrorist acts. Consequently, harbour protection has become a debated topic due to the difficulties inherent in safeguarding such a complex environment. In this paper we will discuss an automatic multi sensor surveillance system designed to detect suspicious events in the harbour area and focus the attention of a remote human operator. Using colour and infrared cameras, the system is able to detect if someone is trying to cross an unauthorized area or if a person or vehicle performs an anomalous pattern of activity. This automatic surveillance system aims to supersede classical CCTV systems by providing effective assistance to the human operator by invoking his attention only when needed. The system can also be used to automatically tag and annotate video streams in order to perform a posteriori automatic query-based retrieval of desired events. Keywords: Video surveillance, event detection, trajectory analysis

1 Introduction Harbour security is the last line of Defence before possibly dangerous cargo is brought inside the homeland. It is therefore important to establish efficient and strict control procedures to check incoming cargo. Nonetheless, terrorist activities may be underway in the port area, despite such controls. Also, facilities and machinery may be the target of sabotage actions. Surveillance of harbour activities is therefore a primary concern, but a daunting goal to achieve by classical means (i.e. closed circuit TV). Monitoring performed by humans could be, in fact, infeasible given the complexity of the domain (many persons and vehicles operating in a huge area) and may lead to high missed alarm rates. Automatic surveillance systems [3, 6, 7] could be a solution to this problem. The main functionalities they provide in the harbour domain are access control and E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

223

224

L. SNIDARO, G.L. FORESTI AND C. PICIARELLI

situation assessment. In the former case, these systems can detect persons or vehicles approaching sensitive areas. The latter case implies a more thorough understanding of the activities underway in the monitored area. In this regard, surveillance systems may assist in recognizing suspicious behaviours that differ substantially from known patterns. In both cases, the system can generate warning/alarm messages for a remote operator only in relevant situations. The operator’s duty is therefore focused only on decision making. Other functionalities such as the automatic storing and indexing of significant events can be provided by the system. This paper will give a brief introduction to automatic video surveillance focusing on how activities in the monitored area can be modeled through the trajectories of moving objects, how common patterns may be automatically learnt by the system, and how anomalous activities may be detected [8, 11, 12].

2 System overview The general system architecture is shown in Figure 1. It is based on multisensor input data and provides scene understanding capabilities oriented at either focusing the attention of human operators or generating an alarm automatically. Image sequences acquired from multiple cameras are processed at a low system level to detect the presence of moving objects in the monitored scene. Detected objects are then tracked to compute their trajectories and to predict their movements in the scene. Finally, at the highest system level, detected objects are recognized and, in some cases, their behaviours are also analyzed to verify whether they are normal or potentially dangerous. Typically, a network of colour and infrared cameras is employed to allow wide area monitoring and 24-hour operation. CAM 1

CAM k

CAM N

OBJECT DETECTION AND RECOGNITION

LOCALIZATION AND TRACKING

EVENT UNDERSTANDING

Figure 1. General architecture of the surveillance system.

AUTOMATIC VIDEO SURVEILLANCE OF HARBOUR STRUCTURES

225

2.1 OBJECT DETECTION AND RECOGNITION Change detection (CD) is performed on two digitized images, I1(x,y) and I2(x,y), yielding a binary image B(x,y) as output. Frame by frame methods consider two successive images of the sequence, while frame to background methods use the current image and a background image, which represents the monitored scene without moving objects. Several CD methods have been developed in the last years. A recent review of the most significant methods can be found in [8]. The basic algorithm computes the absolute difference between the pixel intensities of the input images and thresholds the resulting difference image:

­0 B(x,y) ® ¯1

if D(x,y) < Th otherwise

0 d Th d 255

(1)

where Th is a threshold that may be determined automatically [8]. The output of the change detection algorithm is processed by a focus of attention procedure that pinpoints moving objects with bounding boxes (Figure 2). An object is defined as a set of connected components and is called a blob.

Figure 2. Detection of moving vehicle on the wharf.

Features are then extracted from detected objects and fed to a classification algorithm (e.g. a multilayer perceptron neural network) [9]. 2.2 OBJECT LOCALIZATION AND TRACKING The goal of the object localization and tracking module is to estimate the position and trajectory of the objects in each image frame. In a multi-camera system, if the target is detectable in more than one camera view field, correspondence can be established in a common coordinate frame (e.g. the ground plane) in order to locate the same target in different views; if the cameras are not synchronized, then temporal correspondence must also be estimated. The system needs to maintain tracks for all objects that exist in the scene simultaneously. For each sensor, the

226

L. SNIDARO, G.L. FORESTI AND C. PICIARELLI

system can perform an association algorithm to match the current detected blobs with those extracted in the previous frame. A number of techniques are available, spanning from template matching to feature matching [4] to more sophisticated approaches [13]. Generally, a 2D top-view map of the monitored environment is taken as a common coordinate frame [5]. There are many matching algorithms available in the literature for dealing with the multi-target data assignment problem – especially in the presence of persistent interference: Nearest Neighbour (NN), Joint Probabilistic Data Association (JPDA) and Multiple Hypothesis Tracking (MHT). The choice depends on the particular application. Detailed descriptions and examples can be found in [1, 2, 4] and a recent survey on tracking algorithms can be found in [14].

3 Trajectory clustering In order to make a high-level analysis, objects trajectories must be grouped together in a few distinct classes representing trajectories with a similar structure. This process is called clustering. Clustering allows high-level modules to work with a few models of the common paths detected by the system, instead of dealing with a large amount of trajectories whose information content, taken singularly, is not very meaningful. A strict requirement of the system is that it must work on-line as data is acquired. Many clustering algorithms are described in the literature but they are often off-line systems in which a first phase of simple data collection is followed by a second step of off-line data analysis. Since the proposed system should detect anomalous events in real time, this approach is not an option and the clusters must be constantly built and kept updated while the system is running. The definition of a trajectory clustering algorithm must involve three major steps: x Choosing a suitable way to represent trajectories and clusters x Defining a distance measure between clusters and trajectories x Defining how a cluster is updated with the data of a trajectory that matches it

3.1 REPRESENTING TRAJECTORIES AND CLUSTERS Trajectories are represented in the simplest possible way, as a list of map coordinates of the object acquired at fixed interval times. Each trajectory Ti is thus represented in the form Tk = {tk1, …, tkn} where tki = (xki, yki) where (x,y) are the object’s coordinates as calculated by the mapping module.

(2)

AUTOMATIC VIDEO SURVEILLANCE OF HARBOUR STRUCTURES

227

Clusters are modeled in a similar way, with additional information on local variance: a cluster Ck is represented as Ck = {ck1, … , ckm}

where cki = (xki, yki, ıki),

(3)

where ıki is a local variance. Note that this explicit modeling of trajectories and clusters is possible because the system works on-line and it must keep in memory only the clusters and the developing trajectories. An off-line system could not generally afford such an expensive representation of trajectories because it would lead to a large amount of data in the data collection step. This is why many offline systems use approximation techniques to represent trajectories, most notable of which being vector quantization [10]. Figure 3 shows a graphical example of the model of a cluster.

Figure 3. Representation of a cluster: dots are the main trajectory, circles represent local variance.

3.2 DISTANCE MEASURE AND CLUSTER UPDATING In order to check if a trajectory matches a cluster, a distance measure between trajectories and clusters must be defined. We define this distance as the mean of all the normalized distances of trajectory elements from the nearest cluster element inside a proper temporal window. Formally, the distance between the trajectory T={t1 … tn} and the cluster C={c1 … cm} is defined as: D(T, C)

1 n ¦ d(ti, C) ni1

(4)

where § dist(t i , c k ) · , ¸ k  { ¬(1  į)i¼… ª(i  į)iº} d(t i , C) min¨ k ¨ ı k ¸¹ ©

(5)

dist(ti,ck) is the usual Euclidean distance and į is a parameter that defines the width of the temporal window. Note that the proposed measure is not necessarily a distance in the mathematical sense of the word, it is only an intuitive measure related to the dissimilarity of a trajectory and a cluster.

228

L. SNIDARO, G.L. FORESTI AND C. PICIARELLI

Finally, there is the need for a set of equations to update a cluster with the data of a trajectory. If the trajectory element ti = (x’,y’) matches the cluster element ck= (x,y,ı), the following update formulas are used: x=(1-Į)x + Į x’ y=(1-Į)y + Į y’

(6)

ı=(1-Į) ı + Į[dist(ti,ck)]2 A cluster is thus a dynamic approximation of the position and variance of the matched trajectories, with an exponentially decreasing weight for older trajectories.

4 Building trees of clusters In order to make a high-level analysis of the trajectory data, we propose a method for the detection of common prefixes in groups of trajectories. Common prefixes are useful because they will lead to a very simple handling of predictions about the future movements of an object. Each group of trajectories is thus split into a set of clusters representing the prefixes that trajectories have in common and the clusters are organized in a tree structure, as shown for example in Figure 4. c1

c4

c2

c1

c2

c4

c3

c3

a

b

Figure 4. (a) Trajectories and their clusters, where c1 represents a common prefix for two distinct groups of trajectories; (b) the corresponding trees represent relations among clusters.

When a trajectory is initially detected, its distance from the root nodes of existing trees is computed, and if the trajectory is sufficiently near to one of them a match is found. In this case, the matched cluster is continuously updated with the incoming data. If a match is not found, a new cluster is created. If a trajectory is updating a matched cluster, their relative distance is continuously monitored to detect if the trajectory is exiting from the matched cluster. The trajectory may exit from the cluster in the proximity of the cluster’s end, and in this case a new matching step is performed in order to search for a match among the children of the just-left cluster, otherwise a split is performed, which consists in breaking the cluster into two parts and preserving the common prefix property.

AUTOMATIC VIDEO SURVEILLANCE OF HARBOUR STRUCTURES

229

5 High-level analysis As previously stated, the final aim of a video surveillance system should be to achieve semantic understanding of the activities taking place inside the monitored environment in order to detect and classify events, automatically finding those of interest to the system. Anomalous events can be defined in two different ways: x By explicit modeling x By probabilistic definition The explicit modeling approach is about having a complete description of what an anomaly is. Anomaly detection is thus a matching process among identified activities and known patterns of anomalies (note that the patterns do not need to be explicitly defined by a human operator; they could, for example, be obtained by the unsupervised learning process of an auto-associative neural network). The probabilistic approach is simpler and is based on the assumption that regular events happen most of time, while anomalies are uncommon activities. In this approach, anomaly detection is performed by calculating the probability of an event and searching for events with low probability. The trajectory clustering method proposed in this paper is particularly useful as a support for probabilistic anomaly detections, because it is simple to add probability information to the trees of clusters. If a counter is associated to every cluster, counting how many times the cluster was updated, the probability of going from a parent node p to a child node c is defined as nc nk where nk is the

¦ k

counter of the k-th child of p. The cluster tree can thus have probabilities associated to each branch, as shown in Figure 5. It is thus possible to associate a probability to each trajectory, multiplying all probabilities encountered in the tree starting from the root node and ending in the final cluster. For example, the trajectory starting in c1 and ending in c5 has a probability of 0.9 * 0.3 = 0.27. Trajectories following paths with very low probabilities can be considered anomalous events. c1 0.1

0.9 c2 0.7 c4

c3 0.3 c5

Figure 5. Tree of clusters with probabilistic information.

Note that the probabilistic trees described above can also be used to make predictions on the future developments of a track. For example a trajectory

230

L. SNIDARO, G.L. FORESTI AND C. PICIARELLI

starting in c1 has a probability of 0.27 of going toward c5, 0.9*0.7 = 0.63 of going in c4 and 0.1 of going in c3. Such data could be used in the explicit modeling approach: if a pedestrian is crossing a road outside the zebra crossing, the event should be considered more dangerous if the road has a high probability of being matched by a car (heavy traffic road) rather than a road with low traffic. (Note that we are assuming that a distinct set of clusters is maintained for each kind of object detected by the system; this is reasonable, since the common trajectories for a car are generally very different from those of a pedestrian).

6 Conclusion In this paper, we described how automatic surveillance systems can effectively improve harbour security. These systems can monitor the port activities and signal anomalous events to a remote operator in real-time. Here the focus was given to the analysis of trajectories of moving objects as a means to perform situation assessment. In particular, trajectories that differ significantly from the patterns learnt may be deemed anomalous and pointed out to the operator.

References 1.

Balckman SS. 1986. Multiple-Target Tracking with Radar Applications. Artech House, Boston, MA. 2. Bar-Shalom Y, Li X. 1995. Multitarget-Multisensor Tracking: Principles and Techniques. YBS Publishing, Storrs, CT. 3. Collins RT, Lipton AJ, Fujiyoshi H, Kanade T. 2001. Special Section on Video Surveillance. IEEE Transactions of Pattern Analysis and Machine Intelligence. Vol. 22, n. 8. 4. Collins RT, Lipton AJ, Fujiyoshi H, Kanade T. 2001. A System for Video Surveillance and Monitoring. In: Proceedings of the IEEE. Vol. 89, pp. 1456–1477. 5. Comaniciu D, Ramesh V, Meer P. 2003. Kernel-Based Object Tracking. In: IEEE Transactions on Pattern Analysis Machine Intelligence. Vol. 25, pp. 564–575. 6. Foresti GL. 1999. Object Recognition and Tracking for Remote Video Surveillance. IEEE Transaction on Circuits and Systems for Video Technology. Vol. 9, n. 7, pp. 1045–1062. 7. Foresti GL, Regazzoni CS, Varshney PK. 2003. Multisensor Surveillance Systems: The Fusion Perspective. Kluwer, Boston, MA. 8. Foresti GL, Micheloni C, Snidaro L, Remagnino P, Ellis T. 2005. Active Video-Based Surveillance System. IEEE Signal Processing Magazine. Vol. 22, n. 2, pp. 25–37. 9. Foresti GL, Micheloni C, Snidaro L. 2002. Adaptive high order neural trees for pattern recognition. In: Proceedings of the International Conference on Pattern Recognition (ICPR). Quebec City, Vol. 2, pp. 877–880. 10. Johnson N, Hogg D. 1996. Learning the Distribution of Object Trajectories for Event Recognition. Image and Vision Computing, Vol. 14, n. 8, pp. 609–615. 11. Medioni G, Cohen I, Bremond F, Hongeng S, Nevatia R. 2001. Event Detection and Analysis from Video Streams. IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 23, no. 8, pp. 873–889.

AUTOMATIC VIDEO SURVEILLANCE OF HARBOUR STRUCTURES

231

12. Piciarelli C, Foresti GL. 2005. Toward Event Detection Using Dynamic Trajectory Analysis and Prediction. IEEE International Symposium on Imaging for Crime Detection and Prevention (ICDP 2005). London, pp. 131–134. 13. Poore AB. 1994. Multi-Dimensional Assignment Formulation of Data Association Problems Arising from Multi-Target and Multi-Sensor Tracking. In: Computational Optimization and Applications. Vol. 3, pp. 27–57. 14. Yilmaz A, Javed O, Shah M. 2006. Object Tracking: A Survey. In: ACM Computing Surveys. Vol. 38, n. 4, pp. 1–45.

HETEROGENEOUS SENSORS DATA FUSION ISSUES FOR HARBOUR SECURITY

STEFANO PIVA, MATTEO GANDETTO, REETU SINGH AND CARLO S. REGAZZONI Department of Biophysical and Electronic Engineering, University of Genoa, Italy

Abstract. Nowadays security has globally become a critical issue, especially within communication, transportation, social and business activities key places. As they were in the past, harbours surely are key points for sea towns, significantly contributing to their life but also creating many safety problems. Multi-sensor video processing has a very important role in surveillance. Research in the field of multi-sensor processing for situation assessment and decision support is strictly connected with data fusion aspects. In this work, pros and cons, issues and research open points are explored in this field. The chapter describes how state of the art research faces these problems, to go deeper in the details of the video processing methodologies and solutions to problems of multi-camera tracking, joint calibration and positioning; are also concerned problems of heterogeneous sensors data fusion with the description of a radio/video localization system. Keywords: Data fusion, video surveillance, harbour protection, underwater video camera, situation assessment, tracking, multi-camera association, biometrics

1 Introduction Security has become a critical issue at the global level. The sky, earth and sea are all propitious media of attack. Surveillance is thus becoming increasingly essential to both research and market. Harbour surveillance ([1, 2]), which used to be an interesting topic of discussion, is now a pressing challenge and requires an immediate response. Due to the particular system structure necessary for guaranteeing harbour protection, several processes must be implemented in order to build a fully integrated heterogeneous surveillance system. Different methods based on underwater and surface devices must be combined to provide low false alarm rates and tracking services in a complex coastline environment (shallow water, high ship traffic, etc.).

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

233

234

S. PIVA ET AL.

In some examples [3], a network of underwater devices is used to form a structured harbour protection system: passive linear acoustic arrays are deployed on the sea floor and non-acoustic sensors are used to confirm the presence of midwater threads and active high frequency sonars. In [4], underwater and surface devices are combined to create a more adaptable system: multibeam surveillance sonars are deployed for Sub Sea tracking, steel wire underwater barriers act as sensors, underwater video cameras are used for Sub Sea identification and radar, and infrared image intensified CCD and CCD cameras perform surface surveillance with tracking and classification capabilities. All of these sensors are linked to a control system, which gathers data, performs data fusion and provides situation assessment. Although these preliminary systems exist in the state-of-the-art, many issues are still open for research: a heterogeneous sensor-based system calls for the optimization of single sensing channels and the capability to exploit the diverse nature of collected data in a unique representation tailored to security applications. This article examines some issues that arise in ground harbour area surveillance and protection: the potential of video sensors and the management of heterogeneous data. Section 2 introduces automatic video surveillance systems, starting with their history and recent evolution, Section 3 describes the bases of data fusion techniques, and Section 4 explores the problems of/solutions to combining data from similar sensors (homogeneous sensors data fusion) or from heterogeneous sensors.

2 Automatic video surveillance systems An automatic surveillance system can be defined as a computer system that, based on input data and provided with scene understanding capabilities, is geared on focusing the attention of the human operator, or on automatic alarm generation. Video-surveillance systems have been in development since the 1960s. In the first generation of video-surveillance systems, there was no information processing. In the third generation, with distributed digital signal processing, it is possible to present only the information that is necessary for signalling and describing a dangerous situation to the human operator. Researchers are now studying a new paradigm, called Cognitive Surveillance, for improved perception and reasoning over monitored events, extended communications and interaction capabilities. 2.1 MULTIPLE SENSORS A major limitation of current surveillance systems is the low level of diversity in scene interpretation architecture, which can be resolved by increasing the number and type of sensors monitoring the same scene under different point of views or in different resolutions, and by extracting multiple (possibly independent) features from the same scene. Real time multisensory surveillance systems based on

HETEROGENEOUS SENSORS DATA FUSION ISSUES

235

diversity concepts are unavoidably more complicated than single sensor systems based on centralized architectures; a natural option for remedying this complexity drawback is to take advantage of relatively cheaper processing resources and the lower cost of interconnection potentialities in the shift toward really distributed systems. The advantage of multisensor over single sensor systems can be quantified in terms of an improvement in situation assessment capability [5]. Factors that contribute to the enhancement of quantifiable system performance and to the improvement of overall system utility are: improved system reliability and robustness, extended coverage, shorter response time, improved resolution. Examples can be found in [6, 7].

3 Data fusion Data fusion has been defined [8] as the seamless integration of data coming from disparate sources. A typical application of data fusion techniques is the synergistic use of data received from multiple sensors. In order to be able to estimate the quality of data coming from multiple sensors, it is important to understand the features inherent to different sensors. The application of multiple sensors offers several possible performance benefits over traditional single-sensor based approaches. These benefits must be evaluated while taking into account additional cost, complexity, and interface requirements required for any given application. There are many additional benefits, depending on sensor type, fusion methodology, and the environment the system is operating in. Some of the numerous existing data fusion issues involving surveillance system design are discussed below, with an insight into the benefits of data fusion: video and radio data exploitation for determining the location of humans in surveillance applications, in outdoor and indoor environments. The discussion must take into account that data association can occur at different levels, which can be classified into five groups according to the JDL (Joint Director of Laboratories) model [9]. 3.1 HOMOGENEOUS VIDEO SENSORS FUSION In order to exploit redundant information obtained by multiple cameras, we refer to a classical data fusion approach [10, 11]. in which each JDL fusion level is ordered into three fundamental steps: Data Alignment, Data Association and State Estimation.

3.1.1 Data alignment Data alignment is needed in order to make the data comparable: in dealing with video cameras, the issues encountered in this step are related to temporal and spatial alignment. Temporal alignment: all the data flows coming from each

236

S. PIVA ET AL.

sensor have to be synchronized to compare features referring to the same instant. Temporal alignment can be obtained using a NTP (Network Time Protocol) server. Spatial alignment is obtained by way of the joint camera calibration procedure, which consists in determining correspondences among the image planes by characterizing geometric and optic features of each sensor [12].

3.1.2 Data association The multi-camera association strategy is at the core of the proposed example. Data association is defined as an m-ary decision process among objects in the fields of view of all cameras. The idea is to exploit a number of features to enable the system to adapt the association strategy to situations occurring in the scene. The aim is to optimize the separate association in each single frame. We make use of feature functions based on position, speed, shape factors, and chromatic characteristics. The first two functions are measured in the map reference system that is common to all interested sensors, while shape and colour are features proper to the image plane. We define four independent similarity functions, obtaining from each one an autonomous similarity coefficient, yielding continuous values distributed between 0 and 1. 3.1.2.1 Position Similarity Function f(P) When considering two objects observed by two different cameras (indexes i and j) whose coordinates are reported in the common reference system represented by the area map, we simply compute their Euclidean distance d m,n

( xmi  xnj ) 2  ( ymi  ynj ) 2

(1)

and use this distance in the following exponential function f

m,n

 P e-D d

m ,n

(2)

with position difference amplification factor D t 1 , to obtain a position similarity coefficient. 3.1.2.2 Velocity Function f(V) To obtain a motion direction similarity factor we calculate the vertical and i horizontal components vx,m and v iy , m (for the object m in the image plane of the i-th video sensor) of the speed in the map plane, with notations similar to the position function. We compute the velocity function

HETEROGENEOUS SENSORS DATA FUSION ISSUES

f m,n V = (1 

vi x ,m v j x ,n  vi y ,m v j y ,n )/2 vi m v j n

237

(3)

f V is equal to 0 when the objects move in opposite directions and to 1 when

they follow parallel paths. 3.1.2.3 Shape Factor Function f(S) The third feature in determining the similarity decision parameter is based on a simple property of the video surveillance application, typical object silhouettes: the bounding box containing the shape of a vehicle, such as a car, has different proportions than that of a pedestrian. To make use of this feature in defining the unique similarity factor, we consider the diagonal angle of the bounding box and compare the candidate couples again using the normalized scalar product. If d i is the dimension along the X axis of the m-th object in the i camera xm

image plane, then

f m,n ( S ) =

d i xm d j xn  d i ym d j ym

(4)

d imd j n

is the expression of the normalized scalar product providing proportion similarity. 3.1.2.4 Chromatic Similarity f(C) Chromatic similarity is expressed using the Bhattacharya coefficient to correlate colour histograms of moving objects in different camera image planes.

f m,n  C =

¦

c R ,G , B

h i c ( m) ˜ h j c ( n)

(5)

3

The Bhattacharya comparative coefficient, as well as the other previously described similarity factors, provides results on a continuous scale between 0 (completely different histograms) and 1 (identical histograms), enabling direct comparison and joint use of multiple cameras. 3.1.2.5 Object Similarity Coefficient (OSC) A simple mean value computed on N selected functions can be used to provide the relative similarity probability between two objects m and n in the field of view of video sensors i and j. For our chosen feature set, it is expressed by:

238

S. PIVA ET AL.

OSCm ,n

f

m,n

( P)  f

m,n

(V )  f

m ,n

(S )  f

m,n

(C )

(6)

4

The OSC is the basis of a criterion for choosing the correct associations for the highest values for each object in a camera’s field of view compared to all objects in all image planes with a field of view overlapped with the first. The OSC value is calculated for all possible associations among objects and the maximum result is selected for each object. For an object m, the associated object is therefore selected by computing: A  m, i =max(OSCm ,i )

(7)

i

3.1.3 State estimation Once the data are aligned and the objects are associated, the State Estimation phase performs the actual redundant information exploitation: when single camera positioning (state = position) data are available, they can be fused simply through the use of mean values. In objects’ occlusion phases or when some sensors fail, a more complex strategy must be applied, but this issue is out of our work’s scope. Details can be found in [13]. 3.2 HETEROGENEOUS SENSOR FUSION: VIDEO – WLAN JOINT POSITIONING SYSTEM The procedure intends to provide a solution to allow for sets of heterogeneous sensors, namely CCD video cameras and WLAN 802.11b radio devices, to be integrated in order to extract biometric information (position and ID) regarding objects in a given environment of interest. The integration is achieved in three steps: alignment in space and time, association between heterogeneous data once alignment has been confirmed, and estimation of the information that correctly describes a user. The algorithm is developed to train the system to recognize radio users with respect to video users, thus establishing a relation between them. This problem has been formalized as follows: The logical functional architecture of the proposed system assumes a set of heterogeneous sensors S {S C : c 1,..., N S } that are divided into NS different classes, S C {s ci : c 1,..., N } SC

each of which is composed by a number of N S Computational Units (CU). Each C

sensor is directly connected to a dedicated CU belonging to the set U {ul : l 1,..., N } where N is equal to the total number of CUs and corresponding sensors. Each CU acquires data providing Object Reports (OR) c r i , m ( k ) for each object m found at time k. OR is represented as a

HETEROGENEOUS SENSORS DATA FUSION ISSUES

239

multidimensional vector composed of different features related to the detected object: c

i

r i ,m ( k ) [ f 1 ( k ),… , f

i Nr

( k )]

(8)

where Nr the total number of features f in the report. For each detected physical object, tracks are instantiated and updated:

 Tm ( k ) { rm ( K  k ) : k

0 ,..., K }

(9)

with K the current time and m the detected object. Tracks are sequences of estimated reports r m (i ) derived from integration of heterogeneous ORs:    rm ( k ) [ f 1i ( k ),… , f Ni r ( k )]

(10)

3.2.1 Architecture layout In Figure 1, the overall logic functional architecture of the proposed system is depicted. As can be seen, its structure is inspired by a classical model of Data Fusion systems described in [14] (see Section 3.2). Only two classes of sensors are included in the architecture, Static CCD Video Cameras and 802.11 WLAN Base Stations. Data collected by sensors must be aligned in order to be successfully compared, and two modules – a Video Analysis Module (VAM) that extracts metadata from video sources, and a Radio Analysis Module (RAM) for Base Stations – independently pre-process information coming from different sensors in dedicated CUs (standard Pentium-based PCs). Output Object Reports are respectively addressed as Video Object Reports (VOR) and Radio Object Reports c V,R ˆi (ROR) r i,m (k) . They are associated according to their features f ( k ) and specified association rules. Once the different groups of reports are evaluated by the Data Association sub module, they are fused into estimated reports rˆm (k) by the State Estimator (Figure 1). Tracks Tm(k) are updated or newly instantiated by the Track Manager, and a Prediction sub module feedbacks future values for ORs into the Association step in order to get a closed – loop analysis.

4 Conclusion Some harbour surveillance issues are addressed. Multi-sensor surveillance systems techniques are proposed and described through general data fusion concepts and two instances introducing solutions for homogenous and heterogeneous sensor

240

S. PIVA ET AL.

systems respectively. Video processing methodologies and solutions to multicamera tracking, joint calibration and positioning problems are investigated. An approach towards fusion of Video and Radio Sensors mapped in a general formalism has been presented by using specific algorithms for aligning data in time and space in case of Video and Radio data. SNvv

S1v

R

SNRR

S1

NTP Synchronization

Phtysical level ME

Video ME

Network level ME Alignment

Calibration

rv i,m(k)

rR i,m(k)

Trilateration

VAM

RAM Data Association

Gating

Association

rm(k) State Estimation

Predictor Track Manager

Estimation Tm(k)

Figure 1. Logic functional architecture composed of data alignment, data association and state estimation.

References 1. 2. 3. 4. 5. 6. 7. 8.

Garnier B, “MAP: a new paradigm for costal surveillance and harbour protection”, UDT Asia 2003. “Coastal underwater surveillance and harbour protection”, UDT Asia 2003. Donald JB, “Multi-sensor harbour defence system (M-SHARD)”, UDT Asia 2003. Safe Barrier International LT web site, http://www.safebarrier.com. Varshney PK, “Multisensor data fusion”, Electronic and Communication Engineering Journal, Vol. 9(6), pp. 245–253, 1997. Kettnaker V, Zabih R, “Bayesian multi-camera surveillance”, IEEE CVPR, pp. 253– 259, 1999. Regazzoni C et al., “Distributed data fusion for real-time crowding estimation”, Signal Processing 53, pp. 47–63, 1996. Hyder AK et al., Multisensor Fusion, NATO ARW on Multisensor Data Fusion, Scotland, June, 2000.

HETEROGENEOUS SENSORS DATA FUSION ISSUES 9. 10. 11. 12.

13.

14.

241

Steinberg AN et al., Revisions to the JDL Data Fusion Model, NATO IRIS Conference Proceedings, Quebec Canada, Oct. 1998. Hall D et al., “An introduction to multisensor data fusion”, IEEE Proceedings, Vol. 85(1), pp. 6–23, Jan. 1997. Liggins ME et al., “Distributed fusion architectures and algorithms for target tracking”, Proceedings of the IEEE, Vol. 85(1), pp. 95–107, Jan. 1997. Tsai RY, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses”, IEEE JRA, Vol. RA-3(4), Aug. 1987, pp. 323–344. Piva S et al., “A Multi-Feature Object Association Framework for Overlapped Field of View Multi-Camera Video Surveillance Systems”, IEEE AVSS 2005, Italy, Sept. 2005 Waltz E and Llinas J, “Multisensor data fusion”, ISBN 0-89006-277-3, 1990 Artech House, Norwood, MA.

SHIP DETECTION AND CHARACTERIZATION USING POLARIMETRIC SAR DATA

YANNICK ALLARD1, MICKAEL GERMAIN2 AND OLIVIER BONNEAU2 1

OODA Technologies Inc. Canada Centre de Recherches Mathématiques, Université de Montréal, Canada 2

Abstract. Synthetic Aperture Radar (SAR) sensors offer the coverage rate and allweather capability necessary for wide-area surveillance and can provide very useful information for the detection and characterization of surface targets. Conventional SARs operate with a single polarization channel, while recent and future spaceborne SARs (Envisat ASAR, Radarsar-2) will offer the possibility to use multiple polarization channels, which will enable better detection and characterization of man-made objects. Standard target detection approaches on SAR images consist in the application of a Cell Averaging or 2-parameters CFAR detector and usually produce a large number of false alarms. This large number of false alarms prohibits their manual rejection. However, over the past ten years a number of algorithms have been proposed to extract information from a polarimetric SAR scattering matrix in order to enhance and/or characterize manmade objects. Target decomposition algorithms, such as Cameron’s coherent target decomposition (CTD) and the Odd-Even basis decomposition, provide information for detecting ships and reducing false alarms. On the other hand, information derived from spectral analysis, such as the subaperture coherence, enhance the ship’s signature and allow a better discrimination of the desired target. The evidential fusion of such information can lead to the automatic rejection of the false alarms generated by the CFAR detector. In addition, the aforementioned information can lead to a better characterization of detected targets. This paper presents an approach, as well as results, for automatic target detection using the evidential fusion of polarimetric features and the spectral analysis technique. Preliminary results for ship characterization using polarimetric

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

243

244

Y. ALLARD, M. GERMAIN AND O. BONNEAU

information are also presented. This research is motivated by airborne/spaceborne surveillance applications such as land and coastal surveillance missions using SAR/PolSAR imagery. Keywords: Ship detection/characterization, polarimetric decomposition, evidential fusion, false alarm mitigation

1 Introduction Remote sensing imagery, due to it large spatial coverage, enables the monitoring of large areas. It provides valuable information in the context of area surveillance and activity monitoring. The next generation of sensors that will likely be used for that particular task mainly consist of high-resolution polarimetric SAR (PolSAR), hyperspectral imagery (HSI) and high-resolution optical systems. These sensors will provide a large number of data, and there will be a need to develop tools and methodologies to automatically analyze them and extract meaningful information [1]. For the particular task of target detection and area monitoring, PolSAR sensors have the advantage of being independent of solar illumination and are very sensitive to the presence of man-made objects. During the last decade, many algorithms were developed for point target detection and characterization on polarimetric data. This paper discusses the task of ship detection and characterization using PolSAR imagery for an application in wide area surveillance. In the next section, conventional target detection methodology on SAR imagery is described. Section 3 presents more commonly used polarimetric features for point target detection using polarimetric SAR imagery. A brief introduction on the Dempster-Shafer fusion algorithm is carried out in Section 4 while Section 5 presents the fusion polarimetric features for ship detection and characterization. Finally, conclusions are drawn in Section 6.

2 Target detection using SAR imagery Conventional systems for target detection and recognition on SAR imagery usually consist of five stages (Figure 1).

Figure 1. Conventional stages of a target detection system.

SHIP DETECTION AND CHARACTERIZATION USING POLARIMETRIC DATA 245

The detection stage consists in the determination of the presence of the target signature at a particular position in the image. This task is usually achieved mainly with the application of a CFAR detector. However, the CFAR detector generates numerous false detections, making it impossible to perform manual rejection. On the other hand, target discrimination may be seen as the binary classification in target versus non-target. This paper focuses on these two tasks of the target detection and recognition scheme. Target characterization using polarimetric information will be briefly discussed. 2.1 CONSTANT FALSE ALARM RATE DETECTION The target detection stage selects areas with a large probability of containing targets. The detector must be computationally simple and should provide a high probability of detection while creating at the same time as few false alarms (FA) as possible. One of the most widely used pre-screeners in SAR target detection is the two parameter constant false alarm rate (CFAR) detector, which is based on a normalized test of the pixel intensity versus its local neighbourhood. Figure 2 represents the typical window of analysis of a CFAR detector. The moving window is composed of a test pixel surrounded by a guard ring to prevent any influence of the target on the boundary ring, which is used to compute the necessary statistics.

Figure 2. Principle of a CFAR detector.

However, the discriminating power of the two parameter CFAR is not sufficient to reduce the false alarms to an acceptable level. The incorporation of complementary features, such as polarimetric features, should help target detection systems to provide a more reliable result.

246

Y. ALLARD, M. GERMAIN AND O. BONNEAU

3 Polarimetric features for target discrimination Potential targets, which have been highlighted in the CFAR detection, are passed through a target discrimination process to reject false alarms. Polarimetric features introduce another dimension in the task of target discrimination. Usually, since target discrimination is only applied in regions of interest (ROI) that have been highlighted during detection, more computationally demanding algorithms, such as some polarimetric decompositions, can be applied. Amidst the large number of available polarimetric decompositions, the following are the most interesting for point target detection, discrimination and characterization: x x x x x

The Odd/Even basis decomposition [4] Cameron’s Coherent Target Decomposition (CTD) [2] Polarization anisotropy [7] Symmetric Scattering Characterization Method (SSCM) [6] Subaperture Coherence [5]

For a complete description of these algorithms, the reader should refer to the appropriate publications.

4 Information fusion using evidential reasoning The fusion of the information provided by the polarimetric decomposition is performed using the Dempster-Shafer evidence theory [3]. This framework offers a convenient way to include imprecision, incompleteness and uncertainties in the reasoning process. In the evidence theory, a mass function m(A) is associated to each hypothesis A of the set 24 defined by the 2 N 1 subsets of the frame of discernment 4 . The mass of a given hypothesis is defined as ­° m(‡) 0 m:® ¦ m( A) 1 °¯ A24

(1)

Declarations provided by image processors or other sources of information can be combined using Dempster-Shafer combination rule. If mi is the probability assignment provided by the source SI, the combination m is defined as

m1 † m2 † ... † m p

SHIP DETECTION AND CHARACTERIZATION USING POLARIMETRIC DATA 247

m(‡) 0

if

K z 1 m( A)

where K

¦

¦

– m (B )

B1 ˆ B2 ... B p A 1d i d p

1 K

i

i

(2)

– m (B )

B1 ˆ B2 ... B p ‡ 1d i d p

i

i

where K represents the mass which would be associated to the empty set in the absence of the normalization (division by 1  K ). For this reason K is interpreted as a measure of the conflict between measurements provided by different sources.

5 Evidential fusion of polarimetric features for false alarm mitigation When performing ship detection on a SAR image using a CFAR detector, many false alarms (typically from 3–10 to more than an hundred) are generated. These false alarms are mainly caused by surface clutter due to: the sea state, small fishing boats, icebergs, etc. In these cases, the use of polarimetric information should be beneficial to remove a huge number of false alarms due to a not-sochallenging background (Figure 3). The method we have chosen to demonstrate this is the evidential fusion of polarimetric information in the CFAR contacts to validate or discard the ROI (Figure 4). In order to fuse polarimetric information, it is mandatory to define mass functions on each of the features. We use trapezoidal mass functions for each “continuous” feature (example: subaperture coherence) or a hard confidence if the feature is a hard-decision result of an algorithm (example, binary classification of coherent and non-coherent point targets). The mass functions are assigned using our knowledge about each feature. In the case of continuous features, the overlapping parts of the trapezoidal mass functions allow us to choose their parameters with lesser precision [8]. Care is taken so that ignorance never falls to zero in order to avoid a situation of total conflict.

248

Y. ALLARD, M. GERMAIN AND O. BONNEAU

Figure 3. High level diagram of the ship detection module.

Figure 4. Example of evidential fusion within CFAR contacts.

5.1 TARGET CHARACTERIZATION This task aims at extracting target features and recognizes targets from SAR images. The length extraction and ship characterization is a complex problem to resolve due to various problems such as: uncontrolled environment, variable image acquisition geometry and resolution, focus problem and dependence of the radar scattering to ship’s orientation. We use the Hough transform to compute the ship’s centreline to estimate length and orientation. Polarimetric information can be used to characterize the target. One way to use these features in such a task is to analyze the distribution of elemental scatterers in different portions of the target.

SHIP DETECTION AND CHARACTERIZATION USING POLARIMETRIC DATA 249

To do so, the detected target is segmented in a certain number of parts and the percentage of each elemental scatterer is computed in each section (Figure 5).

Estimated Length : 130m - Real Length : 132.9m Orientation : 34 degrees 100% 90%

Quarter Wave

80%

Narrow Diplane

70%

Cylinder

60%

Dipole Dihedral

50%

Trihedral

40%

Assymmetric

30%

Left Helix

20%

Right Helix Non Reciprocal

10% 0% 1

2

3

4

5

6

7

8

9

10

Scatterer Distribution (CV-580 C-band SAR) Figure 5. Distribution of elemental scatterers (Ville De Quebec).

5.2 PROBLEMS ENCOUNTERED One problem we encountered was that the small number of ship samples had an impact on the system’s target recognition performance. There weren’t enough elemental scatterer distributions available to train a neural network. As a consequence, the length was the most influential parameter in ship recognition. Considering the fusion of polarimetric features for ship detection, the mass functions could not be applied from one scene to another. Some images had focus problems and the difference in acquisition conditions might have had an impact on some polarimetric features. In addition, badly-focused images had an impact on polarimetric feature computation, as well as on ship segmentation and parameter extraction.

6 Conclusion The next generation of spaceborne and airborne imaging sensors will increase the role of remote sensing imagery for wide area surveillance and monitoring. However, due to the growing amount of available data, there is a need for the development of an automated tool that will help the image analyst with his task. The application of image analysis considered in this lecture was directed toward automatic detection and characterization of targets using SAR imagery. As shown, the detection and discrimination performances of an automatic system are better when using polarimetric SAR data than using only its single-channel counterpart.

250

Y. ALLARD, M. GERMAIN AND O. BONNEAU

The polarimetric nature of the data provides additional features of interest for ship and ground-target detection/recognition. The evidential fusion of these polarimetric features within the CFAR contacts can eliminate many of the false alarms generated by the CFAR detector. Dual polarization should be investigated for detection and false alarm reduction, especially using Subaperture Coherence of the HV channel, and maybe other features computed from the HH-HV channel. We plan to pursue our investigation by testing our approach on spaceborne polarimetric data when Radarsat-2 data becomes available, as spaceborne SAR remote sensing will play a key role in wide-area monitoring. We will also pursue work on ship characterization using polarimetric features in order to detect superstructures on ship targets. We plan to study the effect of super-resolution techniques for polarimetric features extraction, ship length and orientation estimation for target characterization.

Acknowledgments This work was supported by the Canadian Space Agency under the Earth Observation Application Development Program (EOADP) (contract 9F028-34910/A) and the Radar Applications and Space Technologies section of Defence Research and Development Canada – Ottawa (DRDC-O).

References 1. 2.

3. 4.

5.

6.

7.

8.

Blacknell D, Contextual Information in SAR Target Detection, IEE Proceedings; Radar, Sonar and Navigation, vol. 148, no. 1, pp. 41–47, 2001. Cameron W, Youssef N, Leung LK, Simulated Polarimetric Signatures of Primitive Geometrical Shapes. IEEE Transactions on Geoscience and Remote Sensing, vol. 34, no. 3, pp. 793–803, 1996. Dempster AP, A Generalization of Bayesian Inference, Journal of the Royal Statistical Society, vol. 30, no. 2, 1968. Novak L, Halversen S, Owirka G, Hiett M, Effects of Polarization and Resolution on the Performance of a SAR Automatic Target Recognition System, Lincoln Laboratory Journal, vol. 8, no. 1, pp. 49–68, 1995. Sourys JC, Henry C, Adragna F, On the Use of Complex SAR Image Spectral Analysis for Target Detection: Assessment of Polarimetry, IEEE Transactions on Geoscience and Remote Sensing, vol. 41, no. 12, pp. 2725–2734, 2003. Touzi R, Charbonneau F, Characterization of Target Symmetric Scattering Using Polarimetric SARs, IEEE Transactions on Geoscience and Remote Sensing, vol. 40, no. 11, pp. 2507–2516, 2002. Touzi R, Charbonneau F, Hawkins RK, Vachon PW, Ship Detection and Characterization Using Polarimetric SAR; Canadian Journal of Remote Sensing (RADARSAT 2 Special Issue), June 2004. Tupin F, Reconnaissance de forme et Analyses de Scène en Imagerie Radar à ouverture Synthétique, Thèse de Doctorat, École Nationale Supérieure des Technologies, Paris, 1997.

QUICK JOINT DETECTION AND FUSION APPLICATIONS IN PASSIVE SURVEILLANCE SYSTEMS

PIERRE BLANC-BENON1, NATHALIE GIORDANA2 AND YVES CHOCHEYRAS 2 1 Thales Underwater Systems, Brest, France 2 Thales Underwater Systems, Sophia-Antipolis, France

Abstract. This paper discusses the use of minimum cost flow solutions for Sonar surveillance, since broadband and narrowband processing outputs overwhelm the operator ability to discriminate between false alarms and potential threats. The proposed automatic function maps a graph to the frequency u bearing u time data and solves it for detection and tracking. A shallow-water experiment is produced to validate this new passive surveillance approach. Keywords: Sonar, data fusion, combinatorial optimization

1 Introduction Within the context of Sonar Systems, Passive Surveillance needs to be revisited on the basis that the available computing power and memory have increased dramatically over the last decade. It is always a rewarding challenge to produce something completely new rather than to revisit existing solutions, and then to load and execute it on up-to-date commercial processors. The purpose here is to combine three ideas in order to build a powerful tool for quickly assessing the acoustic environment in both broadband (BB) and narrowband (NB). This is what we need for harbour protection and, more generally speaking, for global surveillance: x At first, consider an array of sensors that provide an adequate acoustic aperture and beam-width resolution. x As regards memory sizing and real-time specific constraints, base the solution on a 2–3 GHz processor with a 1–2 GByte RAM commercial off the shelf (COTS) processor. x Finally, recall and use processing algorithms that have been partly discarded in the past because they were considered to be too demanding of the limited resources then available: direct signal acquisition, no complex demodulation,

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

251

252

P. BLANC-BENON, N. GIORDANA AND Y. CHOCHEYRAS

panoramic full band and combined NB and BB beam-former processing, and graph theory to detect, track and merge, and display. The following topics will be considered in this paper. First a few considerations will be addressed in order to sketch out the basics of the signal processing to be used in this quick detection and fusion method. A brief description of the array itself, and of the signal coding and processing, will be incorporated. Second, the use of combinatorial optimization will be explained, keeping in mind that what was impossible to do a few years ago is now achievable with off-the-shelf processors. The explanation will be made in parallel with the conventional approach of data detection and association, that is to say gating, measurement selection and false alarm mitigation (e.g. probabilistic data association filters), and one by one target management. Finally an experiment performed in shallow water will conclude the presentation revealing both the simplicity and performance of this new approach for acoustic surveillance.

2 Array description and signal processing 2.1 SENSOR ARRAY DEFINITION The array is linear and consists of 32 hydrophones. The technology used is conventional and covers a frequency range of 100 Hz to 1 kHz. The schematic view below depicts the main characteristics of the array (Figure 1): EdB fHz bearing

Sensor # 1

2

3 …

32 digitized hydrophones, equispaced, d
32

Recorder / Acoust. Processor

Figure 1. Schematic view of the sensor array, beam width, and target.

QUICK JOINT DETECTION AND FUSION APPLICATIONS

253

2.2 SIGNAL PROCESSING Keeping in mind that the processing power and memory available should afford us sufficient breathing room, a simple design for the computation of energy versus time and frequency has been favored – see below a simplified flowchart of the actual implementation (Figure 2). k=1..32 hydrophones

x(t) real signal digitized and sampled

FFT X(f,t) complex spectrum

Y=h+.X

FFT size & overlap to ensure both 'f/f~10-3 and moving targets

Beamformer coefficients

Y(f,t) complex beams

hk,l,f l=1...L beams

|.|² & whitening

BroadBand

6f

NarrowBand

BB & NB matrices : beamu(frequency) utime

Graph mapping & combinatorial tracking

Figure 2. Signal processing, schematic implementation.

3 Graph mapping and combinatorial tracking optimization The objective of the passive surveillance function presented in this chapter is to automatically build groups of associated frequencies, associated in both time and bearing, for multiple threats simultaneously, and therefore our attention is not focused on the estimation problem, e.g. what could be the best estimate given a data set of measurements for a given contact, nor on choosing the best measurements among detected events. Both the broadband and narrowband processing detailed above output a large number of potential contacts for each scan and each beam. A single operator would not be efficient at manually detecting and extracting significant information related to every single potential threat. The idea behind combinatorial tracking can now be summarized, along with an explanation of how it is different from other more or less conventional approaches, from the simplest to most recent ones:

254

P. BLANC-BENON, N. GIORDANA AND Y. CHOCHEYRAS

x Nearest Neighbour Association (NNA) or Probabilistic Data Association (PDA) and their numerous variants (Joint PDA, Interacting Multiple Model PDA, …) consider the tracking problem as measurement selection and single track updating (or up to two tracks in case of crossing in JPDA). Either the “closest” measurement is utilized, according to a maximum likelihood principle, or a probabilistic weighted average of measurements pre-selected by a gating process (PDA). The main difficulties that arise are related to the optimum choice of elementary gates when the false alarm rate is high and the SNR is low. Another problem is that the one-by-one tracking principle is suitable mainly when there is only one target, or targets that are distinguishable enough, but this is not the case in general [1]. Track initiation is also a big issue, which has never been addressed satisfactorily by such filtering methods. x Multiple Hypothesis Tracking (MHT) is a maximum likelihood selection from a family of hypotheses built from ad-hoc rules often based on 2 by 2 comparisons. The key question resides in the enumeration problem, and not in optimum selection [2]. x Multi-Dimensional Assignment (MDA) does not integrate, as an optimization of separate 2-dimensional assignment problems, the full time dependence of the multi-target tracking problem [6]. One should look, rather, toward combinatorial optimization, which is extensively presented in references [3, 4, 5, 8]. Hence, the following steps are undertaken: x Consider the BB matrix to be equivalent to a beam u scan map. To each energy located in one beam and one scan, associate a node in a BB graph. The connection between nodes is a potential target motion. A high connection level means that the graph enables fast target tracking. This is a parameter of the algorithm. The cost cij attached to a link between two adjacent nodes i and j, is a function of the energy. x Minimize the following global cost function: Min xij

6

(1)

c x

i, j^nodes` ij ij

under the following constraints of input/output flow balance:

¦ xij  ¦ x ji j

0, i  ^ nodes `

and capacity

0 d x i, j d 1, i, j  ^ nodes `

.

j

x When these N-best BB-tracks are found, a NB matrix selection will automatically be initiated along the BB bearing track. The same combinatorial optimizations occur for the NB case, replacing the beam u scan map by frequency channel u scan maps. This yields Mi NB-tracks and so 6I = 1,N Mi BB-NB tracks, the NB fusion being a by-product.

QUICK JOINT DETECTION AND FUSION APPLICATIONS

255

This is repeated at anytime in successive batches of 2 or 3 min (tunable parameter).

4 Shallow water experiment 4.1 ENVIRONMENTAL DATA Experimentation was conducted in coastal waters of approximately 100 m in depth and less than 7 nautical miles from the coast. It lasted approximately 1 h. While data were being collected for the experiment, the sound velocity profile was approximately bilinear with both positive gradients in the two layers. Acoustic rays are bent toward the sea surface. The first example depicted below features propagation loss with a simple eigenray propagation model, the array being located at a 50-m depth in the surface layer (Figure 3). SVP

Propagation Loss dB - @ f=600 Hz

0

0

40 < Sonar Depth = 50m 60 80 100 1420

-40

20 T a rg e t D e p th m

D e p th m

20

-50 -60

40

-70 60

-80

80

-90 -100

1430 1440 Sound velocity m/s

1450

0

2

4

6

8 10 12 Target Range km

14

16

18

20

Figure 3. Propagation loss dB for a receiver in the surface layer.

The second example is for an array located in the bottom layer at an 80-m depth with slightly inferior propagation conditions (Figure 4). Propagation Loss dB - @ f=600 Hz 0

20

20 T a rg e t D e p t h m

D epth m

SVP 0

40 60 80

< Sonar Depth = 80m

-40 -50 -60

40 -70 60

-80

80

-90 -100

100 1420

1430 1440 Sound velocity m/s

1450

0

2

4

6

8 10 12 Target Range km

14

16

Figure 4. Propagation loss dB for a receiver in the bottom layer.

18

20

256

P. BLANC-BENON, N. GIORDANA AND Y. CHOCHEYRAS

These computations exemplify the complexity of propagation in coastal waters: many paths exist with no single direct path. This causes signal interferences more frequently than in the open sea: combinatorial optimization will demonstrate its ability to track target signature even in the presence of strong interferences whereas conventional tracking algorithms listed above in §3 and tailored to sonar tracking, are not able to do so. 4.2 QUICK DETECTION AND FUSION RESULTS A typical run of the algorithm is reproduced in Figure 5, after approximately 20 min. Successive batches of BB-NB tracking and fusion are separated by a vertical line, lasting 2 min each, on the left BB part of the figure. The right side contains the updating process of NB displays. This multi-beam spectrum analysis shows the NB associated tracks for each target, superimposed over the raw energy. Colours are used periodically to follow each contact over time, and keep the BBNB association information visible. One noticeable feature is the red contact moving from 60° to 140°. It is the ability of such a tracking and fusion process to prevent any misunderstanding with any acoustic interference: these have been discarded and are not tracked as real sources. Propagation Loss dB - @ f=600 Hz 0

20

20 T a rg e t D e p t h m

D epth m

SVP 0

40 60 80

< Sonar Depth = 80m

-40 -50 -60

40 -70 60

-80

80

-90 -100

100 1420

1430 1440 Sound velocity m/s

1450

0

2

4

6

8 10 12 Target Range km

14

16

18

20

Figure 5. BB and NB panoramic fusion display, raw signals and tracks superimposed.

5 Conclusion This paper discusses a new approach to acoustic surveillance based on minimum cost flow solutions. Here, acoustic surveillance was revisited using a combined track and fusion, and former processor intensive algorithms. This fully automatic function is decomposed into the repetition of three steps: x First, to compute extended broadband and narrowband normalized energy according to a panoramic bearing scale.

QUICK JOINT DETECTION AND FUSION APPLICATIONS

257

x Second, to associate a huge graph to this global information: one node of the graph being frequency u bearing u time. A cost function is associated to node connections within the graph. x Third, to solve the optimum cost flow problem and to threshold the tracks. Real signal analysis corresponding to a shallow water experiment has demonstrated the feasibility and performance of such an approach for a passive surveillance function operating on a commercial processor.

References 1. 2. 3. 4. 5. 6.

7.

8.

Bar Shalom Y, Fortmann T. 1988. Tracking and Data Association. Academic, New York, Vol. 179. Blackman SS. 2003. Multiple Target Tracking with Radar Application. Artech House, Norwood, MA. Schrijver A. 2003. Combinatorial optimization. Vol. A, Paths, Flows, Matchings. Springer, Berlin. Papadimitriou CH, Steiglitz K. 1982. Combinatorial Optimization: Algorithms and Complexity. Prentice-Hall, Englewood Cliffs, NJ. Bertsekas DP. 1992. Linear Network Optimization: Algorithms and Codes. MIT Press, Cambridge, MA. Popp R, Pattipati K, Bar Shalom Y. 2001. m-Best S-D Algorithm with Application to Multitarget Tracking. IEEE Transactions on AES, Vol. 37, No. 1, January 2001, 22– 39. Poore A. 1995. Multidimensional Assignments and Multitarget Tracking. DIMACS Series in Discrete Mathematics and Theoretical Computer Science. Vol. 19, American Mathematical Society. pp. 169–196. Poore A, Rijavec N. 1993. A Lagrangian Relaxation Algorithm for Multidimensional Assignment Problems Arising from Multitarget Tracking. SIAM Journal on Optimization, Vol. 3, No. 3, pp. 544–563.

THE QUIET INTERLUDE PROCESSING SYSTEM (QUIPS) – AN AUTOMATED DATA FUSION CAPABILITY FOR LITTORAL ENVIRONMENTS

JAMES H. WILSON AND ARVID MELKEVIK Naval Postgraduate School, USA, Norwegian Defence Research Establishment (FFI), Norway

Abstract. Producing accurate ship tracks from multi-source automated data fusion in near real time is a difficult but very important task of designing an effective system for maritime security. The QuIPS project funded by the Office of the Secretary of Defence (OSD) is to develop an automated data fusion capability for littoral environments, which includes very shallow water from depths of 300 m to the shoreline. The QuIPS specific objective is to accurately track surface ships in the crowded littoral environments by accurately using heterogeneous sources, which include national sensor satellite data, as well as local sensors, such as radar. This paper describes the initial capabilities developed in this project. Keywords: Anti-submarine warfare, shallow water, multi-source ship tracking, motion/ behaviour pattern analysis

1 Statement of the problem The Office of the Secretary of Defence (OSD) recently funded the development of QuIPS to develop an automated data fusion capability for littoral environments to include very shallow water from depths of 300 m to the shoreline. QuIPS capabilities apply to both traditional anti-submarine warfare (ASW) submarine and “asymmetric” small submersible and surface ship threats, and required a single processing system to accommodate a wide range of acoustic arrays to address asymmetrical threat frequencies from 5 Hz to over 20 kHz. The QuIPS processor is an inexpensive, portable LINUX cluster computer, where the 3Darray geometry table and the data sample rate are variable inputs to address threats over a wide frequency band requiring very different wet-end arrays. A second QuIPS requirement was to accurately track surface ships in the crowded littoral environments accurately using heterogeneous sources of national sensor satellite data, as well as local sensors, such as radar. In this way, an acoustic track that has absolutely no corresponding surface ship track will rouse a E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

259

260

J.H. WILSON AND A. MELKEVIK

“red flag” as a possible submerged threat. Surface ships exhibiting unusual behaviour can also be potential threats (e.g., the small boat that damaged the USS Cole, a barge lingering inside cruise missile range preparing to launch missiles at a critical onshore site, etc.). A third QuIPS requirement was to develop a sonar beamformer that performs well in very shallow water, since all of the operational Navy sonars have plane wave beamformers developed for deep-water applications.

2 Background Producing accurate ship tracks from multi-source automated data fusion in near real time worldwide is a difficult task, but has many important applications, such as ASW, mine warfare (MIW), expeditionary strike force, Joint intelligence, force protection worldwide, and homeland security operations. OSD’s Undersecretary Wynne has eloquently articulated the concepts of “Metrification of the Littorals (MoL)” where DOD must develop the capability to detect, localize, track, classify, and destroy asymmetrical threats, as well as traditional ASW submarine threats. Examples of asymmetrical threats are minisubmarines, autonomous underwater vehicles (AUVs), swimmer delivery vehicles (SDVs), remotely operated vehicles (ROVs), and swimmers. The asymmetrical threats generally operate in very shallow water, inside the 300 m contour to less than 5 m, and have threat emissions that span a wide frequency band from 5 Hz to well over 20 kHz. QuIPS was funded by OSD to address both MoL and ASW issues and has five unique capabilities that cannot be found in any system other than QuIPS: 1. The Global ELINT Tracker (GET) is far more accurate in the automated mode than currently used electronic intelligence (ELINT) correlators based on ground-truth results. 2. The QuIPS non-acoustic ship-track data-fusion module using national sensor data from four separate “stovepiped” surface ship track sources combines the strength of each source and reduces the weaknesses of each system by automated data fusion. 3. Acoustic beamforming in very shallow water at low frequencies using matched phase, matched field processing (MPMFP) is required for very nearshore and harbour applications. 4. Data fusion of acoustic tracks from three sonar beamformers are run simultaneously in real time over multiple frequency bands match the threat emission spectra. – – –

Narrowband (NB) beamformer with M of N Tracker Broadband (BB) correlator with M of N Tracker MPMFP beamformer with Latitude/Longitude versus time tracker

THE QUIET INTERLUDE PROCESSING SYSTEM (QUIPS)

261

5. Data fusion of non acoustic and acoustic tracks with fast track-data retrieval from the QuIPS relational data base, QUIPSQL, for display on FORCEviz, the QuIPS 4D visualization display. Many of the above QuIPS capabilities require computer intensive calculations, and a unique run-time software module, called QuIPSWare, was developed to run the QuIPS applications on a LINUX cluster computer. More intense QuIPS applications can be accommodated by simply adding inexpensive LINUX cluster 1U boards without significant software changes. It is noted that commercial LINUX cluster runtime software packages, such as Message Passing Interface (MPI), were tested with the QuIPS applications and failed totally and completely in the real time mode. MPI and other software packages were not designed for QuIPS type computer intensive applications.

3 System overview QuIPS is a massively parallel, LINUX Cluster system with a 4D visualization display called FORCEviz, and is shown in Figure 1. QuIPS runtime software is called QuIPSWare, and is designed to run the LINUX Cluster as near design hardware speed as possible for any computer intensive application. QuIPSWare has vastly improved performance over the popular commercial software package Message Passing Interface (MPI), which failed catastrophically during the QuIPS development phase. QuIPSWare is the high-speed software package of the future for massive parallel LINUX Cluster processing. QuIPSWare can easily be configured to be compatible with any commercial relational data base, and MYSQL freeware was adapted into QuIPS for the first application described below, and is called QuIPSQL. The first QuIPS software application is sensor data processing and multi-source data fusion, and the QuIPS objectives accomplished are: x Broadband, narrowband, and matched phase, matched field processing (MPMFP) sonar beamforming for an acoustic array of any geometry and design frequencies from 0 to 75 kHz. x Multi-source ship track non-acoustic data fusion using national sensor data ship platform tracks from the Global Command and Control System – Maritime (GCCS-M), the Joint Maritime Information Exchange (JMIE) (soon to be MAGNET) ship track data base, local surface search radar tracks, and the Automated Information System (AIS) tracks. x Acoustic ship track data fusion from a split array broadband correlator, narrowband beamformer, and matched phase, matched field processor (MPMFP). x Acoustic and non-acoustic surface ship and submerged contact data fusion. x Wireless, long range broadband sensor data transmission using 802.11b/g wireless technology, Iridium SATCOM, and Unmanned Aerial Vehicles (UAVs) as a communication relay station.

262

J.H. WILSON AND A. MELKEVIK

x 4D visualization display (FORCEviz) of surface ship track and submerged contact data worldwide with standard GCCS-M technology. x Operator friendly FORCEviz to QuIPSQL relational data base system to retrieve and display track history and contact attribute data quickly after an operator’s “click” on the track’s current position. x Web centric transmission of tracks via GCCS-M using the new GO-1 standard and PMW-157 middleware. x The QuIPS• data fused tracks are input into the Dynamic Ambient Noise Prediction System (DAPS) to estimate ambient noise for ASW and AntiTerrorist Force Protection (AT/FP) applications. The QuIPS data fusion tree processing schematic is shown in Figure 2, and the QuIPS track data is the basis for a near real time threat classification at the Command and Control (C2) Center where the final data fusion occurs. The QuIPS CONOPS schematic shown in Figure 3 is oversimplified, but describes the QuIPS mission quite well. Remotely sensed data from satellites/national sensors is fused with locally sensed data in near real time in the C2 Center, and, if a track is classified as a threat based on all the fused data and information available for that track, assets will be dispersed to kill the threat. Figure 4 shows the QuIPS multisource, automated non acoustic data fusion engine currently being installed at the FLEET METOC Advanced Concept Lab (FMACL) in San Diego under PEO C4I and Space PMW-150 sponsorship. A detailed description of each of the six modules of QuIPS is given in the following sections, and data processing results are described in a series of appendices attached to this report.

Figure 1. The QuIPS processor and 4D display.

THE QUIET INTERLUDE PROCESSING SYSTEM (QUIPS)

TRACKER/CLUSTER DATA FUSION LEVELS

Apply QuIPS multiple array algorithm todata fuse tracks from multiple acoustic arrays

Classify & Display

Level 5 CONTACTS

Level 4

Swimmer Detection from Two QuIPS arrays with - WQX Very high frequency (VHF) (1 kHz to 65 kHz) data

Spectral Levels

Contact Ellipse

QuIPS-COMwireless

ACOUSTIC TRACKS True Bearing, Range, Freq

Level 3

Merge passive and active Acoustic & ELINT/RADINT/COMINT/ IMINT/MASINT

Contact Yes/No

LOCAL SENSOR IMPUT

Multistatic Active

Merged PW Super CLUSTERS

ELINT NonAcoustic Tracks

Relative Bearing, Freq, Magnitude NPW Merged Super CLUSTERS

Non-Acoustic Track Merge

Level 2 Merged NB Super CLUSTERS

Other Non-Acoustic Ship Track Data RADINT, COMINT, MASINT, IMINT data fusion

BB Super CLUSTERS

Level 1

BB Super Clusters

NB Super Clusters

RB Super Clusters

ELINT HITS

Global ELINT Tracker (GET)

Level 0 Peaks & Surfs

Peaks & Surfs

NB NB NB NB

BB BB BB

Narrowband Beamformers

Broad band Split Array Correlator

RB RB RB ELINT Matched Phase Matched Field Processor

Non-Acoustic Remote Sensing Processor

Figure 2. QuIPS Prater data fusion tree schematic.

Local Sensor to Shooter Connectivity using QuIPS-COM Local QuIPS Sensors

Remote Sensing Data from Satellites

QuIPS-COM

Command and Control Center – QuIPS

Data Fusion Figure 3. QuIPS CONOPS.

Kill Threat

263

264

J.H. WILSON AND A. MELKEVIK

4 The QuIPS non-acoustic surface ship tracking module The QuIPS processing schematic shows that the motion of surface ships must utilize multi-sensor national and local sensor data to accurately estimate surface ship track and speed, especially in crowded coastal environments. There is no one “magic silver bullet” source of ship track data. The success of the QuIPS Global ELINT Tracker (GET) has been the critical breakthrough technology that gives QuIPS a unique and enhanced capability over other operational ELINT trackers, such as the Gaussian ELINT Correlator (GEC) in GCCS-M or the Asymmetrical Statistical Platform (ASP) tracker in GALE Lite. GET outperformed GEC and ASP by a wide margin in the automated mode in side by side comparisons in Fleet Battle Experiment – Echo (FBE-E) based on ground truth tracks for 20 ships. GET is a worldwide source of accurate track data used as the base input to the QuIPS non acoustic surface ship track module shown in the attached figure. GET is now being developed and tested by NSA/Community ELINT Management Office (CEMO), and a new release of GET will be available in September 2005. QuIPS next receives a near real time feed of Platform Track (PT) data from GCCS-M, where the GCCS-M ELINT data is discarded totally because GEC performs so poorly in the automated mode. PT and GET surface ship tracks are then merged. A third source of surface ship track data is the Joint Maritime Information Exchange (JMIE) feed from the USCG at ONI. JMIE contains over 20 sources of surface ship track data, such as LA/LONG versus time, ship name, length, draft, propeller depth, cargo, etc. It is at this point that QuIPS starts to analyze motion and behaviour pattern. Although not yet automated, two behaviour examples noted in recent data are cited. QuIPS detected a merchant ship that was stationary outside Los Angeles harbour for 22 h. The ship than began a Trans Pacific route for no apparent reason. QuIPS raises an alert for this behaviour. Another example is the tuna fishing boat departing San Diego, but then heading to a known poor fishing area and rendezvousing with a merchant ship and returning to port. Again a QuIPS alert is broadcast. A fourth source of surface ship track data is the local surface search radar, and QuIPS inputs the Southern California (SOCAL) Offshore Radar Estimate (SCORE) on San Clemente Island (SCI). QuIPS has worked with SCORE to produce a time tagged track in OTH-Gold format so it can be fused with GET/PT/JMIE merged tracks. Any surface search radar available in the operations area can be used also, just as SCORE is used currently. However, QuIPS needs to write software to input the radar’s motion track indicator (MTI) output into a timetagged track.

THE QUIET INTERLUDE PROCESSING SYSTEM (QUIPS)

265

Dynamic Ship Tracking At The FMACL ELINT GET

File IBSSO

ELINT TRACKS

Noise Estimates M.I.F.C Platform Tracks Elint Tracks (call to delete)

Alameda

Platform File

Merge

DANM-DSM Real Time

Merged Platform Plus GET ELINT Tracks JMIE Tracks plus

File Of Ship Tracks

attributes SCORE TSPI Time Tagged Tracks

Merge attributes

Merged GET, Platform and JMIE Tracks

Merge

TEDServices -bathymetry -geoacoustic database

Convert To Danm input format

Final Ship Track File

QuiPS database And display 4D

Figure 4. QuIPS non-acoustic surface ship track data fusion engine being developed at the FLEET METOC advanced concept lab (FMACL) in San Diego.

5 Conclusion In the near future, QuIPS will be analyzing the Automated Information System (AIS) and commercial SAR data, which will be very useful in advancing the current state of the art in motion/behaviour pattern analysis. Given the GET/PT/JMIE/SCORE merged tracks as a base estimate of surface ship track patterns, AIS will not only supply additional track data to merge with this base pattern set, but will identify those ships NOT complying with International Maritime Organization (IMO) regulations. QuIPS will broadcast an alert for all large ships not conforming to AIS IMO regulations. Commercial SAR data is difficult to acquire in near real time, but Neptune Sciences, Inc. has proposed a way to use a mobile SAR ground station to obtain the data. The spatial resolution of RADARSAT II will be on the order of 3 m, so SAR’s usefulness in ship tracking can no longer be neglected. If a SAR hit does not correlate with an existing QuIPS track, this means the ship is NOT operating radar, comm. equipment, is not a PT, but may be detected by local surface search radar, if in range. Again, this behaviour would trigger a QuIPS alert. Geospatially dispersed data, such as weather hindcasts, NOWCASTs or forecasts, good fishing areas by season and time of year, historical shipping lanes, etc., must be collected to help the assessment of motion/pattern behaviour, but these data are easily accessible.

SENSOR DATA PROCESSING FOR TRACKING UNDERWATER THREATS USING TERASCALE OPTICAL CORE DEVICES

JACOB BARHEN AND NEENA IMAM Center for Engineering Research, Oak Ridge National Laboratory, USA

Abstract. A critical aspect of littoral surveillance (including port protection) involves the localization and tracking of underwater threats such as manned or unmanned autonomous underwater vehicles. In this article, we present a methodology for locating underwater threat sources from uncertain sensor network data, and illustrate the threat tracking aspects using active sonars in a matched filter framework. The novelty of the latter paradigm lies in its implementation on a tera-scale optical core processor, EnLight™, recently introduced by Lenslet Laboratories. This processor is optimized for array operations, which it performs in a fixed point arithmetic architecture at tera-scale throughput. Using the EnLight™ 64D prototype processor, our results (i) illustrate the ability to reach a robust tracking accuracy, and (ii) demonstrate that a considerable speed-up (a factor of over 13,000) can be achieved when compared to an Intel XeonTM processor in the computation of sets of 80K-sample complex Fourier transforms that are associated with our matched filter techniques. Keywords: Optical-core processor, sensor net, active sonar, time-difference-of-arrival (TDOA), matched filters, underwater threat localization, underwater threat tracking

1 Introduction Over the past few decades, a great deal of effort has been devoted to the extraction of spatio-temporal information from an array of spatially distributed sensors [9, 10]. In the area of anti-submarine warfare (ASW), much attention has focused on adaptive beam-forming, primarily in the context of towed arrays [5, 20]. The basic emphasis of such research was to achieve robust detection and direction-of-arrival (DOA) estimation under requirements for auto-calibration of the arrays [21, 22]. Notwithstanding the considerable progress reported over the years, today’s leading paradigms still face substantial degradation in the presence of realistic ambient noise and clutter [13].

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

267

268

J. BARHEN AND N. IMAM

In recent years a number of innovative alternative schemes have been proposed. The concept of vector hydrophone was introduced to capture the vector characteristics of impinging underwater acoustic fields [19]. In contradistinction to conventional arrays, where the DOA is embedded in the spatial phase offsets of the distributed sensors, here the DOA is captured through the intrinsic directionality of each component of the vector hydrophone. Consequently, this technology requires no a-priori information on the signal frequency, and avoids complications related to possible near-field curvature effects. The spatio-temporal inverse filter [18] is a focusing technique developed primarily for medical imaging but with clear underwater acoustics applicability, in which the propagation operator relating (in space and in time) the signal source to the sensor array is inverted in the Fourier domain. Potentially, some of the most promising new ideas of relevance to ASW are emerging from the field of source localization in multimedia applications [14]. There has also been a rapidly growing interest in near–real–time remote detection and localization of under-water threats using information provided by dynamically evolving sensor networks. This interest has been driven by the requirement to improve detection performance against much stealthier targets using ever larger distributed sensor arrays under a variety of operational and environmental conditions. Figure 1 illustrates a typical mission, depicting a submerged threat (here a submarine), a patrol aircraft searching for it, and a field of GPS-capable sonobuoys. The buoys are passive omnidirectional sensors that provide sound pressure measurements of the noise of the target and of the ambient conditions. Once the buoys are placed, the aircraft monitors their transmissions and processes the data to detect, classify and localize the threat. However, demanding calculations need to be performed to achieve source localization, and their complexity is known to increase dramatically with the size of the sensor array. The same observation applies to tracking using active sensors and specially designed waveforms. Both applications, in turn, result in substantial processing power requirements that cannot readily be met with standard, off-the-shelf computing hardware.

Figure 1. Patrol aircraft monitoring GPS-capable sonobuoys.

SENSOR DATA PROCESSING FOR TRACKING UNDERWATER THREATS

269

The focus of this NATO advanced research workshop is on protecting sea ports. We see this activity as a critical aspect of littoral surveillance, which involves, in particular, the localization and tracking of underwater threats such as manned or unmanned autonomous underwater vehicles. This article comprises two parts. First, we discuss methodologies for locating underwater threat sources from uncertain sensor data, which assumes the availability of wavefront time-differences-of-arrival (TDOA) at each array element of a distributed sensor network. Then, we illustrate the target tracking aspects using active sonars in a matched filter framework. The novelty of the latter paradigm lies in its implementation on a tera-scale optical core processor, EnLight™, recently introduced by Lenslet Laboratories [15]. This revolutionary digital optical core processor is optimized for array operations, and offers tera-scale computing capabilities in a limited (native 8-bit) precision, fixed-point arithmetic architecture. The specific objectives of our effort were to (i) demonstrate the ability to achieve robust tracking, and (ii) estimate the speed-up achieved when using an EnLight device as compared to a leading-edge Intel XeonTM processor.

2 Underwater threat localization with a sensor network For discussion purpose, we consider here an array of N sonobuoy sensors, randomly seeding the sea surface and drifting with the wind and currents. The sensors are GPS-capable, and are hence self-localizing. In most conventional threat source localization approaches, one typically makes a number of critical (methodology-limiting) assumptions. Usually, far-field geometry is assumed, which is justified by the nominally small aperture of the sensor array. This allows the use of the plane wave approximation. The second assumption regards the spacing between sensors. Most often, a regular grid is considered, with a grid resolution in excess of half the wavelength of the emitted sound, which prevents the localization of sources emitting at higher frequencies. In a moving sensor field (as depicted in Figure 1), where each individual sensor is subject to random motion, such design assumptions are no longer valid. Here, the sensors may have arbitrary spacing between them, and the aperture of the distributed array may be considerable compared to the distance to the source. Thus, an approach based on time-difference-of-arrival (TDOA) will be considered. It is encapsulated in Figure 2. In a related article [2], we have provided details on how to implement algorithms for estimating the TDOAs on the EnLight™ processor. Here, for completeness, we only recall the basic definition of this concept.

J. BARHEN AND N. IMAM

270 S1

S2

SN

TDOAs for each pair of sensors

Sensor data acquisition

Foundational step for all methodologies Source localization methodologies M1 – M3

M1

M2

M3

• maximum likelihood

• closed form solution of SL obtained directly

• constrained Lagrangian optimization under uncertainty (NOGA)

• iterative least squares

Figure 2. Methodologies for threat source localization.

Let W n m (p ( s ) ) denote the TDOA between sensor n and sensor m for a wavefront originating from a source with position coordinates p ( s ) ( p1( s ) , p2( s ) , p3( s ) ) . Note that the superscript ~refers to transposition. The TDOA is defined as:

W n m (p ( s ) )

| p ( n )  p ( s )| | p ( m )  p|( s )  c c

(1)

where p(n) denotes the position of the n-th sonobuoy, and c represents the speed of sound in the medium. In practice, one usually has to estimate the TDOAs for each sensor pair from signals sn (t) measured at each sonobuoy n. Moreover, the signals at each sensor are sampled, and estimates have to be obtained from finite length sequences. In our implementation [2], a cross correlation method based on frequency domain analysis [11] was used. Once TDOAs are in hand (see Figure 2), several paradigms are available for actually localizing the threat source. Here, we highlight only three localization methodologies (M1–M3). Each of them requires, as a necessary first step, that accurate estimates of TDOAs for each combination of sensors be obtained. 2.1 MAXIMUM LIKELIHOOD LOCALIZ ATION The first methodology (M1) is discussed in [7]. It finds an estimate for the source location given the TDOAs and the sonobuoy positions using either maximum likelihood (ML) [1, 7] or iterative least squares [16] optimization procedures. For computing an ML estimate of a threat source location, let a denote the vector of all non-random parameters that need to be estimated. Let r be the vector of length NWformed from the noisy TDOA measurements. If q (a ) represents the actual values of the TDOAs, as computed by Eq. (1), the conventional observation model is written as

r

q (a )  K

(2)

SENSOR DATA PROCESSING FOR TRACKING UNDERWATER THREATS

271

where K is also a vector of length N W , each element of which represents the zeromean, additive white Gaussian noise. If the measurement noises are uncorrelated, the covariance matrix 6 for noise vector K is diagonal. The likelihood function of r can be written as

L (r | a )

(2S )  N

W

/2

|6|

1 / 2

1

exp{ [r  q (a)] 6 [r  q (a)]} 

(3)

2

By definition, the ML estimate of a is the vector aˆ that maximizes the log likelihood, that is,

aˆ ML

arg a max{L (r | a)}

(4)

with

L (r | a )

1

 [r  q (a)] 6 [r  q (a)] 

(5)

2

To date, the optimization problem of Eqs. (4–5) has been solved using local methods (e.g., Levenberg-Marquardt in [14]), notwithstanding the fact that Lu (r | a) is a highly nonlinear function of a . In work reported to date, this has required the expensive computation of approximate initial guesses. An alternative option would be to explore the use of the TRUST [3] global optimization method, which has been demonstrated as faster (by one to three orders of magnitude in terms of the number of function evaluations required to reach the global optimum) than algorithms previously reported in the open literature. 2.2 CLOSED FORM SOLUTION The second methodology (M2) attempts to directly obtain a closed form solution of the source location. Recently reported results [12] indicate that excellent accuracy can be achieved under minimal operational constraints of sensor noncolinearity. In order to obtain a 3D source location in closed form using N  1 sensors (where one needs at least N t 3 ), an arbitrary sensor from the array (indexed for convenience as sensor-0) is chosen as a reference sensor. Denoting the unknown coordinates of the source by pˆ ( s ) , it is shown in [12] that

pˆ ( s )

u  v Rs

(6)

272

J. BARHEN AND N. IMAM

where

u

v

  ( RW R ) RW z

(u1 , u2 , u3 ) ~

(7a)

  ( RW R ) RW U

( V1 , V2 , V3 ) ~

(7b)

1

1

1

1

1

1

In Eq. (6), Rs denotes the Euclidean distance between the source and the reference sensor, W is a weighting matrix that captures uncertainties in the measurement of range differences U . The latter are defined as Un Rns  Rs , where

Rns denotes the Euclidean distance between sensor n and the source. Finally, the regressor matrix R and the auxiliary vector z are specified as

§ p1(1) ¨ R ¨ # ¨ p( N ) © 1

p2(1) # p

(N ) 2

p3(1) ·

¸ # ¸ p3( N ) ¸¹

§ ( p1(1) )  ( p2(1) )  ( p3(1) )  U ¨ z ¨ # ¨ ( p( N ) )  ( p( N ) )  ( p( N ) )  U 2 3 © 1 2

2



2



2

2

2



N

· ¸ ¸ ¸ ¹

(8)

Inserting Eq. (6) into the definition of Rs , one obtains a quadratic equation in Rs , which allows one to express Rs in terms of the components of u and v . This, in turn, enables one to determine pˆ ( s ) . Note that [12] provides detailed conditions for which a closed form solution can be found. 2.3 NOGA-BASED ALGORITHMS We now briefly introduce a third methodology (M3). Its novelty for threat source localization resides in the fact that it enables the simultaneous estimation of uncertain TDOAs and the threat source location. Our approach builds upon the NOGA (Normalized Orthogonal Gradient Algorithm) methods [4] that were developed for the uncertainty analysis of nonlinear systems. The model parameters and inputs are collected in the vector a . Now, a includes not only the (unknown) source coordinates, but also the coordinates of all sensors. Using the same notation as for methodology M1 above, the TDOAs calculated by Eq. (1) are denoted by q , whereas the TDOA estimates obtained from sensor measurements are represented by r. In the NOGA nomenclature, q and r will be called (model predicted and sensor based) system responses. The nominal uncertainties in the parameters a are quantified by specifying their covariance matrix,

SENSOR DATA PROCESSING FOR TRACKING UNDERWATER THREATS

Ca

'a'a

273

(9)

In Eq. (9), 'a denotes a vector of standard deviations; in particular, 'a j represents the standard deviation (uncertainty) of parameter j, and the brackets imply expectation, i.e., integration over a joint probability density function (PDF). Many uncertainty analysis methods rely on specific forms of the PDF. We depart from this tradition and adopt a more general viewpoint by specifying the (typically unknown) PDF only in terms of its moments. Here, we consider only the first two, i.e., the mean value and the covariance matrix. They can be calculated directly from empirical observations without resorting to a specific PDF form. In particular, if only the standard deviation for each parameter is known (or estimated), the initial covariance matrix would be diagonal. Within the NOGA framework, sensitivities provide a systematic way to propagate uncertainties in complex, non-stationary, nonlinear models. They may be efficiently computed using adjoint operator formalisms or automated differentiation technology. For example, to first order in a stationary system, the sensitivity of the calculated response n to parameter i evaluated at the nominal parameter values used in the model is given by:

S ni

wq n

(10)

wa i a

Using the sensitivity matrix S , we can readily calculate the nominal covariance matrix of the model predicted (calculated) responses. By expanding about the centroid of the joint PDF of system parameters, we obtain, again, to first order:

Cq

'q 'q

S C a S

(11)

Note that even if the nominal covariance matrix of the system parameters is diagonal, the covariance matrix of the computed model responses is a dense matrix. To locate the threat source, we seek joint best estimates for the parameters and responses. We denote these quantities by aˆ and qˆ . They can be obtained by consistently combining model predictions and their associated nominal uncertainties with the corresponding sensor-derived information and its associated uncertainties. The values of aˆ and qˆ are related to the nominal estimates by the sensitivities. To first order:

qˆ  r q  r  S  aˆ  a

(12)

274

J. BARHEN AND N. IMAM

It is convenient to define two new vectors of auxiliary variables. Let x = aˆ - a, and y = qˆ - r. The discrepancy between calculated model predictions and the corresponding sensor-based information is then expressed as e = q - r. Using the new variables, the local functional relationship assumed in Eq. (12) between the system parameters and responses becomes y = S x + e . In order to simultaneously estimate the TDOAs and the threat source coordinates, we carry out a Lagrangian optimization of a generalized Bayesian loss function Q that simultaneously minimizes the differences between (i) the best estimate responses and the sensor based responses, and (ii) the best estimates and the calculated parameters [4]. The optimization process is carried out in a nonEuclidean space, i.e., a space where the metric tensor used in computing distances is proportional to the inverse of a generalized total covariance matrix, which includes bloc covariances. These blocs are Cr for the sensor based responses, Ca for the model parameters, and Cp for their cross correlations. The Bayesian loss function must be constrained in terms of the functional relationship between parameters and responses. In Eq. (12) above, this relationship was locally approximated to first-order only. To be specific,

ªCr «  Q > y x ..@ C « p «¬ ...

Cp ...º Ca ...»» ... ... »¼

-1

ªy º «x ». « » «¬ ..»¼

(13)

In the above expression, the three-dot patterns denote additional covariance and cross-covariance contributions that may be included, and which may account, for example, for biases (i.e., systematic uncertainties). One can now construct an augmented Lagrangian, L , in the form

L

Q  O T >S x - y + e@

(14)

where O denotes a vector of Lagrange multipliers. The necessary conditions for a locally-optimal solution are obtained by requiring that the partial derivatives of the augmented Lagrangian with respect to x and y be zero. We solve these equations in conjunction with the constraints satisfaction to obtain the optimal values of x and y, which determine the best estimates for both the parameters and responses. We can now specify the explicit solutions of the source localization problem. Assuming no bias contribution to the optimization metric, we find after some lengthy algebra not shown here: aˆ

  C S ) O a  (C a p

(15)

SENSOR DATA PROCESSING FOR TRACKING UNDERWATER THREATS

275

where

O

  C S  SC S )1 e . ( Cr  SC a p p

(16)

The vector aˆ includes the best estimates of the unknown threat source coordinates. In a similar fashion, the best estimates for the TDOAs are obtained from the expression

qˆ r  ( C r  Cp S ) O

(17)

The covariance matrix corresponding to the best estimates of the system parameters is given by

Caˆ

 p  Cp S  S Ca S ) 1 (Cp  SCa ). (18) Ca  (Cp  Ca S )( Cr  SC

For the TDOAs, the uncertainties are computed from

Cqˆ

 ). Cr  (Cr  Cp S )( Cr  S Cp  Cp S  S Ca S ) -1 (Cr  S C p

(19)

Note that the NOGA methodology is entirely based on matrix-matrix or matrixvector operations. This makes it ideally suited for implementation on array processors such as the EnLight™ platforms, which will be described in Section 4.

3 Tracking underwater threats The algorithm presented in this section involves matched filter (MF) correlation processing via fast Fourier transform (FFT) of broadband Doppler-sensitive waveforms (DSW). The implementation of MFs for broadband DSWs on the EnLight platform is presented in Section 5 with the aid of numerical simulation and actual optical hardware runs. The keys to reliable target tracking are proper waveform selection, accurate signal and system modeling, and efficient real-time signal processing using an MF bank implementation. The common waveforms used in underwater threat detection via sonar signal analysis have diverse and complimentary characteristics. For example, constant frequency (CF) pulses provide superior range-rate estimation but poor range resolution capabilities [6]. The reverberation clutter power vs. Doppler shift of a CF pulse is also more concentrated than that of linear frequency modulated (LFM) signals, another common candidate waveform in sonar tracking systems. To resolve the inherent conflict between reliable detection and good range resolution, signals other than the simple CF pulses have to be used [17].

276

J. BARHEN AND N. IMAM

3.1 AMBIGUITY FUNCTION To improve the range resolution capabilities while preserving reliable detection performance, which is determined by the total amount of energy in the echo signal reflected from the target, it is necessary to employ signals with increased bandwidth and significant energy distribution over a wide range of frequencies. Signals with large time bandwidth products are classified as broadband signals. In most long-range active sonar receivers, a MF is used to reject ambient noise. The MF is a correlator whose output is represented by the ambiguity function (AF). The two-dimensional AF, F s (M ,W ) 2 , maps the correlator output power against range (determined by time delay IJ) and Doppler frequency shift ij>21]. The expression of F for an arbitrary waveform s(t) is f

F s (M , W )

³ s (t ) s (t  W ) exp *

2 jSM

dt .

(20)

f

The AF specifies the output of the matched filter in the absence of noise. Thus the two-parameter correlation function of the signal waveform, with respect to both time displacement W and frequency displacement ij, is a measure of the relative response of the correlator to the echo and reverberation signals. The region of the W and ij domain, within which the AF function still has a significant peak relative to the central peak, is called the region of signal ambiguity. The AF is a useful tool for MF system analysis and provides a starting point for waveform synthesis. The sonar operating environment is extremely hostile due to ambient noise and reverberation interference. Reverberation is the sum of signals returned from unwanted paths, and produces large amplitudes in the correlator output at the W and ij values where the target echo signal is to be manifested. A common scheme for doing reverberation suppression is to spread the energy of the transmitted pulse over a broad bandwidth in the case of Frequency Modulated (FM) signals. If the FM function of a waveform is periodic, the spectrum of the waveform will consist of many individual spectral lobes spaced at multiples of the repetition frequency. The energy of the pulse will therefore be spread out in the frequency domain. The periodic sidelobes of the AF will enclose a smaller portion of the reverberation distribution than a CF pulse. This leads to a corresponding decrease in the reverberation level. One of the most common types of comb spectrum arises from a periodic sinusoidal frequency modulated (SFM) waveform. The MF implementation for a benchmark SFM signal is presented next.

SENSOR DATA PROCESSING FOR TRACKING UNDERWATER THREATS

277

3.2 MATCHED FILTER IMPLEMENTATION The MF is central to ASW applications. Fundamentally, the MF is a correlator that compares the received signal with a hypothesized signal. The output of the matched filter gives a measure of how well the hypothesized signal matches the received signal as function of a set of parameters, usually the range and velocity of targets [8]. A matched filter may be considered a time-sequence array that isolates a signal in signal space in the same sense that a space array isolates a direction in real space. Modern-day sensor platforms employ a bank of matched filters, the ith of which is tuned to a particular Doppler frequency. The target range and radial velocity can be obtained by passing the received signal through an MF bank, where each filter in the array is matched to a different target velocity as shown in Figure 3. A sufficient number of filters must be employed to span the range of probable target velocities. The output of each filter is then input to a simple threshold detector. When the threshold is exceeded, the time delay provides target range estimates. The estimated velocity is that for which the correlation peak magnitude of the filter output has the maximum value. The output of the correlator is calculated via Fast Fourier Transform (FFT), followed by inverse FFT. The vectors representing the discrete replica and echo signals can have considerable size in the case of broadband signals and the size of the FFT can easily exceed 100K complex samples. Hence, it was anticipated that the very computationally expensive implementation of broadband matched filtering would be expedited on the EnLight optical processor. The next section introduces the main architecture and performance details of the EnLight device.

Figure 3. Schematics of a matched filter.

4 The EnLight optical core processor Recently, Lenslet Inc. introduced the novel EnLight™ processing platform [15]. The EnLight™256 is a small-factor digital signal-processing chip (5 × 5 cm) with

278

J. BARHEN AND N. IMAM

an optical core. The processor is optimized for array operations, which it can perform in fixed point arithmetic at the rate of 16 TeraOPS at 8-bit precision. This is substantially faster than the fastest field-programmable gate arrays (FPGA) or digital signal processors (DSP) available today. The optical core performs matrixvector multiplications (MVM), where the nominal matrix size is 256 × 256. The system clock is 125 MHz. At each clock cycle, 128K multiply-and-add operations per second (OPS) are carried out, which yields the peak performance of 16 TeraOPS. Before starting production of the EnLight™256 processor, Lenslet built the EnLight™64D board, shown in Figure 4. This is a prototype demonstrator for the optical processing technology, with a reduced size 64 × 64 optical core. In our proof-of-concept effort, we have used the 64D for all hardware tests. To project scale-up capabilities, we invoked an EnLight™256 bit-exact simulator.

Figure 4. The EnLight™64D demonstrator.

The EnLight™64D is specified as follows. Its clock operates at 60 MHz. The optical core has 64 input channels (configured as 256 vertical cavity surface emitting lasers, bundled in groups of 4 per channel). The optical core performs the MVM function at the rate of 60 × 106 × 642 × 2 = 492 Giga operations per second. Each of the 64 data components in the input and output channels has an 8-bit accuracy, which results in a data stream of 60 106 × 64 × 8 bits/s = 30.7 Giga bits/s.

5 Results In this study, we are interested in demonstrating the ability of the EnLight computing platform to robustly track an underwater threat source. For the purpose of the numerical simulations and hardware implementation, a number of operational simplifications are made. In particular, we assume that: only a single target is present during the detection process; the speed of sound is constant along the propagation path; and the active sonar location is known. In active sonar systems, the proper selection of the transmitted waveform is crucial for target detection and parameter estimation, especially with the existence of reverberation. Several new classes of DSW pulses have been proposed, which theoretically provide superior reverberation processing to CF pulses by virtue of their

SENSOR DATA PROCESSING FOR TRACKING UNDERWATER THREATS

279

comb-like spectra. The simplest case of a signal with comb-like spectrum is the SFM signal where the modulating waveform itself is also a sinusoid. Thus,

s(t)

w(t)exp

j2 S f o t  j E sin(2 S f m t )

(21)

The function w(t) is the amplitude window of the pulse, f0 is the center frequency, fm is the modulation frequency, and Eis the modulation index, which defines the bandwidth of the SFM signal as B = 2 fm (1 +E). The power spectrum of the SFM waveform is a symmetrical comb centered on f0 with a frequency spacing of fm. By comparison with a CF pulse, where the energy is concentrated in a single mainlobe, the signal energy of the SFM pulse is distributed between several peaks. The result is a corresponding reduction in reverberation interference. For this benchmark study, the source was assumed to transmit a SFM waveform at a center frequency f0 = 1,200 Hz, with a 400 Hz bandwidth B. The pulse duration used was T = 1, the frequency was modulated at 5 Hz, and the Doppler scale ranged from -5 to +5 m/s. The sampling frequency fs assumed was 5,000 Hz, resulting in a sampling interval length of Ts = 0.0002 s. To benchmark the EnLight performance, two computer codes were written, one using the Intel Visual FORTRAN-95 compiler, the other in MATLAB. The former was needed to enable the fastest possible execution on an Intel IA-32 dual Xeon processors system. The later readily interfaces with the software of the EnLightTM256 simulator, which is used to design the actual algorithm that was run both on the EnLightTM64D hardware platform, or is used to project the scaled performance for the EnLightTM256.

4

Outputs of Filter#1

6

x 10

Outputs of Matched Filter banks for received signal velocity matched to Filter#1

4

2 0

0

2000

4000

6000

8000 Time (ms)

10000

12000

14000

16000

2000

4000

6000

8000 Time (ms)

10000

12000

14000

16000

2000

4000

6000

8000 Time (ms)

10000

12000

14000

16000

4

Outputs of Filter#2

4

x 10

2

0

0

4

Outputs of Filter#3

2

x 10

1.5 1 0.5 0

0

Figure 5. Matched filter bank output for a filter bank of 32 Doppler bins. Not all filters are shown. Filter 1 has the closest match to the target velocity.

J. BARHEN AND N. IMAM

280

For the MF simulation, a synthetic echo is generated for a particular target range and velocity. The echo signal is correlated with a bank of replicas. Spectral techniques are used. The correlation with the highest magnitude provides an estimate of the Doppler velocity bin. The location of the maximum within that correlation yields the time delay of the echo, and thus provides an estimate of the range. For this benchmark study, the assumed target range and (incoming) velocity were 3 km and -5 m/s respectively. The speed of sound was taken to be 1,500 m/s. A MF bank with 32 Doppler bins was implemented, with each filter performing an 80K-sample complex FFT to calculate the cross-power spectrum and an 80K-sample inverse FFT to obtain the cross-correlation output of the filter. As can be seen in Figure 5, the output of the first filter has the closest velocity match to the received signal. The estimated target delay is 4 s and hence the estimated target range is 3 km, with correct incoming direction and target velocity. In this proof-of-concept study, an EnLightTMD hardware prototype was compared to a dual-processor Intel Xeon (2 GHz) system. A speed-up factor of over 13,000 per processor (for a series of 80K-sample complex FFTs corresponding to 32 Doppler cell banks and 1 target echo) was achieved. The computation on the Xeon system was performed in Intel Visual FORTRAN-95, to achieve the highest possible performance. Figures 6 compare the MATLAB simulation of the first filter output with the EnLightTMD hardware runs. As can be seen in high resolution Figure 6, the numerical accuracy of the hardware runs compares very favorably with the high precision MATLAB simulation and with the theoretical results. -30 MATLAB Alpha

output of filter #1, dB

-35

-40

-45

-50

Figure 6. High resolution plot of matched filter bank output (for filter 1) obtained via hardware run on the EnLight D demonstration board.

SENSOR DATA PROCESSING FOR TRACKING UNDERWATER THREATS

281

6 Conclusions and future research To achieve the real-time performance required for underwater threat source localization and tracking, many existing algorithms need to be revised and adapted to the emerging revolutionary computing technologies. These include FPGAs, emerging multicore architectures, and optical devices. The EnLight terascale optical core processor represents one such revolutionary advance. In that context, our future efforts will focus on demonstrating computational speed and parallel implementation efficiency on multicore architectures for relevant maritime sensing applications; quantifying the speed-up achieved per processor as compared to a leading-edge conventional processor or DSP; determining the scaling properties per processor as function of the number of sensors present in the detection, tracking, and discrimination network; characterizing the SNR gain and detection improvement as function of array size and geometry. An innovative approach to high performance, massively parallel computing remains a key factor for progress in science and National Defence applications. In contrast to conventional approaches, one must develop computational paradigms that exploit, from the onset (1) the concept of massive parallelism, and (2) the physics of the implementation device.

Acknowledgments Funding for this work was provided by the Office of Naval Research. The authors acknowledge the support and guidance of Dr. M. Wardlaw (ONR) and useful discussions with A. Averbuch and M. Berlin (Lenslet, Inc). Oak Ridge National Laboratory is managed for the United States Department of Energy by UTBattelle, LLC under contract DE_AC05-00OR22725 .

References 1.

2.

3. 4. 5. 6. 7.

Ajdler T, Kozintsev I, Lienhart R, Vetterli M. 2004. Acoustic source localization in distributed sensor networks. Asilomar Conference on Signals, Systems and Computers. Pacific Grove, CA, CD ROM IEEE Press. Barhen J, Imam N, Vose M, Averbuch A, Wardlaw M. 2007. Underwater Threat Source Localization: Processing Sensor Network TDOAs with a Terascale Optical Core Device. NATO ASI book series, editors Levebre E et al. pp 56–68, IOP Press. Barhen J, Protopopescu V, Reister D. 1997. TRUST: a deterministic algorithm for global optimization. Science, 276, 1094–1097. Barhen J, Protopopescu V, Reister D. 2004. Consistent uncertainty reduction in modeling nonlinear systems. SIAM Journal of Scientific Computing, 26, 653–665. Burdick W. 1984. Underwater Acoustic System Analysis. Englewood Cliffs, NJ: Prentice Hall. Burdick W. 1984. Underwater Acoustic System Analysis, New York: Prentice Hall. Chan Y, Ho K. 1994. A simple and efficient estimator for hyperbolic location, IEEE Transactions on Signal Processing, 42(8), 1905–1915.

282 8.

9. 10. 11.

12.

13.

14.

15. 16. 17. 18.

19.

20.

21.

22.

J. BARHEN AND N. IMAM Doisy Y, Deruaz L, Beerens SP, Been R. 2000. Target Doppler estimation using wideband frequency modulated signals. IEEE Transactions on Signal Processing, 48, 1213–1224. Klemm R. 1998. Space – Time Adaptive Processing. The Institution of Electrical Engineers. London: IEE Press. Klemm R, ed. 2004. Applications of Space – Time Adaptive Processing, The Institution of Electrical Engineers. London: IEE Press. Knapp C, Carter G. 1976. The generalized correlation method for estimation of time delay. IEEE Transactions on Acoustics, Speech and Signal Processing, 24(4), 320– 327. Mellen G, Pachter M, Raquet J. 2003. Closed-form solution for determining emitter location using time-difference-of-arrival measurements. IEEE Transactions on Aerospace and Electronic Systems, 39(3), 1056–1058. Nuttall A, Wilson J. 2000. Adaptive beamforming at very low frequencies in spatially coherent, cluttered noise environments with low signal-to-noise ratio and finiteaveraging times. Journal of the Acoustical Society of America, 108(5), 2256–2265. Raykar V, Kozintsev I, Lienhart R. 2005. Position calibration of microphones and loudspeakers in distributed computing platforms. IEEE Transactions on Speech and Audio Processing, 13(1), pp. 70–83. Sariel A, Halperin A, Levit S. U < RL: www.lenslet.com> . See also: Control Engineering at http://www.controleng.com/article/CA491825.html> . Schmidt R. 1996. Least squares range difference location. IEEE Transactions on Aerospace and Electronic Systems, 32(1), 234–241. Stewart JL, Westerfield EC. 1959. A theory of active sonar detection. Proceedings of the IRE, 47, 872–881. Tanter M, Aubry JF, Gerber J, Thomas JL, Fink M. 2001. Optimal focusing by spatiotemporal inverse filter: I. Basic principles. Journal of the Acoustical Society of America, 110(1), pp. 37–47. Tichavsky P, Wong KT. 2001. Near-field/far-field azimuth and elevation angle estimation using a single vector hydrophone. IEEE Transactions on Signal Processing, 49(11), 2498–2510. Tichavsky P, Wong KT. 2004. Quasi-fluid-mechanics-based quasi-Bayesian CramerRao bounds for deformed towed-array direction finding. IEEE Transactions on Signal Processing, 52(1), pp. 36–47. Van Buren A. 1991. Near-field transmitting and receiving properties of planar nearfield calibration arrays. Journal of the Acoustical Society of America, 89(3), 1423– 1427. Viberg M, Swindlehurst AL. 1994. A Bayesian approach to auto-calibration for parametric array signal processing. IEEE Transactions on Signal Processing, 42(12), 3495–3507.

A NEW NEURAL APPROACH FOR PATTERN RECOGNITION IN SPACE IMAGERY

VICTOR-EMIL NEAGOE1 AND ARMAND-DRAGOS ROPOT2 1 Department of Applied Electronics & Information Engineering, Polytechnic University of Bucharest, Romania 2 UTI Systems, Bucharest, Romania

Abstract. This paper presents multispectral space image classification using the new neural model, called Concurrent Self-Organizing Maps (CSOM), representing a winner-takes-all collection of small modular self-organizing neural networks. The performance of this classifier is compared with the performance of Bayesian classifiers. The implemented neural/statistical classifiers are evaluated using a LANDSAT TM image with 7 bands composed by a set of 7-dimensional pixels, a subset of which contains labelled pixels corresponding to 7 thematic categories. The best experimental result leads to a recognition rate of 95.29%. The model has potential applications for harbour protection. Keywords: Neural networks, pattern recognition, multispectral space imagery, concurrent self-organizing maps

1 Introduction Satellite image processing has many applications and can be used to generate various kinds of maps: maps of vegetation, maps of mineral resources of the Earth, land-use maps (civil or military buildings, agricultural fields, woods, rivers, lakes, and highways), etc. The standard approach to satellite image classification uses statistical methods. A relatively new and promising category of techniques for satellite image classification is based on neural models [1, 2]. Conclusions drawn as the result of the research on utilizing neural networks for classification of satellite imagery are the following: x Neural classifiers do not require initial hypotheses on data distribution and are able to learn non-linear and discrete input data. x Neural networks can be successfully used for texture classification. x Neural classifiers are generally more accurate than statistical classifiers. x The architecture of neural networks is very flexible and can be easily adapted to improve the performance of a particular application. E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

283

284

V.-E. NEAGOE AND A.-D. ROPOT

The Self-Organizing Map (SOM) (or Kohonen network) [3] is an artificial unsupervised network characterized by the fact that its neighbouring neurons develop adaptively into specific detectors of different vector patterns. The neurons become specifically tuned to various classes of patterns through a competitive, unsupervised or self-organizing learning process. The spatial location of a neuron in the network (given by its coordinates) corresponds to a particular input vector pattern. We present here a new neural recognition model, called Concurrent SelfOrganizing Maps (CSOM, which uses the SOM as a cell characterizing a specific class only. This new CSOM model is built as a collection of small SOMs by using a global competition strategy. It has been successfully applied in [2], [4–7] for image and speaker recognition. We further evaluate the applicability of this model to multispectral satellite image recognition and compare its performance with the SOM and the well known Bayes statistical classifier.

2 Pattern classification An a Concurrent Self-Organizing Map (CSOM) is a collection of small SOMs that uses a global winner-takes-all strategy. Each unit network (SOM) is employed to correctly classify the patterns of one class only and the number of networks is equal to the number of classes. The CSOM training technique is a supervised one, while each individual SOM is trained on a specific training set independently in a competitive non-supervised manner on a specific training set. The CSOM training model is shown in Figure 1(a). At the recognition step, a pattern is question is used in parallel as input to every previously trained SOM. The winner is determined based on the least quantization error and its index is the class index that the pattern belongs to see Figure 1(b).

Input Pattern

DATABASE

Pattern Set “1”

Pattern Set “2”

Pattern Set “n”

SOM “1”

SOM “2”

SOM “n”

Minimization of Quantization Error

SOM “1”

SOM “2”

(a)

SOM “n”

CLASS

(b)

Figure 1. (a) CSOM model (training phase); (b) CSOM model (classification phase).

A NEW NEURAL APPROACH FOR PATTERN RECOGNITION

285

3 Classification of multispectral satellite imagery 3.1 THE SATELLITE IMAGE DATABASE To train and test the proposed CSOM classification model as well as the classical SOM and the Bayes classifier used for comparison we utilize a LANDSAT TM image with 7 bands (Figure 2(a)–(g)) totalling 368,125 pixels (7-dimensional) 6,331 pixels of which are classified by an expert into 7 thematic categories (classes): a – urban area, b – barren fields, c – bushes, d – agricultural fields, e – meadow, f – woods, g – water.

Figure 2. (a) Spectral band 1, (b) spectral band 2, (c) spectral band 3, (d) spectral band 4, (e) spectral band 5, (f) spectral band 6, (g) spectral band 7, (h) calibration image.

3.2 EXPERIMENTAL RESULTS OF CSOM SATELLITE IMAGE CLASSIFICATION Each multispectral pixel (7 bands) is characterized by a corresponding 7dimensional vector containing the pixel projections in each band. These vectors are used as input to the neural/statistical classifier. We experimented with the following neural and statistical techniques in the classification of satellite images: x The new CSOM model x The classical SOM classifier x The Bayesian classifier with normal a priori distribution for each 7 classes The results of simulation are given in Tables 1–6. Two classified multispectral images are given in Figure 3 and the corresponding histograms are shown in Figure 4. The recognition rates for the training lot and for the test lot are shown in Figure 5.

286

V.-E. NEAGOE AND A.-D. ROPOT

Figure 3. (a) Classified multispectral pixels (7 categories) using a circular CSOM architecture with 7 x 112 neurons (recognition rate 95.29%); (b) classified multispectral pixels (7 categories) using a circular SOM architecture with 784 neurons (recognition rate 94.31%). Histogram of the SOM classified image

Histogram of the CSOM classified image A: B: C:

7.21% : Urban area 32.76% : Urban area 20.17% : Bushes

A: B:

D: E:

10.36% : Agricultural fields 21.61% : Meadows

D:

F: G:

5.21% 35.53% 19.73% 13.13% 19.29% 4.58% 2.61%

C: E: F: G:

4.75% : Woods 3.11% : Water

: Urban area : Urban area : Bushes : Agricultural fields : Meadows : Woods : Water

Figure 4. (a) Histogram of the classified multispectral LANDSAT TM image given in Figure 3. ((a) using CSOM); (b) histogram of the classified multispectral LANDSAT TM image given in Figure 3. ((b) using SOM). TABLE 1. Experimental results of multispectral satellite image classification with CSOM, SOM and Bayes classifiers (7 thematic classes; input vector space of dimension 7). Nr

Type of classifier

Total number of neurons

Number of networks

Recognition score for the training lot (%)

Recognition score for the test lot (%)

1

Circular CSOMs (7 x 112) Circular SOM Linear CSOMs (7 x 112) Linear SOM

784

7

98.71

95.29

784

1

96.49

94.31

784

7

98.64

95.10

784

1

97.06

94.12

Rectangular CSOMs [7 x (14 x 8)] Rectangular SOM (28 x 28) Bayes classifier

784

7

97.98

95.07

784

1

96.53

92.80

95.83

94.22

2 3

4 5

6

7

784

A NEW NEURAL APPROACH FOR PATTERN RECOGNITION

287

TABLE 2. Comparison of the best pixel classification scores obtained by SOM and CSOM for the training lot as a function of the number of neurons. Number of neurons Recognition rate (%)

49

98

196

392

784

SOM

91.60

93.53

95.10

95.86

97.06

CSOM

93.27

95.01

96.62

97.76

98.71

Bayes 95.83

TABLE 3. Comparison of the best pixel classification scores obtained by SOM and CSOM for the test lot as a function of the number of neurons. Number of neurons Recognition rate (%)

49

98

196

392

784

SOM

92.04

93.87

93.62

94.34

94.31

CSOM

92.86

94.79

93.71

94.85

95.29

Bayes 95.17

TABLE 4. Confusion matrix for the circular SOM with 784 neurons (test lot). Assigned class Real class A B A’ 80.0 0.08

C 1.97

D 0.00

E 0.00

F 0.21

G 0.62

Total (%) 1.96

B’

8.57

99.41

0.66

0.00

0.00

0.00

0.00

37.54

C’

5.71

0.17

73.68

0.33

0.48

4.95

3.73

4.80

D’

0.00

0.00

1.97

96.45

0.00

9.28

0.00

29.00

E’

0.00

0.00

0.66

0.00

98.55

0.00

0.00

6.48

F’

0.00

0.00

14.47

2.77

0.00

84.74

1.86

14.57

G’

5.71

0.08

4.61

0.00

0.00

0.41

93.79

5.21

Unclassified

0.00

0.25

1.97

0.44

0.97

0.41

0.00

0.44

Total (%)

2.21

37.54

4.80

28.50

6.54

15.32

5.09

100.00

TABLE 5. Confusion matrix for the circular CSOMs with (7 x 112) neurons (test lot). Assigned class Real class A’ B’ C’ D’ E’ F’ G’

A 90.00 2.86 4.29 0.00 0.00 0.00 2.86

B 0.25 99.58 0.17 0.00 0.00 0.00 0.00

C 0.00 0.00 84.87 1.32 0.00 5.26 8.55

D 0.22 0.00 0.44 95.79 0.00 3.55 0.00

E 0.00 0.00 1.45 0.00 98.55 0.00 0.00

F 0.21 0.00 6.80 6.39 0.00 85.77 0.82

G 0.00 0.00 2.48 0.00 0.00 0.00 97.52

Total (%) 2.18 37.44 5.62 28.34 6.45 14.41 5.56

Unclassified

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

Total (%)

2.21

37.54

4.80

28.50

6.54

15.32

5.09

100.00

288

V.-E. NEAGOE AND A.-D. ROPOT

TABLE 6. Training time required by the best SOM and CSOM as a function of the number of neurons (for multispectral pixel classification). Number of neurons

49

98

196

392

784

Training

SOM

276

545

1140

2040

4872

time (s)

CSOM

56

93

171

423

1020

Rerognition rate [%] CSOM SOM Bayes

98 97 96

Legend CSOM SOM Bayes

Rerognition rate [%]

Legend

99

97 96 95

95

94

94 93 93 92

92

91

91 49

98

196 392 784 Neurons

(a)

49

98

196

392

784 Neurons

(b)

Figure 5. (a) Best pixel recognition rates on the training lot as a function of the total number of neurons; (b) best pixel recognition rates on the test lot as a function of the total number of neurons.

4 Concluding remarks x The new CSOM model comprises a collection of SOMs, with each network trained to correctly classify the patterns of one class only. The decision is based on a global winner-takes-all strategy. x Both the neural classifiers (the new CSOM and the well known SOM) and statistical classifier (Bayes) perform well for recognition of vegetation in multispectral LANSAT images. However, the CSOM model produces slightly better results for all considered variants in comparison to SOM and Bayes classifiers. x The best results (a pixel classification rate of 95.29% for the test lot) are obtained using a CSOM model containing 7 circular SOMs with 112 neurons in each of them. Taking into account the varying architectures of CSOM, the best architecture for this application is circular, followed by linear and rectangular. x The CSOM model requires significantly less training time in comparison to the single SOM and does not require a priori knowledge about pixel distribution as Bayesian classifiers. x The CSOM model may be extended for use in ship recognition with image sensors like Synthetic Aperture Radar (SAR) or Forward Looking Infra-Red

A NEW NEURAL APPROACH FOR PATTERN RECOGNITION

289

(FLIR). For such applications, a specific preprocessing stage is required for feature extraction.

References 1.

2.

3. 4.

5.

6.

7.

Benediktsson JA, Swain PH, Ersoy OK. 1990. Neural network approaches versus statistical methods in classification of multisource remote sensing data. In: IEEE Trans. Geosci. Remote Sens., Volume GE-28, no. 4. p 540–552. Neagoe V, Ropot A. 2002. Concurrent self-organizing maps for classification of multispectral satellite imagery. Proceedings of 6th World Multiconference on Systemics, Cybernetics and Informatics Volume XIV; 2002 July 14–18. Orlando (FL). p 126–131. Kohonen T. 1995. Self-Organizing Maps. Berlin: Springer. Neagoe V. 2001. Concurrent self-organizing maps for automatic face recognition. Proceedings of the 29th International Conference of the Romanian Technical Military Academy; 2001 Nov. 15–16. Romania: Technical Military Academy, Section 9. p 35– 40. Neagoe V, Fratila I. 1999. A neural segmentation of multispectral satellite images. In: Reusch B, editor. Computational Intelligence, Theory and Applications. Berlin/New York: Springer. p 334–341. Neagoe V, Ropot A. 2002. Concurrent self-organizing maps for pattern classification. Proceedings of First IEEE International Conference on Cognitive Informatics, ICCI2002; 2002 Aug. 19–20. Calgary (CA). Neagoe V, Ropot A. 2004. Concurrent self-organizing maps - a powerful artificial neural tool for biometric technology. Proceedings of World Automation. Calgary (CA).

CONCURRENT SELF-ORGANIZING MAPS – A POWERFUL ARTIFICIAL NEURAL TOOL FOR BIOMETRIC TECHNOLOGY

VICTOR-EMIL NEAGOE1 AND ARMAND-DRAGOS ROPOT2 1 Department of Applied Electronics & Information Engineering, Polytechnic University of Bucharest, Romania 2 UTI Systems, Bucharest, Romania

Abstract. In this paper, we present Concurrent Self-Organizing Maps (CSOM), a new artificial neural classification model representing a winner-takes-all collection of small SOM units. We consider two significant areas of CSOM applications in Biometric Technology: face recognition and speaker recognition. For the ORL face database of 40 subjects, CSOM yields a recognition score of 91%, while a single, large SOM yields a score of only 71%! For a speaker database provided by 25 talkers, a recognition score of 92.17% was obtained using CSOM, compared to the recognition rate of 79.63% yielded by the SOM. This model may be applied in access control applications for harbour protection. Keywords: Concurrent Self-Organizing Maps, biometric technology, face recognition, speaker recognition

1 Introduction The prospect of new large-scale identity systems dominated the headlines in 2004, and there is no sign that this is about to change. Governments and private companies worldwide are putting biometric technology at the heart of ambitious projects, ranging from access control and company security to high-tech passports, ID cards, and driving licenses. Two major areas of biometric technology are face recognition [1] and speaker recognition [2]. The Self-Organizing Map (SOM) known also as the Kohonen network [3, 4], is an artificial unsupervised network characterized by the fact that its neighbouring neurons develop adaptively into specific detectors of different vector patterns. The neurons become specifically tuned to various classes of patterns through competitive, unsupervised or self-organizing learning. The spatial location of a neuron in the network (given by its coordinates) corresponds to a particular input vector pattern. Similar input vectors correspond to the same neuron or to neighbouring neurons. One important characteristic of the SOM is that it can E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

291

292

V.-E. NEAGOE AND A.-D. ROPOT

extract the statistics of input vectors and simultaneously performs classification. Starting from the idea that the SOM is a cell characterizing a specific class only, we present and evaluate Concurrent Self-Organizing Maps (CSOM) described in [5–7] is a new artificial neural classification model comprising a collection of small SOMs, which uses a global winner-takes-all strategy. Each SOM is used to correctly classify the patterns of one class only and the number of networks equals the number of classes. We further examine the performance of the CSOM model in two applications significant to Biometric Technology: face recognition and speaker recognition.

2 Concurrent self-organizing maps for pattern classification 2.1 DESCRIPTION OF THE MODEL Concurrent Self-Organizing Maps (CSOM) is a collection of small SOMs that use a global winner-takes-all strategy. Each network is utilized to correctly classify the patterns of one class only and the number of networks equals the number of classes. The CSOM is trained in a supervised manner while each individual SOM is trained in a competitive non-supervised manner on a specific training set independently. The CSOM training model is shown in Figure 1(a). Input Pattern

DATABASE

Pattern Set “1”

Pattern Set “2”

Pattern Set “n”

SOM “1”

SOM “2”

SOM “n”

(a)

SOM “1”

SOM “2”

SOM “n”

Minimization of Quantization Error

(b)

CLASS

Figure 1. (a) The CSOM model (training phase). (b) The CSOM model (classification phase).

3 CSOM for face recognition Face recognition is a specific topic of computer vision that has been studied for 25 years and has recently become a hot topic. However, face recognition continues to be a difficult task due to the variation of factors such as lighting conditions, viewpoint, body movement and facial expressions. Face recognition algorithms have numerous potential applications in areas such as visual surveillance, criminal identification, multimedia and visually mediated interaction.

CONCURRENT SELF-ORGANIZING MAPS

293

3.1 FACE DATABASE In experimenting with the proposed CSOM model, we used “The ORL Database of Faces” provided by the AT&T Laboratories of Cambridge University, which contains 400 images corresponding to 40 subjects (namely, 10 images for each of the 40 classes). We divided the entire gallery into a training set (200 pictures) and a test set (200 pictures). Each image is sized to 92 x 112 pixels with 256 levels of grey. For the same subject (class), images were taken during different time of the day and various lighting conditions and facial expressions with or without glasses. For each class, five images are chosen for training and five for testing (see Figure 2). 3.2 EXPERIMENTAL RESULTS OF FACE RECOGNITION The face recognition task comprised of two steps: x Feature extraction using the Principal Component Analysis (PCA) x Pattern classification using CSOM We software implemented the proposed technique and tested the model using the previously mentioned face database of 400 images.

Figure 2. (a) Training images, (b) test images corresponding to the training images given in (a).

3.2.1 Feature extraction with PCA The original 92 x 112 images were resized to 46 x 56 with corresponding input space dimension of 2,576. The Principal Component Analysis stage is equivalent to the computation of the Karhunen-Loeve Transform [2, 3]. We computed the covariance matrix of the entire training set of 200 vectors X  R2576, the eigenvalues and the eigenvectors. We ordered the eigenvalues Ȝ 1 • Ȝ 2 • Ȝ 3 • . . . • Ȝ 2575 • Ȝ 2576, and computed the energy preservation factor E by retaining only n

294

V.-E. NEAGOE AND A.-D. ROPOT

eigenvalues. For example, we can reduce the space dimension from 2,576 to 158 while preserving 99% of the signal energy. In Table 1, the energy preservation factor is given for various n. The cases where n = 158 (E = 99.02%) and n = 10 (E = 65.28%) were chosen for the simulation. TABLE 1. Energy preservation factor for various n. Number of features (n)

199

158

117

100

50

10

Energy preservation factor E (%)

100

99.02

97.03

95.79

88.84

65.28

3.2.2 CSOM versus SOM for face classification At the second processing step of face recognition, we performed a neural classification using the following techniques: x The new CSOM model x The SOM classifier The results of the simulation are given in Tables 2 and 3. TABLE 2. Experimental results of face classification with CSOM versus SOM (40 classes). No

Number of Type of retained classifier principal components n

Total number Number of networks of neurons

Recognition score for the training lot (%)

Recognition score for the test lot (%)

1

158

Linear CSOMs (40 x 4)

160

40

100

91

2

158

Linear SOM

160

1

98

71

3

158

Rectangular CSOMs [40 x (8 x 5)]

1600

40

100

88

4

158

Rectangular SOM (40 x 40)

1600

1

100

3

5

10

Linear CSOMs (40 x 3)

120

40

100

85

6

10

Linear SOM

120

1

97

68

7

10

Rectangular CSOMs [40 x (8 x 5)]

1600

40

100

85.5

8

10

Rectangular SOM (40 x 40)

1600

1

100

16

CONCURRENT SELF-ORGANIZING MAPS

295

TABLE 3. The best face recognition rates obtained by CSOM on the test lot as a function of total number of neurons (n = 158 principal components). Number of neurons

40

80

120

160

200

1,600

Recognition rate (%)

SOM

64.0

70.0

70.0

71.5

60

3

CSOM

84.5

89

89.5

91.0

90.5

89.5

4 CSOM for speaker recognition 4.1 SPEAKER DATABASE We utilize the SPEECHDATA database in our experiments with the CSOM model for a speaker recognition application. The SPEECHDATA database consists of 20 repetitions of 12 words (“one”-“nine,” “zero,” “nought” and “oh”) uttered by 25 speakers. The speech signal of each word is processed in 15 overlapping Hamming windows (frames), where a frame pattern consists of 18 coefficients: 8 cepstral coefficients, a coefficient representing the Mel-scale filter bank output, and 9 delta coefficients. Thus, each word is represented by 270 feature coefficients. The goal is to recognize the correct speaker; there are 25 classes, equal to the number of speakers. 4.2 EXPERIMENTAL RESULTS OF SPEAKER RECOGNITION The results are given in Tables 4 and 5. TABLE 4. Experimental results of speaker recognition obtained with CSOM, SOM and Bayes classifiers (25 classes; the dimension of the input vector space is 270). No

Type of classifier

1

Total number of neurons

Number of networks

Recognition score for the training lot (%)

Recognition score for the test lot (%)

Circular CSOMs (25 x 48)

1,200

25

99.87

91.83

2

Circular SOM

1,200

1

87.73

80.13

3

Linear CSOMs (25 x 48)

1,200

25

99.83

92.17

4

Linear SOM

1,200

1

87.80

79.63

5

Rectangular CSOMs [25 x (8 x 6)]

1,200

25

99.33

90.43

6

Rectangular SOM (40 x 30)

1,200

1

85.80

74.33

7

Bayes classifier

64.07

61.30

V.-E. NEAGOE AND A.-D. ROPOT

296

TABLE 5. The best speaker recognition rates obtained by CSOM on the test lot as a function of total number of neurons. Number of neurons

150

300

600

900

1,200

Recognition rate (%)

SOM

39.87

63.20

76.03

79.90

80.13

CSOM

74.00

86.57

89.53

91.67

92.17

Bayes 61.30

5 Concluding remarks x The proposed CSOM model comprises a collection of small SOMs. Each SOM is tasked with correctly classifying the patterns of one class only. The combined decision of the SOM is based on a global winner-takes-all strategy. x Based on the experimental results of considered applications (face recognition and speaker recognition) we can show the advantage of the CSOM over the SOM both in recognition rate and training time. x From the training time standpoint, the advantage of CSOM over the SOM is obvious. Theoretically, for M classes, the CSOM training time must be about M times less than that of the corresponding SOM with the same number of neurons. During the training of the CSOM, each input vector is used to train a specific small SOM corresponding to the vector class only. Therefore it is only necessary to compute 1/M of the number of distances computed for the SOM. Moreover, the radii of neighbourhoods are smaller for CSOM components than for a single, large SOM. 5.1 FACE RECOGNITION (40 CLASSES)

5.1.1 For n = 158 principal components x By retaining only 158 components in the transformed space (instead of 2,576 components of vectors in the original space), 99.02% of the signal energy is preserved. x Using a CSOM containing 40 small linear CSOMs, each with 4 neurons, a face recognition score of 91% is obtained (for the test set) versus a recognition score of only 71% for a single linear SOM with the same total number of neurons (160). x For the CSOM model, the best face recognition rate in the test set increases by increasing the number of neurons until it reaches an optimum (for example, 91% for a number of 160 neurons), at which point the recognition rate decreases. x The linear architecture of the CSOM modules leads to better results than the rectangular architecture.

CONCURRENT SELF-ORGANIZING MAPS

297

5.1.2 For n = 10 principal components x By retaining only 10 components in the transformed space, we preserve 65.28% of the signal energy contained in the original space of dimensionality 2,576. x Using a set of 40 small linear CSOMs, each with 3 neurons, a recognition score of 85% is obtained versus a recognition score of only 68% for a corresponding linear SOM with the same total number of neurons (120). 5.2 SPEAKER RECOGNITION (25 CLASSES) x We can evaluate a significant advantage of the speaker recognition score obtained using CSOM by comparison to the SOM and Bayes. x The best results (a classification rate of 92.17%) are obtained using a CSOM model containing 25 linear SOMs with 48 neurons each, while the corresponding large SOM with the same number of neurons leads to a 79.63% recognition rate! x Taking into account the architecture variants for the components of the CSOM, for this application the best variant is linear, followed by circular and rectangular. 5.3 HARBOUR PROTECTION APPLICATIONS x The model may be utilized in access control applications for harbour protection from the land border, using a sensor fusion system (face and voice recognition). x Exciting potential applications of the proposed CSOM model can be obtained by applying experience obtained with face recognition to ship recognition as well as experience obtained with to ship sound recognition.

Acknowledgements Part of this research was supported by the Romanian Academy under Grants Nr. 171/2003 and Nr. 152/2004.

References 1. 2.

Hallinan PW, Gordon GC, Yuille AL, Giblin P, Mumford D. 1999. Two- and ThreeDimensional Patterns of the Face, A K Peters, Natick, MA. Rabiner L, Juang BH. 1993. Fundamentals of speech recognition, Prentice Hall, Englewood Cliffs, NJ.

298 3. 4. 5.

6.

7.

V.-E. NEAGOE AND A.-D. ROPOT Bishop CM. 1995. Neural networks for pattern recognition, Oxford University Press, New York. Kohonen T. 1995. Self-Organizing Maps, Springer, Berlin. Neagoe V. 2001. Concurrent Self-Organizing Maps for Automatic Face Recognition, Proc. 29th International Conference of the Romanian Technical Military Academy, November 15–16 2001, Bucharest, published by Technical Military Academy, Romania, Section 9 (Communications), ISBN 973-8290-27-9, pp. 35–40. Neagoe V, Ropot A. 2002. Concurrent Self-Organizing Maps for Pattern Classification, Proc. First IEEE International Conference on Cognitive Informatics, ICCI 2002, 19–20 August 2002, Calgary, Alberta, Canada, ISBN 0-7695-1726-9, pp. 304–312. Neagoe V, Ropot A. 2004. Concurrent Self-Organizing Maps - A Powerful Artificial Neural Tool for Biometric Technology, Proc. World Automation Congress WAC’04, Seville, 3, ISBN 1-889335-20-7, IEEE Catalog 04EX832C.

SENSOR-FUSION IN NEURAL NETWORKS

GUNTHER PALM AND FRIEDHELM SCHWENKER Neural Information Processing, University of Ulm, Germany

Abstract. The problem of sensor-fusion arises in many applications. We have studied the problem primarily in the context of cognitive robotics. An autonomous robot has to create a meaningful representation of its position in the world and, more generally, the situation in which it has to perform an appropriate action. Such robots usually have several different sensors (sonars, laser range finders, cameras, microphones, infrared sensors, bumpers) and they have to combine the different kinds of information provided by these sensors. Neural networks seem to be particularly well suited for the combination of inputs from completely different sources. So we will be using them and the corresponding learning and training strategies for these problems. In the context of pattern recognition, a community has emerged that is analyzing these problems and has accumulated a considerable amount of practical and theoretical knowledge in the theoretical framework of multiple classifier systems. There is a strong overlap between this community and our neural networks community, and we have also contributed to the application of multiple classifier systems built from neural networks for sensor-fusion, for example, for the classification of multivariate biological time series. One question to be addressed here is the use of uncertainty measures for the combination of classifiers. Here we use the framework of multiple classifier systems to compare several methods of sensor- or information-fusion. Keywords: Artificial neural networks, sensor fusion, multiple classifier systems

1 Introduction Sensor-fusion, data-fusion, information-fusion or decision-fusion are closely related problems, which occur whenever a decision has to be made based on data or information from different sources. An important aspect of decision making lies in the possibility that different sources of information may have different reliabilities. These reliabilities have to be estimated and accounted for in the final decision making process. Another aspect relevant to decision making is the level of information fusion (see Figure 1). At the lowest level, data from different sources or from different sensors having different data-formats have to be fused or

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

299

G. PALM AND F. SCHWENKER

300

combined to make a decision. At the highest level, decisions from different experts (or so-called decision profiles) have to be combined. Intermediate levels of combination can also be defined, for example by distinguishing the levels of original elementary data, (pre-) processed data, extracted features, preliminary, local or weak classifications, and the final decision. data

features decision

low-level fusion

data

features decision

mid-level fusion

local data features decision decision

high-level fusion

multi-level fusion

Figure 1. Information fusion at different system levels (low-, medium-, high-, and multi-level).

In general, information fusion schemes may encompass combining information from different sources or sensors, information from one source taken at different times (temporal fusion) and even decisions based on identical information made by different experts (with different specialized expertise). In the case of object or situation classification, the underlying logic of the fusion problem is always the same (Figure 2): one has a single object from which different sources of information are derived (data dj) and these have to be recombined to estimate the identity of the object.

SENSOR-FUSION IN NEURAL NETWORKS

301

d1 d2

c

c^

object dn

Figure 2. From an object c, n different data streams are observed. In the recognition phase these observations d have to be fused to an overall estimate cˆ .

A seemingly natural requirement for the analysis of this situation is the conditional independence of the information sources, i.e. p(di ,dj_c)= p(di_c)p(dj_c). Clearly, this requirement represents an idealization; it will generally not be exactly fulfilled in practical situations. Artificial neural networks have become a standard paradigm for pattern recognition. They are also well-suited for the formulation of sensor-, information-, or decision-fusion scenarios. Typically, three-layer networks are used for classification: The first layer or input layer contains the data to be analyzed. Here, it is usually assumed that the data are given as real vectors of fixed dimensionality (perhaps after some appropriate pre-processing). The second layer or hidden layer contains non-linear combinations of the data, which can be regarded as features that are formed by neural-network learning or training. The third layer or output layer represents the decision of the neural network. For a classification problem with k classes there are typically k output-neurons. Ideally, the network should respond with the j-th unit vector to the j-th class. In practice, the output value of the j-th unit (typically between 0 and 1) is regarded as the strength of evidence for or belief in the object belonging to class j. After proper training of the network, the output of the j-th unit should actually be an estimate of the a posteriori probability p(cj_d) of class j. Thus, the data, features or decisions are uniformly represented as real vectors in artificial neural networks. This uniformity allows for arbitrary hierarchical combinations of neural classifiers, because the features or outputs of one classifier can simply be regarded as inputs of another classifier, and outputs of several classifiers can be combined by concatenation of the real output vectors. Information fusion is a very common problem in human decision making. Artificial, more or less autonomous, agents will often have to combine information from different sources to make a decision. Most of our human information fusion is done unconsciously, when we combine our different senses to localize and identify an object. Often we combine tactile, proprioceptive and visual information when we grasp an object or when we localize ourselves in a room, outside on a field or in the woods, or we combine visual and acoustical information when listening to someone.

302

G. PALM AND F. SCHWENKER

The unconscious combination of information may be revealed when non-matching visual information on face and mouth movements is displayed with auditory speech (as in dubbed movies, or in the McGurk effect [1]). We also do it on a conscious level, for example, when we combine opinions from experts and friends to form our own decisions. In democratic organizations, we use voting majorities to combine individual decisions. Similar decision schemes can also be used in artificial (multi-agent) systems. Such schemes have been analyzed in the scientific literature on multiple classifier systems. Most interesting in this context are schemes that employ some method of expressing and using the degrees of uncertainty associated with different decisions or classifications. There are a few theoretical frameworks for dealing with uncertainty [2–4] and they lead to different fixed rules for combining decisions [5]. Usually, such certainties, fuzzy membership values, or probabilities are represented by numbers between 0 and 1, and perhaps the two most natural combination schemes are simply the arithmetic and the geometric mean. These ideas always lead to so-called fixed fusion schemes. Another, less formal approach to the fusion problem is simply to use more data to develop the combination, for example, by experimentally determining “weights” for the importance of individual classifiers in a weighted average or perceptron approach, or by training an additional neural network for the combination, which takes as inputs the certainty vectors provided by the individual classifiers (now simply regarded as data). This leads to adaptive fusion schemes.

2 Practical comparison of fusion schemes In 2001, Kuncheva et al. [6] compared several differing fusion schemes on two data-sets (SATIMAGE and Phoneme from the ELENA database1). Table 1 reveals some of their results. It is apparent that there is no clearly superior method even for these two data-sets. More generally, for any two of the more popular methods, a data set probably exists on which the first is better than the second and vice versa. Nevertheless, the class of methods called decision templates seem to be among the best for classification [7, 8, 5] on both data-sets and also on other common data-sets. Moreover, we have compared decision templates with a few other fusion methods on a data-set composed of orthoptera songs (cricket and katydids) [9, 10]. Our results are shown in Table 2. Again, the decision templates appear to be slightly superior to other methods. We have also developed an additional small improvement [11] on the decision template idea (multiple decision templates), which is explained in the remainder of this contribution.

1

ftp.dice.ucl.ac.be/pub/neural-nets/ELENA/databases

SENSOR-FUSION IN NEURAL NETWORKS

303

TABLE 1. Average test accuracy (in %) for the SATIMAGE data (left) and the Phoneme data (right) (extracted from [6]). SB: best single classifier, MAJ: majority, NB: naïve Bayes combination, BKS: [12], MIN: minimum probability, AVR: average probability, PRO: product of probabilities, DS: Dempster Shafer rule, LDC: linear discriminant classifier, DT:x: versions of decision templates, E: Euclidean distance, I: fuzzy set inclusion, S: fuzzy set similarity, OR: ‘oracle’, best single result for every case SATIMAGE (6 classes)

Phoneme (2 classes)

Size

100

1,000

2,000

100

1,000

2,000

SB

77.5

80.7

80.6

74.3

75.5

75.2

MAJ

80.9

82.1

82.2

75.5

76.4

76.1

NB

77.2

83.5

83.7

74.1

74.1

73.6

BKS

68.2

79.9

81.6

72.7

77.5

78.0

MIN

82.5

84.4

84.6

75.5

77.8

75.5

AVR

82.6

83.8

83.9

75.8

76.4

75.9

PRO

83.0

84.4

84.5

75.8

76.3

75.9

DS

83.0

84.4

84.5

75.1

76.8

77.1

LDC

42.0

32.3

42.5

37.6

37.4

54.0

DT:E

83.0

84.5

84.6

75.0

76.7

77.1

DT:I1

83.1

84.3

84.4

75.0

77.1

77.5

DT:S1

83.1

84.3

84.4

75.0

77.1

77.5

OR

90.1

90.5

90.5

97.4

97.7

97.6

TABLE 2. Average test accuracy (in %) for single features (distance and duration of pulses, frequency of pulses, energy contours, time-encoded-signals (TES) and maximal amplitude of pulses (MAP)) and for two fusion methods (For details see [10]) Crickets (31 classes)

Katydids (22 classes)

Pulse distances

63.9

63.0

Pulse durations

39.1

65.7

Frequency of pulses

26.4

19.4

Energy contours

39.9

47.6

TES

22.3

38.3

MAP

18.9

33.0

AVR

65.6

68.7

DT:E

93.7

86.0

304

G. PALM AND F. SCHWENKER

3 Multiple decision templates Decision templates belong to the class of adaptive fusion schemes, in which the fusion mechanism is trained by means of additional data for the individual classifiers to be fused. One can view decision templates as particular neural networks that take as inputs the concatenated class-probability-vectors, which are the outputs of the individual classifiers. These networks (the decision-templatenetworks or DTN) are very similar to k-means-networks [13] and are usually trained as follows: one generates typical input vectors for each class (the class specific cluster centers); to this end, all vectors of class j are used to compute their average or (equivalently) are used in a 1-means clustering. More generally, in our MDT algorithm, they are used for k-means clustering to generate k typical cluster centers for class j. For classification, the data point is simply given the class label of the nearest cluster center. The results of multiple decision templates are compared to simple decision templates on our cricket data set (31 classes) in Tables 3–5. TABLE 3. Average test accuracy (in %) for the single features D, N, L and F of the bioacoustic data set (31 different cricket species). D: pulse distances, N: normalized pulse distances, L: pulse length, F: pulse frequency (See [11]) Single feature

Accuracy (in %)

D

63.8 r 1.32

N

48.5 r 0.44

L

26.6 r 0.61

F

25.2 r 0.68

TABLE 4. Average test accuracy (in %) for the feature pairs (D, N), …, (L, F) Combined feature

Accuracy (in %)

(D, N)

64.0 r 1.49

(D, L)

72.2 r 1.14

(D, F)

85.6 r 0.67

(N, L)

62.9 r 0.45

N, F)

62.5 r 1.11

(L, F)

60.0 r 0.89

AVR

84.0 r 1.05

305

SENSOR-FUSION IN NEURAL NETWORKS

TABLE 5. Average test accuracy (in %) for the bioacoustic data set (31 cricket species = 31 classes) for different overlaps of training and validation sets given by different values of p and different decision template algorithms. DT: single decision templates, CMDT k: multiple decision templates with k-means and k templates per time series Algorithm

p = 0.0

p = 0.5

P = 1.0

DT

89.0 r 0.63

90.0 r 0.52

89.9 r 0.26

CMDT1

88.5 r 1.30

90.8 r 1.05

91.2 r 0.55

CMDT2

89.0 r 1.11

91.4 r 0.84

91.3 r 0.74

CMDT3

88.3 r 0.77

91.9 r 0.59

90.0 r 0.61

The resulting algorithm for decision templates or multiple decision templates is described below. Given: Data set D

T ‰ V,

each x  D labelled by an Z  :

^Z , ...Z ` and n 1

L

feature extraction methods Fi 1. Choose base classifiers Bi (one per feature type) and learning algorithm. 2. Initialize and train Bi on training set Fi (T) . 3. Use the validations set Z

Oi

4. Fix

Z

Fi (V ) of class

Z

and calculate outputs

Z

Bi (Fi (V )) . k and

run k-means clustering procedure on the concatenated classifier Z

Z

Z

Z

output vectors O : O1 O 2 ....O for the data of each class Z separately. Output: L sets (one per class Z ) of cluster centres.

Of course, more complex neural network methods could be used to perform classification: learning vector quantization (LVQ), to generate class specific cluster centers, and RBF networks, where the final decision is made by simple perceptrons (one for each class) that are trained on the distance or similarity values obtained from cluster centers. For more intricate training methods for RBF networks see [13]. A data point is finally evaluated by computing the distance (e.g. p-norm) to each of the centers. Each RBF unit typically computes a decreasing function of this distance d (often exp (-d2)), which can be regarded as a measure of similarity. A simple linear perceptron combines these similarity values for the final classification. However, more complex adaptive fusion methods typically need more parameters to be adapted and therefore require more training data. If the available data are limited, these methods will be faced with the well-known bias-variance dilemma [14].

306

G. PALM AND F. SCHWENKER

References 1. 2. 3. 4. 5. 6. 7.

8. 9.

10.

11. 12.

13. 14.

H. McGurk and J. MacDonald (1976). Hearing Lips and Seeing Voices, Nature, Dec. 1976:746–748. D. Dubois and H. Prade (1980). Fuzzy Sets and Systems: Theory and Applications. Academic Press. D. Dubois and H. Prade (1988). Possibility Theory: An Approach to Computerized Processing of Uncertainty. Plenum Press. R. Kruse, F. Klawonn, and J. Gebhardt (1995). Fuzzy Systeme. Teubner. L.I. Kuncheva (2004). Combining Pattern Classification. Wiley-Interscience. L.I. Kuncheva, J.C. Bezdek, and R.P.W. Duin (2001). Decision templates for Multiple Classifier Fusion. Pattern Recognition, 34(2):299–314. J. Kittler, M. Balette, J. Czyz, F. Roli, and L. Vanderdorpe (2002). Decision level fusion of intramodal personal identity verification experts. In F. Roli and J. Kittler (eds.): Proc. 2nd International Workshop on Multiple Classifier Systems, Vol. 2364 of Lecture Notes in Computer Science, Springer, 378–387. G. Giacinto, F. Roli, and L. Didaci (2003). Fusion of multiple classifier for intrusion detection in computer networks. Pattern Recognition Letters, 24:1795–1803. F. Schwenker, C. Dietrich, H.A. Kestler, K. Riede, and G. Palm (2003). Radial basis function neural networks and temporal fusion for the classification of bioacoustic time series, Neurocomputing, 51:265–275. C. Dietrich, G. Palm, K. Riede, and F. Schwenker (2004). Classification of bioacoustic time series based on the combination of global and local features, Pattern Recognition Journal, 37(12):2293–2305. C. Dietrich, G. Palm and F. Schwenker (2003). Decision templates for the classification of time series, International Journal of Information Fusion Journal, 4:101–109. Y.S. Huang and C.Y. Suen (1995). A method of combining multiple experts for the recognition of unconstrained handwritten numerals. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17:90–93. F. Schwenker, H.A. Kestler, and G. Palm (2001). Three phase learning for radial basis function networks, 14:439–458. S. Geman, E. Bienenstock, and R. Doursat (1992). Neural networks and the bias/ variance dilemma. Neural Computation 4:1–58.

TARGET IDENTIFICATION BASED ON DSMT

ALBENA TCHAMOVA*,1 AND JEAN DEZERT2 1 Bulgarian Academy of Sciences, Institute for Parallel Processing, Sofia, Bulgaria 2 Office National d’Études et Recherches Aérospatiales, Châtillon, France

Abstract. An approach providing a rapid reduction of total ignorance in the target identification process is proposed. It is based on a combination of the new DezertSmarandache theory (DSmT) for plausible and paradoxical reasoning, and fuzzy set theory. This approach utilizes information from the adjoint sensor and additional information obtained from a priori defined objective and subjective considerations. As a result, the pignistic probabilities for a target’s nature are obtained and analyzed. Keywords: Target identification, Dezert-Smarandache theory, DSmT, fuzzy logic, attribute data fusion

1 Introduction The protection and preservation of harbours is an extremely important social and environmental issue in peacetime or during a conflict. The provision of multiple sensors to enable the continuous monitoring of areas against intruders could help in the critical task of distinguishing friendly forces from unknown or hostile ones. This provides critical time to counter an attack or prevent fratricide between friendly forces. In this chapter, the process of target state recognition by an IFF (Identification Friend/Foe) sensor is examined. The information received from the sensor pertains to a single attribute: ‘friend target’ (‘F’). The absence of evidence (so-called response), however, does not a priori ensure 100% reliability for the hypothesis ‘hostile target’ (‘H’) and the problem of possible wrong target recognition arises. This problem is especially complicated when the moment of decision-making cannot be postponed. In this case, total ignorance renders the support, as well as the probability for an alternative hypotheses of ‘friendly target’ *

This work is partially supported by MONT grants I-1205/02, I-1202/02 and by the Centre of Excellence BIS21++. E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

307

308

A. TCHAMOVA AND J. DEZERT

or ‘hostile target’, equally ambiguous and plausible. As a result, alternative decisions made on this basis pose an equal degree of risk. One way out of the described problem is to incorporate additional attribute information at a different level of abstraction from another disparate sensor [1, 2]. Hence the IFF sensor is often adjoined with a radar or infrared sensor (IRS). Evidence from the additional sensor should help resolve this dilemma. Unfortunately, this information does not always provide an implicit answer at the time the question is posed (due to the sensors’ technical particularities). A more expensive solution is to increase the number of sensors [2]. In this paper, an approach combining DSmT [4, 6] and fuzzy sets theory [5, 8] is applied in order to utilize all available information – the a priori defined objective and subjective considerations, concerning relationships between the attribute components at different levels of abstraction and the attribute information from the adjoint sensor (radar).

2 Dezert-Smarandache theory The DSmT of plausible and paradoxical reasoning [4, 6] proposes a new general mathematical framework for solving fusion problems. This theory overcomes the practical limitations of the Dempster-Shafer theory (DST) [3] that result essentially from its inherent constraints, which are closely related to the acceptance of the law of the third excluded middle. The foundations of DSmT are to refute the law of the third excluded middle and to allow imprecise/vague notions and concepts between elements of the frame of the problem 4 . It can be interpreted as a general and direct extension of probability theory and DST. 2.1 FREE DEZERT-SMARANDACHE MODEL Let 4 ^T 1 ,...T n ` be a set of n elements that cannot be precisely defined and separated. A free-DSm model, denoted as 0 f 4 consists in assuming that all elements

T i ,i 1,..., n of

4 are not exclusive. The free-DSm model is an

opposite of the Shafer model 0 0 4 , which calls for the exclusivity and exhaustivity of all elements T i ,i

1,..., n of 4 .

2.2 DEFINITION OF THE HYPER-POWER SET AND CLASSICAL DSM RULE OF COMBINATION The hyper-power set D T is defined as the set of all composite possibilities built from 4 with ‰ and ˆ operators, such that: 1. ‡ ,T 1 ,...T n  D 4 2. A  DT , B  DT ,  A ‰ B  DT ,  A ˆ B  DT .

TARGET IDENTIFICATION BASED ON DSMT

3. No other elements belong to

309

D 4 except those obtained from rules 1 or 2.

From a general frame of discernment 4 with its free-DSm model, we define a map m. : D T o >0,1@ associated with a given source of evidence that can support paradoxical or conflicting information, as follows:

m‡ 0 and

¦ m A

(1)

1

ADT



The quantity m A is called A’s general basic belief assignment (gbba) or the general basic belief mass for A. Belief and plausibility functions are defined for A D T : Bel  A

¦ mB ; Pl  A

¦ m B

BDT ,B Ž A

(2)

BDT ,B ˆ A z ‡

The DSm classical rule of combination (DSmC) is based on the free-DSm model. For k t 2 independent bodies of evidence with gbbas m1 . , m 2 . , , …

mk . :

m 0 f 4  A

with m0 f 4 ‡

k

¦ – m X 4

i

(3)

i

i 1 X 1 ... X k D X ˆ ...ˆ X k A

0 by definition. This rule is commutative and associative and

requires no normalization procedure. 2.3 DEFINITION OF THE DSM HYBRID MODEL A DSm hybrid model 04 [6] is defined using the free-DSm model 0 f 4 by introducing integrity constraints on some elements A  D 4 in accordance with the 4 exact nature of the problem. An integrity constraint on A  D consists in 0

forcing A { ‡ through the model. The Shafer model 0 0 4 can be considered as the most constrained free DSm model.

310

A. TCHAMOVA AND J. DEZERT

2.4 DSM RULE OF COMBINATION FOR THE HYBRID MODEL The DSm hybrid rule of combination (DSmH) associated to a given DSm hybrid model 0 z 0 ‡ for k t 2 independent sources of evidence is defined for all

A  D 4 as m 0 4  A I  A >S 1  A  S 2  A  S 3  A @ where

I  A

is the characteristic non-emptiness function of set A,

I  A 1 if A  ‡ and I  A 0 otherwise.

(4) i.e.

Here:

S1  A

¦

k

– m X

i 1 X 1 , X 2 ,... X k D 4 X 1 ˆ X 2 ˆ...ˆ X k A

i

i

(5)

corresponds to the DSmC,

S 2  A

¦

k

– m X

X 1 , X 2 ,... X k ‡ i 1 >U A @› >U ‡ š  A I t @

i

i

(6)

represents the mass of all relatively and absolutely empty sets that are transferred to the total or relative ignorance,

S3  A

¦

k

– m  X ,

i 1 X1 , X 2 ,... X k D 4 X1 ˆ X 2 ˆ...ˆ X k ‡ X1 ‰ X 2 ‰... X k A

i

i

(7)

which transfers the sum of relatively empty sets to non-empty sets.

3 Description of the approach 3.1 DEFINITION OF THE A PRIORI DATABASE The a priori database is realized as a fuzzy relation. It takes into account the defined objective considerations connecting the components of some attributes expressed at different levels of abstraction (for example the fuzzy relation t‘arget type – target nature’). For this purpose, it is defined that:

TARGET IDENTIFICATION BASED ON DSMT

x The set X

DT

311

^x , x ,..., x `, which corresponds to the hyper-power set 1

2

n

2 2 1

D T relative to the level of abstraction of the adjoint sensor (the object type Oi ,i 1,2 ,..., n ). Because object types are exclusive, the Shafer model is valid

and the hyper-power set is reduced to the Dempster-Shafer (DS) power set

x1

x n 1 x The set ( Y ^y1

O1 ; x 2

; xn O2 ;…

., O1 ‰ O2 ;…

On , O1 ‰ O 2 ‰ ... ‰ O n .

x 2 n 1

(8)

Y corresponds to the level of abstraction of the base sensor F riend , y 2

x The matrix

H ostile `).

R : X o Y is a fuzzy relation with membership function

(MF): P R  xk , yl  [ 0 ,1 ] ; k

1,2,...,2 n  1 ; l 1,2 where n is the number of

considered target types. The conditions that MF must satisfy according to the DSmT are:

P R  x k , y l t 0 ; 2¦1 P R xk , yl n

1,l 1,2 .

(9)

k 1

3.2 SEMANTIC TRANSFORMATION The information granule m X pertaining to the object’s type is transformed in a corresponding fuzzy set S X :

P S  x k m X  x k ,k 1,..., 2 n  1. . X

3.3 APPLICATION OF THE ZADEH COMPOSITIONAL RULE The image of the fuzzy set S X through the mapping R is received. The output fuzzy set

TY corresponds to the target’s nature by means of: P T  yl Y

^ >P  y , x , P x @`.

sup min xk  X

R

l

k

SX

k

(10)

TY represents the non-implicit attribute information extracted from the measurement.

A. TCHAMOVA AND J. DEZERT

312

3.4 INVERSE SEMANTIC TRANSFORMATION The fuzzy set

TY is transformed into an information granule mYR through a

normalization of membership values with respect to the unity interval. 3.5 APPLICATION OF THE DSM HYBRID RULE OF COMBINATION The DSmH (2) is used to combine two evidences:

m X and mYR . This

aggregation immediately reduces the total ignorance with regard to the target’s nature. 3.6 DECISION-MAKING BASED ON PIGNISTIC PROBABILITIES The Generalized Pignistic Transformation [6] is used to make a rational decision about the target’s nature within the DSmT framework: P ^A `

¦

X DT

C M X ˆ A m X C M X



for A  D T

(11)

The decision is directed by the maximum of the pignistic probability function

P^` .

4 Simulation scenario and results Two sensors – radar and IFF – are available on board the patrolling warship, serving as a harbour guard against a hostile air raid. Sensor evidence defines a frame of discernment for the target’s type: 4 ^O1 , O2 , O3 ` , O1 -‘fighter’, O2 ‘airlift cargo’, O3 -‘bomber’, and the target’s nature: H  ^O1 , O3 ` , F  ^O ` . The 2

attribute components corresponding to these target types are the angular sizes A of target blips measured on the radar screen. In order to define the influence of these components on this problem it is sufficient to know the specific features of their probabilistic b‘ ehaviour’ and to assi gn fuzzy values to them. It is supposed that the average A1 of the angular size A1 corresponding to O1 is the minimal value ( A1 depends on the size of the elementary radar’s volume P  A1 ! AV | 0 ;

AV );

A2 corresponding to O2 is the maximal value; P  A2  AV | 0 ;

A3 for O3 obeys the relation A1  A3  A2 ; P  A3  AV , P  A3 !A2 cannot be neglected.

TARGET IDENTIFICATION BASED ON DSMT

313

The worst case is when a hostile target is observed and the obtained respective radar blip has a medium angular size. It can originate from a target of any type, i.e. T T 1 ‰ T 2 ‰ T 3 . The information granule is defined as: mX

^m T X

1

0.2, mX T 2 0.2 mX T3 0.3 mX Ti ‰T j 0.0 mX 4 0.3` (12)

The evidence in this case is obtained from the IFF sensor (the IFF sensor has not received a response from the observed target):

mY

^mY F

0, mY H 0, mY 4 1`.

(13)

If the DSmC m X † mY is used to fuse these two evidences, the result will not change the target nature estimate because of the effect of vacuous belief assignment. 4.1 STEP 1 For the considered example, the sets X and Y are:

X

^O1 ,O2 ,O3 ,O1 ‰ O2 ,O1 ‰ O3 ,O2 ‰ O3 ,4` , Y ^F ,H`.

The a priori defined relation

(14)

R: X o Y (database) is described in Table 1.

TABLE 1. A priori defined relation

R

y1

F

y2

H

x1

O1

P R O1 , F 0

P R  O1 , H 0.3

x2

O2

P R  O2 , F 0.8

P R  O2 , H 0.3

x3

O3

P R  O3 , F 0

P R  O3 , H 0.3

0

… x5

4

P R 4 , F 0.2

0 P R 4 , H 0.1

A priori defined objective considerations are taken into account. It is presumed that the information obtained from some particular schedule of civilian and military aircraft flights excludes flights of friendly fighters and bombers but allows planned flights of friendly civil passenger aircrafts and friendly military airlift operations.

314

A. TCHAMOVA AND J. DEZERT

4.2 STEP 2 The evidence from the radar sensor m X is transformed in a fuzzy set

SX .

4.3 STEP 3 The image of the fuzzy set S X is defined through the mapping R. The fuzzy set TY infers the target’s nature with PT  y1 Y

0.2 and P TY  y 2

0 .3 .

4.4 STEP 4 Normalization step: mY R

^m F YR

0 . 4, m Y R  H

`

06 . mYR contains the

non-implicit information about the target’s nature in the radar measurement. 4.5 STEP 5 DSmC (1) is used to combine the evidences m X and mYR (see Table 2). TABLE 2. Combination of evidences

m YR  F

O2

0 .4

mYR  H

O1 ‰ O3

mX  O1 0.2

mO1 ˆ O2 0.08

m O1

012 .

mX  O2 0.2

m O 2

mF ˆ H 0.12

mX  O3 0.3

mO2 ˆ O3 0.12

m O 3

0.18

mX  4 0.3

mO2 012 .

m H

0.18

0.08

0.6

In accordance with the true nature of the problem, the following integrity constraints are introduced: O1 ˆ O2 ‡ , O2 ˆ O3 ‡ , F ˆ H ‡ . By applying DSmH (2), the updated vector of probability masses mupd is obtained below: mupd  O2 0.2 mupd  O3 0.18 mupd  H 0.18½ ­ mupd  O1 0.12 ® ¾ ¯mupd  O1 ‰ O2 0.08 mupd  O2 ‰ O3 0.12 mupd  F ‰ H 0.12 ¿

(15)

TARGET IDENTIFICATION BASED ON DSMT

315

4.6 STEP 6

In conclusion, pignistic probabilities (3) are calculated in order to make decisions about a target’s nature: P  H 0.66, P  F 0.34. Other pignistic probabilities of interest are: P  O1 0.29, P  O3 0.37. It is obvious that the evidence supporting propositions ‘target type is O1 ’ and ‘target type is O3 ’ increase support for proposition ‘H’. But the evidence for a target being ‘H’ does not increase support for the proposition ‘target type is O 1 ’ or ‘target type is O3 ’. To ensure the completeness of this study, two other possible radar measurements are considered: measurement of target type ‘fighter’ (‘bomber’) or ‘airlift cargo’ (‘bomber’): mX' " X

m

^m O ^m O X

1

X

1

0.3 mX  O2 0.2 mX  O3 0.2 mX  O1 ‰O3 0.3 mX  4 0` (16) 0.2 mX  O2 0.3 mX  O3 0.2 mX  O2 ‰O3 0.3 mX  4 0` '

The measurement m X supports the probability ‘hostile fighter’, additionally

increasing the pignistic probability P  H 0.84 and decreasing P  F 0.16. The "

measurement m X supports ‘a friend’s airlift cargo’, additionally increasing the pignistic probability P  F 0.38 and decreasing P  H 0.62. It can be noted that both probabilities tend toward each other due to the lack of more categorical evidence supporting ‘H’. The small difference remaining between them is due to the ambiguous evidence O2 ‰ O3 . The benefits of the proposed approach are also demonstrated when comparing the results with those obtained by direct utilization of the mentioned database and DSmH. For this purpose, the database is said to H

F

consist of two separate data basis ( mDB and mDB ) pertaining to propositions ‘F’ and ‘H’ respectively. Each database contains two columns (1,2 and 1,3 respectively). These granules can be used for direct updating of m X so as to check H both alternatives: mX … mDB

H F mupd , mX … mDB

F mupd . The pignistic probabilities

obtained for both alternatives are: H H 0.685 , PupdH F 0.315 ; PupdF H 0.34 , PupdF F 0.66 Pupd

(17)

The obtained probabilities thus show some improvement from the initial total ignorance, however this improvement does not suffice in practical application due H to the high similarity of the results ( Pupd H 0.685 , PupdF F 0.66 ).

A. TCHAMOVA AND J. DEZERT

316

Possible solutions to this problem could be found through future research investigating the application of the new advanced Proportional Conflict Redistribution Rules (especially PCR5) [7] which proportionally redistribute the conflicting mass (total or partial) to non-empty sets according to all integrity constraints using a given strategy.

5 Conclusion The new Dezert-Smarandache theory for plausible and paradoxical reasoning is used for reduction of total ignorance in the process of target identification for the purpose of protecting and preserving harbour security. Combined with fuzzy set theory, it utilizes the information from the adjoint sensor and additional information obtained from a priori defined objective and subjective considerations. DsmT allows the fusion of sources in the free-DSm model and in more complex/restricted ones, such as the Shafer model. It can successfully deal with cases where the conflict between sources heightens. The decision-making process is based on the generalized pignistic transformation and allows for the construction and analysis of the pignistic probabilities for the target’s nature {Friend, Hostile}. The generated output results simultaneously take into account all available information concerning the considered stochastic events in the database. One of the major advantages of such an onboard system is that protection is always available, independently of harbour location.

References 1. 2.

3. 4. 5. 6. 7.

8.

Blackman SS. 1986. Multiple-Target Tracking with Radar Applications. Artech House, Dedham, MA. Bogler PL. 1987. Dempster-Shafer reasoning with applications to multisensor target identification systems. IEEE Transactions on Systems, Man and Cybernetics, Nov./Dec. 1987, SMC-17(6), p. 968–977. Dempster AP. 1968. A generalization of Bayesian Inference. Journal of the Royal Statistical Society, Series B, 30, p. 205–247. Dezert J. 2002. Foundation for a new theory of plausible and paradoxical reasoning. Information & Security Journal, ed. Semerdjiev T. Vol. 9, p. 13–57. Mendel JM. 1995. Fuzzy logic systems for engineering: a tutorial. Proceedings of IEEE. pp. 345–377. Smarandache F, Dezert J. eds. 2004. Advances and Applications of DSmT for Information Fusion. American Research Press, Rehoboth Beach, DE. Smarandache F, Dezert J. 2005. Information Fusion Based on New Proportional Conflict Redistribution Rules. Proceedings of Fusion 2005 Int. Conf. on Inf. Fusion. Philadelphia. Zadeh LA. 1973. Outline of a new approach to the analysis of complex systems and decision processes. IEEE Transactions on Systems, Man and Cybernetics, SMC-3.

PARTICLE FILTER APPLICATION TO LOCALIZATION

EVGENI KIRIY1, HANNAH MICHALSKA1 AND GUY MICHAUD2 1 McGill University, Centre for Intelligent Machines, Canada 2 Lockheed Martin Canada, R&D, Canada

Abstract. A simple particle filter algorithm with adaptive resampling is implemented showing that land-based range-only radio beacons can provide adequate localization within a port facility. Each beacon transmits its identity along with the range, thus solving the data association problem on the hardware level. The operation of the algorithm is assessed using a single set of experimental data with a 1,000 particle implementation. The associated computational load can be handled on-line by a standard PC. In the same environment the GPS absolute localization is likely to fail. Keywords: Particle filter localization, range-only land-based beacons, data association, proposal density

1 Introduction An application of a particle filter (PF) algorithm to a localization problem is presented. The use of the ground-based radio-beacons as means for the absolute localization (AL) make the approach suitable for application in environments where space-based AL systems, such as the Global Positioning System (GPS), do not function reliably due to sky occlusion and signal reflections. As a result of the many objects present (containers, ships, buildings) a port facility is one such environment. The localization system described in this paper can be used in port facility monitoring as an integral part of harbour protection. In a harbour environment it is critical to detect a potential threat situation before it develops. One of the tools for detecting a potential threat situation is automated port facility monitoring carried out by special purpose autonomous robots or by the sensor suites and processing devices installed on existing port machinery and equipment. Precise localization of the mobile equipment is a prerequisite for such automated monitoring.

E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

317

318

E. KIRIY, H. MICHALSKA AND G. MICHAUD

2 The localization purpose, technology and approach 2.1 THE PURPOSE Localization of an unauthorized object (intruder or container) or detection of an unauthorized location or displacement of a legitimate object on the port premises is an important task in the framework of port facility protection. 2.2 THE TECHNOLOGY Mobile autonomous robots instrumented with appropriate sensors may be used for port facility monitoring and perimeter security.1 In order to track the object it is necessary to localize the mobile sensing platform itself using relative2 and absolute3 localization. The position of the mobile platform is related to known features in the environment using absolute localization. Space-based absolute localization systems such as the Global Positioning System (GPS) and its analogs (Glonass, Galileo) require direct line of sight to the antenna and are unreliable in some areas of the port, for example in the container yards, where the view of the sky is limited and an insufficient number of satellites (if any) is seen for long periods of time. Alternative methods for absolute localization are required. For the absolute localization of an autonomous platform we propose to use short-range land-based beacons. Various implementations of this method of absolute localization were already shown to be effective [2, 6, 8]. To test our algorithm, we use field data collected by Dr. Sanjiv Singh’s group [8]. 2.3 THE APPROACH The localization problem is nonlinear and can either be solved by applying the Extended Kalman Filter (EKF) algorithm4 or by using a more robust Particle Filter (PF) method, the application of which will be described in this chapter. Either approach fuses the measurements of the proprioceptive sensors (on-board INS and wheel encoders) giving relative localization, with the 1 Mobile platforms may be economically efficient and sufficient from the security standpoint alternative to the extensive network of stationary sensors. A smaller number of mobile platforms would be required to survey the same area; they can be allocated according to the imminent situation, allowing for efficient sensor management. The mere presence of a mobile robot may have a deterrent effect on illegal activity stronger than that of stationary sensors. 2 Relative localization integrates previous on-board measurements (from encoders, gyroscopes, accelerometers, for example.) 3 Absolute localization relates the object’s position to some known features of the environment without use of previous sensor information. 4 EKF is computationally efficient due to the compact two-parameter density representation, allowed by the Gaussian assumption, but has limited adaptivity and is unsuitable for representation of multimodal, asymmetric distributions.

PARTICLE FILTER APPLICATION TO LOCALIZATION

319

measurements of the exteroceptive sensors (receiver antennas for the land-based beacons in this case).

3 Particle filter framework Particle filters [3, 1] find increased use in modern tracking and localization systems due to their robust performance and adaptivity. Since the filter represents the true posterior distribution (uncertain state distribution of the state given the measurements) of the system state by a finite set of support points with associated weights (see Eq. (1)) realistic multi-modal non-Gaussian distributions may be used explicitly. N

p ( x0:k | z1:k ) | ¦wk( i )G ( x0:k  xk( i ) ) i =1

(1)

Here G (˜) is the delta Dirac function, x is the state of the system, subscripts indicate the time, superscripts in brackets designate particle indices. There is a constant5 number of particles N in the filter. Each particle i is associated with a raw weight wˆ k(i ) , which is recursively updated at the receipt of a new measurement

zk according to wˆ k( i ) =

^ `

The raw weights wˆ k(i )

^x

(i ) 0:k

, wk(i ) `

N

N

p ( zk | xk( i ) ) p( xk(i ) | xk(i)1 ) ( i ) wˆ k 1. q ( xk(i ) | xk(i)1 , zk )

(2)

are normalized, according to Eq. (3), so that the set

i =1

can be interpreted as a random measure characterizing the posterior

i =1

PDF p ( x0k | z1k ) :

wk(i ) =

wˆ k( i ) N

¦wˆ k( j )

(3)

j =1

Due to the direct incorporation of the nonlinearities, a PF is robust enough to use unthresholded6 measurements.

5 6

Adaptive adaptation of the particle population size is described by Fox [6]. Raw, without any gating procedures or pre-processing.

320

E. KIRIY, H. MICHALSKA AND G. MICHAUD

3.1 TYPICAL PARTICLE FILTER OPERATION The description of a generic particle filter cycle (once the filter is initialized with a meaningful, application-specific prior distribution [14]) begins with an N unweighted measure xk(i ) , N 1 at time k representing the distribution of the

^

`

i =1

system state conditioned on past measurements:

^x

(i ) k

, N 1`

N

i =1

| p( xk | z1:k 1 ).

(4)

After a new measurement is received, the importance weights (for each particle) are updated using the information available at the time k . This results in the weighted measure approximating the distribution of the system’s state conditioned on all available measurements (including the current):

^x

(i ) k

, w( i ) `

N

i =1

| p( xk | z1:k ).

(5)

Subsequently, the resampling step selects the best particles, resulting in the unweighted measure approximating the same distribution, so that

^x

(i ) k 1

, N 1`

N

i =1

| p ( xk 1 | z1:k ).

(6)

Finally, the prediction (or sampling) step propagates the particles according to the system model. The resulting unweighted measure is an approximation of the distribution of the current state conditioned on all available measurements. A typical iteration of a particle filter is shown in Figure 1.

Figure 1. One iteration of a typical particle filter.

PARTICLE FILTER APPLICATION TO LOCALIZATION

321

3.2 PARTICLE FILTER STAGES The splitting arrows in Figure 2 show the alternative paths of the algorithm implementation. The filter is initialized by a prior distribution: a uniform distribution over the expected region is used if no prior information is available. A distribution shaped according to the available information would be better suited.

Figure 2. Particle filter stages.

The resampling step is required because the use of a suboptimal proposal density [12] results in the degeneration7 of particle weights. It is thus essential to carry out resampling by employing Importance Sampling. The resampling algorithm maps a random weighted measure into a measure with uniform weights8 to counteract the effects of sample degeneracy9 that occur due to the use of suboptimal proposal density q ( xk( i ) | xk(i)1 , zk ). Even though resampling reduces the effect of degeneracy, it introduces additional Monte Carlo variation due to sample impoverishment.10 Also, resampling precludes distributed processing. Therefore, resampling is applied adaptively,11 only when the monitored (Eq. (7)) effective sampling size Nˆ eff falls below some experimentally chosen threshold.

Nöeff =

1 N

(7)

( ( wk(i) )2

i=1

7 Degeneration phenomenon: only a few non-negligible weight particles remain after several measurement updates. 8 For example, it could use a uniform random number to select particle indices from the cumulative weights distribution. The particles with larger weights are selected more often than the ones with lower weights. 9 Even though resampling has been an integral part of the particle filter algorithms since their introduction, it remains a heuristic procedure. It is possible to forgo the resampling step completely by using an optimal proposal or by using a Gaussian Particle Filter [9]. However, these alternatives are either not sufficiently developed or lack the generality of using the generally-shaped distribution. 10 Sample impoverishment: the same state hypothesis may be represented by several particles. 11 In difference to systematic resampling performed at every iteration of the algorithm.

322

E. KIRIY, H. MICHALSKA AND G. MICHAUD

In order to counter the effects of sample impoverishment, a Markov Chain Monte Carlo (MCMC) rejuvenation step could be introduced,12 however, it is unsuitable13 for real-time applications and thus will not be used here. 3.3 PROPER PROPOSAL DISTRIBUTION SELECTION The proposal distribution selection determines how fast the particle weights will degenerate. Depending on the distribution of system and measurement data, Maskell et al. [12] recommend the following scheme for the choice of a proposal distribution (see Figure 3): a)

b)

c)

Figure 3. Scenarios for proposal selection (solid line: dynamics; dashed line: likelihood function).

1. Whenever the likelihood function is more diffuse than the dynamics – ignore the measurement, use the dynamics alone:

q ( xk(i ) | xk(i)1 , zk ) = p ( xk(i ) | xk(i)1 )

(8)

2. If the dynamics is more diffuse than the likelihood function – use one analytic filter (an EKF or an Unscented Filter (UF)) for all particles:

q ( xk( i ) | xk( i)1 , zk ) = p ( xk( i ) | zk ) v p ( zk | xk( i ) )

(9)

3. If the dynamics and the likelihood function are peaking in the different regions of the state-space – use one analytic filter (EKF or UF) per particle: 12

Resampling and rejuvenation stages are, in principle, optional [10]. The algorithms proposed for MCMC implementation are so far unsuitable for real-time application. Rejection sampling is inefficient due to the rejection rate proportional to the fraction of the volume of the proposal over the volume of the desired multidimensional densities; the rate grows exponentially with the dimension of the distribution. Gibbs sampling requires a burn-in period in order to start producing usable samples [4, 11]. 13

PARTICLE FILTER APPLICATION TO LOCALIZATION

323

q ( xk(i ) | xk(i)1 , zk ) = p( xk(i ) | xk(i)1 , zk )

(10)

Particle filters with proposals that take into account the measurements (cases b and c) are called adaptable [11]. In the considered application, the likelihood function is more diffuse than the dynamics, so the measurement is ignored and the system dynamics is used to generate the particles from the proposal distribution. Thus, approach (a) is employed, resulting in the simplified calculation of the raw particle weight (Eq. (11)):

wˆ k(i ) = p ( zk | xk( i ) ) wˆ k( i)1.

(11)

The filter invariably involves 1,000 particles. The algorithm is simple and robust since the range measurements contain their source ID, eliminating the need for data association. The localization output was calculated as an instantaneous weighted mean of all the particle states. Other methods may be useful for multimodal cases: best particle, robust mean (mean of the particles in the vicinity of the best particle) [13].

4 The experiment The PF localization algorithm was applied off-line using a single experimental data set that was collected at the Robotics Institute of the Carnegie Mellon University (USA). The test setup is described in detail by Kurth et al. [8]. Here a short description of the experiment is presented. A mobile robot equipped with wheel encoders, a single-axis gyroscope and range-only radar was programmed to repeatedly follow a path over the area where nine operating radio beacons were placed and surveyed for position. The ground truth for the localization was obtained from the 2-cm Differential GPS fused with high quality gyro and wheel encoder data. The inertial navigation system of the mobile platform provided a high rate (100 Hz) and accurate information for a short time horizon relative localization. The absolute localization range measurements, along with the beacons’ identities, were asynchronously obtained at a frequency of about 5 Hz with the standard deviation of about 1.5 m. Typical measurements consisted of one or two contacts. 4.1 RESULTS – BIRD’S EYE VIEW The grey particle clouds are shown in Figure 4 at 10 s intervals to avoid clustering. The shapes of the clouds illustrate the changes in the uncertainty at different locations along the path. The clouds have the characteristic banana-shapes that can

324

E. KIRIY, H. MICHALSKA AND G. MICHAUD

be attributed to higher uncertainty in orientation as opposed to uncertainty along the path.

Figure 4. Bird’s eye view on the path with selected particle clouds. Distances are measured in meters.

4.2 ZOOMING INTO THE DISTRIBUTION A close-up of one of the particle distributions is shown in Figure 5(a); its weighted mean state estimate is shown in Figure 5(b), where the dark dot represents the weighted average of particles. Even though there are a greater number of particles to the right of the ground truth trajectory, the mean is placed to the left of the ground truth (close to it) due to weighting.

Figure 5. Zooming into the distribution. The ground truth is shown in black. Grey arrows represent instantaneous directions of particle motions.

PARTICLE FILTER APPLICATION TO LOCALIZATION

325

4.3 TRANSVERSE AND TANGENTIAL ERRORS In mobile robot operation, the errors tangential and normal to the path may have different significance; the mean absolute errors (MAE) (Eq. (12)) were calculated in perpendicular to the track and along track directions (see Table 1 and Figure 6). K

¦ | err | i

MAE =

(12)

i =1

K

Figure 6. Normal (transverse) and tangential errors (m) with respect to the direction of motion. TABLE 1. The localization mean absolute errors Mean

Standard deviation

Normal to the path error

0.40 m

0.29 m

Tangential to the path error

0.43 m

0.31 m

Total error

0.65 m

0.33 m

5 Discussion and conclusion The system presented in this chapter allowed for stable localization with an average accuracy of less than 0.65 m when GPS absolute localization would likely fail. A simple particle filter algorithm with adaptive resampling was implemented showing that land-based range-only radio beacons could provide adequate localization within the port facility. The 1,000 particle implementation is able to process approximately 20 min of real-time data in less than 5 min off-line, on a Pentium 4, 3.2 GHz, 512 MB of RAM, in Matlab, so the filter is fast enough to be

326

E. KIRIY, H. MICHALSKA AND G. MICHAUD

on-line. However, the optimality of this implementation has only been verified in comparison to an Extended Kalman Filter. The particle filter implemented is a simple but representative example of this family of algorithms. It is envisaged that the PF will be applied in other scenarios involving highly nonlinear system dynamics (such as target tracking). The ability to draw samples from, essentially, an arbitrary density allows the use of a truncated density to naturally incorporate hard constraints, such as road networks, natural terrain obstacles, and blind sensor zones. The improved robustness and versatility of the particle filtering approach is however burdened with an increased computational cost.14

References 1.

Arulampalam S, Maskell E, Gordon N, Clapp T. 2002. A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking. IEEE Transactions on Signal Processing, 50(2):174–188. 2. Durrant-Whyte HF. 1996. An autonomous guided vehicle for cargo handling applications. The International Journal of Robotics Research, 15(5):407–440. 3. Doucet A, de Freitas N, Gordon N, editors. 2001. Sequential Monte Carlo Methods in Practice. Statistics for engineering and information science. Springer, New York. 4. Fox D. 2003. Adapting the Sample Size in Particle Filters Through KLD-Sampling. International Journal of Robotics Research, 22(12):985–1004. 5. Gustafsson F, Gunnarsson F, Bergman N, Forssell U, Jansson J, Karlsson R, Nordlund PJ. 2002. Particle filters for positioning, navigation, and tracking. IEEE Transactions on Signal Processing: Special Issue on Monte Carlo Methods for Statistical Signal Processing, 50(2):425–437. 6. Kiriy E. 2002. A localization system for autonomous golf course mowers. Master’s thesis, McGill University, Montreal, Quebec, Canada. 7. Kotecha JH, Djuriü PM. 2003. Gaussian Particle Filtering. IEEE Transactions on Signal Processing, 51(10):2592–2601. 8. Kurth D, Kantor GA, Singh S. 2003. Experimental results in range-only localization with radio. In 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS ’03), volume 1, pages 974–979. 9. Liu JS, Chen R, Logvinenko T. 2001. A theoretical framework for sequential importance sampling and resampling. In A. Doucet, N. de Freitas, and N.J. Gordon, editors, Sequential Monte Carlo in Practice. Springer, New York. 10. Lee DS, Chia NKK. 2002. A particle algorithm for sequential Bayesian parameter estimation and model selection. IEEE Transactions on Signal Processing, 50(2):326– 336. 11. Pitt MK, Shephard N. 1999. Filtering via simulation: auxiliary particle filters. Journal of the American Statistical Association, 94(446):590–599.

14

An illustrative explanation of the two orders of magnitude increase in the computational burden relative to an EKF is given by Gustasfsson et al. [14].

PARTICLE FILTER APPLICATION TO LOCALIZATION

327

12. Maskell S, Briers M, Wright R. 2004. Distribution in a particle filter. IEEE Target Tracking 2004: Algorithms and Applications, pages 23–31, March, 23–24 2004. 13. Rekleitis I. 2002. Cooperative Localization and Multi-Robot Exploration. PhD thesis, McGill University, Montreal, Canada. 14. Ristic B, Arulampalam S, Gordon N. 2004. Beyond the Kalman Filter. Artech House, Norwood, MA.

RESTRICTIVE ESTIMATION IN TRACKING PROBLEMS

ANDREY MAKSHANOV SPIIRAS, St. Petersburg, Russia

Abstract. This report presents a model for tracking manoeuvring objects based on recurrent restrictive estimation and prediction procedures. One of the principal problems in statistical estimation is how to take into account various forms of prior information on unknown parameters. The proposed mathematical techniques make it possible to introduce into recurrent estimation, tracking and prediction procedures a large amount of prior information of different kinds. Keywords: Tracking, Bayes filter, Kalman filtering generalization, Bayes risk minimization, nonlinear optimization

1 Introduction One of the principal problems in statistical estimation is how to take into account various forms of prior information on unknown parameters. In particular, in the broad area of digital trajectory measurement, estimation problems are usually stated with certain restrictions, namely conditions on parameters, which must lie within a limited region established by a system of equalities and/or inequalities. For example, while tracking naval objects, a standard prediction procedure may localize objects either on the shore or on sandbanks. To avoid such undesirable effects, it is necessary to introduce into the prediction routine geographical information on the possible boundaries of object location. Similar problems may arise when tracking objects whose maneuvering space is limited, for instance, by a minimal circulation radius. The Bayes approach provides many possible methods to overcome such limitations. The estimate is formed by minimizing the prior or posterior Bayes risk, but in recurrent estimation procedures the step by step application of underlying techniques is often limited by the volume of calculations. This fact calls our attention to less complicated alternatives.

2 Kalman filtering – least squares approach and generalizations Consider the stochastic dynamical system described by linear state and observation difference equations. State equation E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

329

330

A. MAKSHANOV

x(n + 1) = )(n + 1,n) .x(n) + *(n + 1,n) .w(n + 1)

(1)

describes system dynamics, and observation equation y(n) = H(n) .x(n) + v(n)

(2)

determines the mechanics of results generated by measurement. In Eqs. (1), (2): x x(n) is the m-dimensional vector of state. x )(n+1,n) is the m u m transition matrix. x w(n) is a random Gaussian k-dimensional vector of disturbances having zero mean and covariance matrix M[w(n)wT(j)] = Q(n).G(j,n). x * (n + 1,n) is the m u k transition matrix of system disturbances. x y(n) – s is the dimensional vector of measurements at the n-th step. x H(n) is the s u m transition matrix of observations. x v(n) is a random Gaussian s-dimensional vector of observation disturbances having zero mean and covariance matrix M[v(n)vT(j)] = R(n).G(j,n). x G(j,n) is the Croneker symbol. The following information becomes available for constructing the estimate xˆ ( n  1) after the measurement y(n + 1) is obtained:

x The extrapolation by dynamics Eq. (1) of the previous step estimate x ( n  1) ) ( n  1, n) ˜ xˆ ( n) , having 6 ( n  1) as covariance matrix. x The measurement y(n + 1). The regression problem (below) must be solved according to conditions: ­ x (n  1) x  K , ® ¯ y (n  1) H (n  1) ˜ x  v;

MK

0;

Mv 0;

ªK º U (n  1) cov« » ¬v ¼

~ ª6(n  1) 0º « ». ¬ 0 R(n  1)¼

(3)

Theorem 1. The solution of the regression problem (3) by generalized leastsquares (Gauss-Markov estimate) provides the standard Kalman filtering (KF) equations: xˆ (n  1)

) (n  1, n) ˜ xˆ (n)  K ( n  1) ˜ [ y ( n  1)  H ( n  1) ˜ ) ( n  1, n) ˜ xˆ (n)]; ~ ~ 1 K (n  1) 6( n  1) ˜ HT (n  1) ˜ [ H (n  1) ˜ 6(n  1) ˜ HT (n  1)  R (k  1)] ; ~ T 6( n  1) ) (n  1, n) ˜ 6( n) ˜ ) ( n  1, n)T  *( n  1) ˜ Q (n  1) ˜ *(n  1) ]; ~ 6(n  1) [ I  K ( n  1) ˜ H ( n  1)] ˜ 6( n  1).

(4)

Let U ( n  1) ' ˜ ' , ' – upper triangular matrix (Choleski factorization). Using the usual whitening techniques, we denote T

RESTRICTIVE ESTIMATION IN TRACKING PROBLEMS

'

1

~ ªK º  1 ª x ( n  1) º ˜ « » W; ' ˜ « » ¬ y (n  1) ¼ ¬v ¼

z; '

1

º ªI ˜« » ¬ H (n  1)¼

331

(5)

B,

and obtain the simplest regression model z = B.x + W, MW = 0, cov(W) = I. The 6 ( n  1)

least

squares

( BT ˜ B )

solution

is

xˆ ( n  1), so xˆ ( n  1)

(6) T ( B ˜ B)

 1

˜ B ˜ z; T

1 .

The so-called Bayes filter is obtained by substituting values of B ɢ Z. The matrix inversion lemma, (A + B.D.BT )-1 = A-1 - A-1.B.[ BT.A-1.B + D-1 ]-1.BT.A

(7)

yields relation (4). This result has been accepted and is currently used in lectures. It is a plausible method for obtaining different generalizations of Kalman techniques [5]. Remark. The Bayes filter requires the inversion of matrices of high dimension at every step, so in calculating it yields to KF. The advantage of the Bayes filter is that because it omits infinite values in 6 it is used as an alternative to the KF in the initial steps of recurrence or in areas of high uncertainty. Regression problems similar to (3) or (6) may be stated and solved for different metrics and under different supplementary conditions. The corresponding optimization routine yields point estimates xˆ ( n  1). Pitman estimates [6] are a possible solution to obtaining stepwise estimates of 6 (3). This technique is applied in procedures of conditionally Gaussian robust filtering [4]. The problem (3) may also be stated under corresponding uncertainty proposals, by substituting least squares by least absolute values or using the Remes algorithm [1] for metrics: max | zi  Bi ˜ x |o min

(8)

i

3 Recurrent filtering based on Bayes risk minimization The idea of using the Bayes approach with minimization of posterior risk leads to severe computational difficulties in calculating multiple integrals of large dimension, and may be effectively implemented only for problems with a moderate number of parameters. Here are some of the stages in this approach:

332

A. MAKSHANOV

When the estimate Tˆ n( X ) of parameter T is to be elaborated from a sample X, the prior Bayes risk is R M M W(T,Tˆ ) where W(T,Tˆ ) is the loss function: R is T

X

n

n

obtained by averaging W over the prior distribution of T and sampling X. Theorem 2 [6]. Let the loss function be W (T ,Tˆ ) (Tˆ  T )2 . The minimum of n

the prior Bayes risk R is Tˆn Tˆ n

n

MT (T | X ), or:

³ T ˜M (T , X )dT , R ³ M (T , X )dT 4

³ T ˜T ˜M (T , X )dT  Tˆ ˜Tˆ ³ M (T , X )dT T

4

min

n

4

T

(9)

n

4

where M(T,X) is the joint probability density of T and X, and 4 is the variation area of T. We will now designate the estimates of minimal Bayes risk (9), referred to as MBR-estimates. To better understand estimate (9), consider the following example [2, 3]. Let X = {x1,…, xn} be i.i.d. random variables subjected to the normal distribution N(T,1) where T is independent of a prior distribution X that is uniform at [a,b]. We must form a point estimate Tˆ n of T that is optimal with respect to Bayes ordering and is determined by the prior distribution and loss function: W (T ,Tˆn) (Tˆn T )2. From (9), we obtain: b

Tˆ n

1 ba

n

1 2 ³ T ˜ exp{ 2 ˜ ¦ ( xi  T ) } i 1

a

b

1 ba

n

2 ³ exp{ ˜ ¦ ( xi  T ) }

.

1 2

a

i 1

(10)

Integration in the numerator and denominator yields: Tˆ n x 

1 exp[ n / 2 ˜ (b  x ) 2]  exp[ n / 2 ˜ (a  x ) 2] , ˜ 2S n )[ n (b  x )]  )[ n (a  x )]

(11)

where ) is a Laplace function. To make sense of this formula, we investigate three cases: x  a, a d x d b, x ! b. In the first and least case, the second term causes a shift of Tˆ n from x to the nearest bound of [a,b]. As nof, Tˆ n oa, or correspondingly, Tˆ n ob. When a d x d b, Tˆ n shifts from x to the center of [a,b] but the correction is exponentially small and tends to 0 as nof. Computation results based on (11) for different mutual placing of x , a, b and for n = 3, 5, 10 are displayed in Figure 1.

RESTRICTIVE ESTIMATION IN TRACKING PROBLEMS

333

Estimation with minimal Bayes risk 1.5 Lower bound 1

n =10

0.5 MBR estimate

n =5 n =3 0

0.5

Sample mean

1

Upper bound

1.5

1.5

1

0.5

Figure 1. Shift of the sample mean

x

0 Sample mean

0.5

1

1.5

when taking into account prior information x  [-1,1].

By solving regression (3) with (9) and using Rmin as 6(n+1), we obtain the MBR recurrence at conditionally Gaussian statement. This method leads to the elaboration of similar procedures for other loss functions, and more complicated connections of prior distribution and observations such as, for instance, when the arbitrary joint distribution of (X, T ) is available. All of these estimates have similar properties that are illustrated in Figure 1 (see Eq. (9)).

4 An example: propagation in the band with two repelling bounds Let the object propagation model (state equation) be of the following polynomial form: X (n  1) ) ˜ X (n)  W (n  1), X (n) )

ª1 W º «0 1» , ¬ ¼

W

ª wx º « w » , MW ¬ v¼

ª x ( n) º « v ( n) » , ¬ ¼

ª0º «0» , Q ¬ ¼

cov(W )

(12) 0º ªV «0 V 2 » , ¬ v1 ¼ 2 x1

with the observation equation Y ( n)

ª1 0 º «0 1» ˜ X (n)  1 (n), 1 (n) ¬ ¼

ªK xº «K » , M 1 ¬ v¼

ª0º «0» , R ¬ ¼

cov( 1 )

ªV x22 0 º «0 V 2 » . ¬ v2 ¼

(13)

W and N are independent. Limitations correspond to conditions placed on position and velocity parameters, which must lie within a bandwidth of 2H. Figure 2 reveals the results of tracking the position parameter x(n) (as it moves along the sinusoid

334

A. MAKSHANOV

with step W = 0.1) using two filters: a standard KF without any restriction and a restrictive filter obtained by stepwise minimization of the prior Bayes risk – the recurrent MBR-filter. When tracking only two parameters for 100 iterations, the computing time was 13.6 s (Pentium-IV, 2.4 MHz) and exponentially increased with the number of parameters. 2 1.5 1 0.5 0

0.5

1

1.5

2

2 1.5 1 0.5 0

0.5

1

1.5

2

Kalman filter Trajectory Measurements Kalman filter Upper bound Lower bound

0

10

20

30

40

50

60

70

80

90

100

Kalman filter with inequality consrraints Trajectory Measurements Restrictive Kalman filter Upper bound Lower bound

0

10

20

30

40

50 60 Step number

70

80

90

100

Figure 2. Results of tracking using a standard KF and recurrent MBR-filter.

5 Solving (3) with equality restrictions The least-squares solution of (3) contributes (4), which is a well-known procedure. We introduce into (3) the equality restriction and denote restrictive estimates by the lower index G. G(n).x(n) = b(n)

(14)

Theorem 3. The generalized least-squares based solution of (3) with step by step restrictions (14) supplements formulae (4) with the following corrections:

S (n  1) [G (n  1) ˜ 6(n  1) ˜ G (n  1)]; xˆG (n  1) xˆ (n  1)  6(n  1) ˜ G T(n  1) ˜ S (n  1) ˜ [b(n  1)  G (n  1) ˜ xˆ (n  1)]; T

(15)

6G (n  1) 6(n  1) ˜ [ I  G T(n  1) ˜ S (n  1) ˜ G (n  1) ˜ 6(n  1)].

After solving (3) with restrictions (14), we obtain the minimization of the Lagrange function:

RESTRICTIVE ESTIMATION IN TRACKING PROBLEMS

335

~ 1 L ( x, O ) [ ~ x (n  1)  x] T ˜ 6(n  1) ˜ [ ~ x (n  1)  x]  (16) 1 T  [ y (n  1)  H (n  1) ˜ x] ˜ R(n  1) ˜ [ y (n  1)  H (n  1) ˜ x]  O ˜ [G (n  1) ˜ x  b(n)],

which yields (15). Remark. Such an approach is plausible, for example, when the trajectory is limited by the graph of available fairways.

6 Solving (3) with inequality constraints The chief shortcoming of the above MBR-filter is that at each step a large number of multiple integrals must be calculated. The following method appears to be a plausible solution for reducing the amount of calculations. Returning to (3), we must digitally minimize at each step the quadratic form: ~ 1 x ( n  1)  x] T ˜ 6( n  1) ˜ [ ~ x ( n  1)  x]  [~ 1 T  [ y ( n  1)  H ( n  1) ˜ x] ˜ R ( n  1) ˜ [ y ( n  1)  H (n  1) ˜ x]

(17)

under given constraints, with equalities as well as inequalities. The quadratic optimization routine supplies a necessary point estimate xˆ ( n  1). The difficulty is in determining its covariance 6(n + 1). Simulations revealed that, if constraints concord with the physical realm, covariance Eqs. (4) may be used without variations. Therefore, correction concerns only point estimates and the mechanics of elaborating the transfer coefficient so as to be left without alternations. Figure 3 shows the results of tracking the position parameter x(n) (as it moves along the sinusoid with step W = 0.1) using two Kalman filters: a standard filter without restriction and a restrictive filter with conditions placed on position and velocity parameters, which must lie within a bandwidth of 2H, based on stepwise digital optimization. The computing time was four times less than that of the MBR-filter and only increases polynomially with the number of parameters. 2 1.5 1 0.5 0

0.5

1

1.5

2 0 2 1.5 1 0.5 0

0.5

1

1.5

2 0

Kalman filter Trajectory Measurements Kalman filter Upper bound Lower bound

10

20

30

40

50

60

70

80

90

100

Restrictive Kalman filter Trajectory Measurements Restrictive Kalman filter Upper bound Lower bound

10

20

30

40

50

60

70

80

90

100

Step number

Figure 3. Tracking the trajectory using a standard KF and restrictive KF with stepwise digital optimization.

336

A. MAKSHANOV

Remark. In this case, estimate generation differs from that of the MBR-filter. If restrictions contradict the KF estimate, the result is the projection of the estimate on the bound of restrictions zone. By taking into account nonlinear restrictions, more common procedures of nonlinear optimization can be used to solve similar problems in non-Euclidean metrics.

7 Conclusion This paper deals with the standard conditions of the Kalman state and measuring model and proposes working procedures for recurrent restrictive estimation and prediction elaborated on the basis of: x Step by step minimization of prior Bayes risk (MBR-estimates) x Stepwise least squares with restrictions in the form of equalities x Step by step nonlinear optimization with restrictions in the form of equalities and/or inequalities Some application examples of elaborated algorithms and their modifications for non-quadratic loss functions are discussed. With the proposed approaches, a large amount of prior information can be introduced into recurrent estimation, tracking and prediction procedures.

References 1. 2. 3. 4. 5. 6.

Demianov VF, Malozemov VY. Editors. 1977. Questions of Theory and Elements of Software for Minimax Problems. SPb University, SPb. Makshanov AV, Musaev AA. 1983. Minimax Methods of Statistical Synthesis of Information Systems. SPb, VICU. Makshanov AV. 1983. Dynamic Models in Tracking and Surveillance Problems. SPb, VICU. Masreliez CJ. 1975. Approximate Non-Gaussian Filtering with Linear State and Observation Relations. IEEE Trans. Vol. AC-20, pp.107–110. Meditch JS. 1969. Stochastic Optimal Linear Estimation and Control. McGraw Hill, New York. Zacks S. 1971. The Theory of Statistical Inference. Wiley, New York/London/ Sydney/Toronto.

FINDINGS OF THE NATO WORKSHOP ON DATA FUSION TECHNOLOGIES FOR HARBOUR PROTECTION

ELISA SHAHBAZIAN1, MICHAEL J. DEWEERT2 AND GALINA ROGOVA3 1 OODA Technologies Inc. Canada 2 BAE Systems Spectral Solutions, Hawaii 3 Encompass Consulting, USA

Abstract. The NATO Security Through Science Program and the Defence Investment Division requested and sponsored the organization of a NATO Advanced Research Workshop (ARW) on the topic of Data Fusion Technologies for Harbour Protection, which was held June 27–July 1, 2005 in Tallinn, Estonia. The goal of the workshop was to facilitate knowledge exchange between technology experts and security policy makers for a better understanding of goals, functions and information requirements of the decision makers as well as of how data fusion technology can enhance the security of harbours. In addition to presentations by experts from the research community on detection and fusion technologies in practice as well as in policy, the workshop program included daily breakout sessions in which participants were given an opportunity to brainstorm on workshop topics in smaller interdisciplinary teams. The working groups: (i) chose a scenario, including threat stages, threat types, threat methods and ranges, and response constraints due to the particular harbour environment; then (ii) identified: (a) requirements (objectives, functions and essential elements of information), (b) technologies (available and future), (c) information available and necessary through sensors and other sources, such as agencies and jurisdiction, (d) methods: detection, identification, situation assessment, prediction. This paper describes the main issues and proposed approaches that were identified by the working groups.

1 Introduction The protection of harbours has always been essential throughout the history of human civilization. With time, this task has become more and more difficult due to increasing cargo and passenger traffic and increasing sophistication, speed and types of marine vessels. Volume II of the Defence Science Board (DSB) 2003 Summer Study on DoD Roles and Missions in Homeland Security issued in May 2004 states “a major terrorist incident in a U.S. port would cost the U.S. economy on the order E. Shahbazian et al. (eds.), Harbour Protection Through Data Fusion Technologies. © Springer Science + Business Media B.V. 2009

337

338

E. SHAHBAZIAN, M.J. DEWEERT AND G. ROGOVA

of a trillion dollars.” Naturally there is an increasing requirement for technology to monitor, and at all times to be fully cognisant of any entity approaching on the sea surface, underwater, in the air, and from the landward side of the harbour, to establish situation awareness, to predict how the situation will evolve in time, as well as to select an appropriate action to deal with the threats. Until recently (less than 10–20 years ago) this meant the development of sensors of increasing sophistication with humans analyzing the information to establish the situation awareness. To cope with the current influx of information and the increased sophistication of potential threats, humans require decision support technologies such as Data/Information Fusion. Data/Information Fusion is essential to facilitating the decision making process to reduce the volume of information available to only that which is relevant, and to ensure that the right decision is made. To facilitate knowledge interchange between technology experts and security policy makers and to impart a better understanding of how data fusion technology can enhance security in harbours, the NATO Security Through Science Programme and the Defence Investment Division requested the organization of a NATO Advanced Research Workshop (ARW) on the topic of Data Fusion Technologies for Harbour Protection, which was held June 27–July 1, 2005. The Unites States (US) Office of Naval Research as well as Defence research and Development Canada (DRDC) also sponsored this ARW. The workshop program included invited presentations by distinguished experts from the research community in detection and fusion technologies in practice as well as in policy. Speakers cognisant in the following topics participated: (a) Harbour security operation, logistics, standards, etc. (b) Description of existing capability in Harbour protection (c) Information sources: 1. Imaging, Image processing and Remote sensing 2. Underwater Technologies 3. Intelligence and other Information sources –

Data and Information Fusion (i) Level 1 fusion for each type of Information source and/or combination of these sources (ii) High Level (2,3,4) DF capability analysis and development

This forum also included breakout sessions with discussions, addressing various aspects of threat and technology to be used for unconventional threat prediction, prevention and response: (a) Threat 1. 2. 3. 4.

Threat components (intents, opportunity and capability) The various stages of threat Ranges of threat Threat type

FINDINGS OF THE NATO WORKSHOP ON DATA FUSION TECHNOLOGIES 339

(b) Technology and Information 1. Information available and necessary through sensors and other sources (e.g. open, agencies and jurisdiction) 2. Technologies for detecting threats (c) Fusion 1. 2. 3. 4.

Existing technologies to support fusion requirements Future technologies to support fusion requirements Opportunities for fusion Possible solution methods

There were six working groups with ten participants in each group (shown in Table 1) who: (a) Chose a scenario, including threat stages, threat types, threat methods and ranges, response constraints due to the particular harbour environment; then (b) Identified: 1. Requirements (objectives, functions and essential elements of information) 2. Technologies (available and future) 3. Information available and necessary through sensors and other sources, such as agencies and jurisdiction 4. Methods: detection, identification, situation assessment, prediction The sections below summarize some conclusions of the working group discussions.

2 Working group discussions and results 2.1 SCENARIOS The working groups mainly chose two types of scenarios. (a) Scenarios of the first group were characterized by intent of a vessel either regulated or not by the Ship and Port Facility Security (ISPS) code (less than 500 t) containing some type of an explosive (conventional or CBRN) to collide with a selected high value vessel (or other target) in the harbour, causing significant loss of life and/or psychological and/or economic damage and creating a number of secondary effects. Scenarios differed by threat stages. While some groups concentrated on information fusion based technologies to prevent attack, the other groups considered post-disaster response. (b) Some type of an explosive (conventional or CBRN) or a Weapon of Mass Destruction (WMD) capable of causing significant loss of life and/or psychological and/or economic damage and creating a number of secondary effects were hidden in/on a container. The groups designing this type of scenario considered information fusion based technologies either to prevent disaster or to increase overall container based supply chain security.

340

E. SHAHBAZIAN, M.J. DEWEERT AND G. ROGOVA TABLE 1. Working group participation

Name Elisa

Surname Shahbazian

Title President

Maria

Rey

Scientist

Gunter Lauro

Palm Snidaro

Professor Scientist

Mike

DeWeert

Bruce

Jones

Sr. Principal Scientist Consultant

Aivar

Lõhmus

James

Douglass

Andrzej

Jacuch

Jiri Eloi

Patera Bosse

Hasmik Arvid Roman

Atoyan Melkevik Nawra

Jacob Barhen Dominique Cant Ervin

Piikmann

Aivar

Usk

John

Regan

Ulo Jaaksoo Alexander Sokolov

Jean Jim Mike

Couture Llinas Moskal

Toomas

Sullakatko

Bernard

Delsupexhe

Matteo

Gandetto

Carlo

Regazzoni

Stephen

Pope

Rein

Haavel

Institute Lockheed Martin Canada, CRM University of Montreal, and OODA Technologies Inc. Canada Defence Research and Development Canada (DRDC), DSTM Ulm University, Germany Department of Mathematics and Computer Science (DIMI), University of Udine BAE Systems Spectral Solutions

NATO; Specialising in international affairs and defence issues Lieutenant Deputy Director General Estonian Board of Colonel Border Guard Chief Deputy National Co-ordinator, Ports Superintendent Policing (Protective Security) Staff Officer Staff officer of the NATO Planing Board for Inland Surface Transport (PBIST) Professor CRM, University of Montreal Section Head Defence Research and Development Canada (DRDC) Valcartier MSc Student Ecole Polytechnique, Montreal, Canada Scientist FFI Norway Head of Section NATO Planing Board for Inland Surface Transport (PBIST) Scientist Oak Ridge National Laboratory Manager Security Manager, P & O Ports, Antwerp, Belgium Captain Major Head of Dept., Coast Guard Department, Estonian Board of Border Guard Dept. Head Department of Navigation Systems (EKTA), Cybernetica AS, ESTONIA Specialist (Former) CSI Liaison to ICE Attaché United States Embassy, London Engineer Cybernetica AS, ESTONIA Captain Deputy Director, “TRANSAS Eurasia” St. Petersburg, Head of Crisis Management System (CMS) Division Scientist Lockheed Martin Canada Professor State University of New York Engineer Calspan-UB Research Center, Inc., Buffalo, NY, USA Senior Head of Department of Communications Lieutenant and Information Technology, Estonian Navy Staff PBIST Port NATO Planing Board for Inland Surface Expert Transport (PBIST) Student Dept. of Biophysical and Electronic Eng., University of Genova, Italy Professor Dept. of Biophysical and Electronic Eng., University of Genova, Italy Operations UK Resilience and Decontamination Agency Director Project Manager Cybernetica AS, Project Manager – Radio Surveillance Systems,

gr. Country 1 CA 1

CA

1 1

GE IT

1

USA

1

UK

1

ES

1

UK

1

PL

1 2

CA CA

2 2 2

CA NO PL

2 2

USA BE

2

ES

2

ES

2

USA

2 2

ES RU

3 3 3

CA USA USA

3

ES

3

FR

3

IT

3

IT

3

UK

3

ES

FINDINGS OF THE NATO WORKSHOP ON DATA FUSION TECHNOLOGIES 341 Sergey

Novikov

Captain

Head of ISPS Department, “TRANSAS Eurasia” St. Petersburg Evgeni Kiriy PhD Student McGill University Krzysztof Gromadowski Port Authority Director, Port Development and Integration with European Union Agoustino Bruzzone Director McLeod Institute of Simulation Science Director, U of Genoa, It Calspan-UB Research Center, Inc., Buffalo, Alan Steinberg Scientist NY, USA Kiril Alexiev Senior Bulgarian Academy of Sciences, Institute Researcher for Parallel Processing Jaanus Kaugerand Junior Operational Department, Estonian Navy Lieutenant Staff Mati Iila Councillor Emergency Management Department, Ministry of Economic Affairs and Communications Tom Cousins Scientist Head of CBRNE Working Group, the DRDC PI on CBRN Monika Oit Dept. Head Cybernetica AS, Head of Department of Information Security OODA Technologies Inc. Canada Yannick Allard Scientist Bassam Kakish Consultant Was Director General of Aqaba Port Authority, Jordan Albena Tchamova Senior Bulgarian Academy of Sciences, Institute Researcher for Parallel Processing Joachim Biermann Scientist FGAN/FKIE Neuenahrer Strasse 20, D-53343 Wachtberg-Werthhoven Pierre Blanc-Benon Scientist Thales Underwater Systems, France Facilitator between QinetiQ and Bill Borrowdale QinetiQ ports’ police Consultant Eric Noor Supervisor Port Supervision Department, Maritime Inspector Safety Division, Estonian Maritime Administration Yuval Alon Head/Senior G.S. SEALS Ltd (Associate of the Advisor Inetnational Policy Institue for Counter Terrorism (ICT), Israel) Ruth Woodward Scientist Scientific advisor and terahertz specialist, HT Consultants Ltd., Magdalen Centre, Oxford, UK Mart Rüütel Project Manager R-Süsteemid OÜ, Mustamäe tee 5, Tallinn 10616, Estonia JOSÉ J. P. Centeno Captain Captain of the Port of Lisbon de Ruslan Sorokin Senior Staff St. Petersburg Institute for Informatics and Scientist Automation of RAS Jean-Remi Duquet Scientist Lockheed Martin Canada Consultant Encompass Consulting, Honeoye Falls, NY, Galina Rogova USA Simon Julier Scientist Contractor for Naval Research Laboratory Marc Owen Scientist SPAWAR Scientist Planning Systems, Inc., 3834 Vista Azul, Jim Wilson San Clemente, CA, USA Nathan Tonnisson Deputy Head Aviation and Maritime Department, Dept. Ministry of Economic Affairs and Communications

3

RU

3 4

CA PL

4

IT

4

USA

4

BU

4

ES

4

ES

4

CA

4

ES

4 4

CA JO

5

BU

5

GE

5 5

FR UK

5

ES

5

IS

5

UK

5

ES

5

PT

5

RU

6 6

CA USA

6 6 6

USA USA USA

6

ES

342

E. SHAHBAZIAN, M.J. DEWEERT AND G. ROGOVA

Erik

Orbeck

Carlos

Rodolfo

PBIST Port Expert Rear Admiral

Chris

Fendt

Sgt

VictorEmil

Neagoe

Professor

NATO Planning Board for Inland Surface Transport (PBIST) President, AFCEA Portugal, AIP - EC Junqueira, Praça das Industrias, Lisboa, Portugal Marine Intelligence Team @ Wapping, Marine Support Unit 0207 275 4431 Faculty of Electronics, Telecommunications, and Information Technology, “Politehnica” University of Bucharest, Romania

6

NO

6

PT

6

UK

6

RO

Both of these threats assumed to have a non-negligible possibility of achieving their objectives due to high vulnerability of the ports: (a) There are a wide variety (fishing, leisure, etc.) and very large number of small ships (less than 500 t). As these do not have to report to the Automatic Identification System (AIS), it was feasible to assume that their activities would not all be detected by Harbour security systems. At the same time, the vessels regulated by ISPS code have an opportunity to achieve their goal due to the enormous amount of information to be processed by the port security personnel, the large number of agencies collecting and processing this information, and the lack of more sophisticated monitoring equipment. (b) The United States General Accounting Office, GAO Report: GAO-04-325T, “Preliminary Observations on Efforts to Target Security Inspections of Cargo Containers”, UNCLASSIFIED, 16 December 2003 stated that more than 80% of goods traded world wide were transported by sea, with much of them in containers employing thousands of container vessels. Each year there were more than 250 million container movements, representing approximately 90% of the world’s cargo moves. Due to such a huge volume of container traffic it was possible that a carefully planned tampering of a container would be missed by the security systems. 2.2 THREAT CHARACTERISTICS Overall the unconventional threat had been characterized as an integrated whole of intent, capability and opportunity graphically shown on Figure 1. The threat can achieve its objective only if it has all three of these characteristics.

FINDINGS OF THE NATO WORKSHOP ON DATA FUSION TECHNOLOGIES 343

Ca pa

bi lit y

y nit rtu po Op

Intent

Figure 1. Threat as integrated whole of capability, opportunity, and intent. (From Little et al., “Theoretical Foundations of Threat Ontology (ThrO) for Data Fusion Applications”, DRDC TR-2005 -269.)

For this analysis, intent was treated as a constant, that was, an adversary desired to use the weaknesses in the ISPS system (in tracking smaller vessels), high port vulnerability, or the container supply chain to deliver a threat to its destination with a high degree of certainty of outcome and with maximal damage, as planned. Because of the variable nature of the security system vulnerabilities, capability was treated as a variable, primarily based on the level of sophistication required to exploit vulnerabilities in the systems, or the level of collusion that would be required to do so. A definition of Opportunity was proposed as the factor having the greatest weight on whether or not vulnerability could be exploited. Opportunity includes, but is not limited to: (a) Time to prepare for the attack and increase capability (b) Methods to: 1. Obtain explosives or WMD without being detected 2. Gain control of a vessel of an optimal size and capability or allow unrestricted access to the container or its contents 3. Choose the time and place that will make the most impact 4. Ensure execution processes that require only a small number of individuals to execute 2.3 THREAT STAGES AND RANGES Threat stages have been defined as: (a) (b) (c) (d)

Potential including Strategic Warning Imminent Attack assessment and response Consequence management, including secondary threats detection and management (e) Lessons learned and refining response plans

344

E. SHAHBAZIAN, M.J. DEWEERT AND G. ROGOVA

Threat range impacts the reaction time of the harbour Defence systems to detect a threat and plan a response. It is mainly a function of geographical characteristics of a Harbour (e.g., a threat at a North American open ocean port has generally longer range than at a Baltic or a Mediterranean port); however various vulnerabilities of the security system in a port can also impact the threat range. During the breakout sessions some groups chose to consider one stage of threat while others included several or all stages in the scenario. A short summary of specifics of each group’s threat scenario with inferences that can be generically applied to the whole harbour security problem are listed below. Group #1 considered of a scenario in which one small and one medium vessel collided with a cruise liner leaving Honolulu harbour. They focused primarily on the potential and imminent threat stages of the attack, since it was judged more important to prevent the attack. However, for the identification of fusion opportunities group #1 also looked at the post-attack phase. Group #1 produced a detailed list of technologies (available, evolving, and future) facilitating effective response to an attack in potential, imminent as well as post-attack phases. Group #2 focused on a pre-attack scenario, in which conflicting intelligence indicated a potential short-range terrorist attack at the annual harbour summer festival scheduled for this weekend. The goal of was to detect the threat with a high degree of confidence and prevent the attack. The focus of group #6 was also on the identification of potential threat and attack prevention. The approach was to consider different phases of the scenario simultaneously from the point of view of terrorists planning an attack as well from the point of view of port protection agencies. The phases of the scenario considered from the terrorist attack planning point of view were analyzed in order to understand how terrorists might build capabilities and look for opportunities. The phases of the scenario considered from the port protection agencies were analyzed in order to understand possible port vulnerability and necessary information requirements for discovering terrorists’ intent and capabilities. Group #5 considered the pre-attack and the post-attack of the scenario, in which enemy intent was selected as an attack on a cruise ship moving through a harbour with a local fishing boat loaded with explosive material. Group #4 considered the scenario of a small boat filled with a mixture of explosives and CBR. That scenario showed various difficulties in threat detection for this kind of boat and concentrated on attack assessment and response. At the same time group #4 presented a good candidate threat assessment and response system, shown in Figure 2, discussing all stages of threat that would be suitable for many harbour terrorist attack situations described by the other groups. This figure also lists sensor data, fusion and information capabilities involved in these phases.

FINDINGS OF THE NATO WORKSHOP ON DATA FUSION TECHNOLOGIES 345 Strategic Warning Regional/ Specific Threat Warning

National Intelligence

Indications & Warning

•Deny access •Interrogate •Arrest •Monitor

•ID Check •Magnetometer •X-ray

Personnel Security Screening

•Boat Registry •CBR Detectors •Trackable features

Boat Portal Screening

•Interdict •Board •Arrest/Destroy •Monitor

Attack Assessment and Response •Surface Search Radar •Sonar •Cued Radar (Sector Scan) •Patrol Helicopter •Patrol Boats

Consequence Assessment

Boat Detection & Tracking

Threat Assessment

Threat Response

•Maneuver Detection •Track Projection •Attack Opportunity (Time-to-go for potential attack: target/weapon)

•Interdict •Destroy

•Helicopter •Patrol Boats

•Forensic Analysis •Responsible Organization •Revised Threat & Attack Models •Vulnerability Assessment •Sensor & Fusion/Analysis Assessment •Response Assessment

Figure 2. Candidate threat assessment and response system presented by group #4.

The working group #3 performed a good analysis of the threat assessment process for the container scenario. Figure 3 presents their analysis of vulnerabilities in a container supply chain. This figure also includes information that can be fused to detect the threat. Process Container Depot (Empty Container Storage)

I

Threat C

O

H

H

M

H

H

H

H

M

M

H

H

H

H

L

M

H

L

L

H

L

H

H

L

L

H

M

L

H

H

H

Empty Container Transport

Container Loading/Stuffing

Container Transport to Originating Port Terminal Container Acceptance, Storage and Handling @ Port Terminal

Container Loading – Ship

Sea Transit

Container Unloading – Ship

Container Acceptance, Storage and Handling @ Port Terminal Container Transport to Warehouse/Distribution Facility

Potential Vulnerabilities Outside Attachment Container Substitution Cargo Substitution Inside Placement Outside Attachment Container Substitution Cargo Substitution Inside Placement Outside Attachment Container Substitution Cargo Substitution Inside Placement Outside Attachment Container Substitution Cargo Substitution Inside Placement Outside Attachment Container Substitution Cargo Substitution Inside Placement Outside Attachment Container Substitution Cargo Substitution Inside Placement Outside Attachment Container Substitution Cargo Substitution Inside Placement Outside Attachment Container Substitution Cargo Substitution Inside Placement Outside Attachment Container Substitution Cargo Substitution Inside Placement Outside Attachment Container Substitution Cargo Substitution Inside Placement

Rating H H N/A H H H N/A H M L M M H H H H L L M M L L L L L L M M L L L L L L M M H H H H

Available Information/Sensor Data Container ID Number, Transactional Data (Container order, billing information, etc.)

Transactional Data, Limited Technical Data

Inventory Records,Bill of Lading

Transaction Records No time details

Video Surveillance Terminal Management System Data Intrusion Detection Systems Data Ships Load Plan, Terminal Managmement System Data Ships Manifest AMS-ATS Data/Targeting (US Bound Only), VACIS Data RADAR/AIS RADAR/LRIT RADAR/ELINT Travel Time 4, Coastal/AIS/RADAR Ships Load Plan, Terminal Managmement System Data Ships Manifest AMS-ATS Data/Targeting (US Bound Only), VACIS Data Video Surveillance Terminal Management System Data Intrusion Detection Systems Data Transaction Records No time details

Figure 3. Group #3 assessment of the threats and vulnerabilities (I – the explosive or WMD concealed in the container, C – cargo or container substitution, O – attached outside of the container).

346

E. SHAHBAZIAN, M.J. DEWEERT AND G. ROGOVA

Analysis of all scenarios, which together covered all stages of threat demonstrated benefits and identified technology and information processing requirements discussed in the next section.

3 Technology and information Group #1 leveraged the findings of IST Task Group IST-038/RTG-016 on Information Fusion Demonstration, IST-055 specialist meeting to help put together a comprehensive list of sensors and information sources as well as systems technologies that could potentially provide information to facilitate the detection of and response to the attack. Information Sources include: (a) Sensor technologies: IR, Radar, Sonar, MADs, Video, Hyperspectral, Polarimetric Imagery, Satellite imagery, explosive and CBRN detectors, Terahertz detectors, Heart-beat detectors, CO2 detectors, synthetic olfactory detector, LLTV, underwater optics, frequency scanning and DF, tactical ELINT and SIGINT (b) Other information available and necessary: information from all jurisdictions, i.e., Police, Customs, Immigration, Security Services, Internal services, External services, National signals intelligence agency, National geospatial agencies, Coast-guard, Intelligence, military (Navy, Air force), fire department, port administration, private sector (terminal operators, freightforwarders, shippers, manufacturers, insurance companies, ship brokers, ship owners), registries, regional authorities, local HAZMAT team and area bomb squad team reps., environmental protection agency, trade union, technical personnel (i.e., utilities water, electricity), medical unit staff, etc. The discussion on current and future/evolving systems technologies shows that quite a few of them apply data fusion techniques to derive the information. The lists below include such systems as well. (a) The currently existing system and technologies for extracting and processing information include: visualization technologies, geomatics (spatial context), reconfigurable dummy and covert applications, surface ship classification, robotic and unmanned systems, automatic video surveillance, diver detectors, biometrics, smart seals, smart containers, data mining, tools such as Lloyd’s Sea Searcher and vigilance software, and technologies for looking for hidden information in files, images and messages (steganography, information hiding). (b) Future/evolving technologies and methods include: technologies for the formation and dissemination of the common, specific and discrete tactical picture; multi-level security and security enforcement; communications; technology to support interoperability; across-provider knowledge and information management/ mining/ discovery; social network analysis techniques, behaviour modeling, knowledge discovery from unstructured data within distributed

FINDINGS OF THE NATO WORKSHOP ON DATA FUSION TECHNOLOGIES 347

information sources; situation assessment tools; visualization tailored to the specific needs of each decision maker; ship stopping technologies; local directional jamming; modeling and simulation; operational analysis; monitoring of information flow and domain/jurisdictional/competency space (for bridging information gaps procedurally and technically); derivation of formally structured and computationally tractable domain representation; methods for the prediction of threat and intent. Modeling and simulation was identified as a very critical technology in understanding the threat and vulnerabilities of the harbour and in responding to threats. The simulation of events and creation of scenarios is necessary to train and validate the data fusion architecture and its components. It is also required for training the operators, communications system and system response coordination to determine the effectiveness of the data fusion architecture. This is a multilayered approach both in terms of system integration and standards implementation. The problem of modeling and simulation is a difficult one due to the high dimension of possible scenarios. Modeling and simulation along with methods of intelligence gathering and analysis are particularly essential in pre-attack phase to understand, train, and evaluate methods for threat prediction, detection, prevention and consequence management. Pre-attack modeling and simulation of threat scenarios needs to: (a) (b) (c) (d) (e)

Identify fusion needs in advance Model effects of prolonged alert (fuel, stores, personnel, cost) Model consequences of losing a sensor or system Get human component into simulation Model human response & Standard Operating Procedures (SOPs) How can changing an SOP affect response effectiveness?

Pre-attack modeling and simulation should be an on-going effort, which adapts itself to world situations. This ongoing process should get all stakeholders involved (tugs, fireboats, port authority, law enforcement, shippers, etc.). It should be designed to: (a) (b) (c) (d)

Identify possible asynchronous gaps Improve timing & accuracy, quality of information, speed of information Improve response Implement hardware & software to address needs identified in Modeling

4 Fusion opportunities, issues and approaches As it was mentioned above, the analysis of all scenarios considered during the breakout sessions identified opportunities for fusion, proposed possible approaches and discussed possible benefits of utilizing data and information fusion for harbour protection. From the analyses of the working groups it is apparent that the

348

E. SHAHBAZIAN, M.J. DEWEERT AND G. ROGOVA

information that needs to be fused to establish satisfactory situation awareness in the harbour comes from a very large variety of disparate and very dissimilar sources of very different levels of confidence and quality. Much of this information is available to only some of the jurisdictions, and the processes for sharing this information as well as for sharing the decisions/recommendations developed by the jurisdictions (or any decision making entity or automated fusion systems) are not currently developed. It is apparent that there are many procedural issues to resolve. These are assumed to be outside of the ARW scope. Assuming the information collection and sharing procedures are developed and agreed with between the jurisdictions, there are fusion system architectures and processes to understand and develop. For example, there is too much and too disparate information to be fused in one fusion system. It would probably be more appropriate to distribute the fusion processing in multiple nodes, which exchange their decisions and act on the recommendations of these systems as appropriate. An example model for distributing the fusion processing was proposed by working group #2. They proposed that security procedures in a Harbour are achieved through a collaboration of multi-jurisdiction Decision Centers (DCs), shown in Figure 4. In this figure, the multi-jurisdiction organizations of Canada are depicted, however the overall model is to distribute processing and decision-making based on jurisdiction and capability, which is generic and could be applied in any country. In this model, the information from one or more jurisdictions is fused within a Decision Centre (DC). The DCs share information between each other and depending on a specific situation the DC, which is best equipped to deal with the situation takes control. DM

ew an

s tie ali ip ic un M

M un ic ip ali tie s

Te Yu rr ko ito n ry

British Columbia

CF

d

t vu na Nu

Municipalities

RCMP

DFO

Mun icip alit ies

New foun dlan

ies palit nici Mu st we rth ries No rrito Te

EC AAFC

Municipalities

RCMP DND CFIA

Prince Edward Island

NRCAN

GOC

CF

DFAIT

PCO

es

HC TC

iti pal nici Mu

CF

RCMP

Real-time Sensing

Br N un e sww ic k

RCMP

PSEPC

va No tia Sco

-SIGNINT -SIGNINT -HUMINT -HUMINT -ELINT -ELINT -…etc. -…etc.

Municipalities Mu nici palit ies Ontario Qu ebec

s tie

Intel Reports

es aliti icip Mun a itob Man

ali ip ic un M

Analysis, fusion & Analysis, fusion & Exploitable Info Products Exploitable Info Products tailored tailoredtotouser user

Mu nic ipal itie s M Alb un erta ic ip Sa ali sk tie atc s h

SA

Decision Centers Multi-Jurisdiction Organizations

Operational Data (e.g. AIS)

Plans Plans&&Tasks Tasks

Distributed Execution Distributed Execution (information sharing, (information sharing, resources allocation,..etc.) resources allocation,..etc.)

Synchronization Synchronization

Events, Events,Containers, Containers, harbour harbourinfrastructures, infrastructures, goods, goods,people, people,…etc. …etc.

Harbour

Actions/Effects Actions/Effects

Figure 4. Multi-jurisdiction Decision Centers co-operation model presented by Group #2.

FINDINGS OF THE NATO WORKSHOP ON DATA FUSION TECHNOLOGIES 349

In terms of specific fusion processing in each node, working group # 6 presented more detailed analyses. The goal of group #6 was to demonstrate the applicability of data and information fusion to designing a decision aid to support situation awareness. Group #6 adopted a general fusion processing initially designed for post-disaster response management and demonstrated its applicability to fusion processing for the identification of potential threat and prevention of an attack. Elements of this fusion processing included: (a) Structured representation of domain and domain dynamics (spatial and temporal relationships between items of interest, e.g. 1. Relationships within and between items on watch lists (people, financial inst., training centers, char. organizations) 2. Port personnel, owners, brokers, ships (routs, cargo, crew…) (b) Construction of a knowledge base (c) Dynamic data collection and fusion (d) Reasoning about observations: dynamic construction of arguments pro and contra hypothesis about suspicious behaviour (e) Argument combination and belief in each hypothesis accumulation (f) Decision on whether the confidence that the information obtained from observation is high enough to make a decision Independently of how fusion processing is distributed between nodes in the overall harbour protection fusion system, all of the above processes need to be addressed in each fusion node, based on the specific types of information it will fuse and the specific types of decision and knowledge (fusion result) it will have to derive from this information. One other very important aspect of the overall harbour protection fusion system is information, decision and knowledge exchange protocols between nodes, including timing constraints as well as the quality, pedigree (from whom and what it is composed of) this information. All of the groups agreed that the most important goal of harbour protection is to identify and prevent potential threat during the pre-attack phase. One of the most critical roles of fusion processing at the pre-attack phase is to identify the “normal” behaviour and detect “abnormalities,” which can be used to identify suspicious behaviour and terrorist intent. Normal behaviour can be modeled in advance as part of domain representation (a-priori knowledge). This a-priori knowledge includes, for example, vessel profiles, passenger profiles, crew and port staff profiles, financial transactions and travel patterns. The result of this modeling can be used to identify suspicious behaviour and terrorist intent. Abnormal behaviour can then be detected by monitoring the behaviour of interest and comparing it with defined earlier normal behaviour. A reasoning process should follow abnormality detection to confirm threat and avoid false alarms. The small group discussions concluded that due to the enormous amount of data produced by existing and future data collection methods, the exploitation of

350

E. SHAHBAZIAN, M.J. DEWEERT AND G. ROGOVA

fusion methods could be beneficial for prediction, prevention, and consequence management of terrorist attacks. The opportunities and requirements for fusion as well as fusion architectures and methods selected may depend on geopolitical situation, scenario, etc. At the same time, there are many fusion opportunities and requirements that are common in most scenarios and threat stages.

5 Conclusions The presented paper summarizes the results of the NATO Advanced Research Workshop on Data Fusion Technologies for Harbour Protection. The goal of the workshop was to facilitate knowledge exchange between technology experts and security policy makers for a better understanding of goals, functions and information requirements of decision makers as well as of how data fusion technology can help to enhance the security of harbours. In addition to participants’ presentations, the workshop program included daily breakout sessions in which participants were given an opportunity to evaluate the applicability of sensor and fusion technologies to harbour protection by designing and analyzing various threat scenarios. Working Group discussions demonstrated the benefits of the utilization of Information/Data Fusion and sensor technologies for identification, prevention, and managing terrorist attacks. The workshop identified the information and processing necessary to develop a fusion system capable of enhancing the decision-making capabilities of security experts. They identified the technology and processing required for designing such a fusion system. The workshop discussions and presentations demonstrated that the technology and processing required for designing an effective fusion system for harbour protection include traditional fusion processing and technology with special attention to: (a) Methods of expert knowledge elicitation and representation, e.g., methods of effective representation of all procedures and regulations related to ships, cargo, and people movements. (b) Means to support cooperation between various agencies and technology enabling secure information exchange between different agencies and databases. (c) Methods of comprehensive and computationally tractable domain representation. (d) Methods of identifying unusual behaviour and reasoning processes for assessment of the causes of such behaviour and recognition of potential threat. (e) Evaluation methods (how to define success?). (f) Small target detection, tracking and identification. (g) Adaptation of existing sensor fusion techniques to fusion of new and future harbour protection-specific sensors.

FINDINGS OF THE NATO WORKSHOP ON DATA FUSION TECHNOLOGIES 351

(h) Decentralized fusion architecture addressing multi-jurisdictional, multinational cooperation. There is a significant on-going effort aimed at data and information fusion capability for specific military systems (Naval, Air, Army systems and interoperability between these), which assumes mostly the conventional threat. At the same time, much less work has been carried out in understanding the fusion requirements for developing systems dealing with asymmetric threat. The workshop identified numerous information/data fusion related research issues to be investigated specifically for asymmetric threat, such as assessment of vulnerability, intent, capability, opportunity, situation assessment and management, institutional issues (inter-agency cooperation), and made the first step towards addressing these issues.

SUBJECT INDEX

A Action, 37, 41, 61, 74, 75, 90, 107, 110, 142, 143, 164, 193, 195, 299, 338 Aggregate, 154, 189, 210 Ambiguity function, 276 Anomalous activities, 224 Anomaly detection, 91, 229 Anti-submarine warfare, 259, 267 Architecture, 156, 157, 179, 199, 202, 206, 207, 239, 240 conceptual, 125 service-oriented (SOA), 90, 206 Artificial intelligence, 59, 108, 172, 191, 192, 198 Asymmetric threat, 111, 114, 115, 131, 141, 171, 177, 351 Attack, 33, 48, 62, 97, 99, 100, 141, 144, 174, 177, 197, 233, 307, 339, 343, 344, 346, 347, 349 cyber, 199, 203, 205 nuclear, 62 terrorist, 142, 144, 197, 200, 344, 350 Automated Targeting System (ATS), 63 Automatic Identification System (AIS), 110, 120, 125, 342 surveillance system, 223 target detection, 209, 243 Autonomous robots, 317, 318 B Battlespace, 130, 131, 149, 153 Bayesian classification, 211 loss function, 274 Behaviour, 47, 59, 97, 102, 111, 113, 114, 117, 127, 130, 131, 132, 133, 136, 150, 163, 164, 167, 169, 177, 187, 203, 215, 260, 264, 265, 312, 346, 349, 350 analysis, 215 default, 131

model, 127, 131 normal, 112, 114, 177, 349

Belief Propagation, 95, 96, 106, 108 Bhattacharya coefficient, 237 Bioinformatics, 199, 206 Biometric, 72, 202, 203, 238, 289, 291, 292, 298 British Institute of Standards, 73 C C4ISR, 119, 120, 121, 122, 124, 125 Capability, 35, 47, 53, 54, 56, 57, 59, 60, 97, 98, 99, 100, 102, 103, 109, 110, 113, 144, 150, 156, 165, 171, 172, 173, 174, 175, 176, 182, 184, 204, 205, 206, 212, 215, 219, 234, 235, 243, 259, 260, 264, 338, 342, 343, 348, 351 Cargo, 8, 9, 10, 11, 20, 23, 25, 29, 37, 41, 48, 61, 62, 63, 68, 71, 89, 110, 113, 120, 181, 189, 200, 201, 203, 204, 205, 223, 264, 312, 315, 326, 337, 342, 345, 349, 350 Categorization, 26, 48, 49, 133, 134, 135, 210 CBRN, 2, 3, 89, 144, 145, 153, 156, 339, 346 CCTV, 27, 29, 30, 34, 59, 223 CFAR, 243, 245, 246, 247, 248, 250 Change detection, 225 Characterization, 73, 97, 113, 117, 149, 243, 244, 245, 246, 248, 249 Symmetric scattering, 246 Chemical Radiological Biological Nuclear and High Explosives, 4 Chromatic Similarity, 237 Civil emergency planning, 1, 2, 3, 4, 17, 18, 20 Classification, 39, 40, 46, 48, 120, 130, 134, 211, 219, 225, 234, 245, 247, 262, 283, 284, 285, 286, 287, 288, 289, 291, 292, 293, 294, 297, 299, 300, 301, 302, 304, 305, 346 Closed world representation, 127 353

354

SUBJECT INDEX

Cluster, 153, 154, 177, 226, 227, 228, 229, 259, 261, 304, 305 tree, 229 Cognitive process, 127, 130, 136, 168 robotics, 299 Work Analysis (CWA), 113, 118, 159 Coherent Target Decomposition, 243, 246 Collaborative, 77, 139, 140, 153 environment, 139, 140 workspace, 139, 140 Collation, 127, 128, 129, 133, 135, 136 Comb spectrum, 276 Combinatorial optimization, 251, 252, 254, 256 tracking, 253 Combinatory logic, 95 Command and Contro (C2)l, 43, 44, 45, 46, 50, 103, 127, 128, 132, 147, 159, 173, 178, 261, 262 Communications and operations, 48, 49 Commutated transducers, 83 Concurrent Self-Organizing Maps, 283, 284, 291, 292, 298 Consequence management, 5, 19, 21, 111, 142, 145, 343, 347, 350 Consistency, 111, 114, 167 Constant frequency pulses, 275 Container threat, 61, 62 monitoring, 16 Security Initiative, 16, 20, 62, 64 shipping, 87, 92 Context, 44, 108, 139, 167, 178 Co-registration, 211 Crisis Management, 30, 31, 32, 118 Critical Infrastructure, 19, 20, 45 Cyberterrorism, 191 D Dangerous goods, 17, 20, 61, 181 Data alignment, 235 association, 92, 93, 105, 149, 231, 235, 252, 317, 323 clearinghouse, 66 fusion, 28, 30, 32, 87, 88, 92, 93, 96, 107, 108, 118, 121,

125, 126, 159, 168, 175, 179, 181, 190, 199, 208, 233, 239, 240, 241, 259, 337, 338, 343, 350 mining, 50, 91, 101, 102, 192, 205, 346 security, 53, 55, 155 sources, 53, 54, 59, 61, 65, 66 Database, 11, 24, 26, 28, 55, 56, 66, 92, 123, 156, 175, 176, 177, 183, 186, 187, 203, 210, 285, 291, 293, 295, 302, 310, 313, 315, 316 Decision, 16, 21, 31, 32, 43, 44, 45, 50, 66, 88, 90, 92, 93, 100, 109, 110, 112, 113, 114, 116, 117, 118, 119, 121, 122, 128, 130, 135, 139, 140, 146, 147, 148, 149, 152, 153, 156, 157, 158, 161, 162, 164, 166, 168, 193, 195, 202, 224, 233, 236, 237, 247, 288, 296, 299, 301, 302, 303, 304, 305, 307, 312, 316, 337, 338, 347, 348, 349, 350 making, 43, 44, 118, 122, 130, 147, 158, 166, 224, 299, 301, 338, 348 support, 31, 32, 92, 119, 139, 140, 148, 152, 153, 154, 156, 158, 233, 338 templates, 302, 303, 304, 305 Dempster-Shafer, 175, 211, 244, 246, 308, 311, 316 Detection, 43, 44, 45, 74, 91, 118, 177, 211, 212, 213, 218, 225, 230, 231, 243, 244, 250, 251 ship, 244, 247, 249 Deterrence, 43, 44, 45, 47, 49, 50 Disaster, 3, 18, 118, 150, 191, 339, 349 ecological, 191

Display, 15, 59, 164, 166, 218, 252, 256, 261, 262 Distributed data fusion, 181, 182, 183, 185, 186, 189 operations, 181 sensor arrays, 26 Doctrinal templating, 131 Domain awareness, 33, 34, 35, 43, 50, 92, 109, 119, 181

SUBJECT INDEX

355

knowledge, 88, 110, 111, 116 representation, 111, 117, 134, 135, 136, 347, 349, 350 Doppler-sensitive waveforms, 275 Double counting, 185, 186, 189 Dual-band MWIR, 213

Global

E Eigenray propagation model, 255 Electronic intelligence, 110, 260 ELINT, 260, 264, 346 Emergency intervention, 152 management, 45, 50, 139, 140, 142, 143, 146, 153, 158, 207 operations centers, 143 planning, 1, 2, 3, 4, 17, 18, 20 response, 30, 43, 140, 142, 150, 152, 153 situation, 139, 140, 152 enforcement agency, 33 Euro-Atlantic Disaster Response Coordination Centre, 3 Event detection, 223 prediction, 97, 101 Evidential fusion, 243, 244, 247, 248, 250 Expert knowledge, 115, 131, 135, 350 system, 123, 191, 193

H Harbour protection, 37, 69, 71, 87, 89, 90, 91, 92, 110, 111, 112, 113, 115, 116, 117, 119, 120, 136, 140, 159, 171, 172, 178, 181, 190, 191, 192, 196, 197, 223, 233, 234, 240, 251, 283, 291, 297, 317, 347, 349, 350 Heartbeat Detector, 15 Higher-level fusion, 97, 109, 110, 129, 130, 131, 132 Homeland security, 19, 72, 73, 74, 76, 209, 260, 337 Horseshoe concept, 156 Hough transform, 248 Human computer interface, 161, 168, 173 intelligence, 110 operator, 59, 130, 132, 134, 136, 161, 163, 223, 229, 234 resources, 27, 49, 202 Hyperspectral imagery, 244 imaging, 217 Hypothesis, 99, 100, 101, 102, 105, 115, 186, 187, 213, 246, 307, 321, 349 evaluation, 101, 115 generation, 99, 101, 102, 115 selection, 101

F Face Recognition, 289, 291, 292, 293, 294, 295, 296, 297 False alarm, 75, 91, 116, 217, 233, 244, 245, 247, 250, 252, 254 mitigation, 244, 247, 252 FLIR, 120, 183, 289 Focus of attention, 225 Formal ontology, 113 Frequency domain analysis, 270 modulated, 276, 282 Functional model, 95 Fuzzy logic, 133, 208, 307, 316 G Gating, 252, 254, 319 Geo-information system, 191, 192

Command and Control System, 261 Positioning System (GPS), 317, 318 Graph mapping, 253

I Identification, 10, 16, 20, 28, 54, 58, 59, 61, 63, 64, 72, 74, 76, 77, 96, 105, 111, 117, 119, 120, 125, 130, 149, 203, 205, 234, 292, 307, 316, 337, 339, 344, 349, 350 and warning, 97 Image sequence, 224 IMO, 11, 24, 25, 39, 41, 55, 63, 265

356

SUBJECT INDEX

Impact, 45, 47, 51, 75, 76, 100, 109, 110, 131, 150, 152, 161, 181, 199, 201, 202, 203, 205, 206, 207, 249, 343, 344 analysis, 203 assessment, 96, 100, 150 Imperfect information, 127, 131, 132, 133 Imprecision, 113, 164, 246 Incident, 47, 48, 49, 60, 142, 143, 144, 145, 146, 153, 156 Incomplete data, 187 Inertial navigation system, 323 Information acquisition, 53, 59, 60, 164 extraction, 133 management, 133, 139, 140, 146, 152, 154, 158, 211, 346 mining, 110 overload, 146 quality, 116 requirements, 111, 113, 114, 117, 337, 344, 350 sharing, 65, 110, 117 sources, 119, 148, 158, 164, 174, 301, 346, 347 superiority, 152 technology, 146, 152, 209 Infrared camera, 59, 223, 224 imagers, 110, 120 imaging, 209, 219 long-wave, 211 sensor, 299, 308 Inland Surface Transport, 1, 2, 4, 17, 18, 19, 20 Integrated security, 140 Intelligence, 16, 33, 35, 59, 63, 66, 70, 97, 100, 109, 110, 127, 128, 129, 130, 131, 132, 133, 135, 136, 143, 172, 182, 184, 188, 190, 191, 192, 211, 215, 260, 344, 346, 347 cycle, 128, 129 data, 136, 188 process, 128 Intelligence, surveillance and reconnaissance, 70, 215 Intent, 47, 61, 92, 93, 95, 97, 99, 100, 102, 103, 109, 113, 120, 124, 150, 162, 171, 173, 174, 177,

178, 182, 187, 339, 342, 343, 344, 347, 349, 351 Interactive fusion, 127 Inter-agency cooperation, 351 Interdiction, 35, 43, 44, 45 Intermodal Hubs, 19 International Chamber of Commerce, 71 Ship and Port Facility Security (ISPS) code, 4, 7, 8, 9, 10, 11, 12, 13, 14, 20, 23, 24, 25, 26, 29, 31, 33, 34, 37, 38, 39, 40, 41, 58, 62, 63, 64, 68, 71, 76, 120, 121, 122, 126, 208, 339, 342, 343

Interoperability, 3, 45, 49, 68, 117, 152, 153, 158, 159, 189, 202, 207, 346 Intrusion detection, 66, 213 ISAR, 120 ISR, 70, 215, 221 IST, 4, 5, 134, 137, 200, 346 J JDL Model, 95, 96, 108, 118, 125, 149, 159, 179, 210, 235, 241 JMIE, 261, 264, 265 Joint Maritime Information Exchange, 261 probabilistic data association, 226 probability density function, 273 Jurisdictions, 50, 144, 145, 153, 156, 346, 348 K Klman filtering, 329, 330 extended, 318, 326

Knowledge discovery, 151, 192, 346 exploitation, 139, 140, 148, 158 processing, 192 representation, 95, 131, 132, 191, 192 L LADAR, 183 Lagrangian optimization, 274

SUBJECT INDEX Learning, 33, 101, 107, 118, 119, 122, 124, 125, 132, 136, 150, 172, 192, 213, 214, 229, 230, 284, 291, 299, 301, 305 Level 1 fusion, 116, 175, 182, 338 2 fusion, 121, 182 3 fusion, 150, 182 Linear acoustic arrays, 234 frequency modulated, 275 analysis, 134 information, 110 Logistics, 31, 66, 198, 199, 200, 201, 202, 204, 206, 207, 208, 338 M Machine Learning, 96 Man-made objects, 243, 244 Marine Security, 40, 44 Maritime Common Operational Picture, 182 domain awareness, 44, 61, 87, 92, 181, 182 information, 35, 261, 264 picture, 53, 55, 58, 59, 87 security, 4, 7, 13, 24, 25, 28, 34, 37, 54, 88, 89, 90, 91, 93, 209, 221 Markov Chain Monte Carlo, 322 Mass function, 246 Matched filter, 267, 269, 275, 276, 277, 280 matched field processing, 260, 261 Matching procedures, 133 Maximum likelihood, 254, 270 MCOP, 182, 184, 188, 189 MDA, 92, 182, 187, 188, 190, 254 Mediterranean Memorandum of Understanding, 38 Modeling and simulation, 50, 191, 199, 201, 202, 204, 347 Motion analysis, 203 track indicator, 264 Multiagency operations, 140, 152, 153

357 agent system, 93, 192 beam sonar, 79, 85 dimensional Assignment, 254 hypothesis target tracking, 213 modal seaport, 43, 44, 45 national operation, 139, 140 sensor, 105, 125, 213, 215, 220, 231, 233, 264 image fusion, 213 spectral imaging, 211, 217

Multiple camera, 224, 235, 237 classifier, 299, 302 Hypothesis Tracking, 226, 254 N National Security, 9, 18, 25, 43, 139, 140, 147, 148, 158, 159 Nearest Neighbour Association, 254 Neural model, 213, 283 networks, 283, 298, 299, 301, 304 NOGA, 272, 275 Nonlinear systems, 272, 281 Non-Vessel Operating Common Carriers (NVOCC), 63 O Object Similarity Coefficient, 237 Tracking, 230 Ontological constraint, 97, 102 status, 114 Ontology, 90, 95, 96, 99, 100, 103, 104, 109, 113, 114, 117, 132, 172, 191, 192, 193, 194, 196 OODA Loop, 128 Operational plans, 33, 35 Opportunity, 32, 47, 99, 100, 103, 113, 164, 171, 173, 174, 178, 182, 187, 202, 337, 338, 342, 343, 350, 351 Optical processing, 278 Organizational structure, 113, 114 P Particle filter, 317, 320, 321, 325, 326 Pattern, 81, 83, 100, 101, 105, 130, 131, 135, 136, 184, 187, 198, 223,

358

SUBJECT INDEX

230, 264, 265, 283, 284, 289, 291, 292, 295, 298, 299, 301 analysis, 136, 230, 265 matching, 133 recognition, 184, 283, 298, 299, 301 PBIST, 1, 2, 4, 5, 17, 18, 19, 20 PBOS, 1, 2, 4, 19 Perceptron, 225, 302, 305 Performance, 45, 64, 82, 84, 157, 162, 163, 164, 165, 166, 169, 200, 201, 202, 205, 211, 235, 249, 252, 257, 261, 268, 276, 277, 278, 279, 280, 281, 283, 284, 292, 319 Plausible, 114, 115, 307, 308, 316, 331, 335 Polarimetric decomposition, 244, 246 feature, 243, 244, 245, 249, 250 SAR, 243, 244, 249 Polarimetry, 250 Polarization, 243, 250 Police, 14, 34, 35, 36, 88, 145, 202 Policy, 43, 48, 89, 118, 159, 200, 204 Port facility, 7, 13, 14, 15, 16, 24, 26, 27, 28, 38, 39, 40, 317, 318, 325 facility security plans, 14, 39 logistics, 199, 201 security, 4, 7, 12, 13, 25, 38, 39, 40, 41, 43, 44, 45, 46, 47, 48, 51, 200, 215, 342 traffic, 7 Predicate calculus, 103, 104, 105 Predict, 95, 110, 111, 122, 124, 191, 210, 224, 338 Preparedness, 2, 43, 47, 142, 145 Probabilistic Argumentation System, 115 data association, 226, 254 Probability, 106, 107, 115, 131, 133, 157, 176, 195, 216, 217, 229, 230, 237, 245, 246, 273, 301, 303, 304, 307, 308, 312, 314, 315, 332 density function, 273

Q Quality, 44, 45, 53, 54, 59, 74, 91, 93, 111, 114, 116, 117, 134, 149, 157, 193, 203, 235, 316, 323, 347, 348, 349 Queuing models, 205 Quiet Interlude Processing System, 259 QuIPS, 259, 260, 261, 262, 263, 264, 265 R Radar, 29, 34, 183, 250, 257, 264, 316, 346 Radio direction finder, 59 Reasoning, 95, 104, 109, 111, 112, 113, 114, 115, 117, 118, 119, 122, 123, 125, 127, 131, 132, 133, 136, 150, 234, 246, 307, 308, 316, 349, 350 abductive, 115 default, 127, 132, 136 dynamic, 111, 114 evidential, 125, 133, 246 heuristic, 131, 132 perceptual , 119, 120, 122, 125 Recognition, 100, 117, 150, 184, 215, 225, 230, 244, 245, 249, 250, 283, 284, 285, 286, 288, 289, 291, 292, 293, 294, 295, 296, 297, 298, 299, 301, 307, 350 speaker, 284, 291, 292, 295, 296, 297 video, 215 Recovery, 43, 47, 143, 144, 199, 200, 205 Redundant information exploitation, 238 Regional Maritime Safety, 27, 28 Relation, 103, 104, 107, 112, 113, 130, 172, 203, 205, 238, 310, 311, 312, 313, 331 Relationship, 46, 98, 105, 135, 141, 274 Reliability, 45, 53, 54, 56, 59, 91, 110, 113, 114, 115, 116, 129, 149, 161, 164, 165, 167, 170, 183, 235, 307 Remote sensing, 244, 249, 250, 289, 338 Resource Management, 107, 171 Reverberation distribution, 276 ROI, 246, 247 Rule-based system, 191, 192

SUBJECT INDEX

S Safety Of Life at Sea, 13, 37 SAR (Synthetic Aperture Radar), 120, 220, 243, 244, 245, 247, 248, 249, 250, 265, 288 Satellite image, 283, 284, 285, 286 Scenario, 48, 97, 110, 112, 113, 115, 117, 124, 152, 153, 175, 186, 191, 193, 194, 195, 196, 197, 204, 205, 312, 337, 339, 344, 345, 350 ontology, 191, 194 Scene understanding, 224, 234 Screening, 31, 63, 76, 203, 204, 205, 206 Security assessment, 10, 11, 24, 37, 38, 39, 40, 41 control, 49, 117 incident management, 48, 49 measures, 7, 9, 10, 11, 20, 24, 25, 27, 40, 50, 64, 67, 68, 71 perimeter, 10, 73, 318 physical and environmental, 48, 49 plan, 7, 11, 38, 199, 202 public, 44, 45, 139 Semantic, 95, 100, 105, 127, 131, 132, 133, 134, 135, 136, 154, 203, 229, 312 information structuring, 127, 136 matching, 134 processing, 133, 134 understanding, 229 Sensor acoustic , 234 fusion, 108, 125, 190, 209, 220, 238, 297, 299, 350 electro-optics, 209 heterogeneous, 233, 234, 238 millimeter microwave, 15 networks, 268 non-colinearity, 271 seismic, 15, 120 sonar, 59, 69, 70, 79, 80, 81, 82, 83, 84, 85, 86, 120, 186, 187, 256, 260, 261, 267, 275, 276, 278, 282 underwater, 110, 120

359

Shipping, 1, 2, 4, 19, 24, 25, 39, 40, 61, 62, 65, 67, 87, 92, 109, 110, 181, 265 Simulation, 50, 151, 152, 159, 169, 177, 188, 191, 192, 193, 195, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 275, 280, 285, 294, 312, 326, 347 Situation, 28, 29, 30, 32, 54, 95, 96, 97, 98, 99, 100, 101, 103, 104, 105, 106, 107, 109, 110, 111, 113, 114, 115, 116, 118, 119, 120, 121, 122, 124, 125, 127, 130, 131, 132, 133, 134, 135, 136, 139, 140, 143, 146, 147, 148, 150, 151, 152, 157, 158, 159, 161, 162, 165, 168, 171, 172, 173, 177, 178, 182, 187, 202, 203, 204, 210, 215, 224, 230, 233, 234, 235, 247, 299, 300, 301, 317, 318, 337, 338, 339, 347, 348, 349, 350, 351 assessment, 95, 98, 109, 119, 120, 121, 150, 171, 182, 224, 230, 233, 234, 235, 337, 339, 347, 351 awareness, 54, 109, 110, 119, 127, 135, 136, 147, 148, 152, 166, 338, 348, 349 picture, 54, 113

Situational awareness, 59, 125, 146, 148, 158, 159, 168, 215 Situation and Threat Assessment (STA), 111, 112, 113, 114, 115, 116, 120, 117, 171, 172, 173, 174, 177, 178 SOLAS, 4, 7, 12, 13, 23, 24, 26, 29, 37, 38, 39, 68

Spatial-spectral matched filtering, 211 Spatio-temporal constraint, 97, 102 Spectral analysis, 243 Standard communication protocols, 156 State Estimation, 235, 238, 240 Strategic Warning, 111, 343 Stress, 166 Subaperture coherence, 243, 247, 250 Supply chain, 61, 62, 64, 65, 66, 67, 68, 76, 77, 201, 339, 343, 345 security, 339 stratified examination, 63

SUBJECT INDEX

360

Surveillance, 14, 27, 29, 43, 44, 45, 48, 50, 53, 54, 55, 58, 59, 60, 70, 85, 90, 93, 109, 110, 120, 124, 143, 177, 181, 187, 202, 203, 211, 212, 213, 215, 221, 223, 224, 230, 233, 234, 235, 239, 240, 244, 249, 251, 252, 253, 256, 257, 267, 269, 292 covert, 213 littoral, 90, 213, 267, 269 video, 14, 223, 224, 229, 230, 234, 237, 346 wide-area, 243 Swimmer detection, 69 T Target recognition, 79, 84, 96, 97, 98, 249, 307 in-field learning & search, 213 underwater, 121 Terminal structure, 7 Terrorism, 2, 4, 13, 26, 31, 63, 127, 131, 139, 142, 146 Terrorist, 2, 3, 4, 18, 25, 26, 27, 29, 30, 33, 36, 37, 41, 44, 45, 48, 61, 62, 63, 64, 71, 87, 97, 109, 111, 117, 132, 139, 140, 141, 142, 144, 145, 147, 152, 158, 181, 191, 200, 223, 337, 344, 349, 350 activity, 4, 18, 48 incident, 36, 62, 109, 142, 144, 337 threat, 27, 48, 140 Text extraction, 132, 134 unstructured, 133, 136

Threat assessment, 59, 95, 97, 99, 100, 101, 110, 111, 116, 124, 171, 172, 173, 178, 182, 344, 345 imminent, 111, 344 intentional, 100, 102 ontology, 90, 99, 101, 343 unconventional, 111, 113, 172, 338, 342 underwater, 267, 269 unintentional, 99 Tracking, 30, 31, 53, 54, 59, 61, 64, 66, 67, 68, 71, 85, 91, 93, 96, 97,

105, 119, 120, 125, 130, 149, 188, 211, 212, 213, 215, 225, 226, 230, 231, 233, 234, 240, 241, 251, 253, 254, 256, 264, 265, 267, 268, 269, 275, 281, 319, 326, 329, 333, 334, 335, 336, 343, 350 Trajectory analysis, 223 clustering, 226, 229 Trust affective, 163 analogical, 163 analytic, 162, 163 in automation, 161, 162, 165, 168 U UAV, 177, 183, 185 Uncertainty, 71, 72, 95, 103, 104, 110, 113, 115, 116, 117, 119, 272, 273, 281, 299, 302, 323, 331 Underwater unmanned vehicle, 70, 120 video cameras, 234 United States Coast Guard (USCG), 215 Unscented Filter, 322 V Vagueness, 116 Vector quantization, 227, 305

Vessel Traffic, 11, 28, 53, 54, 55, 92, 93 Vision colour, 213 computer, 184, 203, 205, 292 night, 120 Vulnerability, 4, 18, 40, 44, 45, 47, 59, 61, 62, 68, 100, 109, 118, 342, 343, 344, 351 W WAAS, 212 Weapons of mass destruction (WMD), 4, 16, 20 WLAN, 238, 239