Contributions to Management Science
For further volumes: http://www.springer.com/series/1505
.
Tatjana Samsonowa
...
277 downloads
2066 Views
13MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
Contributions to Management Science
For further volumes: http://www.springer.com/series/1505
.
Tatjana Samsonowa
Industrial Research Performance Management Key Performance Indicators in the ICT Industry
Tatjana Samsonowa SAP AG SAP Research Dietmar-Hopp-Allee 16 69190 Walldorf Germany
ISSN 1431-1941 ISBN 978-3-7908-2761-3 e-ISBN 978-3-7908-2762-0 DOI 10.1007/978-3-7908-2762-0 Springer Heidelberg Dordrecht London New York Library of Congress Control Number: 2011940419 # Springer-Verlag Berlin Heidelberg 2012 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Printed on acid-free paper Physica-Verlag is a brand of Springer-Verlag Berlin Heidelberg Springer-Verlag is a part of Springer ScienceþBusiness Media (www.springer.com )
Managing the Performance of Industrial Research in the ICT Industry Assessing Research Goals with suitable Key Performance Indicators Dem Fachbereich Rechts- und Wirtschaftswissenschaften an der Technischen Universita¨t Darmstadt zur Erlangung des Grades eines Doctor rerum politicarum (Dr. rer. pol.) eingereichte
Dissertation vorgelegt von Dipl.-Betriebswirtin (FH) Tatjana Samsonowa aus Arkalyk Erstgutachter: Prof. Dr. Peter Buxmann Zweitgutachter: Prof. Dr. Volker Caspari Tag der Einreichung: Darmstadt, 2. November 2010 Tag der mu¨ndlichen Pru¨fung: 14. Januar 2011
.
Acknowledgement
This thesis would have never been written without the support of many, many people. I hope to be forgiven that I do not mention all the names here: I am truly grateful to the members of the supervisory team of the Information Systems department of the Darmstadt University of Technology. Professor Dr. Peter Buxmann and Professor Dr. Volker Caspari: Your commitment was outstanding! I feel privileged that I was able to work with you and to learn so much from your expertise and guidance! As my thesis was created within SAP Research’s PhD program, I owe huge gratitude to many SAP colleagues, of which I want to mention at least two: Dr. Joachim Schaper and Dr. Wolfgang Gerteis. Joachim, you gave me the courage to start such this endeavor in the first place and you restored my confidence more than once in the process. Besides your constructive feedback and your inspiration I knew that I always could count on you. You have been a true mentor during the entire time! Wolfgang, you supported me in critical phases of my research. You gave me both, merciless feedback as well as constructive ideas. With your patience and dedication you helped me many times to get unstuck and to find my path again. I was very fortunate to have such a tough sparring partner and at the same time a trusted advisor like you! A big THANK YOU! to all others who helped me to complete my research and to make it a success: the students who supported me especially in collecting the data for my case studies: Your energy, your good mood, and your hard and dedicated work was invaluable! The interviewees from the case study companies: Your willingness to share relevant information was crucial to support the major findings of my thesis. And all the participants of my survey: A cordial Thank You for spending your precious time on questionnaires and follow-up discussions and providing me with such a valuable data collection! Last not least I want to express my deepest gratitude to the most important people in my life: my parents Emma and Erhard and my sister Lydia: You have always been here for me with your whole heart during the entire time. I dedicate this work to you! I love you! You are the light of my life! Truly, Tatjana. vii
.
Contents
1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Background and Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Research Questions and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Dissertation Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1 2 4 5
2
Performance Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Introduction and Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Invention and Innovation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2 Innovation Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.3 Research and Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Performance Management – Basic Terms and Definitions . . . . . . . . . . . 2.2.1 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Goals and Goal Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 Measures, Metrics, and Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.4 Performance Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.5 Performance Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Purposes of Performance Management/Measurement . . . . . . . . . 2.3.2 The Information and Communication Technologies (ICT) Sector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.3 The Software Industry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9 9 10 14 15 22 22 26 27 32 37 41 42
State-of-the-Art in Performance Management . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Performance Management Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Generic Approaches: Schools of Thought . . . . . . . . . . . . . . . . . . . . . . 3.1.2 R&D Performance Measurement Systems . . . . . . . . . . . . . . . . . . . . . 3.1.3 Research (Only) Performance Measurement Systems . . . . . . . . . 3.1.4 Requirements of a Performance Management System . . . . . . . . . 3.1.5 Characteristics of Research Compared to Development . . . . . . .
53 55 55 58 63 67 74
3
48 50
ix
x
Contents
3.2 Organizational Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Generic Organizational Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 R&D Organizational Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Research (Only) Organizational Goals . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Key Performance Indicators (KPIs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Generic Key Performance Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 R&D Key Performance Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.3 Research (Only) Key Performance Indicators . . . . . . . . . . . . . . . . . 3.4 Conclusions and Research Gaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
83 83 91 93 98 98 101 105 108
4
Performance Management: Analysis Approach . . . . . . . . . . . . . . . . . . . . . . . 4.1 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Design and Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.3 Example Case Study “SAP Research” . . . . . . . . . . . . . . . . . . . . . . . . 4.1.4 Initial Findings and Further Approach . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Performance Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Approach and Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Cluster: Technology Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.3 Cluster: Future Business Opportunities . . . . . . . . . . . . . . . . . . . . . . . 4.2.4 Cluster: Research Portfolio Management . . . . . . . . . . . . . . . . . . . . . 4.2.5 Cluster: Intellectual Property Creation . . . . . . . . . . . . . . . . . . . . . . . . 4.2.6 Cluster: Operational Excellence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.7 Cluster: Talent Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.8 Cluster: Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.9 Cluster: Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.10 Cluster: Presence in Scientific Community . . . . . . . . . . . . . . . . . . 4.2.11 Cluster: Collaboration with Academia . . . . . . . . . . . . . . . . . . . . . . . 4.2.12 Cluster: Collaboration with Partners and Customers . . . . . . . . 4.2.13 Performance Cluster Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Performance Cluster Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Goal Decomposition Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
113 114 114 116 118 127 129 129 132 135 137 140 142 143 145 146 148 149 151 152 157 158 171
5
Survey Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Development of the Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Sample Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Analysis/Description of the Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 General Data: Organizational Context and Research Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.2 Performance Measurement Data: Organizational Goals and KPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
173 173 173 175 177 177 197 221
Contents
6
7
Towards a Systematic Performance Management Approach for Industrial Research in the ICT Industry . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Research Performance Management System . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Suggested Model for a Research PMgS – Levels and Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Recommendations: Process Options to Design a PMgS . . . . . . 6.3 Review of Requirements Against the Created PMgS . . . . . . . . . . . . . . . . 6.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Summary of Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Explicit Answers to the Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.1 Research Question 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.2 Research Question 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.3 Research Question 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.4 Research Question 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Future Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
xi
223 223 224 225 230 235 241 243 243 248 248 250 251 252 252 254 256
Appendix A Content and Media Sector as Part of the ICT . . . . . . . . . . . . . . . 257 Appendix B Key Performance Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 Appendix C List of interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 Appendix D Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.1 ABB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.2 EMC2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.3 IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.4 Intel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.5 Microsoft . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.6 Philips Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.7 Deutsche Telekom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
303 303 311 321 330 340 348 360
Appendix E . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 E.1 Online Questionnaire (English) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 E.2 Online Questionnaire (Russian) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
xii
Contents
List of Abbreviations and Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439 List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443 List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451 Eidesstattliche Erkla¨rung . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
Chapter 1
Introduction
Abstract This chapter serves as an introduction to the dissertation. It outlines the relevant disciplines that embed the research objectives of this work. The central research question is presented to frame the scope of the work. Furthermore, research questions are formulated and finally the research structure is presented. Keywords Background and problem definition in performance management in industrial research environments • Introduction to performance management of industrial research
Industrial research is often seen as an incubator for new technological knowledge. Companies engage in research in order to gain a comparative competitive advantage over their competitors. Sustainable long-term market success cannot be achieved without the right investment in innovation. Technical progress (inventions) and subsequent innovations are the focal point for growth, both in employment and prosperity.1 It is recognized that new technological knowledge can be gained through the planned, systematic use of scarce resources. Only a small portion of technological progress is based on chance discoveries, moreover it is “ordered, financed, produced, and paid for.”2 This means that it is an economic question of how to most efficiently organize and manage the unit that is in charge of the generation of inventions within a company. Furthermore, economic managerial analysis is required on the performance and efficiency of the production of such knowledge.3 This work addresses the performance management of industrial research departments within the ICT industry.4
1
Solow (1956), pp. 65–94. Machlup (1959), p. 117. 3 Sch€atzle (1965). 4 ICT – The Information and Communication Technologies. 2
T. Samsonowa, Industrial Research Performance Management, Contributions to Management Science, DOI 10.1007/978-3-7908-2762-0_1, # Springer-Verlag Berlin Heidelberg 2012
1
2
1 Introduction
An organization’s ability to manage its performance, that is, to define, measure, analyze and improve its performance is of outstanding importance for its future development. The measurement of operational issues is an essential foundation for the assessment of past events that influence the starting point for subsequent management decisions. Changes in both the external environment and the inner world of businesses, such as increased global competition and emphasis on new management concepts, such as “open innovation”, have changed the need for information provision and processing to support management. In order to successfully face progressive globalization and increased intense competitive rivalry, an organization’s performance needs to be permanently monitored in all relevant areas. A number of proposals are presented in the literature on so-called Performance Measurement Systems (PMS). A PMS holistically monitors and communicates the objectives and the degree of achievement of an organization.
1.1
Background and Problem Definition
Schumpeter hypothesized innovation, company size and market concentration to be positively related. He also suggested innovation and economic growth are linked. However to date neither hypothesis has been confirmed.5 The pace and extent of technological progress depends not only on research or R&D (rate of invention), but also on the initial application of new knowledge (rate of innovation) and on the general spread of new methods (rate of imitation). It is the company’s decision translated into its strategy based on its capabilities on how to act in the business environment and what to stress in its business model: invention, innovation or imitation. A typical problem for growing industrial organizations in the ICT sector is to introduce strategic technology research to ensure the long-term success of the company. During the creation of a corporate research function, the question arises as to how to assess this function in both the short and long-term. As the literature study will show, there is no dedicated work available that investigates, in detail, the particular situation that research organizations face. Moreover, studies typically look at the R&D organization by amalgamating research and development into one function. This aggregation represents a real problem for the management and measurement of the performance of the functions, especially in large companies where these two functions operate separately and the activities in each function are fundamentally different.6 Brown and Svenson analyzed how to measure R&D productivity and concluded: “Research and
5
Caspari (2001). Seiler (1965), Leifer and Triskari (1987), pp. 71–78, Brown and Svenson (1998), pp. 30–35, Karlsson et al. (2004), p. 180. 6
1.1 Background and Problem Definition
3
development clearly perform different functions and produce quite different outputs”.7 Assessing performance in industrial R&D is difficult, but it is even more difficult for the research function alone. As Loch and Tapper mention, measurement problems in a research group (as opposed to development) are the more severe.8 One difficulty of assessing performance is that the performance management systems often apply inappropriate performance measures, “That what we measure has nothing to do with what we are doing.”9 This is also the reason why, in practice, many scientists and engineers in research organizations do not accept, or are negative about, measuring performance. It seems to be obvious that the performance management systems need to be related to the activities that are actually conducted in the research organization. Another difficulty with the assessment of performance in research is that it is mostly knowledge work that needs to be assessed and it is often not possible to quantify the knowledge. And, if it is possible to express the output/outcome of the knowledge work in measuring units, it is extremely difficult to assess their value. For example, one such output could be an invention, which a researcher produces. However, the outcome of that output might only appear years later in the form of a patent. The process of converting the invention (output) into a patent (outcome) is uncertain and long. Typically, the details of such an invention would be passed internally to the Intellectual Property department who would draft a patent application. This application would then be submitted to a patent office, for example the European Patent Office. After approximately 2–3 years a patent application will eventually be granted or rejected. Looking at that from the performance measurement perspective the production of the invention disclosure itself might only take some weeks or months and so fits in the annual performance measurement cycle of the organization. Ironically, the number of invention disclosures is a very popular key performance indicator (KPI) for industrial research organizations yet this KPI gives little clue as to the number of patents that will finally be granted to the organization. Of course the organization could assume a certain acceptance rate for patent applications, but this means that the KPI (number of invention disclosures produced) is only as good as the accuracy of the acceptance rate used. Of course this rate is likely to vary over time, and not all patents are as valuable as others. Summarizing, we have exemplified some of the problems in performance measurement in industrial research environments: (1) since performance measurement systems are designed for annual evaluation cycles, they normally capture outputs and not outcomes; (2) output KPIs are not necessarily good indicators for
7
Brown and Svenson (1998), p. 34. Loch and Tapper (2002), p. 185. 9 Brent and Pretorius (2009), and statements found in a survey by Kerssens-van Drongelen (2001). 8
4
1 Introduction
the time-lagged outcomes; (3) in order to predict or estimate expected outcomes, output KPIs are needed together with additional assumptions. Building on these issues, a major mismatch in the practice of industrial research departments is concealed by the fact that research goals are often targeting for longterm outcomes. However, they are set annually and the measurement process to assess goal achievement is usually annual. Therefore, the KPIs that are used are typically designed in a way to measure the outputs rather than outcomes. This is the most fundamental problem for performance measurement in industrial research organizations, and greatest area of contention in this work’s main research question: How to assess the performance of an industrial research organization? This work is a contribution to the “Performance Management of Industrial Research Organizations” and to systematically analyzing the various capabilities of research organizations, their goal-setting as well as their setting of performance measures.
1.2
Research Questions and Objectives
By subdividing the central research question introduced above, it is the goal of the thesis to understand in more detail: RQ1: What is the common practice in regard to performance management of industrial applied research organizations? Are there common or similar organizational goals, and are there common or similar key performance indicators? RQ2: Is there a generic performance management approach (PM approach) to describe the common practice of performance management of industrial applied research organizations? How are goals and KPIs reflected in this approach? RQ3: How are the peculiarities of industrial applied research [specifically, and the requirements of performance management systems in general] considered in the developed PM approach? RQ4: Can there any recommendations be derived from the suggested approach to advise on the implementation and further development of a performance management system in an industrial applied research organization? Based on the research questions the following research objectives can be derived: • List the goals which companies aim at within industrial applied research departments; • Create a catalogue of the KPIs which are applied to monitor the performance of the respective goal; • Examine if there are industry best practices with regards to monitoring the performance of a respective goal as well as if there are dependencies between the KPIs identified before;
1.3 Dissertation Structure
5
• Design and build a performance management system that facilitates performance management for industrial research departments; • Develop recommendations for practitioners on how to set up performance management in industrial research.
1.3
Dissertation Structure
The dissertation is divided into seven chapters. Each chapter is now introduced. The overall design of the dissertation and the structure is summarized in Fig. 1.1. Chapter 1 serves as an introduction to the dissertation. It outlines the relevant disciplines that embed the research objectives of this work. The central research question is presented to frame the scope of the work. Furthermore, research questions are formulated and finally the research structure is presented. Chapter 2 introduces basic terms and definitions. First, the nature and characteristics of invention and innovation are examined. The review shows that the commercial success of an innovation does not only depend on the quality of inventions generated in research and/or development and that both invention and innovation represent components of a process of managing innovation that extends over the entire value chain of a company. Furthermore, the following concepts are distinguished: Innovation Management, Technology Management and R&D Management in order to also understand in detail differences between the functions: research and development. The following questions are answered: What is meant by research and by development; what are the roles and tasks of the units; what kind of technological knowledge do they produce; what is the purpose of Performance Management; what are its components; and, what are industrial research departments? Chapter 3 investigates the state-of-the-art in the performance management literature. In this context, the performance management systems, organizational goals and key performance indicators are examined at three levels of granularity: generic (literature reporting about generic approaches covering either one or more functions of a company), R&D (focusing on the specifics of the R&D function), and research function only. The insights in this chapter enable us to formulate the research gaps and refine the primary question with four research sub-questions. Chapter 4 provides an overview on the methodology and introduces the step-bystep development of a performance management system via a series of in-depth case studies. The analysis approach aims at developing a model for performance management systems built on the basic components found in practice such as organizational department goals and individual KPIs. The model is extended with additional components that allow us to compare the various performance management systems and to identify relationships between these components. Performance clusters (i.e. groups of KPIs), which represent one significant component of our developed model, are introduced. Furthermore, the goals are analyzed for the similarity of their sub-components. Altogether, Chap. 4 describes a preliminary
6
1 Introduction
1 Introduction 1.1 1.2 1.3
Background and problem definition Research questions and objectives Dissertation structure
3 State-of-the-art in Performance Management
2 Performance Management 2.1 2.2 2.3
Introduction and fundamentals Performance management – basic terms and definitions Summary
3.1 3.2 3.3 3.4
Performance management systems Organizational goals Key performance indicators (KPIs) Conclusions and research gaps
4 Performance Management - Analysis Approach 4.1 4.2 4.3 4.4 4.5
Case studies Performance clusters Performance cluster analysis Goal decomposition analysis Summary
5.1 5.2 5.3 5.4 5.5
Introduction Development of online survey Sample design Analysis / description of the results Conclusion
5 Survey Findings
6 Towards a Systematic Performance Management System
for Industrial Research in the ICT Industry 6.1 6.2 6.3 6.4
Introduction Research performance management system Review of requirements against the created PMgS Conclusions
7.1 7.2 7.3 7.4 7.5
Summary of the dissertation Explicit answers to the research questions Limitations Future research Concluding remarks
7 Conclusions
Appendix A B C D E
Definition of media ICT sector Examples of KPIs from the literature List of interviews Case studies Questionnaire
Fig. 1.1 Outline of the dissertation
1.3 Dissertation Structure
7
approach on how to deal with the individual elements of performance management, that is, with KPIs, KPI classes, performance clusters and finally organizational department goals. Chapter 5 presents the findings from the large quantitative survey, which tests the findings from the literature review and case studies. Both the qualitative case studies, as well as the results of a survey and its comparison, are the core of this chapter. The result of the evaluation reveals that industrial research organizations seem to have a set of shared organizational goals and a set of shared KPIs, which can be summarized in KPI classes. In a sum, the survey results endorse the relationships between KPIs and KPI classes, performance clusters and organizational research goals. This chapter is therefore dedicated to the examination of the qualitative results from the quantitative perspective. Chapter 6 synthesizes all our findings into a holistic model and suggests a new systematic way to assess the organizational performance of an industrial research department. This chapter responds to the overall research question and is dedicated to our performance management model. Its five levels, and their relationships, are introduced step-by-step. Furthermore recommendations on how to design a Performance Management System (PMgS) are provided, which take into consideration the four elements of performance management: planning, measurement, analysis and review/improvement. Lastly, the general requirements for the design of a PMgS that take into account the specific peculiarities of industrial research organizations are revisited and our PMgS assessed. Chapter 7 is the final chapter. It clarifies the findings of the dissertation and reflects on the research questions. The chapter firstly summarizes the key results of the work and then provides explicit answers to the research question and subquestions. Furthermore, it discusses the limitations of our work and suggests areas for future research. Figure 1.1 illustrates the outline of the dissertation.
.
Chapter 2
Performance Management
Abstract This chapter introduces basic terms and definitions. First, the nature and characteristics of invention and innovation are examined. The review shows that the commercial success of an innovation does not only depend on the quality of inventions generated in research and/or development and that both invention and innovation represent components of a process of managing innovation that extends over the entire value chain of a company. Furthermore, the following concepts are distinguished: Innovation Management, Technology Management and R&D Management in order to also understand in detail differences between the functions: research and development. The following questions are answered: What is meant by research and by development; what are the roles and tasks of the units; what kind of technological knowledge do they produce; what is the purpose of Performance Management; what are its components; and, what are industrial research departments? Keywords Goals and goal setting • Innovation management • Invention versus innovation • Key performance indicators (KPI) • Measures, metrics, and indicators • Performance management and measurement • Research and development (R&D)
2.1
Introduction and Fundamentals
In this chapter, the formal foundations of our study will be developed. The discipline, nature, and area of our research interest will be outlined and the main definitions in the field of performance management will be introduced.
T. Samsonowa, Industrial Research Performance Management, Contributions to Management Science, DOI 10.1007/978-3-7908-2762-0_2, # Springer-Verlag Berlin Heidelberg 2012
9
10
2.1.1
2 Performance Management
Invention and Innovation
Private companies and the public sector1 are seeking ways to increase their profits or stimulate the economy as a whole. Inventions and innovation have become key to improved global competitiveness in many business sectors and are seen as a means of achieving economic wealth. To be able to understand the dynamics of these phenomena, the definitions of innovation and invention will be analyzed, and the differences elaborated upon. Understanding the difference between the two phenomena helps us to comprehend the nature of each individually. This in turn helps us monitor and identify their emergence and to better match them to the place (environment) and time (phase) when they occur. The ability to do this is important because in the thesis, we will separate R&D, which is commonly seen as one single function, into two departments: research and development and examine only the former. The major goal of a research department is to investigate future trends and to foster innovation within a company. In general, the output of research departments is represented by inventions and it is of utmost importance for companies to transform their inventions into innovations. There are a variety of definitions of the term “innovation” in the literature, but a generally accepted standard definition does not exist.2 From the Latin translation of the original term innovatio, the elements of originality, newness and novelty can be derived. The different approaches to defining the concept of innovation all refer to the change and the uniqueness of a state or process.3 Schumpeter,4 who initially in 1911 coined the term ‘innovation’ in the field of economics, gives five pragmatic examples5 of innovations:
1 There are many national, European and international studies about the nature of inventions and innovations by e.g., OECD, NIST and various private consulting companies. Both innovations and inventions are seen as success factors for technological change, which strongly influences economic growth. Innovative economies typically exhibit the following characteristics: higher rates of economic growth, greater growth in employment, higher productivity, greater investment in people and capital and greater capacity for the economy to attract and retain highly-qualified people. 2 Brockhoff (1994), pp. 27–28, Specht and Beckmann (1996), p. 15, Hauschildt (1997), p. 3, Geschka (1983), p. 823. 3 Hauschildt (1997), p. 25, Pleschak and Sabisch (1996), p. 1. 4 Translated from Schumpeter (1964): “. . .so bedeutet Innovation “die neue und andersartige Kombination dieser (vorhandenen) Dinge und Kr€afte” – the “new and different combination of existing things and forces””. 5 Translated from Schumpeter (1964): “F€ unf F€alle der Innovation: (1) Herstellung eines neuen Produkts oder einer neuen Produktqualit€at; (2) Einf€ uhrung einer neuen, in einem Industriezweig noch unbekannten Produktionsmethode, die jedoch nicht auf einer neuen Erfindung beruhen muss; (3) Erschließung eines neuen Absatzmarktes, auf dem ein Industriezweig noch nicht eingef€uhrt war, egal, ob dieser Markt schon vorher existierte oder nicht; (4) Erschließung einer neuen Bezugsquelle von Rohstoffen oder Halbfabrikaten; (5) Durchf€uhrung einer
2.1 Introduction and Fundamentals
11
• Production of a new product or new product quality; • Introduction of a new, production method in an industry sector, which, however, does not need to be based on a new invention; • Development of a new sales market to an industry sector regardless of whether this market previously existed or not; • Tapping into a new source of raw materials or semi-finished products; • Implementation of a re-organization such as the establishment or breaking-up of a monopoly. Therefore, from our view, innovation in business can be seen as the implementation of new technical, business-related, organizational, or societal solutions in companies. Innovation aims to attain company goals in new ways. An innovation can therefore be considered as the logical successor to an invention, which we define for the context of this thesis as the first technical realization of an existing problem as a result of R&D activities. An invention becomes an innovation once it is introduced to the market or is used in the production process. The commercial success of an innovation requires scientific, technological, organizational, financial and commercial activities. The term “innovation” (Table 2.1) implies the creation of something new. In this regard it is often equated with the term “invention”. Fagerberg (2003) states: “An important distinction is normally made between invention and innovation. Invention is the first occurrence of an idea for a new product or process while innovation is the first attempt to carry it out in practice”.6 He acknowledges that the distinction is sometimes unclear because in some cases (e.g. biotechnology) innovation and invention are closely linked. An important characteristic, however, is the considerable time-lag between the two. He specifies other features such as the different requirements for working out ideas and carrying them out in practice, along with different locations: inventions may be carried out anywhere, while innovations occur mostly in firms in the commercial sphere. Furthermore, he lists other requirements such as production knowledge, skills and facilities, market knowledge, a well-functioning distribution system, sufficient financial resources, which a firm has to combine in order to turn an invention into an innovation.
Neuorganisation wie Schaffung Monopolstellung”. 6 Fagerberg (2004), p. 4.
einer
Monopolstellung
oder
Durchbrechen
einer
12
2 Performance Management
Table 2.1 Definitions and comparisons of the terms “invention” and “innovation” Author Invention Innovation “Invention is the first occurrence of an “Innovation is the first Fagerberg idea for a new product or process” commercialization of the idea” (2003)a Norman “An invention is a new manmade “Innovation . . . (is) a better way of (1994)b device or process. A new device doing things. Individuals and which qualifies as an invention may institutions innovate in all their take such forms as a new physical goal-directed behavior, which is product, a new biological life form, defined as an effort of an individual or a new piece of software. A or an institution at achieving process, on the other hand, is a performance as measured by a chemical, physical, or biological criterion, whether objective or chain of events that produces a subjective. With respect to such product or service” goal-directed behavior, a formal definition of an innovation is the creation or implementation of a new alternative which achieves higher performance as measured by the respective criterion” “The term innovation can be Specht and “Invention means as much as a interpreted in a narrow and broad Beckmann discovery/creation of something sense. In the narrow sense it affects (1996)c new. This is the technical the market launch of a new product implementation of new scientific or the start of a new production findings, or new combinations of process. In the broad sense it is the existing scientific findings, or a entire innovation process of combination of the two” invention and innovation”d a Fagerberg (2004), p. 4 b Norman (1994), pp. 4–5 c Specht and Beckmann (1996), p. 15 d Translated from Specht and Beckmann (1996), p. 15, “Der Begriff Innovation kann eng und weit interpretiert werden. Im engeren Sinne betrifft er die Markteinf€uhrung eines neuen Produkts oder das Anfahren eines neuen Produktionsprozesses. Im weiteren Sinne wird unter Innovation der gesamte Prozess der Invention und Innovation i.e.S. verstanden. Wenn von Innovationsprozess gesprochen wird, so ist meist Innovation i.w.S. gemeint”
Table 2.2 is a summary of the main characteristics that distinguish “invention” and “innovation”. To summarize and highlight the difference between invention and innovation definitions,7 one can make the following conclusions: I-1. An innovation does not directly require an invention; I-2. If it does occur, an invention will happen before an innovation; I-3. An invention becomes an innovation only once it is successfully deployed in a commercial context; I-4. Several inventions can contribute to a single innovation. To illustrate the difference between invention and innovation, three examples are given. A current example of a successful innovation is the iPod. The iPod is a
7
Gurel (2007a).
2.1 Introduction and Fundamentals
13
Table 2.2 The main differences between “invention” and “innovation” Element Invention Innovation Order Predecessor (occurs first, in the form of an Successor (first attempt to idea) implement the idea) Nature Often concerns a single product or process Often involves a combination of products and processes Time-lag 5–20+ years Location May occur anywhere (universities, research Most likely occurs in commercial organizations, R&D departments) firms Skills Inventor skills: narrow, deep, domain-specific Innovator skills: broad, entrepreneurial
portable media player designed by Apple that was launched in 2001. The iPod, as an MP3 player, was not invented by Apple and such devices previously existed. It is, however, the mixture of the iPod’s design, the functionality of the iTunes software, and the iTunes store that in combination made the iPod an innovation.8 This example illustrates that an innovation may be based on several existing (quite old) inventions: in this case it was the MP3 from Fraunhofer in 1991, the portable audio device from Kane Kramer in 1979, and the hard drive as a storage technology that were incorporated into a single device. Conversely, there are several examples of great inventions where the inventors failed to successfully deploy their invention in a commercial context (cf. I-3.). In 1947, a group of scientists at the AT&T laboratories created the world’s first transistor. The invention was patented, but the organization was not able to find an application for the new device. They did an outstanding job with the invention but failed to develop the innovation. Precisely for this reason, in 1952, AT&T decided to license out the transistor. For a mere $25,000, companies like Texas Instruments, Sony, and IBM acquired a technology that would produce enormous revenue in the future.9 Xerox is another company to have witnessed their inventions being turned into innovations by other firms. Many of the achievements of modern computer technology can be traced back to the famous Palo Alto Research Center (PARC). At PARC, the first personal computer was developed (years before Apple or IBM), in addition to the first graphical-oriented monitor, word-processing software, a workstation, a laser printer, a local area network, a hand-held mouse, the concept of the laptop (Alan Kay’s Dynabook), one of the first computer games, and many more. With the exception of the laser printer, Xerox management did not recognize the potential of PARC’s inventions. The success was reserved for other companies like Apple and Microsoft, and for employees leaving PARC to found their own companies to market their inventions. The most prominent examples are Bob Metcalfe, founder of 3Com, who marketed Ethernet, and John Warnock, founder of Adobe, who marketed his invention “Interpress” under the name “Postscript”. In the end, Xerox profited from almost none of these breakthrough inventions.10
8
Gurel (2007b). Cohen et al. (2000). 10 Smith and Alexander (1999). 9
14
2 Performance Management
Examples of this kind from industry were important catalysts for the change in management thinking and brought the importance of both invention and innovation to the attention of managers. In this way, innovation management gained in importance in modern management as described in the following section.
2.1.2
Innovation Management
From our examination of invention and innovation, it seems obvious that the commercial success of an innovation does not solely depend on the quality of the invention in research and/or development. Moreover, the effective management of all components that belong to the very complicated process of managing innovation, which embraces all functions in the entire company play a role in determining the success of the innovation. The interdisciplinary and multifunctional management of innovation focuses on the improvement of competitiveness and effectiveness of firms.11 Innovation management covers all of the tasks that are required to create technology know-how and to transform this know-how into marketable innovations.12 Innovation management is often a part of corporate strategy and may refer to products, services, manufacturing processes, organizational structures, management processes etc. In addition, the development and commercialization of non-technological change processes is within the remit of innovation management. In summary, innovation management deals with the design of processes and functions, which are tailored to the creation and commercialization of innovative things.13 As mentioned in the previous chapter when defining the terms invention and innovation the contemporary German literature on innovation management suggests a distinction between innovation management in a broader sense and in a narrower sense, see Table 2.2. While innovation management in a narrower sense deals with the implementation and diffusion of innovation, innovation management in the broader sense embraces the whole process from the early fuzzy front end14 phases to the subsequent recycling and withdrawal from the market. Products are often initiated within research and development departments, especially in an industrial
11
Tidd et al. (2005). Hauschild (1997), pp. 25–27. 13 Specht and Beckmann (1996), p. 15, Uhlmann (1978), p. 1, Little (1997), p. 1. 14 The term “Fuzzy Front End (FFE)” is used in various sources to describe the early phases of the innovation process; examples are: Khurana and Rosenthal (1998): “. . . we define the front end to include product strategy formulation and communication, opportunity identification and assessment, idea generation, product definition, project planning, and executive reviews”. Kim and Wilemon (2002): “Thus, we define the FFE as the period between when an opportunity is first considered and when an idea is judged ready for development.” Reinertsen (1994): “We call the time between when you could have started development and when you actually do, the ‘fuzzy front-end’.” 12
2.1 Introduction and Fundamentals
15
Fig. 2.1 Innovation management, technology management and R&D management15 (Source: Diagram based on Brockhoff 1994 and Betz 1998
environment. Therefore, one of the tasks of innovation management in the broader sense and the specific task of R&D management is to determine the need for technology know-how and to foster its creation. Figure 2.1 illustrates the building blocks of the disciplines: Innovation Management, Technology Management and R&D Management, which literature often addresses separately. Following these considerations the next chapter focuses on the early phases of the innovation process and discusses the (rather blurred) borders between research and development. This leads us to shape the term “Industrial Research”.
2.1.3
Research and Development
Research and Development (R&D) is a widely-used term. However, its contents are ambiguous and the use of the term itself is not uniform. The term “R&D” in everyday language, especially within the industrial context, can be applied in two ways: (1) R&D is an organizational unit; (2) R&D describes a set of activities16 that are conducted in the early phases of the
15 16
Adapted from Brockhoff (1994), p. 51, Betz (1998), p. 27. O’Donnell and Duffy (2005) consider an activity to be a fundamental element of a process, p. 18.
16
2 Performance Management
innovation process. These two interpretations are related in the sense that the organizational unit in general conducts part of the activities described by R&D.17 The spectrum of activities, which the term “R&D” covers according to general literature is much broader than the specific R&D activities conducted by a specific R&D department. Therefore, two different R&D organizations can only be considered comparable if their corresponding sets of R&D activities are fully juxtaposed. For the purpose of this thesis we use the term “R&D” whenever we refer to activities, and the term “R&D department” or “R&D organization” when referring to an organizational unit. We will now take a closer look at definitions of R&D in the literature. According to Specht and Beckmann, R&D is defined as “activities and processes that lead to new tangible and/or intangible artifacts”.18 Since it is essential for this thesis to separate research from development,19 the word “new” in this definition requires further attention. Brockhoff as well as Specht and Beckmann discuss two dimensions of “newness”20: 1. Content dimension: what is new? 2. Two aspects can be examined for the purpose of diagnosing newness: the newness of the very fact and the degree of newness. Subjective dimension: new for whom? The subjective dimension considers the opinion of four possible subjects: (1) any individual or expert, which could be either (2) a manager (from a businessoriented point of view), (3) a manager (from the eco-system perspective, including customers and partners), and (4) groups of experts (often expressed through national patent offices). Hauschildt adds two extra dimensions21 for the whole innovation process: 3. Process-related dimension: where (in the process) does the “new” start and where does it end? Since Hauschildt refers to the overall innovation process, this dimension is of less relevance to us. 4. Normative dimension: does “new” equal “successful”? The normative dimension asks whether the “new” is also successful. Some literature suggests using the term “innovation” for products or processes, which allow for an “improvement” to the status quo. Along with this, the goal system of a user is referred to and as a consequence, the degree of goal
17
This view is also shared by Porter (1985) in his work on corporate strategy. He considers organizations as a bundle of activities to deliver products and services. In every company activities are therefore the means by which the work gets done and performance is accomplished. 18 Specht and Beckmann (1996), p. 16. 19 For the purposes of assessing the performance of industrial research departments, the question of separation between research and especially development will be further discussed in Sect. 3.1.5. 20 Brockhoff (1994,) p. 35, Specht and Beckmann (1996), p. 16. 21 Hauschildt (1997), pp. 7–23.
2.1 Introduction and Fundamentals
17
achievement after deploying the innovation should be higher than prior to the innovation. Hauschildt discusses that it is assumed that a goal system exists in which the goals can be articulated in a way that (1) they can be recognized by third parties, and (2) that these goals can be generalized. In addition, it is imperative that, (3) from the extent of the “goal achievement”, an assessment of the “achieved improvement” can be derived. Furthermore, it is assumed in this context that (4) it is possible to agree upon a uniform success measure and that all evaluators arrive at a similar assessment in terms of such success measure. Hauschildt discusses the consequences of these four assumptions in detail and concludes that it is not possible to use this dimension to characterize innovation. The relevance of this dimension will be later addressed in the context of performance management (see Sect. 3.2) Brockhoff’s two dimensions are supported by the Frascati Manual22 (FM). According to the FM, research and experimental development – abbreviated as “R&ED”23 – comprise creative work undertaken on a systematic basis in order to both increase the amount of knowledge, including knowledge of man, culture and society, and to extend the use of this knowledge to devise new applications. “New” in the context of the FM is the “. . .appreciable element of novelty and the resolution of scientific and/or technological uncertainty, i.e., when the solution to a problem is not readily apparent to someone familiar with the basic stock of common knowledge and techniques for the area concerned”.24 Within the first part of this definition, the content dimension is reflected (appreciable element . . . technological uncertainty), and the subjective dimension is substantiated in the second part of the clause (someone familiar . . . the area concerned). Using this definition, the FM makes finer distinctions and identifies within the term “R&ED” the following three activities: basic research, applied research and experimental development, as shown in Table 2.3. The FM does not provide a name for phases comprising activities beyond experimental development. We therefore introduce the term product development, which comprises activities that are necessary after ED but before the beginning of the production phase.
22
The Frascati Manual is a Proposed Standard Practice for Surveys on Research and Experimental Development, published in 2002 by the Organization for Economic Co-operation and Development (OECD) dealing with the measurement of scientific and technological activities. The Frascati Manual is based on experience gained from collecting R&D statistics in OECD member countries. It has been developed over the last 40 years on the concept of science and technology indicators and developed a series of methodological manuals known as the “Frascati Family”, which includes manuals on: R&D (Frascati Manual), innovation (Oslo Manual), human resources (Canberra Manual), technological balance of payments and patents as science and technology indicators. 23 The Frascati Manual’s definition of development (the “D” in R&D) does not include product development: instead it only includes experimental development. In order to distinguish between the two definitions of development, R&ED is used wherever a reference to the Frascati Manual is made. 24 OECD (2002), p. 34, clause 84.
18
2 Performance Management
Table 2.3 Definitions of R&ED in the Frascati Manuala Activity Definition Focus The creation of knowledge in Basic research Experimental or theoretical work general (BR) undertaken primarily to acquire new knowledge, or the underlying foundation of phenomena and observable facts without any particular application or use in view Applied research Also original investigation undertaken in The creation of marketable (AR) order to acquire new knowledge. It is, knowledge however, directed primarily towards a specific practical aim or objective Systematic work, drawing on knowledge The development or improvement Experimental of new products, processes, gained from research and practical development systems or services experience that is directed to produce (ED) new materials, products and devices; to install new processes, systems and services; or to substantially improve those already produced or installed Beyond The FM does not provide detail or definitions of activities beyond ED. experimental However, the FM does explore activities that are on the borderline of or development b beyond ED. Examples of these are: Item Treatment Remarks Prototypes Include in R&D As long as the primary objective is to make further improvements Pilot plants Include in R&D As long as the primary purpose is R&D Industrial design and Divide Include design required during drawing R&D. Exclude design for production process Industrial engineering Divide Include “feedback” R&D and and tooling up tooling up industrial engineering associated with development of new products and new processes. Exclude for production processes Trial production Divide Include if production implies fullscale testing and subsequent further design and engineering. Exclude all other associated activities Routine tests Exclude Even if undertaken by R&D staff Data collection Exclude Except when an integral part of R&D Public inspection Exclude control, enforcement of standards, regulations Source: OECD (2002) a OECD (2002), Frascati Manual, p. 41, clause 110 b The FM does not provide a name here; it only discusses activities beyond ED
2.1 Introduction and Fundamentals
19
The FM uses a strictly functional approach for these definitions, i.e., “the nature of the R&D activity of the performing unit, rather than its principal (economic) activity, is examined”.25 In this context, principal activity refers to the organizational role within the performing unit within the innovation lifecycle. Consequently, the FM covers both “formal” R&ED in R&D units and “informal or occasional” R&ED in other units.26 This is in line with the approach that we have chosen for the term “R&D”. A logical temporal sequence for research (basic and applied) and experimental development is assumed. The definition of ED implies the extrapolation of research (basic and applied) results within this phase: “a . . . work, drawing on existing knowledge gained from research. . .” The FM itself acknowledges: “there are many conceptual and operational problems associated with these categories”.27 The phases of knowledge gain are intertwined, partially influence each other, and cannot be considered independently.28 The distinction between these phases is criticized because of the fuzzy demarcation of the phases. The Illinois Institute of Technology Research explicitly illustrated this fact in an empirical study on the overlapping of activities between phases by analyzing five important innovations.29 We would like to point out that this phenomenon is also apparent for the later product development and production phases.30 Nonetheless, Brockhoff31 states that the problem in distinguishing terms (i.e., basic research, applied research and experimental development) leads to a more applicable structure either by eliminating borders or by an even more subtle differentiation, i.e., by creating new borders. Although this approach sounds promising at a first glance, the context-based classification of activities stated in the FM provides a counter-argument to the feasibility of the approach. Since this thesis focuses on research in an industrial environment, it is necessary to put the term ‘industrial research’ into context.32 The role and dynamics of
25
OECD (2002), Frascati Manual, p. 76, clause 236. OECD (2002), Frascati Manual, p. 17, clause 14. 27 OECD (2002), Frascati Manual, p. 79, clause 251, note also that the FM acknowledges that possibly the greatest source of error in measuring R&D is the difficulty of locating the cut-off point between experimental development and the related activities required to realize an innovation. 28 Sch€atzle (1965), p. 19, Hauber (2002), p. 24. 29 Illinois Institute of Technology Research (1968), p. 15, Brockhoff (1994), p. 41. 30 Brockhoff (1994), p. 40. 31 Brockhoff (1994), p. 39. 32 Depending on the business sector, discipline and individual organizational setup of companies, there are often overlaps of different phases that are affiliated to the research function, and there are also different names for these phases. A widely-used term in literature that describes the phase beginning somewhere within industrial research, sometimes starting before the “grey zone” and sometimes after and continuing with product development is “New Product Development (NPD)”. Another name is “Advanced Development (AD)”, or just “Development (D)”. These phases are beyond the scope of this work; the separation from development, however, will be discussed in detail. 26
20
2 Performance Management
industrial research is acknowledged, for example, by the European Commission, as a one of the important facilitating forces behind innovation. The European Commission carefully monitors a variety of research activities in the European Union and pays special attention to statistics and figures such as research investment33 by top European firms. As discussed above, the borders between the phases are blurred, and in terms of this thesis, we must focus our attention on the borders of industrial research in order to identify its activities. This problem has often been discussed in literature. In order to better understand the core questions: “where does industrial research start/end?” and “where the development starts?” we review the literature on the definition of the term “industrial research” in the following sections. The term ‘industrial research’ became popular in the American literature in the 1960s and 1970s. Chorafas identifies industrial laboratories as one of seven types of research establishment. Stating the difficulty of an exact separation of pure research and applied research and development, he says34: pure research consists of the exploration of something previously unknown in order to scientifically formulate physical or technical singularities; applied research consists of a transformation of the discoveries of pure research into practical products. Chorafas continues describing development as the continuous improvement of these products up to the highest degree of perfection and the determination of the most optimal production run. He argues that the difference between industrial research and other research establishments is manifested in the intent of the work rather than the working methods. He also notices that industrial research departments are established to tackle certain research areas and to seek solutions that are industrially exploitable.35 Sch€atzle takes up these considerations and states that exponents of business economics and engineers who manage research activities within companies favor the term ‘industrial research’ instead of ‘research and development’. He concludes that the term ‘industrial research’ is identical to the term R&D, provided that the research is aimed at technology research, and the activities are conducted by industrial companies.36
33
The European Commission collects and analyzes policy-relevant information on corporate R&D through its ‘industrial research and innovation monitoring and analysis activities’ (IRIM) at the Joint Research Centre’s Institute for Prospective Technological Studies (JRC-IPTS), in co-operation with DG Research. The EU Industrial R&D Investment Scoreboard, the EU Survey on Business Trends in R&D, the Digest of Industrial R&D and the Industrial R&D Economic and Policy Analysis are some of the products of this work (for more information on these reports see http://iri.jrc.ec.europa.eu). EC-JRC (2008), p. 5. 34 Reine Forschung bestehe in der Erforschung des bisher Unbekannten, um entdeckte physikalische oder technische Eigenheiten im naturwissenschaftlichen Sinne zu formulieren. Angewandte Forschung dagegen bestehe in der Umgestaltung der Endeckungen der reinen Forschung in brauchbare Produkte. Ihr folge die Entwicklung mit einer stetigen Verbesserung dieser Produkte bis zu einem hohen Grad der Vollkommenheit und der Bestimmung des g€ unstigsten Herstellungsganges. 35 Chorafas (1963), p. 19. 36 Sch€atzle (1965), pp. 11–12.
2.1 Introduction and Fundamentals
21
Bruggmann, for his part, defines industrial research in a narrow sense as technology research, and in a broad sense to also include business-oriented research related to technology research.37 The European Union provides a more precise definition of industrial research: the planned research or critical investigation aimed at the acquisition of new knowledge and skills for developing new products, processes or services or for bringing about a significant improvement in existing products, processes or services. It comprises the creation of components of complex systems, which is necessary for the industrial research, notably for generic technology validation, to the exclusion of prototypes as covered by “experimental development”. These definitions of industrial research are quite inconsistent. They range from basic research, to product development in its broadest definition, to applied research only in its narrowest definition. Additionally, the implied sequence and separation,38 as for example, defined by the FM, rarely exist in practice as – in general agreement – the borders between the phases are rather blurred and somewhat overlapping. The execution of these three types of R&D can happen within the same department by essentially the same staff.39 Hence, the mapping of individual activities to a dedicated phase can be a very difficult task and strongly depend on the context in which the corresponding activities are conducted.40 Furthermore, recent trends such as open innovation and shortened innovation life cycles confirm this observation. Figure 2.2 summarizes the discussion above and integrates the terms
Fig. 2.2 Scope of activities in industrial research (Source: The figure was derived from the literature review by the author)
37
Bruggmann (1957), p. 4f. Refer to Sch€atzle (1965), p. 21, OECD (2002), pp. 77–79, clauses 240–250. 39 OECD (2002), Frascati Manual, p. 79, clause 251. 40 FM discusses this in terms of project contexts, OECD (2002), p. 34, clause 85. 38
22
2 Performance Management
‘research and development’ (R&D), R&ED according to the FM, and our view of industrial research. Our view, which is that industrial research in the narrower sense encompasses applied research and experimental development, is based on the initial ideas of the FM, which we subsequently validated with the case studies (cf. Appendix D). Due to the blurring of the different phases, industrial research in the broader sense may, however, also cover activities starting at the end of basic research and ending within the product development phase.41 In the following thesis, we will always use the term ‘industrial research’ in its broader sense. As a final remark in this section we would like to point out that industrial research sometimes tends to cultivate self-perpetuating dynamics within an organization. This is due to the very different nature of a research department compared to other company departments. A research department, for example, that conducts public-funded research projects has to nourish its own eco-system. This eco-system, according to Beck and V€ olker, includes external groups such as academia, industrial partners, political bodies, media and internal units. Of these internal units, the foremost would be the development department, as well as the communication and marketing departments, top management, etc.42
2.2 2.2.1
Performance Management – Basic Terms and Definitions Performance
The subject of this study is performance measurement and therefore the term ‘performance’ is defined. A review of relevant literature shows that no uniform definition of the term ‘performance’ exists. Management literature, in particular, has many proposals as how to measure performance without precisely defining it first.43 Originally, the term “performance” meant a play or piece of music, according to Wettstein’s44 citing of Andersen and Fagerhaug “. . .performance is believed to have originated in the fifteenth century to mean a play or exhibition of some type”. The Oxford English Dictionary defines performance as: “Performance. The action of performing, or something performed. . . The carrying out of a command, duty, purpose, promise, etc.; execution, discharge, fulfillment. Often antithetical to promise. . . The accomplishment, execution, carrying out, working out of anything ordered or undertaken; the doing of any action or work; working, action (personal
41
Product development comprises activities that are beyond ED activities. However they are necessary ahead of the production phase. 42 Beck and V€olker (2009), p. 34. 43 Krause (2005), p. 17. 44 Wettstein (2002), p. 15.
2.2 Performance Management – Basic Terms and Definitions
23
or mechanical); spec. the capabilities of a machine or device, now esp. those of a motor vehicle or aircraft measured under test and expressed in a specification. . . The observable or measurable behaviour of a person or animal in a particular, usu. experimental, situation. . . The action of performing a ceremony, play, part in a play, piece of music, etc.. . .” The dictionary definition shows that the term has kept its original definition but has gained additional meanings. “Accomplishment”, “efficiency”, “capability”, and “satisfaction” are listed as synonyms. The definitions encompass many different aspects; this is reinforced by the often cited statement of Meyer and Gupta: “there is a massive disagreement as to what performance is and that the proliferation of performance measures has led to the paradox of performance, i.e. that organizational control is maintained by not knowing exactly what performance is”.45 In the performance measurement literature we encountered many different definitions. Different fields use different definitions in different contexts: For example, production management accentuates the activity, the organizational context focuses on fast and optimal cost processes; economics sees performance as productivity; business studies often translate performance into monetary value; management accounting sees performance as an output of a company in financial terms; change management defines performance as generating results and emphasizes stakeholders e.g. shareholders, customers, personnel. Table 2.4 shows some relevant definitions and is ordered by publication date. Performance is not an absolute but a relative measure of success. Hauber46 reports that performance can be assessed for: (a) set objectives (planned/actual comparison, planned/will be comparison), (b) other defined periods (intertemporal comparison), and (c) an object of comparison (competitive comparison/ benchmarking). A comprehensive overview of the different facets of the term “performance” can be found in the work of Krause.47 All definitions above have one common characteristic; they all are related to two terms: effectiveness and efficiency. These two terms are well defined in the literature, with the most common citation being by Drucker: “Effectiveness is the foundation of success – efficiency is the minimum condition for survival after success has been achieved. Efficiency is concerned with doing things right, effectiveness is doing the right things”.48 However, in colloquial language these terms are mis-used as synonyms for profitability or goal-oriented behaviour. The definition of efficiency and effectiveness implicitly presumes the existence of a pre-defined goal as both can only be evaluated against a goal. As a consequence, all definitions examined above are similar with regards to the existence of
45
Meyer and Gupta (1994), p. 309. Hauber (2002), p. 54. 47 Krause (2005), pp. 17–22. 48 Drucker (1974), p. 45. 46
24
2 Performance Management
Table 2.4 Definitions of the term “performance” Source
Definition
Venkatraman and Ramanujam (1986) Cordero (1989)
“Performance is the time test of any strategy”a
Lebas (1995)
Neely et al. (1995) Rolstadas (1998)
Dwight (1999) Hoffmann (1999)
Andersen and Fagerhaug (2002)
Gr€ uning (2002)
Hauber (2002)
Wettstein (2002) EFQM (2003) Krause (2005)
a
“Effectiveness (i.e. measuring output to determine if they help accomplish objectives)” “Efficiency (i.e. measuring resources to determine whether minimum amounts are used in the production of these outputs)”b “Performance is about deploying and managing well the components of the causal model that leads to the timely attainment of stated objectives within constraints specific to the firm and to the situation”c “Efficiency and effectiveness of purposeful action”d “A complex interrelationship between seven performance criteria: effectiveness, efficiency, quality, productivity, quality of work life, innovation, profitability/ budget-ability”e “The level to which a goal is attained”f “The term “performance” describes an evaluated contribution to the attainment of organizational goals. This contribution can be generated by individuals and groups of employees within the organization, as well as by external groups, e.g., suppliers”g “We believe it is sufficient to have reached a point where performance has replaced productivity and is generally accepted to cover a wide range of aspects of an organization – from the old productivity to the ability to innovate, to attract the best employees, to maintain an environmentally sound outfit, or to conduct business in an ethical manner”h Performance is understood as the ability of a company to achieve goals, i.e. meet expectations, and is therefore influenced by results in a wider sense,i but also by the corresponding goal settingj “The term “performance” describes the contribution of specific systems (organizational units of differing sizes, employees, and processes) to attain and validate the goals of a company”k “Performance can be understood as the degree of stakeholder satisfaction”l “Performance is the level of attainment achieved by an individual, team, organization or process”m “Performance refers to the degree of the achievement of objectives or the potentiallypossible accomplishment regarding the important characteristics of an organization for the relevant stakeholders. Performance is therefore principally specified through a multidimensional set of criteria. The source of the performance is the actions of players in the business processes”n
Venkatraman and Ramanujam (1986), p. 802 Cordero (1989), p. 185, note that Cordero hypothesizes that overall performance is a function of both technical performance and commercial performance c Lebas (1995), pp. 29 d Neely et al. (1995), pp. 80–116 e Rolstadas (1998), pp. 989–999 f Dwight (1999), pp. 258–275 g Translated from Hoffmann (1999), p. 33: “Unter Performance/Leistung wird der bewertete Beitrag zur Erreichung der Ziele einer Organisation verstanden. Dieser Beitrag kann von Individuen und Gruppen von Mitarbeitern innerhalb der Organisation sowie von externen Gruppen (z.B. Lieferanten) erbracht werden” h Andersen and Fagerhaug (2002), cited from Wettstein (2002), p. 17 i The formulation “in a wider sense” emphasizes the fact that “results” do not only refer to periodical revenue figures j Translated from Gr€ uning (2002), p. 5: “Performance wird hier als die F€ahigkeit eines Unternehmens verstanden, Ziele zu erreichen, also Erwartungen zu erf€ullen und ist somit sowohl von Ergebnissen im weiteren Sinne, wir aber ebenso durch die entsprechende Zielstellung beeinflusst” (continued) b
2.2 Performance Management – Basic Terms and Definitions
25
Table 2.4 (continued) Source
Definition
k
Translated from Hauber (2002), p. 54: “Unter Performance wird der Beitrag spezifischer Systeme (Organisationseinheiten unterschiedlicher Gr€ oße, Mitarbeiter, Prozesse) verstanden, die Ziele des Unternehmens zu erreichen und zu € uberpr€ ufen” l Translated from Wettstein (2002), p. 17: “Performance kann aufgefasst werden als Grad der Zufriedenheit der relevanten Anspruchsgruppen” m EFQM (2003) n Translated from Krause (2005), pp. 17–21: “Performance bezeichnet den Grad der Zielerreichung oder der potenziell m€ oglichen Leistung bez€ uglich der f€ ur die relevanten Stakeholder wichtigen Merkmale einer Organisation. Performance wird deshalb erst durch ein multidimensionales Set von Kriterien pr€azisiert. Die Quelle der Performance sind die Handlungen der Akteure in den Gesch€aftsprozessen”
one or several goals of which the degree of attainment can be determined. Gr€uning, for example, defines performance as the ability of a company to achieve its goals (cf. Fig. 2.3). Performance depends on the one hand from the results (over or under performance) and on the other hand from the goal setting.49
result1
Over performance Goal setting
result2
Underperformance result3
Time
Fig. 2.3 Performance as goal attainment (Source: Gr€ uning 2002)
Effectiveness and efficiency in this context can therefore be understood in an abstract sense as performance measures that need to be appropriately quantified to evaluate goal attainment: Effectiveness as an indicator of the degree of a goal attainment, and efficiency as an indicator of the resources that were consumed to reach the level of achievement. For an overall evaluation of the performance, the relative importance of each aspect should be appropriately considered. In this thesis, the term “performance” is used as the level/degree of goal achievement of an organization/department rather than of individuals. Individual work performance is very much addressed in the area of applied psychology.
49
Gr€uning (2002), p. 5.
26
2.2.2
2 Performance Management
Goals and Goal Setting
The way in which the term “performance” has been defined in the previous section immediately raises the following question: What are “goals” and/or “organizational goals”? In organizations a number of individuals simultaneously work on different activities with different or at least slightly different interests. In order to bundle the interests and direct them in a strategic direction for the overall organization, an instrument is required. A means to achieve this alignment are explicitly formulated, as are jointly-accepted goals e.g. management by objectives.50 Within this context, how this alignment process takes place is of major significance. We initially revisit definitions of the term “goal” found in the literature. The Merriam-Webster dictionary51 defines the term “goal” as the end towards which effort is directed. Synonyms are “objective”, “aim”, and “intent”. According to Nagel, within the business-organizational context, “goal” should be used when that which is strived for is relevant for action and has a direct connection with the concrete problem and its solution.52 D€ orner points out that individual’s goals can be contradictory due to their different interests.53 He concludes that the major purpose of goal setting is discovering and issue-related handling of conflicts. Hamel cites the first German-speaking investigation by Schwantag in 1951.54 The definition of the characteristics of a “goal” include: reference to the future, assignment of a positive valence, determination of a state, event, process, and effect. Within organizational psychology authors define goals as: • What the individual is consciously trying to do55; • Where levels of performance sought appear to be common elements in attempts to motivate performance; success is associated with goal achievement and failure with performance below the goal level56; • What an individual is trying to accomplish; it is the objective or aim of an action57; • A target state or condition the organization wants to achieve.58
50
Nagel (1992), p. 2626. Merriam-Webster’s 11th Collegiate Dictionary (2004) [goal]. 52 Nagel (1992), p. 2627. 53 D€orner et al. (1983), pp. 37–38. 54 Schwantag (1951) cited from Hamel (1992), p. 2634. 55 Locke (1968), p. 159. 56 Frost and Mahoney (1976), p. 328. 57 Locke et al. (1981), p. 126. 58 Griffin (1990), p. 161. 51
2.2 Performance Management – Basic Terms and Definitions
27
For this thesis we use the term “goal” as per the definition of Hamel59: A goal is an envisaged and intended future state, an anticipated vision of the impact of actions.
Hamel adds to the definition that in contrast to (pure) forecasts, goals show the character of activity; within the goal, the intent of attainment or completion is logically included.60 Nagel61 deals with goal setting in the context of the problem solving process. He develops a hierarchical (top down) approach for goal setting and describes goal setting as a process where goals are cascaded and refined six times for seven levels. Specht and Beckmann also support the process view. They argue that within the phases of problem recognition and the evaluation of alternative solutions the matter of goal creation should be seen as a process across a period of time and not as an act that occurs at a specific point in time.62 Following Specht and Beckmann63 we define the goal setting process as: A systematic reduction of complexity, which can be realized, on the one hand, by the initial decomposition of the goal followed by subsequent structuring in a goal system and, on the other hand, through the iterative involvement of goal creation in the problem-solving process.
2.2.3
Measures, Metrics, and Indicators
The performance measurement literature relies on a variety of definitions to describe metrics that are applied to assess goal attainment in organizations. In this section we analyze the different terms and select which term to be use in this thesis. The following terms, inter alia, have been found in the performance measurement literature: “performance metrics”, “performance criteria”, “performance measures”, “performance indicators”, “key result indicators”, “critical success factors”, “key success indicators”, “indexes”, “strategic measures” and “success
59
Translated from Hamel (1992), p. 2634: Als Ziel kann man folglich “einen vorgestellten und gewollten zuk€unftigen Vorgang oder Zustand, eine antizipierte Vorstellung der Wirkung unseres Handelns” verstehen. Ziele weisen im Gegensatz zu (reinen) Prognosen den Charakter von handlungssteuernden Vorgaben auf; im Ziel ist die Erreichungs- oder Erf€ullungsabsicht definitionslogisch enthalten, also Bidlingmaier (1964). 60 Hamel (1992), p. 2634. 61 Nagel (1992), p. 2627. 62 Specht and Beckmann (1996), p. 18, p. 125. 63 Derived from Specht and Beckmann (1996), p. 125: “Der Zielbildungsprozeß kann als eine systematische Komplexsit€atsreduzierung, die zum einen durch Zielzerlegung mit anschließender Strukturierung in einem Zielsystem und zum anderen durch iterative Einbindung der Zielbildung in den kognitiven Problem-L€ osungs-Prozeß realisiert werden kann, verstanden werden”.
28
2 Performance Management
measures”. Krause notes that the use of the terms: “performance measures”, “performance metrics”, “performance indicators”, and “key performance indicators” has gained in importance recently.64 In order to assess things (e.g. activities, products, services) adequate measurement instruments are required. For this thesis, the definitions and differences between the following terms will be elaborated on: measure, metric, performance indicator and key performance indicator (Table 2.5).
Table 2.5 Key terms in Performance Measurement
Terms to be defined in this chapter Measure Metric Performance Indicator Key Performance Indicator
The Merriam-Webster-Dictionary65 describes the term “measure” as: (a) A fixed or suitable limit (1a (3)); (b) The dimensions, capacity, or amount of something ascertained by measuring (1b); (c) An estimate of what is to be expected (1c); (d) A measured quantity (1d (1)); (e) Amount, degree (1d (2)); (f) A standard or unit of measurement. . . (2b); (g) A basis or standard of comparison (6). The complete dictionary entry contains additional aspects to those listed above; we have only selected those that are most relevant for our context.66 Our selection already hints at many different facets and suggests that the exact meaning of the word depends heavily on the context in which it is used as well as on subjective interpretation. Within our context, definitions (d) and (f) have the best match as they both suggest an “indication of a quantity”. For our work we use the definition closest to (d) and define measure as “a quantifying value”.
64
Krause (2005), p. 21. Merriam-Webster’s 11th Collegiate Dictionary (2004) [measure]. 66 For example music, dance or instruments were not considered in our selection. 65
2.2 Performance Management – Basic Terms and Definitions
29
Geisler67 provides the following definition for metrics: they “may be used generically to describe a system of measurement that includes: (1) the item or object that is being measured; (2) units to be measured, also referred to as “standard units”, and (3) value of a unit as compared to other units of reference”. Comparison of the terms “measure” and “metric” suggests that the major difference between them is that a metric embodies additional information about the referent. A metric puts a measure into a certain context (e.g. the distance between two points in a two-dimensional plane) which is given by an item or an object or a set of those, defines a unit of measure (e.g. meter) and a reference unit (the definition of 1 m). Within the context of performance measurement there is often no one single adequate metric that allows us to exactly determine the degree of goal achievement. Geisler states that “in the social, managerial, and behavioral environments and sciences, the phenomenon under consideration is much less precise. In most instances the phenomenon of interest is in the form of a process, or at least as a set of events”.68 According to Gladen,69 numbers that try to picture complex issues in a simple manner have, in a broader sense, more or less the character of indicators. He states that indicators in narrower sense are not obtained through the consolidation of quantitative information and he defines indicators as “auxiliary metrics, whose characteristics or changes allow some conclusions on the characteristics or changes of another measure which is considered important”.70 He adds that indicators are needed for facts or parameters that are not directly measurable or observable, and mentions that their validity is less than those of the original facts. Following this explanation, we define a performance indicator in the context of organizational performance measurement as follows: A performance indicator is an auxiliary metric that partially reflects the performance of an organizational unit.
The following definitions of “performance indicator” have been found in the literature (Table 2.6):
67
Geisler (2000), p. 34. Geisler (2000), p. 35. 69 Gladen (2005), p. 14. 70 Translated from Gladen (2005), p. 14: “(Indikatoren) Sie sind Ersatzgr€oßen, deren Auspr€agung oder Ver€anderung den Schluss auf die Auspr€agung oder Ver€anderung einer anderen als wichtig erachteten Gr€oße zulassen”. 68
30
2 Performance Management
Table 2.6 Definition of the term “Performance Indicator” Source Definition Ahaus (1994) “Description of a subject, measurement scale and a measurement procedure. A performance indicator is the operationalization of a non-measurable goal”a Neely et al. (1995) “A performance measure can be defined as a metric used to quantify the efficiency and/or effectiveness of an action”b Kerklaan et al. (1996) “An instrument to measure a predefined part of the performance of a process in order to monitor the development of this performance. A complete indicator consists of a measure, a norm, a measurement instrument and a registration technique”c Kerssens-van “A performance indicator is a variable which indicates the effectiveness Drongelen (2001) and/or efficiency of a process, system or part of a system when compared with a reference value”d Gladen (2005) “. . . (in a broader sense) are the quantitative information which have been prepared for the specific needs of business analysis and management” “. . . are alternate parameters, whose characteristic or variation allow the inferring of the characteristic or variation of another parameter which is considered to be important” a Ahaus (1994), p. 143, cited from Kerssens-van Drongelen (2001), p. 74 b Neely et al. (1995), p. 80. In our interpretation, this definition of Neely et al. is more closely related to the concept of performance indicators c Kerklaan et al. (1996), p. 208, cited from Kerssens-van Drongelen (2001), p. 74 d Kerssens-van Drongelen (2001), p. 81
We can conclude from the above definitions that performance cannot be seen as something absolute and it is difficult to capture and quantify performance precisely – thus indicators for performance are needed. Concluding from Gladen’s definition that indicators are alternatives that provide approximations it is obvious that in general performance cannot be sufficiently quantified by means of one single indicator. Sound statements need a set of performance indicators.71 Arguing that the information to be reported to upper management should be presented in a reduced form, Gladen explains information reduction through consolidation (qualitative and quantitative) and that reduction through selection exists. With reduction through selection he suggests the existence of key performance indicators and in this way justifies their use.72 Some definitions found in the literature are listed below (Table 2.7):
71
B€osch (2007), pp. 104–105. Gladen discusses six types of consolidation adopted from Birk (1991): Informationsentlastung durch Verdichtung und Informationsentlastung durch Selektion, Gladen (2005), p. 13. This view will be further analyzed in Chap. 3.
72
2.2 Performance Management – Basic Terms and Definitions
31
Table 2.7 Definition of the term “Key Performance Indicator” Author Definition Dransfield et al. “Tactical measures, or key performance indicators, are a set of enterprise(1999) level measures that collectively capture the overall performance of the enterprise and act as predictors of future success, that is, of future values of the success measures”a Hauber (2002) “Performance measures provide information in a quantified and condensed form about the performance of organizational units, employees and processes and are, therefore, an important basis of information for managers to use when supervising a company”b Meyer (2002) “. . . drivers of financial performance, that is non-financial measures describing internal processes, products, and customers, at the level of the entire firm or its business units (Meyer also calls them aggregate measures)”c Gladen (2005) “(in a narrower sense) are measures, which are deliberately/intentionally/ willfully heavily compacted to absolute and relative numbers so as to being able to report in a concentrated form about a numerically ascertainable facts/data”d Parmenter (2007) “Key performance indicators represent a set of measures focusing on those aspects of organizational performance that are the most critical for the current and future success of the organization”e a Dransfield et al. (1999), pp. 99–150. Furthermore, authors distinguish between “strategic measures” defining performance on an investment level and “operational measures” on the work processes of the enterprise b Translated from Hauber (2002), p. 54: “Performance Measures sind Kennzahlen, die in quantifizierter und verdichteter Form Auskunft € uber die Performance von Organisationseinheiten, Mitarbeitern oder Prozessen geben, und daher f€ ur das Management eine wichtige Informationsbasis zur Unternehmenssteuerung sind”. In our interpretation, this definition of Hauber is more closely related to the concept of key performance indicators c Meyer (2002), p. 9, uses a similar definition to Hauber’s “non-financial” measures term, which expresses his motivation for performance indicators and the notion of holistic view “level of the entire firm” demonstrates the character of key performance indicators. Meyer also notes that the information about performance is obscured by aggregate performance measures. The aggregation conceals the sources where the performance is poor and where it is excellent. They are lumped together and in the end do not indicate where to place corrective actions d Translated from Gladen (2005), pp. 11–12, “Kennzahlen im ‘weiteren Sinne’: Das sind quantitative Informationen, die f€ ur die spezifischen Bed€ urfnisse der Unternehmensanalyse und steuerung aufbereitet worden sind”. Gladen subsumes Indicators falling into this category. “Kennzahlen im ‘engeren Sinne’: Diese sind Maßgr€ oßen, die willentlich stark verdichtet werden zu absoluten und relativen Zahlen, um mit ihnen in einer konzentrierten Form €uber einen zahlenm€aßig erfassbaren Sachverhalt berichten zu k€ onnen” e Parmenter (2007), p. 3. Parmenter distinguishes between three types of performance measures: (1) Key result indicators (KRIs) describing how you have done in a given perspective; (2) Performance indicators (PIs) telling you what to do; and (3) Key performance indicators (KPIs) suggest what you should do to increase performance. His concept will further be discussed in Sect. 3.3.1
The common ground shared by these definitions is that in the “last” step, the focus is on aspects that are deemed critical to the organization. This leads us back to the earlier idea that a reasonably complete impression of the overall performance of
32
2 Performance Management
an organizational unit typically requires a set of Key Performance Indicators (KPIs).73 Building on the terms “measure”, “metric” and “performance indicator” presented above, we define the term “key performance indicator” in the following way: Key performance indicators are a set of performance indicators, which have been selected or defined upfront by management that strongly reflects the critical factors that are of particular interest for performance of an organizational unit.
Below is a summary of the terms defined in this chapter (Table 2.8):
Table 2.8 Definitions: measure, metric, PI and KPI Term Definition A quantifying value Measure Metric
A metric puts a measure into a certain context. The context is given by an item or an object or a set of items or objects. It defines a unit of measure and a reference unit
Performance Indicator
A performance indicator is an auxiliary metric that partially reflects the performance of an organizational unit
Key Performance Indicator
Key performance indicators are a set of performance indicators that are selected upfront and agreed on by management to be the most representative and/or critical performance indicators. A key performance indicator is an element of this set
2.2.4
Performance Management
Simply knowing the level of attainment or performance does not improve the performance itself. The performance has to be actively managed. Performance management and performance measurement are sometimes mistaken for each other. Klingebiel states that the literature is inadequate on the conceptual, contextual and definitional differences between performance management and performance measurement.74 Lebas claims “management could hardly exist without measurement”. He argues that performance management and performance measurement are closely intertwined and therefore inseparable. His clarification regarding their dimensions: “performance management precedes and follows performance measurement, in a virtuous spiral and performance management creates the context for measurement”, suggests performance measurement is a
73 74
Note that all the definitions that we found use the term in the plural indicating “a set of KPIs”. Klingebiel (1999), p. 9.
2.2 Performance Management – Basic Terms and Definitions
33
part of performance management.75 The distinction between the terms is essential for this thesis because they differ in the functions they cover and therefore help us to define the scope of the thesis. Many different definitions of performance management exist in the literature. To help provide a consistent view on performance management, the definition for performance management system will also be reviewed. The interpretation of the term “system” is often ambiguous. Therefore, before studying “performance management” and “performance management system” we briefly analyze the definition of “system”. System can be defined as a uniformly-ordered whole. System concepts provide a useful way to describe many organizational phenomena, including the information system, features of applications and development processes. Organized, purposeful structure regarded as a ‘whole’ consisting of interrelated and interdependent elements (components, entities, factors, members, parts etc.). These elements continually influence one another (directly or indirectly) to maintain their activity and the existence of the system, in order to achieve the common purpose (the goal) of the system.76
On the one hand, the system can be interpreted as the interplay of all integrated components and their interdependencies; on the other hand it is often interpreted as an information system that monitors the overall performance. The following definitions for the terms “performance management” and “performance management system” have been found in the literature. Performance management system definitions will be mentioned at the end. The order is by publication date (Table 2.9). The definitions suggest that performance management encompasses all “management” activities: planning, organizing, co-ordinating, leading, controlling, staffing and motivating. This supports the fact that management is the larger domain and includes performance measurement as a component. Wettstein77 draws on the general concept of corporate management78 deteruhli defines management as the entirety of institutions, promined by R€uhli.79 R€ cesses and tools that provide a basis for will-formation (planning and decision) and will-enforcement (instructions and control) in the context of problem-solving by a community (with complex interpersonal relationships). From the four constitutive elements of planning, decision, instructions/order and control, Wettstein assigns control and partly planning to performance measurement, while decision and instructions/order he clearly assigns to the concept of management. This approach is also supported by Brunner who limits performance measurement to “measurement/assessment” by means of performance indicators. He also highlights
75
Lebas (1995), p. 23. Business Dictionary (2009). 77 Wettstein (2002), p. 27. 78 R€uhli (1985), p. 30 follows Gutenberg (1976) in his definitions and uses the following terms synonymously: F€uhrung ¼ Leitung ¼ Management. 79 R€uhli (1985), p. 28. 76
34
2 Performance Management
Table 2.9 Definitions of the term “Performance Management” Author
Definition
Lebas (1995)
“A philosophy which is supported by performance measurement.Achieving congruence as to the definition of the parameters of performance and the causal model(s) that lead to it is one of the essential functions of (performance) management”a “A company-wide management system which transforms the process of the operationalization of company strategies and objectives into a permanent management system. The achievement of objectives (of the relevant stakeholders) is supported by the combination of strategies, strategic initiatives and the planning, controlling and monitoring of the relevant management quantities”b “An approach to connect value-based strategic planning (financial value) with a measurable strategic implementation in order to resolve currently-existing deficits in strategic management and to point to new ways towards value-based corporate governance”c “Includes techniques which enable managers, in coherence with the overall company objectives, to plan, guide and improve the performance of their employees”d “Performance management is the process of planning, managing and controlling quantified variables that refer to the resources (inputs) and their transformation (throughput) in the performance (outputs) of a company’s specific systems”e “The process of managing an organization’s strategy through a fully integrated system of business improvement methodologies, metrics, processes, software tools and systems that manage the performance of an organization”f “Performance management encompasses all activities that are aimed at the optimization of stakeholder benefits through the constant improvement of the players’ professional competence and social skills, and at the same time, that minimize the financial, physical, temporal, emotional and social effort”g
Brunner (1999)
Gomez et al. (2002) Hoffmann (2000) Hauber (2002)
Cokins (2004)
Krause (2005)
Definition of the term “Performance Management System” Krause (2005) “A performance management system (PMS) is a management system based upon indicators that support the tasks aimed at optimizing the benefits to an organization’s stakeholders. Therefore, effective PMSs must represent the correlation between performance goals, goal achievement indicators, success-critical value-added activities and techniques for the improvement of the performance on all levels and along the entire value chain of an organization”h a
Lebas (1995), p. 34 Translated from Brunner (1999), p. 11: “Performance Management ist ein unternehmensweites Managementsystem, das den Prozess zur Operationalisierung der Unternehmensstrategien und –ziele in ein permanentes F€ uhrungssystem € uberf€ uhrt. Durch die Verkn€upfung von Strategien, strategischen Initiative und der Planung, Steuer und Kontrolle der relevanten Steuerungsgr€oßen wird die Zielerreichung (der relevanten Anspruchsgruppen) unterst€utzt” c Translated from Gomez et al. (2002), p. 426: “Performance Management ist ein Ansatz, der die wertorientierte Strategieplanung (finanzieller Wert) mit einer messbaren Strategieimplementierung verbindet, um dadurch heute bestehende Defizite im strategischen Management zu € uberwinden und neue Wege zu einer wertbewussten Unternehmensf€uhrung zu weisen”. d Translated from Hoffmann (2000), p. 29: “Performance Management beinhaltet Techniken, mit denen Manager in Abstimmung mit den € ubergeordneten Unternehmenszielen die Performance ihrer Mitarbeiter planen, lenken und verbessern k€ onnen”. e Translated from Hauber (2002), p. 56: “Unter Performance Management wird der Prozess der Planung, Steuerung und Kontrolle quantifizierter Gr€ oßen verstanden, die sich auf die Ressourcen (Input) und deren Transformation (Troughput) in Leistungen (Output) von spezifischen Systemen eines Unternehmens beziehen” f Cokins (2004), cited from Krause (2005), p. 38 g Translated from Krause (2005), p. 39: “Performance Management umfasst alle Aktivit€aten, die unter st€andiger Aktualisierung der Fach- und Sozialkompetenz der Akteure auf die Optimierung des Stakeholder-Nutzens gerichtet sind und dabei gleichzeitig den finanziellen, materiellen, zeitlichen, emotionalen und sozialen Aufwand minimieren” h Translated from Krause (2005), pp. 17–21: “Ein Performance Managementsystem (PMS) ist ein indikatorenbasiertes Managementsystem zur Unterst€ utzung der Aufgaben bei der Optimierung des Stakeholder-Nutzens einer Organisation. Daher m€ ussen effektive PMS den Zusammenhang zwischen Performance-Zielen, Indikatoren f€ ur die Zielerreichung, erfolgkritischen Wertsch€opfungsaktivit€aten und Maßnahmen zur Verbesserung der Performance €uber alle Ebenen und entlang der gesamten Wertsch€ opfungskette einer Organisation abbilden” b
2.2 Performance Management – Basic Terms and Definitions
35
“management” and thus the planning, management and control of performance within the performance management term. Performance management can also be explained in terms of management cybernetics.80 Like planning, completion and control, the processes of feed-forward and feed-back belong to performance management. Both feed-forward and feed-back are information loops which contribute to the target state of the system. In order to return the imbalances to a target state within a system, the system during the feedback process81 has to communicate the information about the output. This has to trigger counteraction until the state of the system matches the set goals. This control mechanism leads to a system that always returns to the target state. The control mechanism compares the current values with set goals, and in the case of difference, takes corrective action until the current values match the goals.82 The advantage of a feed-back information loop is that management, as the control authority, does not require much information in order to match the system and goals. The system can be seen as a “black box” and control activities can be confined to optimize input and output relationships. The disadvantage lies in the time lag and consequently the fact that the corrective actions can occur only after the result or output is known. The goals can usually be achieved only after additional loops and therefore with considerable time delay. In the process of feed-forward, the system requires information regarding the anticipated deviation of goals before the result or output occurs. Corrections can be triggered at an early stage until the result complies with defined targets. The main advantage of a feed-forward process, compared to feed-back, is that the detection of imbalances happens at an early stage, and not after the deviation is identified. The intervention of corrective action does not happen as it does with the feed-back process. It happens through early anticipation of negative factors rather than by feed-back. The disadvantage of the feed-forward processes is that management does not have comprehensive information. The precondition for the feed-forward control is the knowledge about the relationships within the system and their inherent cause and effect. According to Gr€ uning83 the performance measurement system translates the cybernetic process control for the multidimensional goal
80
“Management cybernetics is the concrete application of natural cybernetic laws to all types of organizations and institutions created by human beings, and to the interactions within them and between them. It is a theory based on natural laws. It addresses the issues that every individual who wants to influence an organization in any way must learn to resolve. This theory is not restricted to the actions of top managers. Every member of an organization and every person who to a greater or lesser extent communicates or interacts with it is involved in the considerations”, Beer (1959). http://en.wikipedia.org/wiki/Management_cybernetics. 81 Herder-Dorneich (1993), pp. 47–48. 82 Beer (1962), p. 131, Gomez (1981), pp. 246–247 (five steps to design cybernetic process control), Hauber (2002), p. 57. 83 Gr€uning (2002), p. 9.
36
2 Performance Management
system at all levels, from the strategic via tactical to the operational level84 (cf. Fig. 2.4). This view is shared by Dransfield, Fischer and Vogel.85 Fig. 2.4 Performance level model (Source: Bredrup 1995)
Having concluded our outlining and discussion of management concepts, a synthesis of the constitutive elements within performance management can be made. All of the definitions we have reviewed share at least four elements: planning, measurement, analysis and review/improvement (Fig. 2.5).
Fig. 2.5 The elements of Performance Management (Source: The figure was derived from the literature review by the author)
84
Bredrup (1995), p. 174. Some parts of the literature state that the operational and tactical levels are identical, see Gr€uning (2002), p. 9. 85 Dransfield et al. (1999) distinguish three basic zones of measurement arguing that strategic level contains external measures of success. They pool tactical and operational levels together stating that these mostly comprise internal measures.
2.2 Performance Management – Basic Terms and Definitions
37
• Planning, including general planning of strategy, defining goals and escorting them through the entire goal-setting process, defining to-be state or nominal values for later comparisons with actually achieved values, defining key performance indicators, deciding on timeframes for the planned strategy (short term, long term etc.). • The measurement element includes the determination of the current status. Sometimes pure data collection is associated with this activity. This element is not explicitly mentioned in some definitions, e.g. in Hoffmann and Krause. However, in our opinion, the measurement element is implicitly included in their definition. This is because they subsequently refer to the analysis element, which requires an “as-is” state. If necessary, the measurement element can include breaking the KPIs down into the PIs which are actually measured. It can also work in the other direction, consolidating the PIs back into KPIs. • Analysis includes the activities that go beyond pure measurement activities: evaluating, interpreting, projecting and forecasting from the current situation, determining the deviation from objectives and analyzing the effects of corrective actions resulting from interdependencies between goals and actions with ‘what if scenarios’. Within the analysis element it is not only deviations from goal attainment that are detected, but also information is provided with regard to “what happens if” the priorities (of goals or indicators) are changed. • Review/Improvement concentrate on the identification of concrete activities to implement conclusions drawn from analyses. Some examples of short-term decisions are: periodic rewards or identification of necessary training or corrective actions such as budget cuts, travel restrictions or resource reassignments. Longer-term examples include the adjustment and reformulation of organizational goals and KPIs between periodical performance management cycles. The performance management cycle can be seen as applied in both a long cycle (e.g. for a single goal-setting period) and in a shorter-cycle when assessing the goal achievement intermediately and taking corrective actions to improve goal achievement for the overall period.
2.2.5
Performance Measurement
In this section we analyze various definitions of performance measurement and performance measurement system found in the literature with respect to the elements identified above. Since the late 1980s the problem of measuring work results and work performance within the English language management accounting literature has been addressed under the title “Performance Measurement”.86 Although the term
86
B€osch (2007), p. 103.
38
2 Performance Management
“Performance Measurement” is used frequently, its definition is incomplete.87 In their literature review Neely et al., write: “Performance measurement is a topic which is often discussed but rarely defined”.88 In the following section we will list and discuss definitions found in the literature of the term “performance measurement” and “performance measurement system”. Table 2.10 contains definitions of “performance measurement” followed by “performance measurement system” that have been found in the literature. The order is chronological. Table 2.10 Definitions of “Performance Measurement” and “Performance Measurement system” Source
Definition
Definition of the term “Performance Measurement” Anthony et al. (1989) “Performance measurement is the key to effective management supervision and control of people in organizations. But it is also an effective tool for guiding the direction of organizational subunits. The aim of performance measures is to minimize losses and to reward quality performance by comparing actual with desired performance”a Sink and Tuttle (1989) “Performance Measurement is relative measurement. In order to interpret performance measurement data, one must have something with which to compare the measures. Commonly used alternatives are standards, goals, or baselines”b Emmanuel et al. (1990) “A vital part of the control process, and one with which accounting is particularly concerned, is the measurement of actual performance so that it may be compared with what is desired, expected or hoped for. However, it is important to stress that performance measurement is but one stage in the overall control process; it is also necessary to set standards, and to take appropriate action to ensure that such standards are attained”c Carter et al. (1995) “If there is a unifying theme to performance measurement, then it lies in the genuflection to the perspectives of economy, efficiency, and effectiveness, and the production of measures of input, output and outcome”d Gleich (1997) “Performance measurement can be defined as the development and deployment of (often several) quantifiable measurements of various dimensions (e.g., cost, time, quality, innovation, customer satisfaction) which are applied to assess the effectiveness and efficiency of the performance and performance potential of different objects within the enterprise (organizational units of various sizes, employees, and processes)”e Kerssens-van Drongelen and “The acquisition and analysis of information about the actual attainment of company Cook (1997) objectives and plans, and about factors that may influence this attainment”f Evangelidis (1992) “The process of determining how successful organizations or individuals have been in attaining their objectives”g Neely et al. (1995) “Performance measurement can be defined as the process of quantifying the efficiency and effectiveness of action”h Sinclair and Zairi (1995) “The measurement (as a process) of performance at all levels within an organization”i Hauber (2002) “Performance measurement involves the process of quantifying and evaluating the goal achievement of organizational units, employees and processes”j Wettstein (2002) “The term “performance measurement” . . . encompasses the measuring, analyzing and communicating of the performance as well as the planning of actions and measures”k
(continued)
87 88
Sometimes the literature uses the term “Performance Reporting”, Ramin and Fey (1998), p. 287. Neely et al. (1995), pp. 80–116.
2.2 Performance Management – Basic Terms and Definitions
39
Table 2.10 (continued) Source
Definition
Definition of the term “Performance Measurement System” Neely et al. (1995) “A performance measurement system can be defined as the set of metrics used to quantify both the efficiency and effectiveness of actions”l Gleich (1997) “A Performance Measurement System is a planning and control concept, containing monetary indicators which are aligned to the objectives of a company. These indicators pertain, using measurements and ratios to complement each other, to all of the company’s success and performance-relevant levels that influence its longterm financial viability. The indicators are designed to integrate the needs of stakeholders and are focused on continuous improvement and flexibility”m Simons (1999) “Performance measurement systems: information systems that managers use to track the implementation of business strategy by comparing actual results against strategic goals and objectives. A performance measurement system typically comprises systematic methods of setting business goals together with periodic feedback reports” Hauber (2002) “A performance measurement system is a coordinated set of performance measures of various dimensions which are interrelated, and thus, as a whole, provide complete information about the goal achievement of various entities within a company”n Wettstein (2002) “A performance measurement system (PMS) communicates the operational strategy and monitors the overall performance of an organization at all levels. The PMS supports effective communication of the performance to all stakeholders, provides managers with both operational and strategic decision support, gathers knowledge about the organization and simplifies the organizational learning process. To achieve this goal, the PMS defines processes and makes use of appropriate information systems”o Gr€ uning (2002) “A performance measurement system is a system for measuring and managing business performance that is multi-dimensional, characterized by mutual interdependence and integrates strategic and operational aspects, based on a cybernetic process with elements of organizational learning”p Baum et al. (2004) “Performance measurement systems are used for the measurement and management of aspects of the company’s success and its determining factors. These aspects are multidimensional and, by reciprocal interdependencies, are identified as both strategic and operational”q a
Anthony et al. (1989), p. 142 Sink and Tuttle (1989), p. 60 c Emmanuel et al. (1990), p. 31 d Carter et al. (1995), p. 35 e Translated from Gleich (1997), p 112: “Als Performance Measurement kann der Aufbau und Einsatz meist mehrerer quantifizierbarer Maßgr€ oßen verschiedenster Dimensionen (z.B. Kosten, Zeit, Qualit€at, Innovationsf€ahigkeit, Kundenzufriedenheit) verstanden werden, die zur Beurteilung der Effektivit€at und Effizienz der Leistung und Leistungspotenziale unterschiedlichster Objekte im Unternehmen (Organisationseinheiten unterschiedlichster Gr€oße, Mitarbeiter, Prozesse) herangezogen werden” f Kerssens-van Drongelen and Cook (1997), p. 346 g Evangelidis (1992), p. 45 h Neely et al. (1995), p. 80 i Sinclair and Zairi (1995), pp. 145–168 j Translated from Hauber (2002), p. 29: “Performance measurement umfasst den Prozess der Quantifizierung und Evaluierung der Zielerreichung von Organisationseinheiten, Mitarbeitern oder Prozessen” k Translated from Wettstein (2002), p. 19: “Unter dem Begriff Performance Measurement wird . . . das Messen, Analysieren und Kommunizieren der Performance sowie das Planen von Aktionen und Maßnahmen verstanden” l Neely et al. (1995), p. 81 (continued) b
40
2 Performance Management
Table 2.10 (continued) Source
Definition
m
Translated from Gleich (1997), pp. 114–115: “Ein Performance Measurement System ist ein Planungs- und Steuerungskonzept, das monet€are Kennzahlen beinhaltet, die auf allen erfolgs- und leistungsrelevanten Unternehmensebenen mit den Einflußgr€oßen der langfristigen finanziellen Leistungsf€ahigkeit des Unternehmens komplementiert sind, so daß die Maßgr€oßen bzw. Kennzahlen einander erg€anzen, auf die strategische Zielsetzung des Unternehmens ausgerichtet sind, die Anspr€uche der Stakeholder integrieren und auf kontinuierliche Verbesserung und Flexibilit€at ausgerichtet sind” n Translated from Hauber (2002), p. 58: “Ein Performance Measurement System ist die geordnete Gesamtheit von Leistungsmaßen verschiedenster Dimensionen, die in einer Beziehung zueinander stehen und so als Gesamtheit € uber die Zielerreichung unterschiedlichster Objekte im Unternehmen vollst€andig informieren” o Translated from Wettstein (2002), p. 24: “Ein Performance-Measurement-System (PMS) kommuniziert die operationalisierte Strategie und € uberwacht die ganzheitliche Performance einer Organisation auf s€amtlichen Ebenen. Das PMS unterst€utzt die effektive Kommunikation der Performance mit allen Stakeholdern, bietet Managern sowohl operative als auch strategische Entscheidungsunterst€ utzung, sammelt Wissen der Organisation und vereinfacht das organisationelle Lernen. Um dieses Ziel zu erreichen, definiert das PMS geeignete Prozesse und bedient sich geeigneter Informationssysteme” p Translated from Gr€ uning (2002), p. 10: “Ein Performance Measurement-System ist ein System zur Messung und Lenkung der mehrdimensionalen, durch wechselseitige Interdependenzen gekennzeichneten, strategische und operative Aspekte integrierenden Unternehmensperformance auf Basis eines kybernetischen Prozesses mit Elementen organisationalen Lernens” q Translated from Baum et al. (2004), p. 49: “Performance Measurement Systeme dienen der Messung und Lenkung der mehrdimensionalen, durch wechselseitige Interdependenzen gekennzeichneten strategischen und operativen Aspekte des Unternehmenserfolgs und seiner Einflussgr€oßen”
Common to many authors’89 definitions is the attainment of objectives and the process element. Kerssens-van Drongelen and Cook highlight the analysis aspect of the measurement. While Kerssens-van Drongelen and Cook stress the acquisition and analysis of information; Wettstein in addition incorporates the communication and further planning of actions and tactics. This could lead to the assumption that performance measurement is more than just a review and illustration of past achievements. Another important element of performance measurement seems especially to be the process of interpretation of results for improvement and the development and demonstration of actions and action opportunities. In order to consistently derive the key elements for performance measurement, the definitions will be analyzed on the basis of the four elements extracted from performance management. The table shows a particular emphasis on two elements: measurement; analysis.
89
Here: Evangelidis, Neely et al., Sinclair, and Hauber.
2.3 Summary
41
Table 2.11 Comparison of Performance Measurement (system) definitions Constitutive elements mentioned in Performance Measurement definitions Review/ Planning Measurement Analysis improvement Performance Measurement
Source
Anthony et al. (1989) (✓) ✓ ✓ ✓ Sink and Tuttle (1989) (✓) ✓ (✓) Emmanuel et al. (1990) ✓ ✓ ✓ Carter et al. (1995) ✓ Gleich (1997) ✓ ✓ (✓) Kerssens-van Drongelen and Cook (1997) ✓ ✓ Evangelidis (1992) (✓) ✓ Neely et al. (1995) ✓ (✓) Sinclair and Zairi (1995) ✓ Hauber (2002) ✓ ✓ Wettstein (2002) ✓ ✓ ✓ Source Performance Measurement system Neely et al. (1995) ✓ Gleich (1997) ✓ ✓ ✓ ✓ Simons (1999) ✓ ✓ ✓ ✓ Hauber (2002) ✓ ✓ Wettstein (2002) ✓ ✓ ✓ ✓ Baum et al. (2004) (✓) ✓ ✓ ✓ component is explicitly mentioned in the definition (✓) component is not explicitly mentioned, but is implied by our definition (In those cases where the availability of an element has been put in parentheses the existence of the element is not explicitly mentioned in the cited source, nevertheless the definition implies its presence)
We therefore include the following aspects in our definition of performance measurement (see Table 2.11): A process in which: • Relevant data for performance indicators (as-is data) is collected; and • The collected data is evaluated, analyzed and interpreted (comparison of as-is with to-be).
2.3
Summary
Figure 2.6 summarizes all terms that have been introduced in the context of performance management. The overall performance management cycle is based on four constitutive elements. Within the planning element, organizational goals are derived from the company strategy and are operationalized through KPIs. The performance measurement part concentrates on collecting adequate performance indicators and consolidates them into KPIs for the purposes of analysis of the current situation and also prediction. The review and improvement element assesses performance achievements by comparing KPIs as-is with KPIs to be, and draws conclusions on short-term corrective actions as well as on the adjustment of goals and KPIs in the next management cycle.
42
2 Performance Management
Fig. 2.6 Performance Management: putting the terms together (Source: The figure was derived from the literature review by the author)
2.3.1
Purposes of Performance Management/Measurement
According to Geisler90 measurement plays an important role in everyday life: “Measuring the objects and the events in the world around us is not only a scientific necessity, but also the means to make sense of the complexity of natural phenomena. We continually live by measures of our surroundings. We measure the passage of time, the temperatures in our climate, our economic situations, and everything else with which we make contact.” The question of measuring or not measuring the performance of researchers91 is a heavily discussed question in both theory and practice. The fact that rigorous measurement kills creativity is probably the most prominent argument for not measuring. Creativity is seen as an indispensable feature in the search for new ideas, new products, and innovations. This search for newness covers a significant part of the activity of industrial research organizations. Therefore it should be adequately reflected in a PMS in research organizations.
90
Geisler (2000), p. xi. In conducting our case studies we realized that some companies that were pre-selected did not measure performance within their research department. The desire not to constrain the freedom of researchers seems to be a major argument for not measuring their performance. These cases were taken into account, however they were not further considered for our case studies.
91
2.3 Summary
43
There are a number of contributions to the literature focusing on purposes for measuring performance, as outlined in Table 2.13 below. According to Sink and Tuttle,92 “The most important, and perhaps the only really valid, reason for measuring performance . . . is to support and enhance improvement”. In order to improve, information about the status quo is required, which in the end allows us to demonstrate that an improvement has actually been achieved. According to Specht and Beckmann93 this information is also necessary for decision-making by management and has to be available in the right place at the right time, in the right quality and quantity, as well as at the lowest possible cost. According to the literature, there are six major categories of reasons for performance management; two of them – communication and alignment – are strongly related to the context of goal setting. Another two have their background within the evaluation of goal attainment – determining the status quo and predicting attainment for a specific period. The final two have their origin in psychology and focus on motivation – on the individual level as well as on the organizational level. Table 2.12 illustrates these six categories and provides the structure which is used in the remainder of the chapter to classify the purposes of performance management as found in the literature. Table 2.12 Purposes of Performance Management in industrial research Categories of purposes of Performance Management 1. Goal setting (a) Communication
(b) Alignment
2. Evaluating goal achievement (a) Status-quo (b) Prediction
3. Motivation (a) Organizational
(b) Personal
Setting goals is a prerequisite for the assessment of the actual performance. Communication and alignment are two purposes within the goal setting category. Communication ensures that company goals get across into other parts of the organization. According to our definition of the goal setting process (cf. Sect. 2.2.2), the goals formulated at higher levels have to be decomposed and cascaded and communicated to lower levels. Alignment provides assurance that the sum of all cascaded goals reflects the overall goal at the next higher level. During the goal setting process iterative communication and alignment takes place by receiving, checking and giving feedback on whether the cascaded goals meet certain crucial characteristics as discussed by Locke.94 Such information can be used to revise the performance indicators and potentially adjust the cascaded goals. These two
92
Sink and Tuttle (1989), p. 141. Note that Specht and Beckmann (1996), p. 332 discuss the control of R&D and its fundamentals. The analysis and integration of R&D control into performance measurement will be discussed in Sect. 3.2.2. 94 (1) Each goal has to be clear so that people can carry out appropriate actions to reach the goal; (2) goals should not be too easy to attain, but at the same time, they must be attainable; and (3) goals should be accepted by the receivers, goals that are not accepted may have negative impacts on the performance. Locke (1968), p. 125, Tankoonsombut (1998), p. 12. 93
44
2 Performance Management
purposes primarily reflect the planning element of the performance management cycle as the goal setting takes place within this element. Our definition of goal – an envisaged and intended future state; an anticipated vision of the impact of actions – directly motivates category 2 as it requires: determination of the status quo (2a), i.e. to provide the current performance level compared to the level aimed for; and to be able to predict (2b) the future achievable state; especially with regard to being able to support decision-making for corrective actions. Status quo evaluation and prediction respectively cover the measurement and the analysis elements of the performance management cycle. Not only business economics but also other disciplines such as “Organizational and Occupational Psychology” and “Organizational Behavior and Human Performance” examine the performance of individuals in an organizational context. The motivation of employees impacts their individual performance, and consequently, organizational performance. This is a result of a great number of studies from applied psychology examining different behaviors and their impacts on performance. Therefore, it is not surprising that individual motivation is considered to be one of the most effective tools for governing performance. By personal motivation we mean rewards, career planning, etc., and this represents category 3b. The demonstration of the contribution of subordinate goals to superior goals is often used as a means to justify the existence of an organizational unit. This especially occurs when instead of a clear quantitative (economic) outcome an organizational unit’s output can only be vaguely estimated. This aspect (existential justification) represents the organizational facet (3a) of the motivation category. The motivational purposes mainly refer to the review/improvement element of the performance management cycle. Figure 2.7 illustrates the discussed relationships between the constitutive elements of performance management with the described purposes. In the following, by way of example, we discuss three sources that elaborate on purposes for performance management and map their findings to our categories. There are a variety of other sources, which are not further discussed here. The interested reader is referred to Table 2.13 where we list additional sources and indicate the purposes. Landy and Farr95 focus their research on the work performance of individuals. They state that the prediction of performance plays a major role in all personnel decisions and many other types of organizational decisions. They list three kinds of purposes why performance information may be collected and what it is used for: administrative purposes, guidance and counseling purposes, and research purposes. 1. Within the administrative part they mention promotion, lateral transfer, demotion and retention decisions; merit compensation decisions; training program assignments; and the establishment of scores for selection procedures.
95
Landy and Farr (1983), pp. 3–4.
2.3 Summary
45
Fig. 2.7 Constitutive elements of PMS and their relationships (Source: The figure was derived from the literature review by the author)
These purposes are primarily reflected within our “motivation” category as the reasons mentioned mainly concern the implications for decision-making related to the performance of individuals. 2. Landy and Farr identify the improvement of job satisfaction and work motivation as general purposes for guidance and counseling performance information. This is required for providing information regarding the current individual performance level as well as for probable, and possible, future job assignments in the organization. In their opinion, guidance and counseling purposes may include supervisory feedback to subordinate personnel regarding their relative and absolute strengths and weaknesses. They also mention career planning and preparation in this context. The purposes mentioned are also related to individual motivation. 3. The third purpose is obtaining performance information as part of various projects assessing human-resource-related procedures and initiatives. Examples of these are the validation of selection procedures, the evaluation of training programs, and the evaluation of motivation and satisfaction-oriented interventions such as compensation plans and job enrichment programs. The purposes96 described in this part best fit into the ‘status-quo’ aspect of our second category.
96
See Table 2.13.
46
2 Performance Management
Butler97 identifies nine possible reasons for measuring: (1) Improve organizational performance, (2) assist in decision making, (3) provide visibility of results, (4) improve understanding, (5) compare absolute performance, (6) improve motivation, (7) improve communication, (8) improve individual performance, and (9) retention of control. Reasons 4 and 7 cover the communication aspect (1a) of the “goal setting” category. Reason 9 in our context is partially related to the alignment aspect (1b) of “goal setting” and partially to organizational motivation (3a). Reasons 3 and 5 relate to the status quo category (2a) whereas reason 2 is reflected by the prediction aspect (2b) within our categories. Finally, reasons 1, 6 and 8 deal with motivational aspects where 1 refers to the organizational (3a) and the latter two to the personal (3b) category. Kerssens-van Drongelen98 explores purposes and translates them into functions by developing a taxonomy of measurement systems. She identifies seven measurement system functions: 1. Provide insight into (expected) deviations from objectives/environmental factors to support the diagnosis by a manager as to whether, and if so which, steering measures to apply 2. Fuelling learning to improve the predictive model 3. Alignment and communication of objectives 4. Supporting decision making on performance-based rewards 5. Provide insight into (expected) deviations from objectives/environmental factors to support the diagnosis by subordinates as to whether, and if so which, steering measures to apply 6. Justification of existence, decision and performance 7. Motivating people through feedback Functions 1 and 5 match with our status quo (2a) category as they detect the deviations from objectives, whereas function 2 fits to the prediction (2b) aspect. Function 3 reflects the category “goal setting” (1a and 1b); function 6 corresponds with the organizational (3a), and functions 4 and 7 with the individual (3b) motivation category. Table 2.13 Comparison of purposes for Performance Management Categories of purposes of Performance Measurement Author 1. Goal setting
Landy and Farr (1983)a Sink and Tuttle (1989)b
2. Evaluating goal achievement
3. Motivation
(a) Status (a) (b) Communication Alignment quo ✓ (3)
(b) (a) (b) Prediction Organizational Personal ✓ (1, 2)
✓
✓
✓
(continued)
97 98
Butler (1994), p. 21, see Table 2.13. Kerssens-van Drongelen (2001), p. 46, see Table 2.13.
2.3 Summary
47
Table 2.13 (continued) Categories of purposes of Performance Measurement Author
Butler (1994)c Bonsdorff and Andersin (1995)d Dhavale (1996)e Kaplan and Norton (1996)f Kerssens-van Drongelen (2001)g Loch and Tapper (2002)h Godener and S€oderquist (2004)i Schreyer (2007)j a
1. Goal setting
2. Evaluating goal achievement
(a) (b) Communication Alignment ✓ (4, 7) ✓ (9) ✓ (2, 3)
(a) Status quo ✓ (3, 5) ✓ (4, 5)
3. Motivation
(b) (a) Prediction Organizational ✓ (2) ✓ (1, 9) ✓ (6)
✓ (2)
✓ (2)
✓ (3)
✓ (3)
✓ (1, 5) ✓ (2)
✓ (1)
✓ (2, 3) ✓ (4)
✓ (1)
✓ (1)
✓ (2)
✓ (1, 3, 7)
✓ (1, 2, 3, 4)
✓ (4, 5) ✓ (8)
✓ (2)
(b) Personal ✓ (6, 8) ✓ (1)
✓ (1)
✓ ✓ (1)
✓ (6)
✓ (4, 7) ✓ (2)
✓ (3, 5)
✓ (4, 5) ✓ (6)
Landy and Farr (1983), pp. 3–4 Sink and Tuttle (1989), p. 1. “The most important, and perhaps the only really valid, reason for measuring performance . . . is to support and enhance improvement” c Butler (1994), p. 21 d Bonsdorff and Andersin (1995), p. 67 e Dhavale (1996), p. 50: “Performance Measurements, evaluation systems, and reward systems are indispensable management tools. They can help motivate employees to work toward fulfilling the organization’s strategic perspectives. By contrast, poorly designed or poorly implemented performance measurement systems encourage dysfunctional and suboptimal behavior throughout an organization” f Kaplan and Norton (1996), p. 147. “The objective of any measurement system should be to (1) motivate all managers and employees to (2) implement successfully the business unit’s strategy” g Kerssens-van Drongelen (2001), p. 46 h Loch and Tapper (2002), p. 186, highlight four main functions of R&D performance measurement: (1) alignment and prioritization; (2) evaluation and incentives; (3) operational control, and (4) learning and improvement. Furthermore the authors state that the fundamental purpose of performance measurement is to encourage behavior that achieves the goals of the organization i Godener and S€oderquist (2004), p. 197. (1) communicating objectives, agreements and rules; (2) defining corrective actions based on diagnosis and control; (3) allocating resources; (4) deciding on individual promotions, salary increases and other incentives (5) learning/continuous improvement j Schreyer (2007), p. 74: (1) operationalization of the company’s strategy, (2) identification of and focusing on success factors, (3) visualization of the interrelations, (4) planning and controlling of the employment of resources, (5) performance evaluation, (6) employee motivation, (7) communication processes, (8) learning effects b
48
2.3.2
2 Performance Management
The Information and Communication Technologies (ICT) Sector
In comparison with some other industries mentioned, such as pharmaceuticals, chemicals, petrol and gas, steel and some sectors of electronics, the ICT sector is relatively young and very fast growing.99 In an interim report “Benchmarking national and regional policies in support of the competitiveness of the ICT sector in the EU”100 the authors report that the ICT sector overtook the traditional pillars of the European economy, such as the pharmaceutical and the automotive industries, in terms of annual turnover which in 2004 was about EUR 200 billion in Europe and about EUR 1,000 billion worldwide. Although the history of this sector dates back to the 1950s,101 definition of the ICT sector remains a challenge. Looking at the relevant sources it is apparent that a generally agreed upon definition of the ICT industry does not exist. The different interpretations of the term are due to the fact that the sector is widespread, is subject to constant technological changes and is embracing and integrating more and more domains by delivering cross-domain solutions. ICT includes all hardware and software, which are necessary for transmitting, processing and utilizing digital data in any possible form. Data in this sense include texts, sounds and images.102 Currently, there are three from several attempts succeeded to create a standardized form of accounting and measurement of scientific and technological activity within and across countries: OECD103 and UNESCO, the U.S. National Science Board, and Japan’s National Institute of Science and Technology Policy (NISTEP). There are a few different standardization communities that have classified
99
For example: at the end of 2008, from close to zero (only) 10 years ago, in the developing world, mobile phones have reached an estimated average 49.5% penetration rate, ITU (2009), p. 1. 100 Friedewald et al. (2004), p. 7. 101 In particular, discussions about the role of research and development in high technology started around 1950 within the context of technological and economic competitiveness of the ICT producing sector, Friedewald et al. (2004), p. 7. 102 The OECD defines the ICT sector as a combination of manufacturing and service industries that capture, transmit and display data and information electronically. 103 The Organization for Economic Co-operation and Development (OECD) is an international organization of 30 countries that accepts the principles of representative democracy and freemarket economics. It originated in 1948 led by Robert Marjolin of France to help administer the Marshall Plan for the reconstruction of Europe after World War II. Later, its membership was extended to non-European states. In 1961, it was reformed into the Organization for Economic Cooperation and Development by the Convention on the Organization for Economic Co-operation and Development.
2.3 Summary
49
the ICT sector, with their various standards (SIC,104 ISIC,105 CPC,106 NACE,107 and NAICS108). The definitions are revised on an irregular base. After having reviewed two standards: ISIC and NACE, we decided to use the most current definition provided by the OECD in 2007 for the thesis. This is based on the UN International Standard for the Industrial Classification of all Economic Activities (ISIC4). The standard applies the following general principle (definition) to identify economic activities (industries) related to ICT: The production (goods and services) of a candidate industry must primarily be intended to fulfill or enable the function of information processing and communication by electronic means, including transmission and display.109
The list of ICT industries (ISIC Rev. 4) that meet this condition is provided in the table below (Table 2.14). Regarding our exploration and definition of ICT, it is necessary to exclude some of the subsectors. We are examining industrial research departments and their activities and it is clear to us that these departments will not engage in all of the subsectors listed. For example ICT-related research and development activities are not relevant to the subsector of ICT services called “Repair of computers and communication equipment”. For this reason we exclude these subsectors: • 9511 Repair of computers and peripheral equipment, and • 9512 Repair of communication equipment.
104 The Standard Industrial Classification (abbreviated SIC) is a United States government system for classifying industries by a four-digit code. 105 ISIC is the United Nations International Standard Industrial Classification of all economic activities. This classification is the international standard for the classification of productive economic activities. The main purpose is to provide a standard set of economic activities so that entities can be classified according to the activity they carry out. The hierarchical structure of the classification comprises: Tabulation Categories – one letter alpha code A to Q; Divisions – twodigit codes; Groups – three-digit codes; Classes – four-digit codes. The third revision of ISIC is used in the 1993 SNA. 106 The Central Product Classification (CPC) of the United Nations Statistics Division constitutes a complete product classification covering goods and services. It was intended to serve as an international standard for assembling and tabulating all kinds of data requiring product detail including industrial production, national accounts, service industries, domestic and foreign commodity trade, international trade in services, balance of payments, consumption and price statistics, CPC (2002). 107 NACE stands for The Statistical Classification of Economic Activities in the European Community and is a European industry standard classification system consisting of a 6 digit code. NACE is equivalent to the SIC and NAICS system. 108 The North American Industry Classification System or NAICS is used by business and government to classify and measure economic activity in Canada, Mexico and the United States. It has largely replaced the older SIC system; however, certain government departments and agencies, such as the U.S. Securities and Exchange Commission (SEC), still use the SIC codes. The NAICS numbering system is a six-digit code. 109 OECD (2007), p. 15.
50
2 Performance Management
Table 2.14 ICT sector according to ISIC Rev. 4 Definition of the ICT sectora (codes and sectors) ICT manufacturing industries 2610 Manufacture of electronic components and boards 2620 Manufacture of computers and peripheral equipment 2630 Manufacture of communication equipment 2640 Manufacture of consumer electronics 2680 Manufacture of magnetic and optical media ICT trade industries 4651 Wholesale of computers, computer peripheral equipment and software 4652 Wholesale of electronic and telecommunications equipment and parts ICT services industries 5820 Software publishing 61 Telecommunications 6110 Wired telecommunications activities 6120 Wireless telecommunications activities 6130 Satellite telecommunications activities 6190 Other telecommunications activities 62 Computer programming, consultancy and related activities 6201 Computer programming activities 6202 Computer consultancy and computer facilities management activities 6209 Other information technology and computer service activities 631 Data processing, hosting and related activities; web portals 6311 Data processing, hosting and related activities 6312 Web portals 951 Repair of computers and communication equipment 9511 Repair of computers and peripheral equipment 9512 Repair of communication equipment Source: OECD (2007) a Note that there is a growing recognition of inclusion of the “content industries” into the ICT definition. “Content industries” create and distribute content (e.g. text, audio, video), particularly those that create and distribute content to a wide audience. The definition of the content and media sector is provided in the Appendix A
2.3.3
The Software Industry
Having defined the ICT sector as a whole we position the software industry within the ICT sector in the next step. The European Commission initiated extensive studies to evaluate the contribution of information and communication technologies in economies, particularly monitoring the software industry. The software industry is described as “one of Europe’s best vectors of growth in innovation and competitiveness”.110
110
Truffle 100 (2006), p. 1.
2.3 Summary
51
For a good overview of the economic principles, strategies and future trends among software providers and software industry, see Buxmann et al.111 In its definition of software (SW), BITKOM112 distinguishes between system software and standard applications: • System software: System infrastructure software includes system management SW, network management, security SW, storage SW, server ware, system level SW. Application tools include data management SW, middleware, development tools, etc. • Standard applications: consumer, content, collaboration, enterprise applications. • Independently from software, BITKOM defines IT-services as Consulting: planning/design of IT-systems, including IT-related business consulting. Implementation: services such as procurement, configuration, installation, testing and management, development of customized solutions, IT-education and -training. Operations management: management of components of the IT infrastructure for customers including help desk services, asset management services, systems management, network management, software update management, application services providing, web hosting, facilities management, back-up/archiving/business recovery services. Support services: maintenance, telephone support, etc. On a multi-company level, the definition of “software industry” is somewhat difficult. First of all, this is because software can be interpreted as (a) a good and (b) as a service. Secondly, there is a discussion about software publishing and broadcasting industries, in particular whether these industries, or components thereof, should be classified in the ICT services grouping or in the content and media grouping. The OECD states that software publishing (ISIC 5820) covers at least two distinct components: the publishing of productivity software and the publishing of multimedia software (i.e. games). Because the latter type of software is designed to inform, educate or entertain and has more in common with other types of content products such as newspapers, television programs, films or musical recordings, this industry should ideally be classified in the content and media sector according to some members of the board.113 However, due to the fact that ISIC recognizes only one software publishing industry that produces both types of software, this industry was included in ICT services. According to the OECD the productivity software which is designed to facilitate information processing is classified more appropriately with technology-centric services such as telecommunications or hosting services (ISIC 62/631).
111
Buxmann et al. (2008). BITKOM is a German Association for Information Technology, Telecommunications and New Media and stands for Bundesverband Informationswirtschaft, Telekommunikation und neue Medien e.V. 113 The definition is based upon an agreement between the members of the “Working Party on Indicators for the Information Society” (WPIIS). 112
52
2 Performance Management
To summarize, according the latest review of ISIC4, the ICT sector is comprised of ICT manufacturing industries, ICT trade industries and ICT services industries (including ICT repair industries). Regarding the definition of the software industry, it can be concluded that in essence the software producing sector is mainly represented in ISIC 62 Computer programming, consultancy and related activities.
Chapter 3
State-of-the-Art in Performance Management
Abstract This chapter investigates the state-of-the-art in the performance management literature. In this context, the performance management systems, organizational goals and key performance indicators are examined at three levels of granularity: generic (literature reporting about generic approaches covering either one or more functions of a company), R&D (focusing on the specifics of the R&D function), and research function only. The insights in this chapter enable us to formulate the research gaps and refine the primary question with four research subquestions. Keywords Characteristics of research compared to development • Key performance indicators • Organizational goals • Performance management systems (PMS) • Requirements catalogue of a PMS
Having already defined the relevant terms for the context of the thesis and explored reasons for managing performance, this chapter presents an overview of the relevant literature. As it has been shown earlier in Sect. 2.2 there is no consistent definition or conceptualization of performance management and performance measurement in the literature. Therefore, we do not make a clear distinction between authors that position their work in performance management and those that position themselves in performance measurement.1 Moreover, our focus is on the specifics parts that are relevant for this thesis, which authors deal with within their contributions. Due to our focus on performance management within industrial applied research departments in the ICT sector, this literature review starts by exploring general
1 See our discussion about definitions in Sects. 2.2.4 as well as 2.2.5, where we concluded that performance measurement builds a part of performance management.
T. Samsonowa, Industrial Research Performance Management, Contributions to Management Science, DOI 10.1007/978-3-7908-2762-0_3, # Springer-Verlag Berlin Heidelberg 2012
53
54
3 State-of-the-Art in Performance Management
literature on performance management across industries rather than within specific to organizational functions. The focus is subsequently narrowed to our specific research area: performance management in industrial research organization in the ICT sector. Altogether in this chapter, literature which deals with the following aspects is examined: • The holistic view on performance management systems (PMgSs) and their major components: organizational goals and KPIs • The application of PMgS in different organizational functions: in the entire organization (generic); in the R&D function as one single entity; and in the research department as an entity separate from the development department. • The affiliation to the industry, i.e. non-sector specific and industry sectorspecific. In both literature and practice, multiple interpretations of performance management exist and are difficult to classify. Therefore, Sect. 3.1 presents a framework based on existing schools of thought. This framework allows for a categorization of performance management and measurement concepts at an abstract level. Within this framework, the most well-known performance management approaches are shown. The literature review in this section is conducted with the following focus: generic PMgSs, R&D PMgSs and finally, research PMgSs. Having looked at the schools of thought, it is essential to understand the necessary characteristics of a system in order for it to have value. Section 3.1.4 elaborates on the more detailed requirements for a performance management system. Here we analyze four recent requirement catalogues and condense them into a single set of requirements. This section is crucial for a basic understanding of the constitutive elements of a performance management system. This work measures industrial applied research departments. Therefore, Sect. 3.1.5 separates “research” and “development”, which literature has traditionally treated as a single entity. Furthermore, this section identifies the main characteristics of industrial research and derives additional requirements for a PMgS tuned to a specific organization. Organizational goals as a major component of a PMgS are discussed in Sect. 3.2. The literature review in this section presents an overview of different existing goal systems and also highlights the goal-setting process within organizations. The organizational goals are inspected from three different perspectives: the generic company-wide perspective, the R&D function, and the research function (separately from development). The same approach is used in Sect. 3.3 when reporting on the state-of-the-art about key performance indicators. Section 3.4 presents our conclusions on this review and identifies the research gap. This provides a basis for the central research questions to be formulated, which reflect the general intent of our research.
3.1 Performance Management Systems
3.1 3.1.1
55
Performance Management Systems Generic Approaches: Schools of Thought
We begin our literature review by exploring the question “what are the theoretical underpinnings for performance measurement concepts and how can these be classified?” The analyzed literature on performance management often presents isolated approaches that have different foci and that are difficult to sort into a uniform classification scheme. Only one recent contribution (from Krause 2006) that suggests a uniform classification scheme based on different “schools of thought” has been found. Krause classifies the different schools of thought according to the major focus of the respective approach: (1) financing/funding, (2) business strategy, (3) business processes, and (4) employees. Please note that concrete performance management approaches cannot always be mapped to a single category, as they may address aspects of different schools of thought. The four schools of thought will now be discussed briefly. 1. The guiding principle for the finance-focused school of thought is “the most serious potential problem is that we are running out of money”. Krause places approaches within this category in which financing plays a central management role and all other aspects are aligned accordingly. Approaches of this category are: Beyond/Better Budgeting (Hope 2003), Value Based Management (Brunner 1999), Shareholder Value (Rappaport 1998), Individual Classification Systems (GE, PIMS). Criticism is given to budget-oriented approaches for the following reasons: • The lack of flexibility of the budgets to changing business conditions; • The lacking connection of budgets to the corporate strategy; • The dysfunctional and creativity-inhibiting “command-and-control” principle; • The short-term effectiveness in the achievement of financial targets by focusing on an annual quota. 2. The alignment of the strategy of a company throughout its departments is the most important component within the strategy school of thought. Krause expresses its guiding principle by “the capital importance that we know where we want to go and all pull together”. The Balanced Scorecard (Kaplan 1992, 1993, 1996a, b, 2000, 2002, 2004), benchmarking concepts (Camp 1989; Siebert 1998), the Strategic Measurement and Analysis Reporting Technique (SMART), the Performance Pyramid (Lynch and Cross 1995), and Ratios au Tableau de Bord (Lauzel and Cibert 1959) are all mainly orientated towards this school of thought. 3. Business process management and control is the third school of thought portrayed by Krause. The most renowned approaches which he identifies in
56
3 State-of-the-Art in Performance Management
this category are again the Ratios au Tableau de Bord, which is deemed to be the precursor to the balanced scorecard; the Activity Based Profitability Analysis (ABPA), which is part of the Activity Based Costing group (Meyer 2002), SixSigma (Hurry and Schroeder 2000); and Total Quality Management (TQM) (Malorny 1996; Wolter 1997). Both Six-Sigma and TQM overlap with the employee-oriented framework. 4. Approaches which focus on employee contributions to performance are based on self-assessment by employees and encompass the quality award concepts and intellectual capital approaches. “We must above all bring our employees to use their knowledge within the teams” is the guiding principle of the employeefocused school of thought. Specifically, the European Foundation for Quality Management (EFQM) Award, the Malcolm Baldrige Award in the U.S. and the quality award of the Ludwig Erhard Stiftung in Germany are mapped into this last category. The goal of these approaches is the continuous improvement of existing organizational structures through self-adaptation (Fig. 3.1). Critical appraisal of the schools of thought can be drawn from Klingebiels2 research where he discusses the integration problems within the coordination tasks of a performance management system. As a coordination task he understands the goal-appropriate, content-based and time-based alignment of interdependent decisions within existing corporate structures.3 Klingebiel states that for a The framework approaches can be allocated to four schools of thought Beyond/Better Budgeting Individual classification systems (GE, PIMS) Benchmarking
Process Management
Value Based Management / Shareholder Value SMART / Performance Pyramid
Balanced Scorecard
Ratios au Tableau de Bord TQM
Activity Based Costing
Six Sigma
EFQM, Self Evaluation Intellectual Capital / balance of knowledge
„The most serious problem is that we are running out of money“
Financing
„The most important thing is that we know what we want to achieve and that we all pull together“
Strategy
„We have clear & reasonable goals. The crucial point is their realization in optimized processes.“
Processes
„We mainly have to force our employees to use their knowledge in the team.“
Employees
Fig. 3.1 Four schools of thought (Source: Figure based on Krause 2006)
2 3
Klingebiel (2000). Klingebiel (2000), p. 298.
3.1 Performance Management Systems
57
competitive advantage of a company the correct horizontal integration (between functions, products or regions) and vertical integration (between different company levels) is of utmost importance. In his view, the interplay of the following factors determines the competitive advantage of a company: (1) structures, (2) processes, (3) systems, (4) cultures. These should be reflected in the design of a performance measurement system. According to Klingebiel the correct mix of these factors depends on the individual situation of a company. In summary, Klingebiel in his approach embraces at least three of the four schools of thought (Table 3.1). Table 3.1 Comparison of factors determining PMSs: Krause and Klingebiel Four schools of thought, Krause Klingebiel • Business strategy (2nd in the framework) • Organizational structures (1st mentioned) • Processes (3rd) • Processes (2nd) • Employees (4th) • Cultures (4th)
Klingebiel does not mention financial aspects, which is prominently represented as the first school of thought. However, he additionally refers to systems (3rd), which in turn are not considered in Krause’s schools of thought. From this comparison it is clear that “schools of thought” is a useful construct for a theoretical classification even if they are intermingled in practice. As Klingebiel stated, this demonstrates that the performance management system cannot be a pure form of one of the schools of thought. The four schools of thought proposed by Krause allow for the classification of performance management and measurement approaches according to their focus and, therefore, provide our research with a framework of possible directions when trying to understand and sort the recent literature contributions. A great number of literature contributions discussed in this section highlight the relevance and importance of the topic of performance management and measurement systems in organizations today. The literature reports that performance management encompasses many different aspects. It ranges from generic suggestions for holistic approaches to corporate performance management4 to very specific strategic recommendations on how to develop or implement performance management in practice along with dashboards, tools and supporting guidelines.5 On the other hand, many academic contributions have been published with different foci on, for example, specific functions or industries like: • • • • •
4 5
Marketing (Reinicke 2004); Procurement (Bode 2008); Healthcare (Bevan); Automotive (Lampl 2009); Public management (Halligan et al. 2010), etc.
Krause (2006), Oehler (2006), Nyiri (2007), just to name some of the recent ones. Jetter (2004), Cokins (2009).
58
3 State-of-the-Art in Performance Management
Generic, well-known performance management concepts are reported by authors such as: • • • • •
Kaplan and Norton (1992); Lynch and Cross (1995); Gleich (2001); Gladen (2002); Krause (2006).
Since these literature sources attempt to address as many elements of PMgS6 as possible, they should therefore, according to our definition (see Sects. 2.2.4 and 2.2.5), be assigned to performance management approaches. Whereas contributions from Klingebiel (1999, 2000), Wettstein (2002), Hauber (2002) are, according to the authors, considered to be performance measurement approaches, a clear differentiation from performance management is hard to see.
3.1.2
R&D Performance Measurement Systems
In this section, we move along the organizational functions dimension from generic approaches to specific issues addressing the R&D function. Before reviewing the performance measurement literature, a brief discussion of the R&D function is given to help us understand the specifics of the R&D function in comparison to other functions within an organization and consequently the implications for performance management. As discussed, a pure form of performance management does not exist in general, and this fact is even more pronounced in R&D. In the following review, the difficulties of R&D management and measurement are discussed and these show that the pure form of performance management is not applicable. As it follows from the Sect. 2.2.5 where the definition of the “R&D” term has been described, the R&D within a company can have the function of an incubator for new or future business opportunities.7 The significance of R&D for industrial organizations is often described as one of the most important factors affecting future profitability.8 Having in mind the four schools of thought: (1) financing/ funding, (2) business strategy, (3) business processes, and (4) employees, we explore its applicability to R&D.
6
Four constitutive elements of a PMgS have been defined in Sect. 2.2.5: planning, measurement, analysis and review. 7 The R&D function justifies its existence within organizations due to the very dynamic markets with constant newcomers and, thus, potential competitors (increased levels of competition), continually-improving technologies (rapidly-increasing production processes and methods) and the growing transparency and fast-changing preferences of customers, Hauber (2002), pp. 26–34. 8 BCG (2003), p. 5.
3.1 Performance Management Systems
59
R&D is generally meant to prepare a company for the “next generation”, be it the next generation of products, services, customers, competitors, partners or employees. However, how does the management and measurement in the R&D function differ from in the other internal functions? The nature of experimentation, which is inherent to the R&D function, explains this best. Researchers have to “try it out”; they do not know what the result will be and might even have little idea about the desired result and its effect on the planned result. This experimental modus operandi shows that the schools of thought fail in R&D; for example the business processes school of thought is very difficult to apply due to the nonrepetitive nature of the experiment. This uncertainty makes it difficult to plan, manage and measure the activities. The financing school of thought is hard to apply to R&D in the ICT sector. This is because outputs lack tangibility and perceptions about their monetary value are fuzzy. The quantity (of outputs) as such, which is often an important criterion for other departments, is of little importance for assessing R&D. A pre-condition for calculation of profitability, which sets inputs in relation with outputs, is the quantifiability of the outputs. Obstacles to the quantifiability of R&D outputs are reported in the literature9: • Mapping difficulties: although expenditure on R&D is directly linked to future earnings, it is practically impossible illustrate the detail of how each Euro spent on R&D contributes to the company’s income. • Measuring the output: an appropriate methodology for measuring and evaluating R&D output does not yet exist. The output of an R&D project may be the development in the form of hardware and software, of a patent or an engineering drawing. An R&D output could also be the knowledge that a desired solution is not feasible. This output cannot always be assigned to a subsequent product. • Periodical accounting: the typical timing of the R&D projects’ results means a delay between the expense and income dates, and within the classical, periodbased cost accounting system, no direct revenue can be assigned to current expenditure.10 • Linkages with other functional areas: the comparison of the costs of an R&D project and the associated cash flows of the product only allows limited statements about the value-added contribution of R&D. Success within the value-added R&D phase is a necessary, but not sufficient condition, for business success. Assigning economic success to indirect operational areas is an unsolved problem in industrial economics.
9
Hauber (2002), p. 41. Note that author acknowledges that his work applies predominantly to development (D); nevertheless, he uses the term “R&D” because, according to Hauber, on the one hand, the basic philosophy applies also to research, and on the other hand, the strict distinction between research and development is not appropriate. Hauber 2002, p. 24. 10 This problem is addressed in the literature under the term “time lag“. Schainblatt also observes that the time lag between fundamental discoveries and new industries can be from 15 to 50 years.
60
3 State-of-the-Art in Performance Management
Another characteristic of R&D is that R&D is “often a shared work for which it is quite difficult to assign a meaningful output to any one individual”.11 For over 90 years, practitioners and academics have devoted their thoughts to the matter of measurement of work performance. The question of measuring R&D productivity remains very controversial. In 1988, Brown and Svenson found that the major problem of R&D performance measurement is its acceptance. In their study, they state some of the reasons why assessment of R&D is difficult: • Many scientists and engineers think that it is impossible to effectively measure R&D productivity. In fact, the very fact of measurement is thought to discourage creativity and motivation among high-level professionals. • Many feel that management should just have faith that R&D is a good investment without trying to measure it. • Scientists and engineers feel negatively about measuring R&D performance for fear that such systems may expose their own inadequacies and lack of productivity. • Many attempts to measure R&D performance have resulted in dismal failure. These failures have led many to believe that all measurement systems don’t work.12 Kerssens-van Drongelen states three major problems of R&D measurement: • It is difficult to accurately isolate R&D’s contribution to company performance from the other business activities because it is always the intertwined efforts that eventually result in outcomes in the marketplace. • There is a problem of matching specific R&D inputs (in terms of money or manhours) and intermediate outputs (research findings, new technologies, materials, etc.) with final outputs (including new or improved products and processes). • The time lag between R&D efforts and their pay-offs in the marketplace represent the third major problem.13 Previous approaches to performance management in R&D have a common defect: they deal only with selected aspects while trying to assess the performance of R&D. A similar observation is made by Hauber (2002): “None of the models fully succeed in encapsulating R&D performance as none of them take all relevant factors into consideration”.14 Kerssens-van Drongelen also notes that there is no confirmed theoretical model for a performance measurement system. The question of the characteristics of the various dimensions of performance measurement is especially important in order to understand all of the influencing factors. Questions such as, “what is the measured object?”, “what kinds of measurement levels are
11
Schainblatt (1982), p. 10. Brown and Svenson (1998), pp. 31–32: following five reasons of failure have been identified: too much emphasis on internal measurement; too much focus on behavior; measuring outputs of questionable value to the organization; measurement system is too complicated, and measurement system is too subjective. 13 Kerssens-van Drongelen (2001), pp. 9–11. 14 Hauber (2002), p. 119, Gaiser and Servatius (1990), p. 13, Zeidler (1986), p. 321. 12
3.1 Performance Management Systems
61
there?” and “what are the (content) areas for measurement?” are important and have to be clarified before going further in setting up a performance measurement system and deciding on performance measures. A widespread view is that process improvements are the only levers for increasing performance.15 Few articles in the performance management literature were found that summarize the range of performance measurement dimensions. Often, individual aspects are singled out and discussed in isolation without considering the overall relationship. Contributions focusing on isolated objects can be found, and discussions distinguishing possible organizational levels are especially prevalent. For example, Ranftl (1977), p. 70, reports on five possible R&D organizational levels: an individual, a group, a major design, an entire program, and a larger organizational segment; and Griffin and Page (1993) discuss a single product, a program, and the total firm. Patterson (1983), Robb (1991), Szakonyi (1994) focus on an R&D department, while Kerssens-van Drongelen (2001), p. 55, gives an overview of measurement subjects including combining functional and departmental views. Hauber (2002), p. 158, develops three performance measurement layers: R&D department, product projects and projects without a concrete link to the market. Lynch and Cross (1995) presented a four-level pyramid, which makes a link between strategy and operations by translating strategic objectives from the topdown and measures from the bottom-up. These four levels are: corporate, business units, business operating systems, and departments and work centers. An attempt to summarize the different dimensions of R&D performance analysis has been undertaken recently by Ojanen and Vuola (2006).16 The authors present five basic dimensions of R&D performance analysis, which are given in Table 3.2. Table 3.2 The basic dimensions of R&D performance analysis Measurement The purpose of Measurement perspectives measurement level R&D type
Process phase
FOR WHOM? Customer Internal
WHAT? Basic research Exploratory research Applied research
WHEN? Input In-process
Product development Product improvements (incremental)
Outcome
WHY? Strategic control Justification of existence Benchmarking
Financial, shareholders Other stakeholders Resource allocation
WHERE? Industry Network Company SBU/department
Process Development of activities/problem areas Others Motivation, rewarding Project, team Individual Source: Ojanen and Vuola (2006) Learning
15 16
Hauber (2002), p. 119, Hronec (1993). Ojanen and Vuola (2006), pp. 279–290.
Output
62
3 State-of-the-Art in Performance Management
The first dimension addresses different measurement perspectives on strategy and objectives. Ojanen and Vuola list the following possible stakeholder groups: customer, internal, financial shareholders, other stakeholders, learning, and others. Beck and V€olker (2009)17 present a research and R&D-specific stakeholder analysis. The second dimension represents the purpose of R&D performance analysis. Strategic control, justification of existence, benchmarking, resource allocation, development of activities/problem areas, motivation, rewarding, and others are assigned to this category.18 The third dimension distinguishes different organizational levels of R&D performance analysis. The following levels have been identified: industry, network, company SBU/department, process, project, team, and individual. The type of R&D is the fourth dimension suggested. A distinction has been made between basic research, exploratory research, applied research, product development, and product improvements (incremental).19 The last dimension is focused on the phase of the R&D process to be measured. Input, in-process, output and outcome are the proposed characteristics for this dimension. This set of dimensions is one possible way to structure an overview of the different measurement perspectives. However, these examples are sometimes misarranged. For example, the “learning” of the first dimension implies that a second dimension has the purpose of measurement. The dimensions-table concept presented by Ojanen and Vuola helps to structure the multi-dimensional, multicriteria and multi-person task of measure selection, which is often difficult to execute effectively. Specifically, the understanding and recognition of the combinations of correct measurement dimensions are of value in selecting the appropriate R&D indicators. Due to the special characteristics of R&D shown at the beginning of the section, which outlined problems related to R&D, many of them are caused by the time-lag that belongs to a fifth dimension: the process phase as per Ojanen and Vuola. Two recent publications on R&D performance measurement, which design a performance measurement system are: a thesis from the University of Twente by Kerssens-van Drongelen (2001) and a thesis from the University of Mainz by Hauber (2002). While Kerssens-van Drongelen’s work is rather theoretical, Hauber’s research originated in an automotive industry and is therefore praxisoriented. Kerssens-van Drongelen in her research develops a Performance measurement system Systematic Design Approach (PSDA). In her work she specifically enhances taxonomies of measurement system functions, R&D measurement subjects, strategic R&D control models, R&D work types, and measurement
17
E.g., externally: academia, industrial partners, political bodies and media, and internally: development department, as well as the communication and marketing departments and top management. Beck and V€ olker (2009), p. 34. 18 We presented an overview of purposes of performance management in Sect. 2.2.5, pp. 37–43. 19 Detailed differentiation between types of R&D activities is provided in the definition Sect. 2.1.3, pp. 8–15.
3.1 Performance Management Systems
63
methods and metrics. She also proposes tools and guidelines for the development of measurement systems in R&D environments. The reader is referred to her work for the detailed analysis and relevant literature sources as it does not make sense to repeat it in our work. Hauber proposes a performance measurement concept with three layers. The first layer is based on a balanced scorecard with four perspectives: finance, customers, innovation and knowledge, and employees representing the management mechanism for R&D managers. The second layer represents a subsystem for product projects.20 Two processes are central to this layer: the first process is decision-making in terms of the start, design and goal-setting of each project. This process concerns the portfolio selection and strategy finding, i.e. effectiveness. The second process is system control in terms of management and control of product projects. The quality of this process demonstrates the efficiency. Finally the third layer considers of projects without a clear link to the market. The important difference between projects in the second and the third layer is that third layer project’s have no connection with products, whereas second layer projects do. According to Hauber, performance measurement concepts often ignore the fact the choice of right strategy in the context of the project selection an important factor for the achievement of corporate goals. He states that another weakness of performance measurement concepts, is the unlinked nature of these objects. Hauber concludes that the isolated assessment of an individual R&D project without consideration of other projects or the total R&D organization is not sufficient to capture the entire performance. It is worth mentioning that it is still rather difficult to identify the instruments or concepts in the performance measurement system, which can help to capture the entire performance of the department.
3.1.3
Research (Only) Performance Measurement Systems
The approaches discussed above focus on R&D as a unified entity, i.e. R&D is seen as one single function. In practice however, especially in large corporations, research and development operate as two different organizational units.21 Therefore, the aforementioned approaches are of limited help to practitioners for
20
Note that even though Hauber calls his work "R&D performance measurement", he acknowledges the fact that his work primarily focuses on the development part. Hauber (2002), p. 24. As such, the second layer of his proposed PMsS, which focuses on product projects is typical for the development function but might not necessarily find an equivalent type of projects in research only. 21 Seiler, (1965), Mansfield et al. (1971), Leifer and Triskari (1987), pp. 71–78, Brown and Svenson (1998), pp. 30–35.
64
3 State-of-the-Art in Performance Management
developing and implementing performance management within their particular organizations (either research or development).22 Research and development units have fundamentally different purposes. This can be seen in their different activities and leads to a need for different performance indicators. The differences between the activities in the two units and the implications for performance measurement in remaining relevant to the context of each unit will be shown below. For example, a company that manufactures products differs in its overall company strategy from the one that offers services. Correspondingly, their goals differ and therefore so do their KPIs. Each component is adjusted to the contextdependent environment of the company. Thus, it is obvious that performance measurement systems differ dependent upon whether they are designed to monitor a research or a development organization. In addition, the concrete content, goals and KPIs may vary depending on the industrial sector and are tightly connected to the strategy of the company and thus have to reflect a fit to its current situation.23 In our review of the performance management and measurement literature no sources could be found reporting a holistic performance measurement system. Two case studies were found reporting on the experiences of setting up performance measurement in research. Loch and Tapper (2002) report on a performance measurement system for an industrial research group at a medium-sized diamond producer. Mett€anen’s case study (2005) focuses on the design of a performance measurement system in a non-profit research institute. Loch and Tapper present a case study where the authors implemented a performance measurement system for a research group (rather than development) because research was especially difficult to measure, thus providing a higher test hurdle. In a first step the circumstances for acceptance were created. In the next step the content of the research performance was cascaded from the company’s business strategy. This is regarded in our research work as an important element of performance management. Therefore, special focus has been attached to cascading the strategy. Unfortunately the description has been simplified in order to focus on exposition and to preserve confidentiality of the company. During this process, the type of outputs were identified, a set of performance measures were formulated (this part will be discussed later in Sect. 3.3.3), and a collection of programs (a portfolio) was defined. Both contributions will be discussed in depth in Sect. 3.3 in the context of performance measures. Table 3.3 summarizes the most relevant approaches of performance management and measurement representing organizational function and industry affiliation. The mapping to the schools-of-thought was made by considering the focus of each approach. Not every approach has been discussed in detail, neither has every approach been mentioned within this section. Some relevant approaches will be
22
Samsonowa et al. (2009), p. 158. There is no one set of measures that will remain definitive over time. Performance measures, as with the organization itself, should be flexible to change (cf. Driva et al. 2000).
23
Non-sector specific Non-sector specific Non-sector specific
Non-sector specific
General General General General General
General General General General
General General General
R&D
Beyond Budgeting
Value Based Management Shareholder Value
Individual Classification systems (GE, PIMS)
Ratios au Tableau de Bord
Balanced Scorecard (BSC)
Integrated Performance Measurement
Activity Based Profitability Analysis (ABPA) Total Quality Management Six-Sigma
Hope and Fraser (2003) Brunner(1999) Rappaport (1998) Anthony et al. (1989) Lauzel and Cibert (1959) Kaplan and Norton (1992) Klingebiel (2000) Meyer (2002) Malorny (1996) Hurry and Schroeder (2000) EFQM (1996)
European Foundation for Quality Management Award Malorny (1996) Malcolm Baldrige Award Krause (2005) Performance Management: Eine StakeholderNutzen-orientierte und Gescha¨fts-prozessbasierte Methode Szakonyi (1994) Measuring R&D Effectiveness
Non-sector specific Non-sector specific Non-sector specific
Non-sector specific
Non-sector specific
Non-sector specific
Non-sector specific
Non-sector specific Non-sector specific
Non-sector specific
General
Approach
Source
Functiona Industry affiliationb
Table 3.3 Summary of relevant performance management and measurement concepts
✓
(✓)
✓
✓
✓
✓ ✓ ✓
(✓)
✓ ✓
(✓)
✓
✓ ✓
✓
✓ ✓
(continued)
✓ ✓
✓
(✓) (✓)
Financing Strategy Processes Employees ✓
School of thought
3.1 Performance Management Systems 65
Performance Measurement in Research and Development Systematic Design of R&D Performance Measurement Systems
Hauber (2002)
Non-sector specific Non-sector specific
R&D R&D
Functiona Industry affiliationb
✓
Financing Strategy Processes Employees ✓ ✓
School of thought
Measuring R&D productivity: complementing the R&D Aerospace ✓ picture by focusing on research activities New strategic goals and organizational solutions in R&D Automotive, ✓ large R&D labs telecommunication ✓ ✓ Exploring the differences in performance R&D Aerospace, cosmetics, measurement between research and pharma-biotech, development: evidence from a multiple case chemicals study Loch and Tapper Implementing a strategy-driven performance R only Diamond producer ✓ (2002) measurement system for an applied research group R only Work efficiency non-profit ✓ ✓ Metta¨nen (2005) Design and implementation of a performance organization measurement system for a research organization Note that this list is not exhaustive, neither is it a complete listing of all sources found in relevant literatures. The Approaches are not listed in any particular order. This table presents an intended categorization of literature sources regarding the function, industry affiliation and the school of thought. a General – apply for the entire company or cross-department R&D – Research and Development as a single entity R only – Research department b Non-sector specificIndustry-sector specific
Kerssens-van Drongelen (2001) Karlsson et al. (2004) Cesaroni et al. (2004) Chiesa and Frattini (2007)
Approach
Source
Table 3.3 (continued)
66 3 State-of-the-Art in Performance Management
3.1 Performance Management Systems
67
discussed in subsequent sections as they address specific components of either organizational goals or performance indicators. In the next step we analyze the necessary characteristics of a performance management system. Normally, a requirements analysis describes aspects that are relevant for determining the needs or conditions to be met by a new or altered product. An example of this is to take into account conflicting stakeholder requirements.24
3.1.4
Requirements of a Performance Management System
Most differences between performance management systems can be explained by their initial definition.25 The initial definition determines who (stakeholders)26 are provided with what kind of information.27 This fact influences the design of processes and interfaces within a PMgS and highlights the need for a thorough requirements analysis that considers all intricacies and specifics for a performance management and measurement system. Many isolated literature sources can be found reporting on different requirements, which constitute possible complementary elements of performance measurement systems. In the following, we analyze the four most recent sources28 that present requirement catalogues to get an understanding of the components that are crucial for a performance management and measurement system. • A comprehensive performance measurement study by Klingebiel from 2000;
24
Laplante (2009), p. 13. Requirement – something essential to the existence or occurrence of something else, MerriamWebster’s 11th Collegiate Dictionary (2004) [requirement]. A requirement is a singular documented need of what a particular product or service should be or perform. It is a statement that identifies a necessary attribute, capability, characteristic, or quality of a system in order for it to have value and utility to a user, Wikipedia, 17.02.1020 [http:// en.wikipedia.org/wiki/Requirement] Requirements try to describe the result, in our research the result is a performance management system. In this chapter we describe what the PMgS should do. 26 Hence each and every organization has its unique ecosystem, and its own definition of stakeholder groups. For a listing of possible stakeholders in a typical organization see Hahn (1994), p. 62. 27 Since the interpretation of performance management varies greatly, see Sects. 2.2.4 and 2.2.5, it especially differs in the definition given by different stakeholder groups, which are to be provided with information. As such, the design of interfaces also differs. 28 We are aware of the fact that the four studies we have chosen for the analysis of the PMS requirements focus on different units of analysis. This is not a disadvantage for our examination as they study essentially the same phenomenon. Moreover, the examination of the same phenomenon in different units of analysis and on different abstraction levels, helps to produce a comprehensive view. 25
68
3 State-of-the-Art in Performance Management
• A dissertation about performance measurement with a focus on IT-design by Wettstein from 2002; • A study on performance measurement in R&D by Hauber from 2005; • A study encompassing a holistic performance management approach by Krause from 2006. Klingebiel analyzes and compares relevant conceptual contributions to performance measurement and examines which internal and external company (stakeholder) information need is relevant. An important part of the study is the requirements framework for a performance measurement system. According to Klingebiel, a framework of a performance measurement system should be oriented to the following requirement: to use patterns of thought, which help to identify complex operational dependencies in their multifaceted interfaces and to design optimal processes.29 In his requirement catalogue (cf. Table 3.4) for performance measurement systems, Klingebiel30 differentiates between eleven requirements for the structure of a PMS and twelve requirements for selected indicators. Wettstein31 introduces a catalogue with 45 requirements for a PMsS. Since Wettstein focuses on the integration of a PMsS with an IT-system, he distinguishes two parts: (1) requirements for the PMsSs as such, and (2) requirements for ITsystems supporting the PMsS. He classifies the requirements of the first part into five groups: performance indicators, goals, organization, processes, and operating efficiency. Within the second part, Wettstein distinguishes between: considered entities, functional, not functional, and other requirements32 (Table 3.5). Hauber focuses on performance measurement in research and development and distinguishes, on the one hand, between purposes, and on the other hand between requirements, which are split into formal and contextual requirements33 (Table 3.6). Krause examines performance management concepts and analyzes requirements for a management method, which he sorts into four clusters. The suggested requirement clusters, with 20 requirements in total, are: overall concept, knowledge base, procedure model, and software support34 (Table 3.7). When comparing these four requirement catalogues, it becomes evident that Klingebiel, Hauber and Krause suggest requirements that focus on a generic management approach, while Wettstein presents requirements on a more detailed level. Although Wettstein’s requirements do not consider the specifics of
29
Klingebiel (2000), p. 298. Klingebiel (2000), pp. 34–35. 31 Translated from Wettstein (2002), p. 105. 32 Please note that we have considered in our classifications the requirements which are businessoriented (WET01 – WET 17); remaining requirements . WET18 – WET45 describe the IT-system requirements and were omitted, since the software integration is out of the scope of this work. 33 Hauber (2002), p. 109. 34 Krause (2005), p. 109. 30
3.1 Performance Management Systems
69
Table 3.4 Catalogue of requirements for a “Performance Measurement System” according to Klingebiel The performance measurement structures KLI01 Require a company-specific adaptation with a tight dependence on strategic direction to fulfill the delivery of information KLI02 Are an integral part of an aligned and self-consistent information supply of a company’s different organizational units and levels based on performance indicators KLI03 Enable the deduction of actions to improve the current performance situation (no scoreboard) by means of the structure of performance indicators KLI04 Are targeted at continuous improvement, which, in individual cases, could also be an abrupt improvement, especially with regards to the elimination of non-value-added activities KLI05 Are targeted at the coordinated balance of relevant stakeholders’ interests KLI06 Build a system of performance indicators that are oriented according to specified areas of responsibility KLI07 Consider the independent information needs of decentralized units KLI08 Feature, in private sector companies, a system-oriented, enterprise value-oriented holistic architecture KLI09 Are targeted at strategic information demand according to value chain KLI10 Incorporate constantly changing information needs in terms of lifecycle KLI11 Have an evolutionary system focus The selected performance indicators KLI12 Focus on the most important aspects and, therefore, are few in number KLI13 Build a consistent and complementary arrangement in terms of content with monetary/ non-monetary orientation KLI14 Are primarily aimed at improving future performance (e.g. emphasis on innovation and learning) KLI15 Have an intentionally high stability throughout the reporting KLI16 Derive the rhythm of their reporting mainly from the corporate level KLI17 Support the evidence of recent methodological approaches’ success through a high process orientation KLI18 Are characterized by good communicability, which strengthen self-motivation KLI19 Are coupled with a parallel system of incentives and support personal motivation KLI20 Take into account the competitive requirements of cost, time and quality KLI21 Represent the current/expected performance in the core competencies KLI22 Also include, after critical examination, increasingly recent financial-oriented target values (e.g. EVA,a shareholder value) and their value drivers KLI23 Consider behavior-related aspects during the selection process of performance indicators Source: Translated from Klingebiel 2000, pp. 34–35. In order to assure an accurate mapping we classified each requirement by numbering it sequentially after the first three letters of the author’s name. This procedure was applied to all four sources under consideration a Economic Value Added, the value of an activity that is left over after subtracting from it the cost of executing that activity
companies in a particular sector, or in a specific function, his concept considers organizational goals and links them to performance indicators. Even though Hauber designed an approach for a specific R&D function, his requirements are nevertheless of quite a generic nature. Examples of this include: creation of transparency, decision support or motivation within the purpose; simplicity, operability and
70
3 State-of-the-Art in Performance Management
Table 3.5 Requirements for a “Performance Measurement System” and for “IT-System for a Performance Measurement System” according to Wettstein Performance indicators WET01 Consideration of • Financial and non-financial • Leading and lagging • Internal and external performance indicators WET02 Structuring according to impact, execution and premise indicators WET03 Indicators are operationalized goals WET04 Set target (target value of a performance indicator) Goals WET05 Are operationalized by means of indicators WET06 Are derived from the strategy WET07 Only those goals are considered which have a stakeholder Organization (embedding, structure, involved parties) WET08 Goals are connected with organization and employees WET09 Individual structure can be established WET10 PMS can be integrated into existing management systems WET11 Administration of performance relevant data is carried out centrally Processes WET12 For the creating of a PMS WET13 For the acquisition of company internal and external data WET14 Stakeholder communication of the performance relevant data WET15 For the development of the ex ante causal relationships and for their ex post review WET16 For the maintenance (review) of the PMS Operating efficiency WET17 Verifiability of the cost-benefit situation of a PMS Considered entities WET18 Performance indicators WET19 Goals WET20 Company organization structure WET21 Employees WET22 Stakeholder WET23 Goal targets (required values) of performance indicators for a certain point in time or period WET24 Causal relationships Functional requirements: Data acquisition WET25 Manual entry of data WET26 Internal and external data sources WET27 Appropriate mechanisms for the automatic generation of performance-relevant data from operational systems WET28 Coupling with external sources Data storage WET29 Centralized, uniform data structure WET30 Storage of performance indicators at different granularity levels WET31 Set goal targets WET32 Long-term storage of performance data (continued)
3.1 Performance Management Systems
71
Table 3.5 (continued) WET33 Performance data should be available to other management systems within the company Data analysis WET34 Tools for analyzing the cause-effect chains WET35 The system enables the calculation of trends WET36 The information system calculates performance gaps between actual and target values Data communication WET37 Free navigation in the data structure WET38 Stakeholder-oriented communication of performance-relevant data WET39 Representation of the performance gap Not functional requirements WET40 Security concept, high data security WET41 Based on a standardized technology (e.g. Internet) WET42 Good response times WET43 Good expandability and scalability WET44 High availability and short recovery time Other requirements WET45 Far-reaching independence from the applied performance measurement approach Source: Translated from Wettstein (2002) Table 3.6 Requirements for a “Performance Measurement System” according to Hauber Purposes of performance measurement systems HAU01 Creation of transparency HAU02 Decision support HAU03 Cybernetic model HAU04 Motivation Formal requirements HAU05 Simplicity and operability HAU06 Comparability HAU07 Objectivity of performance indicators to influence the results, and not the behavior HAU08 Reliability to avoid dysfunctional impact HAU09 Acceptability by affected people Contextual requirements HAU10 Relevance, focus on KEY performance indicators, congruency with corporate goals HAU11 Focus on the future in terms of planning effectiveness (enabling early-warning) HAU12 Multiple dimensions, consideration of not just a single goal but rather a bundle of goals of a department and their performance indicators HAU13 Multiple levels, consideration of the corporate levels of a company Source: Translated from Hauber (2002)
comparability within formal requirements; KPIs-focus, consideration of multiple levels. These listed requirements are not function-specific and can be applied in a generic approach. Generally, performance measurement systems should support individual organizations in their individual environments. This often requires a definition or adjustment of context-related artifacts; e.g., within his knowledge base cluster, Krause subsumes among others the provisioning of techniques and templates,
72
3 State-of-the-Art in Performance Management
Table 3.7 Requirements for a “Performance Management Method” according to Krause Overall concept KRA01 Holistic approach KRA02 Solution integration KRA03 Openness Knowledge base KRA04 Represent content and structure of existing PMS KRA05 Provide existing techniques and templates KRA06 Indicator catalogue KRA07 Reference models Procedure model KRA08 Task-related PMS development KRA09 Method to capture actual profile and to develop target profile KRA10 Method for cause and effect net transparency KRA11 Method for process-oriented indicator definition KRA12 Indicator data model KRA13 Creation of PMS triangular structure KRA14 User participation via reverse flow principle Software support KRA15 Knowledge management in terms of experience backup KRA16 Knowledge use KRA17 Development of new PMS KRA18 Alignment of PMS content and structure KRA19 Analysis about PMS content and structure KRA20 Representation of the value chain Source: Translated from Krause (2005)
indicator catalogues, or reference models. Such information, which sometimes may simply not be available and have to be generated, is needed in order to populate a general performance framework to make it work for a specific organization or department. By analyzing the four requirements catalogues a synthesis can be made. Within all approaches the following four main areas can be identified: 1. The organizational structure, design and embedding of a performance management system; 2. Processes required within a PMgS; 3. Goals; 4. Performance indicators. Each area is described by a number of different requirements, summarized from Klingebiel, Wettstein, Hauber, and Krause (Table 3.8). The first area is concerned with the overall performance management characteristics: the holistic approach, adherence to company strategy, integration with the overall existing management systems, balance of relevant stakeholders, and the existence of templates, techniques and catalogues. The second area is process-related, indicating the importance and need for well-defined processes. The third area describes
3.1 Performance Management Systems
73
Table 3.8 Our classification of requirements for a “Performance Management System” Condensed requirements
Literature sources
PMS embedding and design
Klienge- Wettstein biel (WET) (KLI)
General characteristics: A PMS needs to be: • Derived from and aligned with the overall company strategy • Built on a holistic approach • Open and flexible for potential changes/ enhancements Specific PMS components: A PMS needs to include: • Reference models • An indicator catalogue and data model • Existing best practices, techniques and templates Model to capture and analyze cause and effect relationships: This model needs to allow for: • Identifying measures/potential for continuous improvement • Coordinating and balance interests of relevant stakeholders • Reviewing the effectiveness of the PMS itself • Evaluating the efficiency of the PMS itself (including cost-benefit analysis) The PMS needs to be coupled with a reward system The PMS needs to be integrated with: • Information demand according to the value chain • Existing management systems • Existing organizational structure(s) (e.g. decentralized units) • Existing tasks and business processes • Existing IT infrastructure PMS processes are required for Capturing company internal and external data Communicating performance data to stakeholders Capturing as-is profile and to develop to-be profile Ensuring user participation Creating, reviewing and evolving the PMS itself (including goals and performance indicators) Capturing and analyzing cause and effect relationships Goals have to Be derived from the strategy and reflect core competencies Be operationalized by means of indicators Have a stakeholder Be tied to organizations and employees to ensure responsibility Performance Indicators Are operationalized goals
KLI01 KLI08
Hauber (HAU)
Krause (KRA)
HAU01 HAU05
KRA01
HAU06 HAU09
KRA02 KRA03 KRA05 KRA06 KRA07 KRA12
KLI04 KLI05
WET15
KLI10
HAU03
KRA10
HAU08
KRA14
KLI19
KLI09 KLI07
WET07 WET08
HAU12
KRA13
KLI17
WET09
HAU13
KRA08
HAU09 HAU07
KRA11 KRA09
WET11 WET12
KLI11 KLI10
WET12 WET14 WET13
KLI21
WET06 WET05 WET07 WET08
KRA14 KRA11 HAU08
HAU02 HAU10
HAU09
WET03
(continued)
74
3 State-of-the-Art in Performance Management
Table 3.8 (continued) Condensed requirements
Literature sources
Focus on the most important aspects/critical success factors Measure the current/expected performance of core competencies
KLI12
Are low in number/comparable
Refer to results and not behaviors Have good communicability Are continuously part of periodical reporting of the department/company Consider measuring the following aspects: • Financial values (cost, EVA, shareholder value) • Non-financial values (time, quality) Consider measuring the following perspectives: • Leading and lagging • Internal and external performance indicators • Impact, execution and premise indicators Allow actions to be derived to improve current performance Strengthen self-motivation and guide behavior
KLI06 KLI13 KLI21 KLI12
HAU02 HAU08
KRA13
WET04
HAU01 HAU06 HAU07 HAU08
KLI18 KLI02 KLI15 KLI16 WET01 KLI20 KLI22 WET01
HAU11
WET02 KLI03 KLI13 KLI18 KLI19 KLI23
KRA13 HAU04
requirements for organizational goals; and finally, the fourth area summarizes the requirements for performance indicators, describing their qualities within performance management systems. When allocating different requirements from the four catalogues, the mapped source has been sorted to the principal item in each category. Table 3.8 gives, on the one hand, a comprehensive overview of fundamental components (i.e. four areas) that a performance management system should have. On the other hand, it describes qualities and characteristics of these components by means of the collected requirements.
3.1.5
Characteristics of Research Compared to Development
After having analyzed the requirements of performance management, we will examine specific requirements for R&D in the next step and then for research separately from development as per the focus of this work. First of all we assess the specifics of industrial research reported in the literature in order to derive and formulate requirements specifically for this function. In this context we
3.1 Performance Management Systems
75
consequentially discuss the current state-of-the-art in the literature regarding the separation of these two functions. The Research and Development functions within an organization have traditionally been treated as one single function called R&D. For assessment purposes, these functions need to be separated as will be shown in the following discussion. By assessing a research department’s performance we are better able to understand the relationship between the individual components that are assessed and how they impact the organization as a whole. This requires both knowledge about research activities that directly influence the department’s performance and knowledge about factors that affect its ability to produce significant results. The issue of assessing the performance, especially for R&D in industry, is a difficult one, and it becomes even more complicated when moving to the very beginning of the value chain within a company, which often starts in its research department. The need to treat research and development separately has been identified and proposed long ago. For example Brown and Svenson35 analyzed the reasons for failure of R&D measurement and evaluations systems and derived five reasons from which they derived six recommendations36 for the design of a workable PMS. Their sixth recommendation is to separate the evaluation of research and development. They argue that: “Research and development clearly perform different functions and produce quite different outputs”. The authors define: “primary research output” as information or knowledge relevant to the business of a company; and “development output” as enhancements, or completely new products and processes. The authors suggest that research output is an input for development and vice versa. Because of these major differences, the authors strongly recommend to track different KPIs for research and development. The authors suggest some examples for contrasting measures,37 however they do not mention goals or objectives related to these measures. The authors explicitly emphasize that the functions must be considered separately, “. . .research and technology development need their own measures, which are different from the measures of product development”. The managerial implications of the Leifer and Triskari study suggest that both units should not be managed or controlled in the same way. The differences stem from their contextual, organizational and communication characteristics. The results suggest different orientations inherent in performing research versus development activities. “These results suggest that . . . research and development units should not be lumped together either in research or in the management literature”.38 Surprisingly, this suggestion of separation has only recently been followed by
35
Brown and Svenson (1998), pp. 30–35. The following six recommendation have been made on how to design a PMgS: (1) focus on external vs. internal measurement; (2) focus on measuring outcomes and outputs, not behavior; (3) measure only valuable accomplishments/outputs; (4) make the measurement system simple; (5) make the measurement system objective; and (6) separate R&D evaluation. 37 The term “Contrasting” refers to the different functions for which the measures are relevant. 38 Leifer and Triskari (1987), p. 76. 36
76
3 State-of-the-Art in Performance Management
researchers. Loch and Tapper (2002), Karlsson et al. (2004), Mett€anen (2005), Chiesa and Frattini (2007) have all investigated the differences between the departments or in particular the research function within an industrial environment. Karlsson et al.39 attempt to define areas of research and development to demonstrate why their performance should not be jointly measured. They observe that Research in industrial practice is most often separated from Development organizationally. Authors cite empirical studies from Seiler (1965) and Mansfield et al. (1971) noticing that they show identical results reporting that 58% of the companies studied had separated Research from Development. Mansfield’s hypothesis was that size of expenditure was the major explanation for the cases where research is not separated from development. For example, when either research or development units are smaller than 26% or larger than 76% companies tend not to separate the two.40 With their empirical investigation Chiesa and Frattini stress the need for differentiation when designing performance measurement systems according to the type of measured activity. The authors, however, do not investigate the concrete characteristics in terms of contextual goals and performance indicators of both types of units. The individual assessment of research departments was, for a long time, regarded as an obstacle since creativity is seen as a major part of the work of researchers.41 Accordingly, KPIs assessing research activities are a highly controversial topic of discussion.42 To better understand the particularities industrial research the following extracts the characteristics of industrial research departments from the literature. The first characteristic identified is: the nature of activity realized in research. Leifer and Triskari43 analyzed the main differences between research and development and derived the main disparities from the tasks and activities of research and development units.44 The authors note that research tasks consist of technology expansion and are primarily concerned with expanding scientific knowledge and assessing its feasibility. In this sense, technology expansion places an emphasis on the inventive process, that is, expansion of a technology base. Development tasks, on the other hand, involve the application of the technology base to operational requirements with the intent of bringing a new (or modified) product into existence. Research tasks are aimed at generation or expansion of knowledge whereas development tasks have as their objective the physical output of new products or processes.45 Similar to Leifer and Triskari’s approach, Karlsson et al. suggest that the activities of both functions to be artifacts to examine: “It is our
39
Karlsson et al. (2004), pp. 179–186. Karlsson et al. (2004), p. 180. 41 Refer to Schainblatt (1982), p. 10, Brown and Svenson (1998), pp. 30–35. 42 The examination of KPIs within industrial research (only) will be addressed in Sect. 3.3.3. 43 Leifer and Triskari (1987), pp. 71–78. 44 We are aware of the military setting of this research, nevertheless the hypotheses examined in this study are perceived as valid for our considerations. 45 Leifer and Triskari (1987), p. 71. 40
3.1 Performance Management Systems
77
belief that research activities, according to these variables,46 are very different from development activities, making it hard to measure R&D as a unified entity”.47 Another very important and defining feature of the research function is the time lag between when outputs are produced in research and then outcome which represents the collective efforts of many different units of a company at the end of the value chain. This feature, in our opinion, applies to both research and development functions. Loch and Tapper48 state: “. . .success can be assessed only after long delays, or it accrues to other units of the organization”. A very generic and comprehensive illustration of the performance measurement process can be found in Brown and Svenson.49 They describe a model for R&D (see Fig. 3.2) where they introduce the R&D laboratory as a system within the macrosystem of the entire organization.
2. PROCESSING SYSTEM
• • • • • • •
People Ideas Equipment Facilities Funds Information Specific Requests
4. RECEIVING SYSTEM
R&D Lab
1. INPUTS
Activities • • • •
Researching Developing Testing Reporting Results
A A) IN-PROCESS MEASUREMENT AND FEEDBACK
3. OUTPUTS • • • • •
Patents Products Processes Publications Facts / Knowledge
B) OUTPUT MEASUREMENT AND FEEDBACK
• Marketing • Business 5. OUTCOMES Planning • Manufacturing • Cost • Engineering Reduction • Operations • Sales Improvement • Product Improvements • Capital Avoidance
B
C) OUTCOME MEASUREMENT AND FEEDBACK
C
Fig. 3.2 The R&D Laboratory as a system (Source: Brown and Svenson 1998)
The model consists of two components: the processing system and the receiving system. The R&D lab receives inputs and transforms these inputs into outputs, which are taken up by the receiving system (consisting of other units of the same organization) that in turn produces outcomes. The actions of transforming are called activities. We refer to inputs, activities, outputs and outcomes as measured artifact types since they represent types of things that are measured. The distinction between output and outcome is the important aspect of this model. In their model
46
The authors refer to four factors: time, originality, organization and knowledge depth. Karlsson et al. (2004), p. 180. 48 Loch and Tapper (2002), p. 185. 49 Brown and Svenson (1998), p. 31. 47
78
3 State-of-the-Art in Performance Management
they also introduce measurement and feedback loops related to activities, outputs and outcomes. This illustration (Fig. 3.2) shows figuratively the time lag between A (in-process measurement), B (output measurement), and C (outcome measurement). As Brown and Svenson did not apply their model to a specific industry sector they could not give any indications as to the time of feedback loops. The next characteristic, which shows its unique legitimacy for the Research function, is the fact that outputs of research need not always necessarily be positive. Hauber50 distinguished between the following six types of outputs of R&D51: • • • • • •
Output-type 1: “product innovation” Output-type 2: “shelve product” (non-launched technology) Output-type 3: “support services” Output-type 4: “technical know-how” Output-type 5: “competencies/expertise” Output-type 6: “know-how”
Research that attempts to find solutions for a particular problem fails for nonavoidable reasons. This can be called a negative research output: something that should have worked does not. According to Prechelt52 due to the current publication climate such negative results are usually camouflaged as positive results by nonevaluating or mis-evaluating the research or by redefining the problem to fit the solution. He indicates that such publication behavior hampers the progress by suppressing some valuable insights, producing spurious understanding and misleading further research efforts. The validity of positive as well as negative outputs needs to be accepted as equally incorrect. One prominently discussed characteristic in the literature is the uncertainty factor in the research. The research outcome is uncertain to result in economically successful products or services. Feltham and Xie53 highlight in their work the uncontrollability and un-observability and name three factors: first, the actions and strategies implemented are not observable directly – the direct compensation for the input into the firm is therefore not possible; second, the full consequences of managers’ actions are not observable – because the impact of those actions extend beyond his subunit and time as its manager; and third, uncontrollable events influence the consequences that are observed. Loch and Tapper54 state same facts like Feltham and Xie: “. . .effort levels may not be observable, project success is uncertain, influenced by uncontrollable factors. . .” Slightly different aspect of uncertainty (not results but rather the environment) were analyzed by Leifer and
50
Hauber (2002), p. 207. Note that not all R&D output types are also applicable to research. Instead of type 1 and 2 research, outputs could be concepts or feasibility studies. 52 Prechelt (1997). 53 Feltham and Xie (1994), pp. 429–453.s 54 Loch and Tapper (2002), p. 185. 51
3.1 Performance Management Systems
79
Triskari.55 They operationalized uncertainty using the environmental uncertainty component measures of complexity/dynamism and predictability/controllability. Different stakeholders and different communication methods is the next characteristic of industrial research function identified in the literature. Beck and V€olker mention that compared to other functions, research seems to interact with different stakeholders.56 Depending on the type of projects,57 research departments might interact differently to other units with external parties like academia, industrial partners, political bodies and media. Also, interaction with internal units, foremost with the development department, as well as the communication, marketing and top management is notable. Heterogeneous stakeholder groups require special communication of research topics.58 Leifer and Triskari show very surprising findings. They found that development units coordinate more with other organizational units than research units do and that members of development units process more information to and from people outside their own unit than do members of research units.59 This can only be explained due to the military environment and timeframe of the study. It should be mentioned that contemporary research suggests that the organization and execution of research projects is influenced by new trends such as open innovation. The next research characteristic is originality. It is not meant to describe the nature of topics and problems dealt with, as by definition research tackles new and unsolved problem. Moreover, Karlsson et al.60 use “originality” to illustrate the differences in terms of continuity/discontinuity. They cite Stahl and Steger61 concerning the fact that research often makes discontinuous “jumps” in knowledge, which results in solutions that can be called innovations. “The innovations are not aimed at a specific product, but merely display the possibilities within a technology area.”62 Development on the other hand, is a more continuous evolution of existing ideas. The next characteristic identified is knowledge depth. Karlsson et al.63 state that the knowledge regarding the specific area of interest has to be deeper in the case of research. General and specific competences are distinguished by the knowledge
55
Leifer and Triskari (1987), p. 73, note especially their second hypothesis. On both these measures as well as an overall measure of environmental uncertainty there was no statistical difference between the Research and Development units, meaning that both research and development groups perceived the environment similarly. The authors, however, did not examine the uncertainty of the output artifacts of each unit. 56 Beck and V€olker (2009), pp. 28–35. Amelingmeyer (2005), p. 353. 57 For example public funded projects, depending on a form, can include different disciplines like academia, industrial companies, standard bodies, etc., and can be comprised from more than 20 different organizations. 58 Beck and V€olker (2009), p. 35. 59 Note that the sixth hypothesis was not commented here as it was not relevant to our research. 60 Karlsson et al. (2004), pp. 179–186. 61 Stahl and Steger (1977), cited from Karlsson et al. (2004), p. 181. 62 Karlsson et al. (2004), p. 181. 63 Karlsson et al. (2004), p. 182, Roussel et al. (1991), cited from Karlsson et al. (2004), p. 182.
80
3 State-of-the-Art in Performance Management
depth aspect. The need for specialists, with highly specialized knowledge, is greater in research, and the need for generalists, with cross-functional understanding, is greater in development. Structure and processes have been identified as one characteristic of research. Karlsson et al. examine how the type of activities in research and development are organized. They conclude that research projects are often conducted in semi-controlled chaos in a uni-disciplinary environment. Development, on the other hand, is often conducted in project form and in a multi-disciplinary environment. Leifer and Triskari examined, in their third hypothesis, whether inter-unit dependence for development units is higher than for research units.64 The authors found inter-unit dependence in research units was significantly greater that in development units. Furthermore, they found that research units were significantly more centralized than development units and that there was partial support for the hypothesis that development units are more loosely structured. The authors do acknowledge, however, that the military setting of the research is of fundamental importance. Chiesa and Frattini analyzed and compared the structure of the PMS in research and development departments. Their empirical investigation shows that the design of the structure of a performance measurement system is significantly influenced by the type of activity it is applied to. They found that in research activities, typical control objects are single researchers, research projects and the scientific or technical department. Conversely, development departments do not apply individual and department measures, but tend to apply teams and projects as controlled objects. Furthermore, the authors stress that research is generally structured according to an input-oriented approach favoring high specialization in knowledge production, whereas development is organized mainly on the basis of output-oriented solutions. The authors conclude that in terms of structure the departmental dimension is more relevant for research, while the team dimension is more important for development. In their investigation, the authors notice a wide spread of matrix structures that reconcile the advantages of input- and output-oriented structures and therefore the project dimension is critical in both research and development functions. These above mentioned characteristics, and especially when we consider the time-lag and output vagueness of research, mean that the research function has to be seen as an investment without tangible results. A key feature of the inventive process is that the result will not be known ahead of the output. It is often difficult to define, describe, and specify the artifact which is expected. In a commercial environment “the view of what the expected output is from a firm’s research activities varies from company to company, as well as within a company over a period of time, much due to market developments and changing customer demands.”65 Karlsson et al. note that the question of “expected output”
64
Inter-unit dependence was operationalized with questions pertaining to the extent that work units depend on others to accomplish work objectives and the extent to which others depend on them. 65 Karlsson et al. (2004), p. 180.
Description
Generally the tasks of researchers are different from development because the researcher is more oriented toward theoretical knowledge whereas the developer is repeating validated methods in order to engineer the product Time-lag (output vs. outcome); In research there is in general a originality significant time lag between output of research and the outcome resulting from this output. The steps turning the output into the outcome are generally not under the control of research Generally, research has the potential to generate disruptive ‘jumps’ in knowledge resulting in innovations whereas development is more a continuous evolution of existing ideas Validity of positive as well as In general the research outputs can be negative outputs positive or negative, and in both cases represent valid research results Output uncertainty (nature of output – Generally the research outcome is uncertain to result in economically complexity/dynamism/ predictability/controllability/ successful product or services. The secrecy) risks are more commercial than technical
Tasks types (routine vs. exceptions)
Characteristic
Table 3.9 Main characteristics of industrial research and requirements
(continued)
Leifer and Triskari (1987)
Hauber (2002), Prechelt (1997)
R and D Brown and Svenson (1998), Karlsson et al. (2004), Chiesa and Frattini (2007)
The PMgS has to consider both results R without penalizing the negative findings The PMgS has to reflect the fact that due R to uncertain conversion of research outcome into successful products the outcome only cannot represent the basis for conducting the performance assessment
The PMgS has to cope with the irregular generation of outputs
The PMgS has to reflect the fact that there is a significant time lag between output of research and the outcome resulting from this output
Leifer and Triskari (1987), Karlsson et al. (2004)
Function Source
The PMgS needs to consider the nature R of research tasks (more nonroutine, more exceptions, more complexity)
Requirement
3.1 Performance Management Systems 81
Investment per se
Function: R&D entity – Research and Development as a single entity R only – Research department
Definition of the output
Quantifiability of the output
Knowledge depth
Different stakeholders and different communication methods
Description
From the time-lag characteristic and difficulty to precisely describe the research output results, there needs to be clear agreement that the research function is seen as an investment without tangible results The research department, depending on the type of research projects it conducts, has a variety of stakeholders: e.g. academia, industrial partners, political bodies, media and internal units such as the development department, communication and marketing departments, and top management being the most important The profile of a typical researcher usually requires a specific competence (specialized knowledge) within an area, which takes long time to acquire, whereas the knowledge from a wide range of areas is required for a profile of a developer (general competence) The results of industrial research departments have no immediate commercial value, it is generally difficult to directly quantify monetary value of research output It is generally difficult to define in advance and therefore measure the “expected output” due to its uniqueness and lack of repeatability
Characteristic
Table 3.9 (continued) Requirement
R
The PMgS has to identify benchmarking R values for the output
The PMgS has to identify reasonable R indicators for the monetary value of research output
The PMgS has to consider the different R skill sets of people working within research
The PMgS has to consider the interactions with different stakeholders
Karlsson et al. (2004)
Chiesa and Frattini (2007)
Karlsson et al. (2004)
Beck and V€olker (2009)
Loch and Tapper (2002), Ojanen and Vuola (2006))
Function Source
The PMgS has to consider the R timeframes of research activities; initially the performance can only be assessed upon activities or outputs, not outcomes
82 3 State-of-the-Art in Performance Management
3.2 Organizational Goals
83
from research activities is a reflection on the definition of research at the company. The authors observe that if there is no clear definition at hand, there will be differences in opinion regarding what the expected output is, thus making the measurement very difficult. Table 3.9 summarizes the main characteristics of research. Every characteristic is then translated into a specific requirement for a PMgS, furthermore an indication for the function R only, or R&D together is provided. In this section we have shown general performance management approaches summarized within four schools of thoughts suggested by Krause and have elaborated on a catalogue of fundamental requirements for a performance management system. The examination of each aspect of a performance management approach is not possible within our investigation and is beyond the scope of our work. For example we exclude the investigation into the technical integration and process of a performance management concept. To stress the context-dependent aspects the subsequent sections focus on the critical components of performance management as identified in Sect. 2.3 and analyze the nature of organizational goals in more detail and thereafter performance indicators to assess the achievement of the goals.
3.2 3.2.1
Organizational Goals Generic Organizational Goals
In Sect. 2.2.2 we defined the term “goal” as an envisaged and intended future action or state, an anticipated vision of the impact of our actions66; we also clarified the role of goal setting in problem solving processes. Furthermore, we found evidence that goal creation should be seen as a process across a period of time and not as an act that occurs at a specific point in time.67 In this section we will specifically analyze the literature on the nature of generic and non-sector-specific organizational goals. It is difficult, on a generic level, to find evidence on concrete goals due to the uniqueness of the operational environment and circumstances of each organization. The literature at this level refers to goal setting, principles, systems and dimensions. Hamel (1992) noticed that organizations, due to their complexity, do not have one specific goal, but rather a bundle of interrelated goals. As such he suggested the presence of an organizational goal structure or system.68 All activities of an organization are directed to ultimately realize the future state desired in the goal
66
Schwantag (1951) cited from Hamel (1992), p. 2635, Bidlingmaier (1964) cited from Hamel (1992), p. 2635, Hamel (1992), p. 2635, Nagel (1992), p. 2626, Specht and Beckmann (1996), p. 18, Hauschildt (1997). 67 Specht and Beckmann (1996), p. 18 and p. 125. 68 Hamel (1992), p. 2635.
84
3 State-of-the-Art in Performance Management
structure. Furthermore, Hamel underlines that the goal itself is a complex issue that consists of several interactive elements and hence has an internal structure. Therefore he directs his goal system analysis towards an intra-goal structure (cf. Table 3.10) as well as an inter-goal structure (Fig. 3.5). He also mentions the “supra-goal” structure, arguing that it is necessary to examine the relationship between the corporate goal system and the corporate eco-system. Table 3.10 Intra-goal structure of corporate goals Corporate goal Substantive goal Formal goal Product 1 Product related Product 2 • Quality Product 3 • Specifications Product 4 • Reusability Product n • Dimensions Source: Hamel (1992)
Organization related Optional Mandatory • Social • Profitability • Liquidity • Legal compliance
For the analysis of corporate goals, Hamel takes the following aspects into consideration: obligatory and optional elements of a goal system, internal and external relations of a goal system, elements of individual goals and their relationships, and static and dynamic components of a goal system. For our work we will confine our investigation to the comprehension of the goal system and will examine in detail the intra and inter-goal structures. Each organization has a corporate goal, which reflects the totality of all activities. This corporate goal is, however, regarding its structure and mission, globally articulated and is not appropriate for directly controlling specific actions. It must instead be “broken down” to allow a direct change from an “existing” into a “desired” state through a more precise and concrete definition.69 Goals are then derived regularly over several corporate levels by way of a “goal hierarchy” Granger (1964) or “means-ends chain” Heinen (1966). Hamel distinguishes between two types of goals: corporate goals70 and decision goals71 that represent two opposite ends of the goal system, where decision goals are at the lowest level of the cascading chain. First, corporate goals are “substantive”72 or “formal.” Substantive goals can be focused on one single “product” or “service,” or a “product family” or “range of services.” Formal goals can be either product related goals or organization related goals.
69
Hamel (1992), p. 2638. “Unternehmensziele“, Hamel (1992), p. 2638. 71 “Entscheidungsziele“, Hauschildt (1970), p. 551, Hamel (1992), p. 2638, Hauschildt (1997), pp. 269–272. 72 “Sachziel“, Kosiol (1966), or “Leistungsziel“ Schmidt (1969). 70
3.2 Organizational Goals
85
Second, according to Hauschildt73 and Hamel,74 decision goals represent guidelines for economic activity for the smallest unit: the task. These apply at all organizational levels. These generally require very specific goals in order to enable a direct transformation of the “existing” reality into the other “desired” reality. As every executable action results from a previous decision-making process, the analysis of intra-goal structures can also be focused on decision goals (Table 3.11). Table 3.11 Elements of the decision goal Decision Goal Object Specification Properties P1 P2 Pn Source: Hamel (1992)
Standard S1 S2 Sn
Function F1 F2 Fn
Following Hauschildt, the decision goals are a desired state with an intention to a realization that allows the actors to develop a specific behavior. This requires an appropriate goal articulation, which depends on a sufficient quantity of items to provoke unambiguous action sequences. Both Hamel and Hauschildt list five dimensions, which represent the elements of the decision goal. While Hamel discusses a fifth dimension, “goal complexity,” Hauschildt instead places a different dimension, “time reference” and discusses “goal complexity” on a general goal system level (Table 3.12). We will follow Hauschildt’s approach and discuss the issue of goal complexity on a more detailed level as it represents an important aspect for organizational goal setting. Corporate actions are generally determined by more than one goal. This phenomenon of ‘multiple’ goals is not problematic as long as the individual goals are complementary, i.e. as long as the positive fulfillment of one’ goal also positively fulfills the other. The conflicting goals pose a prioritization problem: how to proceed if an alternative is presented by which a sub-goal can only be accomplished to the detriment of others? In this case priorities are to be set within a goal system; Hauschild75 defines a goal system as the purposive order of several goals, including conflicting goals. • Each complementary goal can be decomposed into a series of subordinate goals via a multiple-level tree structure. Each subordinate goal contributes via a “means-end” relationship to the achievement of superior goals. • Conflicting goals require a different order form. Two concepts can be used to systematically handle conflicting goals in theory and practice: the benefit and the constraint concepts.76
73
Hauschildt (1997), p. 269. Hamel (1992), p. 2638. 75 Hauschildt (1997), p. 271. 76 Thommen and Achleitner (2003), p. 112. See also details for value benefit analysis Weber (1992), pp. 1435-1448, Hauschildt (1997), p. 271. 74
86
3 State-of-the-Art in Performance Management
Table 3.12 Dimensions of the decision goal according to Hauschildt Dimension Description Goal object Identifies a facet (not the entirety) of the reality, which is competently differentiated by the decision-maker. A goal object is represented by the decision field or rather the problem, for which the goal is defineda Goal specifications Goal objects feature individual specifications, each of which has to be separately defined. Other literature provides related terms: goal contents, goal variables, goal attributes, decision criteriab Goal norm The decision maker chooses the alternatives to be realized according their contribution to goal achievement. This contribution has to be quantifiable. What is needed is a goal norm or standard, i.e. a rule that specifies how to dimension and quantify the goal specification Goal function Each alternative which is to be considered in the decision making process provides a certain goal attainment contribution. The rule, by which the decision maker specifies which goal achievement contribution he aims for, is called the goal function Time reference Goals always reach into the future. It is necessary to determine the end date or timeframe in which the goal has to be achieved. Within this dimension, goals are to be defined as shorter-term goals or longer-term goals Source: Hauschildt (1997) a Hauschildt observes that in classical business economics the goal object applies to the entire company. He contradicts the implicit assumption that goals that are defined for the entire company apply to the individual decision fields and states that for every problem a specific goal has to be developed. However, the compatibility of problem-specific goals (decision goal) must remain compatible with the organizational goal b For example, the technical goal variables are often described as content goals and they refer on the one hand to constructiveness of a product or service (construction goal), and on the other hand to the utilization aspects (utilization goal). Economic variables are the target profit and turnover from a product or a service
Hauschildt suggests that the goal ordering axiom is goal clarity: goals are to be clearly and operationally defined in marked dimensions. He postulates the norm of goal clarity as imperative for successful management. Figure 3.3 shows the range from two goals being mutually (totally) exclusive (goal opposition) to two goals being fully complementary, so that the achievement of one goal inevitably leads to the achievement of the second goal (full goal complementary). Less extreme cases are goal competition, where the attainment of the first goal negatively impacts the attainment of the second one; goal neutrality, where goals are independent; and partial goal complementarity, where the attainment of the first goal raises the likelihood of the attainment of the second goal (Fig. 3.4). The complexity of decision goals is due to the multiple properties, norms and functions of the goal characteristics (cf. Table 3.11) The number of characteristics is considered by Hamel to define ‘goal complexity’. Hamel also points out that goal complexity could be a function of the formulation of goal characteristics and their conceptual content is therefore not absolute. In terms of inter-goal structure, Hamel in Fig. 3.5 specifies the relationship between the individual goal levels (corporate goal, division goal, decision goal) as a self-contained unit, in which on the corporate level content and formal goal are integrated into a decision goal.
3.2 Organizational Goals
87
Complementarity Relationship --
-
0
+
++
very negative
negative
none
positive
very positive
Goal Opposition
Goal Competition
Goal Neutrality
Partial Goal Complementarity
Full Goal Complementarity
Achieving one goal means complete failure in achieving the other goal.
Achieving one goal has a negative impact on achieving the other goal.
Achieving one goal has no impact on achieving the other goal.
Achieving one goal has a positive impact on achieving the other goal.
Achieving one goal necessitates the achievement of the other goal.
Instrumental relation
Preference relation
Fig. 3.3 Degrees of compatibility between two goals (Source: Thommen and Achleitner 2003)
Goal system
Goal dimensions
Content
Extent
Goal relations
Time
Competition
Main goal
Neutrality
Secondary goal
Complementarity
Superior goals Subordinate goals
Fig. 3.4 Analysis need in building a goal system (Source: Heinen 1991, p. 16)
The characteristics of decision goals that foster the best possible performance have received much attention from organizational psychologists. It is not our intention to examine this in detail, although the effects on goal-setting itself do merit attention. Organizational psychology looks at the goals’ characteristics from a slightly different angle. In his goal-setting theory77 in industrial-organizational
77
As mentioned already in section 2.2 other disciplines such as Organizational and Occupational Psychology or Organizational Behavior and Human Performance examined individuals’ performance in an organizational context and had to therefore deal particularly with goal setting. Goal setting theory has been approached by Edwin Locke in mid-1960s who had developed this theory upon Aristotle’s speculation that purpose can cause action. Locke researched the impact that goals have on the individual activity of its time performance; he continued researching goal setting for 30 years.
88
3 State-of-the-Art in Performance Management
Corporate Goal Substantive Goal Product 1
Formal Goal Product Related
Product 2 Product 3 Product 4
Organization Related
- quality
optional
- specifications
- social
mandatory - profitability
- reusability
- liquidity
- dimensions
- legal compliance
Product n
Division Goals
Specification
Object
Decision Goal Properties
Standard
P1
S1
Function F1
P2
S2
F2
Pn
Sn
Fn
Fig. 3.5 Inter-goal-structure between corporate and decision goals (Source: Hamel 1992)
psychology, Locke78 analyzes the characteristics of task performance and its effects on goals. According to Locke, hard and specific goals can lead to higher performance.79 He argues that ‘goal setting is most likely to improve task performance when the goals are specific and sufficiently challenging, the subjects have sufficient ability . . . and assigned goals are accepted by the individual.’80 Locke defines difficulty of goals as the degree of proficiency or level of performance sought, and specificity (or clarity) as the degree of quantitative precision with which the aim is specified. Three aspects of Locke’s statement are outlined in the literature81: • To achieve performance improvement, the content of each goal has to be clear so that people can carry out appropriate actions to attain the goal. If the content is not clear and it is misunderstood there might be confusion and frustration on what needs to be done.
78
Locke (1968), p. 161. This hypothesis was also proven by other authors: Champion and Lord (1982), p. 267, Yukl and Latham (1978), Locke and Latham (1990). 80 Locke et al. (1981), p.125. 81 Tankoonsombut (1998), p. 12. 79
3.2 Organizational Goals
89
• Goals should not be too easy to attain, but at the same time, they must be attainable. Setting easy goals may have no effect because people do not have to exert increased effort to reach the goals. Locke82 examines 12 studies on the relationship between goals and task performance. The outcome from the studies reveals that hard goals produce a higher level of performance than easy goals do. He claims, “The results are unequivocal: the harder the goal the higher the level of performance.” Endorsement is provided by other authors: ‘Of course, goals should not be so easily attainable that employees see them as a joke or an insult,’ ‘Goals must be challenging but within reach.83 • Goals that are not accepted may have negative impacts on task performance. The individual rejects the goal and stops trying for that goal, such as when he is assigned a goal which is perceived as impossible.84 According to Tankoonsombut85 employee involvement in the goal-setting process may lead to greater goal acceptance or facilitate greater goal commitment than if goals had been unilaterally assigned. The importance of goals and feedback in an employee’s work has also been stated by Drucker. The effects of feedback on organizational performance have been widely studied86 and the positive effects of feedback on task performance have been widely recognized as essential for learning, motivation, satisfaction and performance. Six categories of feedback sources are suggested87: (1) formal rewards, (2) informal assignments, (3) supervisors, (4) co-workers, (5) comparisons of work to that of others (benchmarking) and (6) information one receives from the task itself (task feedback). Tankoonsombut88 arranges these in two groups from the employees’ perspective: The first group includes feedback sources that are intrinsic to, under the control of, or psychologically closer to the individual. The second group includes feedback sources that are external to, beyond the control of, or psychologically further away from the individual. Feedback from self-comparison of one’s work to that of others (benchmarking) and from the task itself fall into the first category, while feedback from the organization (i.e. formal rewards and informal assignments), supervisors and co-workers falls into the second category. It is more difficult to find evidence in the literature reporting on goal dimensions of a PMS than on concrete articulated organizational goals. Similarly, it is difficult with regard to classifications or any other organization or systemization attempts
82
Locke (1968), p. 162. Griffin (1990), p. 170. 84 Locke (1968), p. 162. 85 Tankoonsombut (1998), p. 13. 86 Locke (1968), Greller and Herold (1975), Ilgen et al. (1979), Earley et al. (1990), Busby (1997). 87 Greller (1980), cited from Tankoonsombut (1998), p. 11. 88 Tankoonsombut (1998), p. 31. 83
90
3 State-of-the-Art in Performance Management
other than the dimension of performance measures. For example Hahn suggests three categories of goals89: • Substantive goals (product/service program with specific quality goals); • Value goals/financial goals (increase of economic value of the organization, dividend continuity, liquidity); • Social goals/human goals (employees, society, environment). Even in conjunction with performance measurement, only very few articles report about concrete organizational goals and goal-setting processes. These will be discussed later. Nevertheless, experiences with performance measurement interventions and consequences are reported. Contrary to the literature examined above, the PMS literature reports an absence of consideration of organizational goals and a lack of continuous systematic breakdown of goals (Table 3.13). Table 3.13 Problem areas in performance management reported in the literature Author Statements regarding Ittner (1998) Difficulties with goal cascading to lower levels Gleich (2001) Poor (or loose) coordination of strategic, operational goals and indicators; alignment between strategy and goals is not sufficiently pronounced Frigo (1999) The link between goals and indicators is practically nonexistent Brunner (1999) No formulated (corporate) strategy exists, strategic goals are not quantified (not specific), employees in lower levels, customer or suppliers are not involved in the goal setting process Tieke (1999) Goal attainment control, for 70% of companies, is of central importance, strategic goals are rarely quantified/unoperationalized Steinle (2001) Quantification of strategic goals is reported to be problematic Schneidermann (1999) Goal values are negotiated and not based on stakeholder requirements or the fundamental borders of processes and skills of the organization Zimmermann (2000) Main problem area – definition of strategic goals Kueng (2001) In spite of ISO-9000 certification only 36% of companies defined goals and measures for process management Bourne (2000) Goal commitment of top-management was observed to positively influence performance measurement (patriarchal structured family businesses were more successful than business units of big enterprises), additional initiatives of holding companies: changed strategies lead to irrelevant goals and metrics as a negative factor T€ opfer (2002) Problem areas reported: (more pronounced in medium-sized businesses in comparison to big companies) coupling of goal attainment with financial incentives, no explicit coupling of strategic goals with budgeting process Habermann (2002) Problem areas: two-thirds of surveyed companies had no clear formulated strategy, 77% had difficulties with definition of adequate measures, too few strategic measures, too many operative measures, 69% had difficulties with the definition of target values
89
Hahn (1994), p. 62.
3.2 Organizational Goals
91
These statements indicate that major problems exist on both the tactical and operational levels. The quantification, breakdown and cascading are activities that deal with the formulation of goals. This translates to the coupling of goals with metrics, and to the alignment with strategy, and includes aspects of communication. The complete absence of goals is rarely reported. It should not be interpreted that those firms do not pursue organizational goals. The assumption can be made, however, that in this case again the communication aspects are to be improved.
3.2.2
R&D Organizational Goals
Only a few studies describe the concrete goals of R&D departments. Examples that have been analyzed in detail are based in the following industrial sectors: automotive (Centro Ricerche Fiat (CRF), Italy), telecommunication (Telecom Italia Lab (TiLab), Italy), and aerospace (Volvo Aero Corporation, Sweden). These studies do not focus on analyzing organizational goals per se, but rather on the implications on goal setting after organizational changes in managerial procedures and strategies. These studies suggest that organizational goal setting for R&D departments within the context of reorganization requires concomitant analysis of organizational strategy, goals and performance-related measures. The request for more accountability of R&D departments reported as being the trigger for organizational changes. The fact of shifting from cost centers to autonomous business units with the goal of producing cash flows in order to justify expenses is reported for CRF and TiLab90 (Cesaroni et al. 2004). In the case of TiLab, the origin of the organizational changes was telecommunication deregulation. New competition on the domestic telecommunication markets and new opportunities on foreign markets changed the role of R&D departments. As such the exploitation of R&D activities become a priority.91 The authors report that the goal of TiLab in 2001 was to ‘. . . create the ability to integrate new realities through the creation of autonomous units, partnership with external subjects and financial participation in new entrepreneurial activities.’ Furthermore, the following goals are mentioned: • • • • • •
90 91
Creation of spin-off companies and incubators; Development of specialized competencies for the marketing of inventions; Development of new ideas, technologies or business opportunities; Technological transfer; Partnerships or acquisitions; Creation of competencies for organizational “Skill Building”.
Cesaroni et al. (2004), pp. 45–56. Cesaroni et al. (2004), p. 47.
92
3 State-of-the-Art in Performance Management
Historically, CRF since its inception in 1976 was an independent center. The reorganization of CRF was a reaction to the automotive industry crisis in 1993, where exploration and exploitation strategies were reinforced. Major restructuring took place in 1998 where CRF responded to objectives of technology transfer and exploration of long-term research when designing the organization. Within the exploration strategy two major goals were identified: • Research transfer: ‘to develop products, processes and technologies to be effectively transferred to the clients’, and • Participation in publicly-funded research programs: ‘to create an international research network to be activated when CRF intends to take part in research programs promoted by the European Union or other governmental authorities’. The exploitation strategy was dominated by the technology transfer strategy ‘to search for “proper” clients’ and ‘to develop “proper” products’, knowledge transfer and the transfer of human resources. All three are strongly interconnected. The most complete form of technology transfer is the transfer of human resources, since the knowledge of researchers is transferred as well.92 For Volvo Aero Corporation, the authors reported that the company operated under great pressure. Being a functionally-oriented company, dependent to a large extent on nationally-funded military projects, the company had evolved into a business-oriented, leading manufacturer in the aerospace market. The authors report that an internal distinction between research and development is made in terms of levels of validation. The validation process contains 9 steps: 1–3 basic research, 4–6 demonstration, and 7–9 development (development starts with a decision to take a product to full-scale production).93 The authors observe that in 1996, all research programs at Volvo Aero were structured into product development research and process development research. Product-related properties, such as performance, environment, lifecycle cost, safety and availability combine to form product development research. On the other hand, increased efficiency of the internal company processes, such as product development, production and product support, are at the core of process development research. The following goals of the technology programs during 1970s are reported: • Military. Develop technologies to achieve increased performance of military engines, while at the same time developing competent engineers. • Commercial. Develop spin-off products from the military competence, such as hydraulics, heaters for cars, trucks, and boats, vehicle components for trucks, etc.94 • Space propulsion. Early investments were made to develop possibilities to compete in this area.
92
Cesaroni et al. (2004), p. 51. Karlsson et al. (2004), p. 182. 94 These were alternative products because Volvo Aero saw a future decrease in military projects. 93
3.2 Organizational Goals
93
During the 1980s, priority was given to the development of the product: ‘to develop product-related technologies for commercial aircraft engines as well as for military engines’. In 1990, reorganization took place in business areas: Vehicle Components, Land and Marine Gas Turbines, Space Propulsion, Commercial Engines, and Military Engines. This shows that the research programs became even more business-oriented. The current goals of the research programs are as follows: • • • • • •
Decrease cost; Decrease lead-time; Increase quality; Increase flexibility; Increase performance of the products (present or future); Increase efficiency of the company.
The studies presented above analyze organizational goals for different purposes. The two first studies highlight the shift of industrial R&D departments from a cost center situation to acting like a profit center. The second study explores factors which affect a performance measurement system. While the first two studies address R&D departments as a whole, the third study seeks evidence on how to separate these two departments and recommends separating them when measuring performance. When analyzing the goals, the following key terms reflect the broader aims of the goals: spin-offs, incubators, inventions, new ideas, new technologies, new business opportunities, new realities, technological transfer, partnerships, acquisitions, people skills, public-funded research, and international network. A conclusion that can be drawn from these studies is that the individual context in which any given research department operates is so specific as to invalidate the comparison of research goals across organizations on a standardized basis.
3.2.3
Research (Only) Organizational Goals
Companies engage in research for a number of reasons; their objectives may include the development of innovative products or services, investigation of new markets, increased efficiency and/or quality, or the creation of capabilities to learn about and to implement change. Identifying motivations of companies for engaging in research helps to examine forces that drive their research activities, such as competition or opportunities for entering new markets. The output of a research department could be an invention, whereas the outcome from the perspective of a company could be an innovation. Therefore, the primary role of a research department is to help a company analyze new markets or seek new technologies, thereby feeding the company with inventions that lead to innovations. Understanding the exact role of a research department within the company helps it formulate its organizational goals. The formulation of organizational goals for a research department is a very difficult task due to the nature of research. ‘To gain increased
94
3 State-of-the-Art in Performance Management
understanding of a phenomenon or to search for new elements of technology is the goal of a research department’ (Karlsson et al. 2004, p. 182; Asimov 1962). ‘It further entails greater uncertainty of outcome and higher technological risks’ (Karlsson et al. 2004, Mansfield et al. 1971). The authors also observe that the customers for these kinds of results are often internal to the company. A very useful study that contributed greatly to the understanding of the distinction between research and development, and especially the elaboration of organizational goals of these units, is by Mesthene and Clintock (1962). Their document “The Nature of Research Goals: Some Necessary Definitions” was prepared for the United States Air Force Project, RAND.95 Mesthene and Clintock examined the nature of three types of scientific research: academic research, mission-oriented research, and technological development. These were examined in two types of organization: academic organizations (universities and university-like institutions) and missionoriented organizations (industries, the military services and other government agencies such as NASA,96 AEC97 and NIH98). They observed that the objective of industrial research was not merely to add to knowledge, but to contribute to the production of a useful item.99. The objective of academic research was to contribute to the growth of knowledge about the world. This difference, in their opinion, implies an important role for research management in a mission-oriented organization. Accordingly, university administrators have no significant research management function beyond that which is implicit in their commitment to support research in the major intellectual disciplines. All research planning and guidance is done by the research scientists themselves. “But the criterion for evaluation and funding research in a mission-oriented organization is more than simply the availability of a competent and balanced research staff. Research must be evaluated and funded instead in the light of its contribution to the extra-scientific goals of the organization.”100 The authors specify different kinds of decisions to be made by management: • • • • •
What lines of research to encourage; What projects to discontinue; What kinds of scientists to hire; How to exploit the results of research for organizational ends; How to modify organizational objectives in ways calculated to take advantage of research findings.
From this listing the authors conclude that in mission-oriented research, management becomes a major and necessary function.
95
RAND – the name is derived from a contraction of the term research and development. NASA stands for National Aeronautics and Space Administration, USA. 97 AEC, abbreviation for Atomic Energy Commission, USA. 98 NIH is the abbreviation of National Institute of Health, USA. 99 As an example authors use the production of a refrigerator or a bomber, noticing that the objective for which research is supported by an army or a firm is a production of a useful item. 100 Mesthene and Clintock (1962), p. 2. 96
3.2 Organizational Goals
95
The authors state that mission-oriented organizations find it easier to formulate goals for development units than for research programs. The reason, in their opinion, lies in a difference between scientific research and technological development. First of all, they state that although both research programs and development programs produce knowledge, each produces a different kind of knowledge. Research, in their view, seeks to find out how nature operates and whether its principles can be translated into laboratory models. Development, on the other hand, aims at a specific useful device. It seeks information about how laboratory models and other existing mechanisms can be modified or combined into pieces of equipment that can accomplish a particular practical objective. Mesthene and Clintock distinguish three different kinds of goals: 1. End items (these are the goals of development); 2. Purely intellectual goals; 3. Useful research goals. The authors use the term “end item” to express a specific hardware101 item with which development ends. “Purely intellectual goals” are at the other end of the R&D spectrum and are wholly within the context of a scientific inquiry. ‘Contrary to what is sometimes loosely believed, the most fundamental research is neither goalless, nor directed merely at very vague, general goals. All science, no matter how basic, seeks specific answers to specific questions.’102 The authors state that this kind of research is often motivated simply by curiosity and though very specific, however, the goals of the research may have no known or intended reference to practical use. The last category represents “useful research goals”.103 These are located between end items and intellectual goals. The term refers to the goals of the research undertaken in the expectation of a practical payoff – a use, or “application” – beyond itself. The notion of “use” differentiates useful research goals from intellectual goals. The authors define useful research goals as ‘. . . intellectual goals with the addition of an extra-scientific or extra-intellectual use-relevance’.104 Although “use” distinguishes useful research goals from academic research, “use” never takes the form of an end-item as in development. The authors examining the use-dimension observe that the research process itself provides no basis for the distinction, which is one reason why it is so difficult to distinguish between
101 In our opinion, it is also possible to apply this term in the software industry with a slight modification. "Hardware" implies a degree product maturity. The authors give examples of hardware: a bomber, a submarine, an engine, a drug, etc. In the case of software providers, we think executable software could also be valid as an end item. 102 Mesthene and Clintock (1962), p. 2. 103 The authors admit that the term chosen is awkward, but must suffice for want of a better one. 104 Note that this definition and distinction is made from the manager’s perspective. The authors note that a different distinction from that which can be made within intellectual goals on the basis of their apparent usefulness is from scientists.
96
3 State-of-the-Art in Performance Management
intellectual goals and useful research goals. Thus they analyze the product of research activities, stating that this (product) takes the form of true general propositions, which perform a dual function. The propositions both describe some natural operation or behavior, and predict some future event, asserting that certain things will happen in the future if certain conditions are met. This predictive aspect is built into scientific (research) propositions. The authors note that the term “built into” is important, meaning logically contained in; logically necessary. The authors claim that the process of achieving intellectual goals is identical with the process of achieving useful research goals. They define the use-dimension of useful research goals as ‘something added to its intellectual component’.105 Furthermore, they state that any attempt to carefully formulate a useful research goal must start with the relevant intellectual goal and then go on to incorporate the use-relevance dimension. They pose two further questions: (1) at which point in the research organization is use-relevance introduced into research goals, and (2) what is the source of this additional element? To answer the second question we reiterate that according to the authors the research process is not a source of use-relevance, but rather the extra-scientific purpose of men. They state that the raw materials from which useful research goals are derived are questions that arise from attempts to solve problems that are not primarily intellectual or purely scientific in nature. Regarding the first question, the authors argue that the use-relevance (based on the answer to the second question) is not always pertinent to the formulation of research goals. They outline three cases when research goals can acquire use-relevance: 1. Before research starts: in the case of research undertaken with the specific expectation of a useful payoff. Useful research goals contain a use-dimension before the research is started. 2. During the course of the research: in the case when research is initiated in search of purely intellectual goals but then takes some new direction before the original goals are achieved. The authors distinguish two reasons for that occurrence: (a) Revealing previously unsuspected possibilities that result in a replacement of the original intellectual goals with a new set of intellectual goals; and (b) Consciously changing orientation because intellectual goals are converted into useful research goals.
105
The authors claim that all research, whether academic or mission-oriented aims at least at intellectual goals. “This is necessary so long as it is to remain respectable research, because it is the intellectual ends of an inquiry that provides its rationale” (Mesthene and Clintock 1962, p. 13).
3.2 Organizational Goals
97
3. .After the completion of the research: the use-possibilities of research findings may either not appear at all, or only after the research is completed. The practical implication of the findings appears only after the intellectual goals have been achieved. The authors summarize that in the first case the goals remain only intellectual. In the other cases, the use-relevance serves to convert intellectual goals into useful research goals. They observe that if the useful research goals precede the initiation of a project, as in the classic case of mission-oriented research, they can be said to have stimulated it. If, in the other case, the use-relevance is added at some time during the course of a project, the research may take a different direction. In both cases non-scientific, practical considerations stimulate and guide research.
Table 3.14 Summary of different goal dimensions addressed in literature Goals Goal object
Characteristic
Goal-setting process Feedback
Goal-description
Operative problems
Author Product-specific Karlsson et al. (2004) Organizational Asimov (1962), Mansfield et al. (1971), Heinen (1991), Hahn (1994), Cesaroni et al. (2004), Karlsson et al. (2004) Individual Locke et al. (1981), Locke and Latham (1990) Operative Kosiol (1966), Schmidt (1969), Hamel (1992), Nagel (1992), Hahn (decision (1994), Hauschildt (1997), Habermann (2002) goal) Economic Hahn (1994) Compatibility Mesthene and Clintock (1962), Thommen and Achleitner (2003) Complexity Habermann (2002), T€opfer (2002) Achievability Locke et al. (1981), Locke and Latham (1990), Griffin (1990) Specificity Asimov (1962), Locke et al. (1981), Locke and Latham (1990), Champion and Lord (1982), Griffin (1990), Cesaroni et al. (2004) Activity-based Cesaroni et al. (2004), Karlsson et al. (2004) Hamel (1992), Nagel (1992), Hauschildt (1997), T€opfer (2002) Locke (1968), Greller and Herold (1975), Latham and Yukl (1975), Ilgen et al. (1979), Greller (1980), Earley et al. (1990), Busby (1997), Tankoonsombut (1998) Schwantag (1951), Bidlingmaier (1964), Mesthene and Clintock (1962), Kosiol (1966), Schmidt (1969), Locke (1968), Griffin (1990), Heinen (1991), Specht and Beckmann (1996) Bourne (2000), Brunner (1999), Frigo (1999), Habermann (2002), Tieke (1999), T€opfer (2002), Kueng (2001), Zimmermann (2000), Schneiderman (1999), Steinle (2001), Ittner (1998), Gleich (2001)
Table 3.14 gives an overview of the relevant organizational-goals literature on the generic level that embraces the entire company, and on the specific company levels: R&D function and Research only. However, it can be said that very few contributions discussing the individual strategies of companies can be found. It is often possible to deepen the understanding of a company’s strategy by looking up mission and vision statements that are usually provided on their web-pages. However, the translation of the strategy into practical organizational goals remains an under-investigated area in the literature.
98
3.3 3.3.1
3 State-of-the-Art in Performance Management
Key Performance Indicators (KPIs) Generic Key Performance Indicators
In the following section we examine non-sector specific literature addressing performance measures, performance indicators, and KPIs.106 Our attention will especially be given to the following questions: 1. What kinds of performance measures/indicators are listed in the literature 2. In which context are they measured, e.g. how are they organized in terms of structure; what is the frequency of measurement; which artifact is being measured (input, output, outcome or activities); what, if any, logic is being applied. As we reported in the section on PMS approaches, the balanced scorecard is perhaps the most used analysis framework. It is a general framework with four dimensions and therefore does not specifically detail performance measures. An alternative framework to the balanced scorecard is Parmenter’s107 framework that includes two additional perspectives. He provides a database, listing 342 measures108 with an indication of possible application. This listing is a collection of different measures with no specific industry focus. The list makes a distinction in following aspects: frequency of measure, balanced scorecard perspective, applicable balanced scorecard teams, applicable sectors and the strategic objective. Regarding the frequency of measure, the range is reported according to specific time frames such as: 24/7, daily, weekly, quarterly, monthly, two, three or four times a year. It is also reported following events such as: when audits are performed, after an employee survey, after the performance review round, after every staff survey. Parmenter organizes all measures within the following six perspectives: • Financial (utilization of assets, optimization of working capital); • Customer (increase customer satisfaction, targeting customers that generate the most profit); • Environment/community (supporting local businesses, linking with future employees, community leadership); • Internal (delivery in full and on time, optimizing technology, effective relationships with key stakeholders);
106
The respective definitions are provided in Sect. 2.2.3. Parmenter (2007). 108 Parmenter (2007), pp. 203–231, the database also contains metrics for R&D which will be examined in detail in the following section. On account of minor relevance the table with 342 measures will not be placed in this section, but can be found in Appendix B. 107
3.3 Key Performance Indicators (KPIs)
99
• Employee satisfaction (positive company culture, retention of key staff, increased recognition); • Learning and growth (empowerment, increasing expertise, and adaptability). The listing assigns each measure to at least one of the following teams: sales and marketing, production, quality assurance, accounting, sales and back office, IT-communications, IT help desk, call centers, service delivery teams, production control, dispatch, project teams, service teams, human resources, operations, public relations, stock control, research and development, payroll, procurement, design, planning, training. Furthermore, Parmenter distinguishes in his list the following applicable sectors: all private sectors, all sectors where customers retain a sum of money until satisfactory completion, all sectors that dispatch goods, other sectors like: manufacturing, service, charity, construction, banking, insurance, critical services sector, retail, professional service firms and tertiary. The last aspect in Parmenter’s listing is the strategic objective. The following objectives are distinguished: increase profitability, provide efficient operations, retain customers, minimize negative comments in the marketplace, establish long-term relationships with profitable customers, increase sales, generate more reliable products, improve employee satisfaction and productivity, create desirable workplace, make employees happy, make customers happy, which makes shareholders happy, work well with the community and environment, reduce environmental impact, create positive public perception, maintain supplier relationships, create innovation, retain skilled and experienced workforce, provide healthy and safe work environment. Parmenter introduces his concept with the following three parameters109: KRIs, KPIs, and PIs, but this distinction is not available within his database. He notices that KRIs have often been mistaken for KPIs and that KRIs provide information that is ideal for the Board. He argues that KRIs typically cover a longer period of time than KPIs. He also suggests separating KRIs from other measures, which in his concept would separate performance measures into those impacting governance and those impacting management. Furthermore, a concrete recommendation is made to have a governance report with a dashboard format, consisting of up to 10
109 “Key result indicators (KRIs) give a clear picture whether you are traveling in the right direction. They do not, however, tell one what is needed to do to improve these results; Key performance indicators (KPIs) represent a set of measures focusing on those aspects of organizational performance that are the most critical for the current and future success of the organization.” Parmenter does not really define performance indicators (PIs) as he only mentions: “In between KRIs and the true KPIs are numerous performance indicators. These complement the KPIs and are shown with them on the scorecard for the organization and the scorecard for each division, department, and team.” Parmenter (2007), pp. 3–7.
100
3 State-of-the-Art in Performance Management
measures providing high-level KRIs for the Board and a balanced scorecard comprising up to 20 measures (Parmenter suggests a mixture of KPIs and PIs) for management. Altogether, Parmenter introduces a 10/80/10 rule and argues that an organization very rarely needs more than 10 KRIs, up to 80 PIs and 10 KPIs.110 For our specific question, which asks which performance indicators are best suited to assess organizational goals in an industrial research department, Parmenter’s database is of little use. From the sector-specific perspective, we do not find indications in the database of the ICT-sector or reference to the industrial research function. A good crossover from the generic literature on performance measures to the literature addressing measures in R&D can be presented by the work of Geisler.111 Geisler, in his book, summarizes metrics for science and technology (S&T). Therefore, his work cannot be completely allocated to generic functions because S&T is only a part of the company’s value chain. Furthermore, his work cannot be precisely allocated to R&D because the area he covers is broader, addressing the whole innovation continuum. In our context the larger part of his work is allocated to the generic function with a gradual transition to R&D. Geisler summarizes 52 metrics within the following eight categories112: • • • • • • • •
Input/investments in S&T; Economic/financial metrics; Commercial and business metrics; Bibliometric metrics; Patents; Peer review metrics; Organizational, strategic and managerial metrics; Stages of outcomes.
Some of the metrics are well-suited to assess the performance of industrial research, for example measures from “patents” are typical metrics to assess the inventiveness of an organization. Bibliometric metrics are also often used to assess publication intensity, or peer-review metrics for the content, which is very domain specific and only a small number of individuals (peers) can judge. Economic or financial metrics are more difficult to apply and are less common in practice when assessing the performance of industrial research departments.
110
Parmenter (2007), p. 8. Geisler (2000). 112 Geisler (2000), pp. 80–86. Please note that the author emphasizes: each category contains illustrative measures. The lists are not exhaustive and are not a complete listing of all measures found in the relevant literatures. Measures are not listed in any particular order. See Appendix B. 111
3.3 Key Performance Indicators (KPIs)
3.3.2
101
R&D Key Performance Indicators
In the following we review literature reporting on performance measures for R&D. Interest in the topic of performance measurement in R&D has increased in the last few decades, but the topic is not new at all. Legitimate interest for the “right” KPIs has been shown, in particular, by managers of R&D laboratories. This is especially the case when there is an increased demand for contributions or value of contributions from R&D and a need to make the impact of results more visible to the overall organization. Therefore, the call for transparency is connected to budget allocation procedures, which often results in a search for the “right” KPIs that justify R&D’s existence. Articles reporting these issues can be traced to the 1970s. One of the most comprehensive studies for evaluation of R&D productivity was published by Ranftl (1977) within the Hughes Aircraft Company. In this study Ranftl analyzed typical characteristics for a high-productive research and development department. The two-stage study,113 which was conducted between 1973 and 1975, identified three levels as starting points for the measurement and improvement of R&D productivity: Productivity of the organization, productivity of the R&D manager and productivity of the employees. The results of the study are presented in the form of checklists (to examine the typical criteria of a productive R&D process) and profiles (to describe a typical profile of a respective unit). Since then, based on his findings,114 many studies have been conducted to find appropriate metrics to assess R&D’s productivity. Schainblatt115 lists more than 40 different kinds of measures, indicators, ratios, etc., from seven in-depth case studies. Brown and Svenson116 present 20 measures distinguishing research and technology development measures and pure product development measures. Brown and Gobeli117 report on 23 measurements collected in a field study and suggest a list of the ten most important indicators covering the primary areas of importance. Tipping and Zeffren118 present 33 metrics within their approach called Technology Value Pyramid, by associating them with five managerial aspects. In their empirical investigations, Chiesa and Frattini found that there is a pronounced specialization in the use of different performance indicators along the R&D process.119 In their
113 The following characteristics are attached to this study: more than 2,300 interviews with top R&D managers, interviews in 59 enterprises of the investment and consumer goods industry, interviews with 28 consulting companies and various seminars and large-scale literature search. 114 An extract of the indicators suggested by Ranftl can be found in Appendix B. 115 Schainblatt (1982), pp. 10–18. Note that we summarized only relevant measures for R&D, and left measures reported for the engineering units out, we also tried to avoid double entries. 116 Brown and Svenson (1998), pp. 30–35. The detailed list of measures suggested by Brown and Svenson is provided in Appendix B. 117 Brown and Gobeli (1992), p. 330. The detailed list of measures suggested by Brown and Gobeli is provided in Appendix B. 118 Tipping and Zeffren (1995), pp. 32–63. The detailed list of 33 measures suggested by Tipping and Zeffren is provided in Appendix B. 119 Chiesa and Frattini (2007), pp. 294. The full list of performance indicators collected by Chiesa and Frattini is provided in Appendix B.
102
3 State-of-the-Art in Performance Management
study, the authors found a tendency for the use of quantitative objective metrics in development, and qualitative subjective measures in research. The authors draw particular attention to the specific degree of uncertainty and complexity that characterizes the activities of the two departments. They argue that this translates into availability of quantitative data to be used to build precise and objective indicators. According to the authors, the uncertainty and the distance in time of research results, make it impossible to use numeric metrics that are independent from experts’ opinion. Their empirical investigation does not provide evidence of the use of quantitative subjective indicators; nevertheless authors argue that they represent a sort of hybrid solution between quantitative objective and qualitative subjective metrics that can be used in the development of PMsSs when it would be too costly to systematically measure objective indicators, or in applied research when uncertainty levels are lower and there is a specific need for numeric information (e.g. for supporting formal project curtailment decisions). Due to the very broad range of activities covered by the R&D function, a great variety of metrics is to be found. Almost all listings distinguish between qualitative and quantitative metrics. Regarding the categorization perspective, the following taxonomy of performance measurement methods has been suggested by Kerssensvan Drongelen120 (Table 3.15). Table 3.15 Summary of measurement methods adapted from Kerssens-van Drongelen Taxonomy of performance measurement methods Subjective, qualitative methods Semi-objective, qualitative methods Subjective, semi-quantitative methods: • Simple: single item/judgmental conversion or averaging of scores for several items into an aggregate metric • Sophisticated: conversion of scores for several items into an aggregate metric using sophisticated formulae or techniques Semi-objective, semi-quantitative methods: • Simple: single item/judgmental conversion or averaging of scores for several items into an aggregate metric • Sophisticated: conversion of scores for several items into an aggregate metric using sophisticated formulae or techniques Subjective, quantitative methods: • Financial • Non-financial Semi-objective, quantitative methods: • Financial • Non-financial Objective, quantitative methods: • Financial • Non-financial Source: Kerssens-van Drongelen (2001)
120
Kerssens-van Drongelen (2001), p. 80.
3.3 Key Performance Indicators (KPIs)
103
From the content perspective, many authors cite people-based metrics. These include a focus on skill and training aspects, motivation, moral etc. Furthermore, the intellectual property aspect is often mentioned with metrics such as granted patents, patent application, cost of inventions, counts of patent disclosures, etc. Publications, papers, books are also present in many listings. Technology transfer to manufacturing or other units and strategic alignment are also often named. Another observation, which is not exactly of the content nature, but more structural, is the fact that many authors try to assign each metric to specific content areas. These areas summarize the metrics and they often represent important areas for the companies evaluated. For example, Tipping et al. (1995) call these areas Managerial Factors121: • Value Creation (VC) demonstrating the value of R&D activities to the positioning, profitability and growth of the organization and to the creation of shareholder value; • Portfolio Assessment (PA) communicating the total R&D program across various dimensions of interest, including time horizon, level of risk, core competency exploitation, and new/old business; • Asset Value of Technology (AVT) indicating the strength and vitality of the firm’s technology (e.g. proprietary assets, know-how, people, etc.) and foreshadowing the potential of the R&D organization to create future value for the company; • Integration with Business (IWB) indicating the degree of integration, the commitment of the business to the R&D processes and programs, teamwork, and ability to exploit technology across the organization; • Practice of R&D Processes to Support Innovation (PRD) indicating the efficiency and effectiveness of R&D processes in producing useful output for the firm. The processes include project management practices, idea generation, communication and other “best practices” in managing R&D. The authors assign each of the 33 metrics to one or a combination of these factors. Brown and Gobeli short-listed ten metrics, summarized in seven categories: (1) resources, (2) project management, (3) people management, (4) planning, (5) new technology study and development, (6) outputs, (7) division results/outcomes. Similarly, Geisler allocates all metrics to the following eight categories mentioned above: (1) input/investment in S&T, (2) economic/financial metrics, (3) commercial and business metrics, (4) bibliometric metrics, (5) patents, (6) peer review metrics, (7) organizational, strategic and managerial metrics, and (8) stages of outcomes. Our observation about the connection between individual measures with more general areas is consistent with the findings of Mertins and Krause. Mertins122 formulated an object structure model that is based on the idea of a goal – measures – process network. Krause and Mertins distinguish four object types in their model shown in Fig. 3.6:
121
Tipping et al. (1995), pp. 32–63. Mertins (1998), cited from Krause, notes that Krause had adapted the structure in the course of his work and therefore the naming has been changed. 122
104
3 State-of-the-Art in Performance Management Critical success factor(s)
Performance indicator(s)
Business process(es)
Performance improvement project(s)
Fig. 3.6 Tetrahedron model adapted from Krause (Source: Krause 2005)
• The first object type is a critical success factor (CSF).123 The concept of the CSF goes back to Daniel in the early 1960s and Rockart at the end of the 1970s.124 The concept is based on the assumption that a limited number of individual variables exist that makes a significant contribution to a company’s success. “Critical success factors . . . are, for any business, the limited number of areas in which results, if they are satisfactory, will ensure successful competitive performance for the organization. They are the few key areas where ‘things must go right’ for the business to flourish. If results in these areas are not adequate, the organization’s efforts for the period will be less than desired. As a result, the critical success factors are areas of activity that should receive constant and careful attention from management. The current status of performance in each area should be continually measured, and that information should be made available. . . ., critical success factors support the attainment of organizational goals.”125 • The second object type in Krause and Mertin’s model is business process. According to Krause, business processes represent sources of added value and hence of performance. Krause argues that the precise knowledge is the basis for the identification of CSFs.
123
Note that a critical success factor (CSF) is not a key performance indicator (KPI). Critical success factors are elements that are vital for a strategy to be successful. KPIs are a set of performance indicators that quantify management objectives and enable the measurement of strategic performance. A critical success factor, following Rockart, is what drives the company forward. 124 Daniel (1961), Rockart (1979). 125 Rockart (1979), p. 85.
3.3 Key Performance Indicators (KPIs)
105
• The third type is represented by the performance improvement project. The authors argue that the performance improvement project impacts the efficiency and effectiveness of business processes. • Performance indicators represent the fourth object type of the model. The justification for including this type into the model is its management support function. Although Krause mentions that CSFs are linked to goals by the performance indicators in his model, the goals are omitted within both the description and in the illustration. These were, however, part of Mertins’s original model. Business processes and performance improvement projects are driven by these goals, and the measured data is consolidated through performance indicators and is usable for the assessment of goal attainment.
3.3.3
Research (Only) Key Performance Indicators
Because companies, for long time, regarded the functions of research and development as one single function, very few literature sources were found on performance indicators that could be applied for a research department only. Two studies were found that reported performance measures that were applied in practice within research departments only (one within an industrial research group and one within a non-profit research organization). Mett€anen’s126 study designed and implemented a performance measurement system in the TTS Institute (Work Efficiency Institute, in Finnish: Ty€ otehoseura). The study focuses on research activities only. The TTS Institute’s research activity involves three research departments: agriculture, forestry and home economics. Mett€anen reports ten success factors allocated to all three departments subsumed under financial or non-financial categories. It is striking to note that the number of non-financial success factors outweighs the financial success factors (Table 3.16). Altogether 24 measures were defined to be applied within a performance measurement system of the TTS Institute. The author stresses the difficulty of data collection for the measures chosen due to special characteristics such as the importance of intellectual capital. Different collection methods are required for intellectual capital measures than for financial measures. Questionnaires are often deployed to gather data and are subjective assessments by the interviewees. The performance measures listed in Table 3.17 attributed to each success factor mentioned above were finally defined for each department. Mett€anen reports that the calculation is done separately for each department (Table 3.17) Another case study which reports on a concrete example from industry is the study from Loch und Tapper.127 They applied a strategy-driven performance measurement system for the process technology research group at GemStone, a medium-sized diamond producer. The authors developed 28 (see Table 3.18)
126 127
Mett€anen (2005), pp. 178–188. Loch and Tapper (2002), pp. 185–198.
106
3 State-of-the-Art in Performance Management
Table 3.16 Case organization’s success factors divided into financial and non-financial factors Financial success Non-financial success factors factors Related to the Related to external Related to internal • Growth • Increasing profitability
• Good productivity
employees • Increasing academic competency • Increasing competencies • Employee welfare
relationships • Positive publicity • Internationalization
structure • Effective distribution of work
• Customer satisfaction, users • Effective project management • Customer satisfaction, • Availability of financers equipment • Finding new financers and • Sharing knowledge keeping old financers between departments
Source: Mett€anen (2005)
Table 3.17 Measures of the case organization Success factor Measure/measures Growth Increase in turnover (%) Increase in working years (number) Increasing profitability Profitability of projects on average (coefficient) Annual profit (€) Positive publicity Number of appearances in media (number) Internationalization Number of employees having attended an international occasion (number) Customer satisfaction, users Percentage of satisfied customers (%) Finding new financers and Percentage of financing other than government subsidy (%) keeping old financers Percentage of private financing (%) Percentage of new financers (%) Customer satisfaction, financers Percentage of satisfied customers (%) Effective distribution of work Percentage of employees who work from 50% to 80% of the theoretical work time on projects (%) Effective project management Mean ratio, calculated based on project evaluation form (number) Availability of equipment Annual investment in equipment (€) Sharing knowledge between Number of joint projects between departments (€) departments Number of joint projects between departments (number) Good productivity Amount of financing per employee on average (€) Number of publications per employee (number) Increasing academic Number of credits for postgraduate studies (number) competence Number of postgraduate degrees (number) Number of other degrees (number) Increasing competencies Percentage of employees who have achieved personal development goals (%) Investments in education (€) Employee welfare Employee satisfaction index (questionnaire) Source: Mett€anen (2005)
Conscientious management of uncertainty (e.g., risk reduction assessment) Professional documentation, peer-review
Level of prototype maturity (e.g. # of major technology revisions after hand-over to development) Quality of documentation to development Professional schedule and budget planning and control
Communication within research and with customers Use of external knowledge, cooperation with partners Source: Loch and Tapper (2002)
Process Measures (project level)
Output Measures (project level)
Innovation score
Market potential of innovations in $ # Of presentations to external customers
Knowledge repository and external reputation # Of requested handbooks published and delivered Quality of research program homepage in intranet: # of external publications, patents, and their impact
Clarity and quality of conclusions in technology assessments (e.g. by peer review) “Project management” of writing handbooks and assessments
Request fulfillment Clear go/no go decision Quality of interaction with the Completeness of literature surveys technical services requestor Documentation
% Of support requests fulfilled # Of training sessions signed off by customer and delivered # Of problem analysis reports requested and delivered Response time Successful study completion
Table 3.18 Performance measures New technologies & breakthrough concepts Customer support Output Measures # Of significant innovations delivered Customer satisfaction index (group level, by customer) Impact of the technologies delivered (qualitative Response time to queries estimation by customer, follow-up to learn)
3.3 Key Performance Indicators (KPIs) 107
108
3 State-of-the-Art in Performance Management
performance measures, which are distributed on the one hand between group level and project level, and on the other hand between three types of research outputs: new technologies and breakthrough concepts, customer support, and knowledge repository and external reputations.
3.4
Conclusions and Research Gaps
This chapter started by explaining the terms that are crucial for a general understanding of this work. Altogether in this chapter an overview is provided of relevant specific streams of writing concerned with performance management and measurement, in general, in R&D, and in industrial research departments. This includes non-sector specific findings as well as industry-sector specific findings. A particular emphasis within the ICT sector with a focus on the software industry is made. Thereafter, Sect. 2.2 dealt with the state-of-the-art of the performance management concept in general, exploring performance measurement and its constitutive elements in particular. Sect. 2.3 discussed the purposes of performance management: we identified and compared them, consolidating them into three main categories; and finally associated the categories with the performance management cycle. Section 3.1.4 listed a requirements catalogue for a performance management system. Section 3.1.5 summarized the main characteristics of an industrial research department and its requirements in terms of performance assessment. Thereafter, the nature of organizational goals and key performance indicators was investigated in Sects. 3.2 and 3.3. As the literature review makes clear, performance measurement in a creative environment, such as a research department, is seen by the majority of contemporary researchers as a matter of fact128 and is as necessary as the management function itself: “The time has come to lay aside the old excuses for not measuring and base-lining R&D effectiveness, and to do it anyway – because it is the right thing to do. Direct measurements can be made. Do they involve uncertainty? Yes. Are they somewhat subjective? Of course. Are they valuable? Absolutely!129; “It’s one thing to recognize that all measurements of research are imperfect. But it’s quite another thing to say that because they are imperfect, we shouldn’t measure at all.”130 Nevertheless, some companies still hold the belief that performance measurement damages the creativity of research output.131
128 Kerssens-van Drongelen has in particular shown in her work the need for performance measurement; see Kerssens-van Drongelen (2001), pp. 1–10. 129 Philip Francis (1992), Vice President and Chief Technical Officer of Square D company, cited from Kerssens-van Drongelen (2001). 130 Walter Robb (1991), Senior Vice President for Corporate Research and Development at General Electric, cited from Kerssens-van Drongelen (2001). 131 One of our analyzed companies argues against the measurement of research.
3.4 Conclusions and Research Gaps
109
In general the review illustrates that performance measurement is a central challenge for research departments in industry.132 It shows that there are several approaches to measuring performance in industrial R&D,133 and that there are few approaches that measure the research environment independently from the development environment.134 Despite the established understanding that measuring research is important and worthwhile, there is a lack of scientific contributions that address the performance management of research departments only. The following research questions always imply the context of the ICT sector without being mentioned explicitly. Consequently, the research question for the present investigation is: RQ1: What is the common practice in regard to performance management of industrial applied research organizations? Are there common or similar organizational goals, and are there common or similar key performance indicators?
The investigation into this question should produce valuable information for the research inquiry in general. Moreover, these findings should provide the basis for more detailed research on the applicability and favorability of performance management practice in industrial research departments. As the literature review reveals, the application of performance measurement is still unexplored in industrial research. Furthermore, the reviewed literature shows that there are fundamental differences between the research function and the development function within industrial organizations.135 As such, the application of an R&D measurement system to the research function would only have a limited validity. Loch and Tapper (2002), Karlsson et al. (2004), Mett€anen (2005) who investigated the performance measurement phenomenon within different industrial sectors confirm this conclusion. Thus, there is a need to further investigate whether a generic approaches exists that describes the common performance management practice of industrial research departments: RQ2: Is there a generic performance management approach (PM approach) to describe the common practice of performance management of industrial applied research organizations? How are goals and KPIs reflected in this approach?
This work investigates the organizational performance of industrial research departments. Therefore the most issue is to understand the characteristics of industrial research departments that substantiate organizational peculiarities and how the performance management system reflects these peculiarities. In this context the literature review provides a limited theoretical base. In our related analysis
132
For example, Loch and Tapper (2002), p. 185. As summarized in Table 3.3. 134 Mett€anen (2005), Loch and Tapper (2002). 135 Refer to Sects. 3.1.4 and 3.1.5. 133
110
3 State-of-the-Art in Performance Management
we presented a detailed overview of the specific characteristics of research in general, and industrial research in particular,136 as there are: • The nature of activities pursued in research137; • The time lag between outputs that are produced in the research and their resulting outcomes138; • The fact that valid outputs of research are not necessarily positive139; • The uncertainty factor (uncontrollability, unobservability, unpredictability) in research activities140; • The different stakeholders and different communication methods141 compared to development and production units; • The originality of research work in terms of discontinuous “jumps” in knowledge142; • The knowledge depth and highly specialized knowledge of research staff vs. more cross-functional expertise143; • The different organization in terms of structure and processes of research in regard to formalization and centralization.144 One feature that reflects the very nature of the inventive process is that the result of research activities is not known ahead of time. It is often difficult to define, describe and specify the artifact or knowledge, which is expected as an output. In a commercial environment “the view of what the expected output is from a firm’s research activities varies from company to company, as well as within a company over a period of time, due largely to market developments and changing customer demands”.145 The authors also note that the question of “expected output” from research activities is a reflection of the definition of research of the company. They observe that if there is no clear definition at hand there will be differences in opinion regarding what the expected output is, thus making the measurement very difficult. Managing that many particularities and the complexity emanating from them requires a comprehensive approach that allows for the flexibility of accommodating all necessary conditions such as, for example, the possibility to track the transformation of research output into the outcomes for the company which typically need a longer observation cycle than just one measurement cycle.
136
In Sect. 3.1.5. Leifer and Triskari (1987), Karlsson et al. (2004). 138 Feltham and Xie (1994), Brown and Svenson (1998), Loch and Tapper (2002), Chiesa et al. (2007). 139 Prechelt (1997). 140 Feltham and Xie (1994), Leifer and Triskari (1987) Loch and Tapper (2002). 141 Leifer and Triskari (1987), Amelingmeyer (2005), Beck and V€olker (2009). 142 Stahl and Steger (1977), Karlsson et al. (2004). 143 Karlsson et al. (2004). 144 Leifer and Triskari (1987), Chiesa et al. (2007). 145 Karlsson et al. (2004). 137
3.4 Conclusions and Research Gaps
111
When organizational performance is being assessed, the organizational performance means nothing other than the accumulated contribution of employees to the organization’s goals. Employees’ actions are anchored within their organizational activities. Activities are herewith the mechanism by which work gets done and performance is accomplished. If the purpose of measurement is to manage the performance of an organization, then the key to effective measurement is to understand corresponding activities. The reviewed literature shows that the activities that people execute can be used as a basis for composing an organizational function. Therefore, orientating performance management regarding activities might be a promising way to tackle RQ2. As this area lacks longitudinal analyses that study the performance management phenomenon, the approach has to consider the peculiarities of the studied unit of analysis and has to be closely related to the following question(s): RQ3: How are the peculiarities of industrial applied research [specifically, and the requirements of performance management systems in general] considered in the developed PM approach?
Although existing frameworks146 are undoubtedly valuable, their adoption is often constrained by the fact that they are simply frameworks. They suggest some areas in which measures of performance might be useful, but provide little guidance on how the appropriate measures can be identified, introduced and ultimately used to manage the business. For any performance measurement framework to be of practical value, the process of populating the framework has to be understood. The answers to these three research questions should produce novel insights concerning the state-of-the-art of performance measurement in industrial research departments. Thus, the research findings of this inquiry should be valuable for the scientific community, but also for practitioners. In order to achieve a high practical relevance there is a need to develop guidelines, in the form of policy recommendations, on how to design, implement and/or further develop a performance management system in practice. In this context the following additional research question is of major interest: RQ4: Can any recommendations be derived from the suggested approach to advise on the implementation and further development of a performance management system in an industrial research organization?
The answers to each of the four research questions will be provided in the chapters as follows: In the Chap. 4 we introduce the step-by-step development of a performance management system, via a series of in-depth case studies, and thereby answer RQ 2 and RQ 3. Chap. 5 describes the results of a large quantitative survey on the developed approach, thereby contemporarily validating the approach. Chap. 6 describes recommendations that can be derived from the developed approach including the goal-setting process and the performance management process.
146
E.g. balanced scorecard, ABC, etc.
.
Chapter 4
Performance Management: Analysis Approach
Abstract This chapter provides an overview on the methodology and introduces the step-by-step development of a performance management system via a series of in-depth case studies. The analysis approach aims at developing a model for performance management systems built on the basic components found in practice such as organizational department goals and individual KPIs. The model is extended with additional components that allow us to compare the various performance management systems and to identify relationships between these components. Performance clusters (i.e. groups of KPIs), which represent one significant component of our developed model, are introduced. Furthermore, the goals are analyzed for the similarity of their sub-components. Altogether, Chap. 4 describes a preliminary approach on how to deal with the individual elements of performance management, that is, with KPIs, KPI classes, performance clusters and finally organizational department goals. Keywords Case studies • Collaboration with academia • Collaboration with partners and customers • Design • Future business opportunities • Image • Intellectual property (IP) creation • Interviews • Operational excellence • Performance clusters • Performance management analysis approach • Portfolio management • Presence in scientific community • Publications • Talent pool • Technology transfer
In this chapter we introduce our approach to developing a model for performance management systems. It is built on basic components found in practice (such as organizational department goals and individual KPIs) as well as on additional components that allow us to compare the various performance management systems and to identify relationships between these components. The first part of the chapter describes the approach in general; the latter part applies the approach to a number of case studies. In this way we develop a methodological base for a large quantitative survey. A mixed-method research approach has been chosen for this work because of the sensitivity of the topic and the lack of extant secondary data available. The area is T. Samsonowa, Industrial Research Performance Management, Contributions to Management Science, DOI 10.1007/978-3-7908-2762-0_4, # Springer-Verlag Berlin Heidelberg 2012
113
114
4 Performance Management: Analysis Approach
often considered to be sensitive for companies because performance management includes planning, assessment, analysis and review of departments and internal goals. A mixed-method approach means that both qualitative and quantitative methods of data collection were used.1 Collis and Hussey state that this approach is known as triangulation.2 The following four types3 of triangulation can be distinguished: data triangulation; investigator triangulation; methodological triangulation; and triangulation of theories. Within this work methodological triangulation was applied. Morse classifies methodological triangulation as simultaneous or sequential.4 For this work, sequential triangulation was applied, including: 1. Extensive literature research which has consistently accompanied all major stages of this investigation; 2. Case study research5 that allowed for the data collection and validation of the relevance of the research area; and 3. Survey research where the findings were eventually validated via a large quantitative survey. The literature review that was presented in Chap. 2 forms a basis for the development of a catalogue of semi-structured questions. These were used for the preparation of the case studies. In the remainder of this chapter we describe the analysis of existing Performance Measurement Systems and develop a model for performance measurement in industrial research.
4.1 4.1.1
Case Studies Design and Preparation
This work required a framework to analyze the sample of industrial research companies. Such a framework could not be found in the literature. The following
1
Eastery-Smith et al. (2002), Kasper (2006). Collis and Hussey (2003) argue that the application of a mixture of approaches is quite usual when conducting business research. “The use of different research approaches, methods and techniques in the same study is known as triangulation and can overcome the potential bias and sterility of a single-method approach” (Collis and Hussey 2003, p. 78). The main advantage of triangulation is the greater validity and reliability of the research results (Denzin 1978). 3 Eastery-Smith et al. (2002). 4 “Simultaneous triangulation is the use of the qualitative and quantitative methods at the same time. In this case, there is limited interaction between the two datasets during the data collection, but the findings complement one another at the end of the study. Sequential triangulation is used of the results of one method are essential for planning the next method. The qualitative method is completed before the quantitative method is implemented or vice versa” (Morse 1991, p. 120). 5 Yin (2006). 2
4.1 Case Studies
115
Fig. 4.1 Framework: research department and its eco-system
model6 (cf. Fig. 4.1) is based on Brown & Svenson’s graph and represents the interactions and flows, in and out, of an industrial research organization. This systemic view shows a research department as a processing system and other parts of a company as the receiving system. Furthermore, inputs, outputs and outcomes are exemplified and the measurement aspect is highlighted with measurement timeframes. This graph, while useful in its simplicity, is a caricature of reality and the granularity of the processes is excessively simplified. Although it is incomplete, this graph has the benefit of presenting the environment around a research department within a company on a high but appropriate level of abstraction. This framework enables us to establish a shared basis for understanding between interviewees about the unit of analysis, which is the research department, and to make the research environment with its inputs, activities, outputs and outcomes more explicit. In a next step it was necessary to understand to whether performance measurement existed and was used in practice. Thereafter, we had to identify the companies that could be used to study the topic. In accordance with the focus of our research we selected the candidate companies for the interviews according to the following criteria:
6
Samsonowa et al. (2009), p. 162.
116
4 Performance Management: Analysis Approach
• Global companies in the ICT sector and similar sectors listed on a stock exchange, and • With a dedicated research organization, and • With high or medium R&D intensity rate, (ratio of R&D spending to total sales) according to the Scoreboard of the European Union.7 According to these criteria following 12 companies were selected: ABB, Deutsche Telekom AG, EMC, Google, Hewlett Packard, IBM, Intel, Microsoft, Nokia, Philips, SAP, Siemens, and Sun Microsystems. Twenty-first-round interviews8 with representatives from the research departments of the companies were made. The selection criterion was primarily a certain understanding and necessity of the organizational performance assessment; and the existence of a functional performance measurement system within the research department. In order to be able to compare the companies it is necessary to collect the same relevant data within each company and to process this data in a common manner. Two matrices (cf. Table 4.1 for the outline) were developed to capture the essence of performance measurement within each case study: a matrix comprising research goals and subgoals (cf. Table 4.2) and their weightings as well as the matrix (PMS matrix compilation) containing performance measures, their weights and mappings to the research goals (cf. Table 4.4). These matrices were used to capture the contribution of individual KPIs to the goal achievement of the entire department. They represent, in summation, the performance measurement system as a set of weighted department goals with each goal being assessed by a collection of weighted KPIs. By means of this matrix we were able to collect ‘comparable’ data for the case study companies. The subgoals in this context were a methodological tool to develop the performance matrix in cases where companies had an underdeveloped performance management system. For example such a system could be characterized as where KPIs were unclear, the assessment of goals was not well-defined, and/or the relationship between goals and KPIs was not explicit. Subgoals helped to breakdown organizational research goals, in an intermediate step, into their measurable constitutive parts with which the KPIs could easily be associated.
4.1.2
Execution
After the first round of interviews, which provided the first insights into performance management practices, we selected a set of companies with which to test our
7 The EU industrial R&D investment SCOREBOARD groups companies into four sectors of R&D intensity: high (above 5%), medium (2%–5%), low (1%–2%) and very-low (less than 1%). 8 The detailed list of the first round interviews is in Appendix C.
4.1 Case Studies
117
research questions. We conducted a series of in-depth case studies to investigate how performance measurement in industrial research organizations is reflected in practice. The empirical investigation employed a multiple case study methodology9 as it allows for in-depth cross-examination of each case in its own complex environment and natural setting. In this section we describe the organization and structure of the case studies and summarize our first findings. To select candidates for the in-depth case studies we added a fourth criterion to the three selection criteria used to choose candidates for the first interviews. This fourth criterion is the existence of performance management within the company. The case studies were conducted as semi-structured expert interviews with several iterations. The following companies were selected: ABB, EMC, Deutsche Telekom, IBM, Intel, Microsoft, Philips and SAP. The interviewees were the heads of the research departments as well as additional people such as managers in charge of controlling research or heads of project management offices responsible for collecting, processing and editing the performance measurement data for review meetings with decision-makers. To be able to capture the data within companies as comprehensively as possible, three people from the research departments of each company were interviewed.10 The case study is divided into three parts as shown in Table 4.1: the first part addresses the organizational context and research environment; the second part deals with the management of performance in the research unit, and the third part captures the peculiarities of each case that were discovered during the case study. Table 4.1 Case study outline
Case study outline 1. Organizational context and research environment: (a) Company profile (b) R&D Characteristics (c) Research organization structure (d) Research process (e) Methods and tools 2. Performance measurement: (a) Research goals/subgoals (b) Research KPIs (c) Mapping goals and subgoals/KPIs 3. Conclusions
Part 1. contains five subsections (1.a–1.e). 1.a starts with a brief introduction of the company with key facts and figures such as: size, locations, sector, market presence, number of employees, products and services, investment in R&D and in
9
Eisenhardt (1989), Yin (2006). Information about all interviews, including names and positions of interviewees, interview dates and methods are summarized in Appendix C.
10
118
4 Performance Management: Analysis Approach
research only. Section 1.b describes the research department in terms of input, activities and output. The section, by way of a diagram, lists the inputs, activities and outputs. This provides a basis for understanding the relationships between inputs, activities and outputs as well as the goals and KPIs related to them. Section 1.c addresses the embedding of the research department within the overall structure of the company, as well as the internal structure of the organization. Section 1.d describes how research is conducted, more specifically, the detailed sequence of activities carried out in the research organization. Finally, Section 1.c describes tools that support the research process, the production of outputs, as well as collaboration with external scientific communities and internal networks. The core of the empirical investigation is Part 2.“Performance Measurement”. The focus in Section 2.a is on the kind of organizational goals that industrial research departments have and the way they are prioritized from an internal research view and from a corporate view. Sections 2.b and 2.c explore the performance indicators that industrial research organizations use to measure and monitor the achievement of their goals, as well as capturing the relationships and contribution of each sub-goal or KPI to research and corporate goals. Peculiarities and key observations from each case study are summarized in Part 3. In the following section we present just one case study to illustrate the type of the data that was collected. The remaining seven case studies are available in Appendix D.
4.1.3
Example Case Study “SAP Research”
4.1.3.1
Company Profile
SAP11 is the world’s leading provider of business software. Based on the SAP NetWeaver Platform, SAP provides its customers with a wide range of business solutions. SAP defines business software as comprising enterprise resource planning (ERP) and related applications such as supply chain management (SCM), customer relationship management (CRM), product life-cycle management (PLM) and supplier relationship management (SRM). These five major enterprise applications form the SAP Business Suite, which is SAP’S core product, which is supplemented by further product offerings and solutions for Small and Midsize Enterprises (SME). SAP is headquartered in Walldorf, Germany and employs about 43,800 people worldwide (as of December 31, 2007), of which more than 14,700 are located in Germany. The company is represented by its employees in more than 50 countries, serving over 47,000 customers in more than 120 countries. The shareholder structure12 of SAP is characterized by a wide distribution of share ownership: Approximately 27.2% from a total of 1,246.6 million shares are in
11 12
Internet source: http://www.sap.com/about/index.epx. Internet source: http://www.sap.com/about/investor/stock/shareholders/index.epx.
4.1 Case Studies
119
free floatation. Approximately 28.8% of the shares are held by the three founders of SAP, their trusts and holding companies (without family members). The company itself holds about 3.9% of the shares. In 2007 SAP achieved total revenues of €10.25 billion, which lead to a profit of €1.94 billion. Approximately 14% (€1.46) of the revenues were spent on R&D activities.13
4.1.3.2
R&D Characteristics
Figure 4.2 is derived from Brown and Svenson14 and Samsonowa, Buxmann and Gerteis15 showing the environment as well as the main inputs, activities, outputs and outcomes of SAP’s research organization.
Fig 4.2 Definition of inputs and outputs of SAP Research (Source: Samsonowa et al. 2009)
SAP Research was established in 1999 and is mainly focused on the execution of collaborative research activities. It distinguished the following four different types of research project:
13
Internet source: http://www.sap.com/germany/about/press/archive/press_show.epx?ID¼4266. Brown and Svenson (1998), pp. 30–35. 15 Samsonowa et al. (2009), p. 162. 14
120
4 Performance Management: Analysis Approach
University grant projects: Research activities are conducted within the scope of a grant project by an academic partner of SAP Research at universities. The content of grant projects is defined in collaboration with SAP Research. Co-innovation research projects: The content is defined jointly by SAP Research and a project consortium and the funding organization (e.g. the European Union). Transfer projects: Transfer projects ensure cutting edge technology in SAP products and allow for research outputs to be transformed into products. Transfer projects are completely, or at least to a certain extent, funded by SAP Business Units. Seed-funded projects: Seed-funded projects are set-up primarily to identify future trends and to enable breakthrough innovation. The content is solely defined by SAP Research in order to build competencies in areas, which go beyond today’s businesses, and have the potential to result in competitive advantage in the future. Funding characteristics (as of 2007): The ratio between internal and external funding at SAP Research in the EMEA and Australia regions is approximately 55:45. The Americas region is totally internally funded. Internal (55): 82% of internal funding is corporate-financed (a fixed budget allocated to the research department); 18% is financed by business or development units. External (45): external contractual research (including publicly-funded projects). 4.1.3.3
Research Organization Structure
The research organization of SAP is as a corporate entity, which is directly accountable to the Executive Board. The Head of SAP Research is a direct subordinate of the Executive Board member, which is in charge of the department of Research & Breakthrough Innovation (R&BI) (Fig. 4.3).
Fig. 4.3 Organizational structure of SAP (Source: SAP Annual Report 2007)
SAP Research comprises Research Centers and Campus-based Engineering Centers, as well as the corporate venturing group: SAP Inspire (Fig. 4.4).
4.1 Case Studies
121
Fig. 4.4 Organizational structure of SAP Research (Source: SAP Research Report 2007)
SAP Inspire collects ideas from internal and external entrepreneurs (employees, customers, etc.), evaluates them, and selects those with the most promising growth opportunities to successfully bring them to commercialization along the whole innovation process in a time-frame of about 18 months. There are two different types of organizational setups for research locations: Campus-based Engineering Centers (CECs) are located close to universities thereby leveraging the talent and expertise of academia; SAP-Labs-based Research Centers (SRCs) are co-located with SAP development labs, which allows the close alignment of research and development (Fig. 4.5).
Fig. 4.5 Distribution of SAP research locations (Source: SAP Research Report 2007)
122
4 Performance Management: Analysis Approach
Six research programs (cf. Fig. 4.6) build the pool of technology expertise in SAP Research. They are distributed across the above-mentioned locations with at least two research programs being conducted at each location. Each research program is lead by a Research Program Manager (RPM) who is responsible for the technology vision of the research program and its implementation by university grant projects, publically-funded projects as well as transfer projects.
Fig. 4.6 Research programs at SAP research (Source: SAP Research Report 2007)
SAP Research is supported by three shared functions: Research Portfolio Office (RPO) dealing with portfolio management and internal business development, Project Management Office (PMO), which is responsible for the administrative management of publically-funded projects and Communications, which supports all kinds of communication, media and events of SAP Research.
4.1.3.4
Research Process
Starting with the identification of emerging trends and technologies, the organization develops a strategic roadmap to build expertise in certain research topics with a time horizon of 3–5 years until the first major results become market-relevant. Research topics are dealt with in the phases shown in Fig. 4.7 where a dedicated research project spans one or two phases. There is no formal transition between the phases other than the production of the outputs associated with each phase. Within the co-Innovation projects, as well as the transfer projects, a very rigid milestonebased approach with concrete deliverables and results is used. Project progress and results are assessed by SAP-external reviewers and/or the receiving organization. The idea selection process of SAP Inspire is oriented alongside a funnel model: submitted ideas have to pass several quality gates before they can proceed to the next phase of the innovation process and be realized within a prototyping project (Fig. 4.8).
4.1 Case Studies
123
Focus SAP Research Basic Research
Applied Research
Industrial Research
Qualified Idea
Demonstrator
Prototype
Market Evaluation Customer Pilot
Production or Spin Off Innovative Project
Collaborative Research Invent
Validate
Transfer
Fig. 4.7 Research process at SAP research (Source: SAP Research Report 2007, p. 29)
Fig. 4.8 Idea selection process at SAP Inspire (Source: SAP Research Report 2007)
4.1.3.5
Methods and Tools
SAP Research has developed the PMO Suite, which supports the whole innovation process from idea generation for project proposals, to project close and transfer. The following tools are available under the umbrella of the PMO Suite: INNOVATION: The InnoSuite manages the proposal process for publicallyfunded projects. This includes initial proposal ideas, initial planning and execution location selection. PARTNER: The Partner Database keeps track of project relationships with partners in collaborative research projects. PLANNING: The CAPA tool is a resource planning tool for managing resources in projects. Records about the progress of the projects are kept, comparisons of target and actual performance are drawn and administrative project information can be shared depending on user roles.
124
4 Performance Management: Analysis Approach
INFORMATION: The SAP Research Net (SRN) provides information about every single researcher, his area of expertise, and the projects he has been engaged in. Furthermore, every research project is listed with the information about the scope of the project, related projects and the relevant stakeholders of the project. Thus, communication between researchers is simplified and enhanced and transaction costs can be reduced. WIKI: The SAP Research Wiki is the information-sharing tool for the SAP internal audience. PUBLICATION: SAP Research is using BibSonomy16 to manage and keep record of the publications of researchers.
4.1.3.6
Research Goals
As the global technology research unit of SAP, SAP Research’s goal is to significantly contribute to SAP’s product portfolio and extend SAP’s leading position by identifying and shaping emerging IT trends through applied research. The research activities are focused on strategically important business areas of the company and are aimed at leveraging the entrepreneurial and inventive talent within SAP. Table 4.2 Relative weighting of SAP corporate research goals Name Description
Weighting
Research Corporate view view 1. Intellectual Create and secure IP to ensure commercially exploitable 10 10 property benefits to maximize competitive advantage and protect innovation leadership 2. Alignment Give input to Business Units/Development to drive 30 40 & transfer incremental innovation into SAP’s existing product portfolio and influence future products 3. Image Create a positive image (internal & external) to ensure 15 5 successful interactions with all partners 4. Operational Ensure efficiency and effectiveness: what is an 5 15 excellence obligation in today’s business processes 5. Technology Achievement of technical goals within a project or 30 20 creation research field 6. Talent Discover, develop and retain talented resources for SAP 10 10 ∑ ¼ 100 ∑ ¼ 100 Source: Compiled based on interviews with representatives of SAP Research (The detailed list of SAP Research interviewees is in Appendix C)
4.1.3.7
Research KPIs
The following table (Table 4.3) shows the list of key performance indicators used by SAP Research together with the metrics that are used to evaluate them.
16
A Social Bookmark and Publication Sharing System.
4.1 Case Studies
125
Table 4.3 Subgoals and KPIs used to assess SAP Research Subgoals KPIs Transfer efforts into business units # Of person days delivered to BU’s Rate of significant contributions to business Subjective evaluation by the BU manager about units the significance of the transferred research results for development Product/service scenarios selected for board & # Of products/scenarios with a positive rating by CTO consideration the Advisory and Investment Board Vision research program in place and reviewed Research program paper of program manager which gets subjective approval of superior manager Research program implementation Extent to which projects meet the vision of the research program (subjective evaluation) Ratio of key technology trends uncovered by % SAP research 3 years ago that are considered strategic for today’s product development EU projects delivered on time and budget % Sponsored research contract results # of projects conducted at universities with a disseminated within SAP fixed grant from SAP research Money provided from outside vs total budget % Quality IDFs submitted to SAP IP Office # Of filed IDFs Patents granted to SAP research vs. SAP all % Different idea submitters # Of different people who submit proposals for research projects Ideas/business cases that pass first quality gate # Of proposals achieving a positive subjective (Steering Committee) and get a positive rating by the committee and by SAP rating by the community employees Business cases approved by SAP Board # Of proposals approved by Executive Board Stimulation of innovative talent No metric in place Attract external senior scientists to join SAP # Of external researchers joining SAP Research Research Research top talent leaving to other SAP areas # Of SAP researchers Image of SAP Research within SAP No metric in place External image of SAP research No metric in place Scientific publications # Publications # Publications in newspapers, customer magazines, etc. Partner and customer events/workshops # Of events where demonstrators or prototypes are shown Company fair contributions # Of internal events where the awareness within the BU’s can be raised Quality of patent filings Ratio of # patents filed vs. # patent granted Ph.D. theses completed # Guest researchers from universities # Source: Compiled based on interviews with representatives of SAP Research (The detailed list of SAP Research interviewees is in Appendix C)
4.1.3.8
Mapping Goals and KPIs
The following table (Table 4.4) shows the mapping of subgoals to goals and their relative weighting that has been developed with SAP during a series of interviews.
Intellectual Property
100
5 10 15 10
100
30 20 15 5 20 10
100
15
10 30 30 15
Alignment & Transfer Image Operational Excellence 40 20
100
50 20 30
Technology Creation
Source: Compiled based on interviews with representatives of SAP Research (The detailed list of SAP Research interviewees is in Appendix C)
KPIs Transfer efforts into business units Rate of significant contributions to business units Product/service scenarios selected for board & CTO Vision research programme in place and reviewed Research program implementation Technologies uncovered 3 years ago that are strategic now EU projects delivered on time and budget Sponsored research results disseminated within SAP Money provided from outside vs. total budget Quality IDFs submitted to SAP IP Office 30 Patents granted to SAP Research vs. SAP all 70 Different idea submitters Ideas passing first quality gate with a positive rating Business cases approved by SAP Board Stimulation of innovative talent Attract external senior scientists to join SAP Research Research top talent leaving to other SAP areas Image of SAP Research within SAP External Image of SAP Research Scientific publications Publications Partner and customer events/workshops Company fair contributions Quality of patent filings PhD theses completed Guest researchers from universities ∑ 100
Goals
Table 4.4 Mapping KPIs to goals and their relative weighting
30 10 100
30 30
Talent
126 4 Performance Management: Analysis Approach
4.1 Case Studies
127
In this regard, the table actually shows the subjective assessment of which goals the KPIs are likely to affect.
4.1.3.9
Conclusions
Most of the different KPIs are clearly assigned to well-defined research subgoals in a formalized framework. The subgoals and KPIs are developed in an iterative process between the research department and the top management and serve the strategic adjustment of the research activities. Where it makes sense, subgoals are further broken down and are part of personal target agreements with individual researchers. Other observations: • • • • •
PMS not mature, but advancing, as the organization grows; No financial evaluation of research results; Mainly objective measurement criteria; Image is set as a goal, but no definite KPI has been defined; In general 4–5 KPIs are used to measure a single goal.
4.1.4
Initial Findings and Further Approach
When conducting the case studies, two kinds of companies were found: (1) companies with a sophisticated performance measurement that could easily answer all our questions related to Part II. of the case study, and (2) companies where performance measurement was weakly developed. The latter cases can be characterized as follows: although goals exist, their priorities are not defined and/or the goals are not really reflected in the KPIs and/or there were no weightings available. If priorities or research goals were missing, the interviewees were first asked for an estimated weighting of the research goals according to the perceived importance. Secondly, the interviewees were asked about the KPIs that they were monitoring and how they would relate them to goals. As mentioned earlier, we used the concept of subgoals to develop this mapping, which resulted in the PMS matrix compilation. The mappings in our case studies do not necessarily reflect the concrete performance management system in place for the research department, but rather indicate the perceived performance measured. When examining the case studies with respect to feedback loops of our theoretical framework, it becomes evident that the feedback loops implemented with respect to performance evaluation only partially match. The feedback loops that are introduced in the theoretical framework (four loops coming back from activities, two from outputs and outcomes) are in reality only formally implemented through performance measurement cycles. The difficulty of holistic measurement is that the performance measurement cycles, including the internal reward cycles, are executed on an annual basis. Therefore, companies generally tend to only reflect
128
4 Performance Management: Analysis Approach
annual and/or shorter term (mostly quarterly) measurement cycles. This annual assessment is a consequence of the nature of the financial year and the necessity for the organizational unit to plan its budget and the allocation of its resources adequately. In the case of activities monitored on a quarterly basis, and outputs that can be measured annually, there is no problem. However, the real value-added outcomes from research, such as additional license revenues or brand value, are only quantifiable after a significant time delay and therefore cannot be measured within a single annual performance measurement cycle. Nevertheless, it is common practice to measure these outputs with KPIs, even though the KPIs may only reflect the longer-term outcomes to a very limited extent. The outcomes are also difficult to measure because the value of an isolated research output within an aggregate company outcome is difficult to assess. This is particularly true where large and complex software systems are produced and the research results may ultimately only represent a small part of the overall system, but the research could still be a critical part of the system. One-hundred-and-sixty-four different KPIs were collected from the eight case studies, as well as 45 organizational goals of which 30 were unique. These numbers by themselves hit at great complexity and that direct comparison of performance measurement between companies with contrasting research goals and KPIs is not possible at this level of granularity. To cope with this situation (164 different KPIs) an abstraction of the KPIs was needed to permit harmonization of KPIs according to our framework which we introduced in Fig. 4.1. Such an abstraction could, for example, be similar to four categories of the balanced scorecard; it needs to reflect the activities of a research department and mirror the content perspective, which it ultimately attempts to assess in terms of the performance. The information gained from case studies served two purposes: (1) the development of an activity-oriented performance management system, and (2) the preparation of a large quantitative survey to validate the findings gained from the qualitative analysis. Altogether, four major analysis steps were carried out: 1. Performance Cluster Definition: In-depth analysis of the performance data collected in the case studies by deriving a set of generic research activities based on a grouping of KPIs and their relationship to goals. The result was that 11 performance clusters representing the performance-related research activities were identified and validated; 2. KPI Consolidation: Classification of the KPIs of each performance cluster into the KPI classes was made; 3. Cluster Analysis: A calculation and comparison of the weights of the performance clusters across the case studies was conducted; 4. Goal Spectral Analysis: An analysis of relationships between individual goals as well as between goals and KPIs of research departments in terms of performance cluster contributions was made. In the following section, we present the results arising from the four analysis steps.
4.2 Performance Clusters
4.2
129
Performance Clusters
Initially we assumed that the direct comparison of the performance management systems of different companies is possible by directly comparing the collected research goals and KPIs. However this turned out to be extremely difficult for two reasons: 1. Although the goals of different companies appear similar at a first glance, when exploring the KPIs that were used to measure these goals, it seemed that the interpretation of the goals by each company is markedly different. Likewise, the opposite case exists: although goals initially sounded very different, the companies seemed to be evaluating similar things from the perspective of KPIs. 2. The direct comparison on the KPI level is simply impossible due to the sheer number of different KPIs.
4.2.1
Approach and Definition
In order to cope with this situation we used a clustering approach that allowed for a harmonization of KPIs according to our framework introduced above (cf. Fig. 4.1). The harmonization approach draws upon a grouping of KPIs from a content perspective that especially reflects activities that are being conducted, or are needed, in order to contribute to the achievement of goals. The following four steps describe the evolution of the performance clustering process. Step one: Clustering of KPIs The initial set of clusters was identified by applying the following procedure to the KPIs collected in the case studies: • The KPIs were analyzed with respect to their similarity regarding the artifact classes (i.e. inputs, outputs, activities and outcomes) that are measured. • The assignment to clusters was done by answering the following two questions: first, what is being measured (artifact), and second, why is it being measured (goal/sub-goal). KPIs addressing similar artifacts (what?) in a similar context (why?) were assigned to the same cluster. In general, there was a clear match between one KPI and one cluster. There were few exceptions: – In some cases the same KPI was assigned to multiple clusters due to different contexts in which they were measured (the same KPI was contributing to different goals); – In other cases the same KPI was split into its constituent parts reflecting the different artifacts that were measured; the KPI constituents were then assigned to clusters. Step two: Cluster validation The clusters were validated by a literature review. In particular it was based on the OECD Frascati family of manuals elaborating the concept of science and technology
130
4 Performance Management: Analysis Approach
indicators, and on the work of Chesbrough17 on the open innovation concept. The result from this analysis was a set of typical research activities related to each cluster. Step three: Identification of underlying process From our observations on our framework (cf. Fig. 4.1) on a research department and its environment, and the literature review, we realized that the activities summarized in the identified clusters can be organized in an underlying process. We examined each performance cluster to refine its activities by either decomposing the identified processes into activities, or by composing processes from the set of identified activities. The underlying cluster process can embrace the entire ecosystem of a research department and thereby even cross the boundaries of the company. Typical processes in this regard are: portfolio management, project management, partnerships/networking management, knowledge management, intellectual property management, idea management, technology asset management. Step four: KPI consolidation As a consequence of the identified underlying cluster processes, we revisited the original groups of KPIs and abstracted from the concrete KPIs to generic KPI classes that are organized according to the underlying cluster processes. By abstracting from concrete company-specific processes of the case studies to more general processes we were able to consolidate the KPIs within the performance clusters by formulating more generic KPIs. To derive more generic KPIs we analyzed the measurability of activities within each performance cluster: what is measurable about the activities (e.g. writing papers: number, quality, etc.), which of them reflect a process, or a part of a process, and if so, which parts of the process are measured? Depending on the process, in some cases the whole activity chain has to be assessed, i.e. also the parts that go beyond the research department, in order to attain the outcome and to be able to assess it. As a result, we were able to derive 37 KPI classes (see definition below) from the original 160 KPIs. As a result of the creation process described above, we are now able to define the term: performance cluster: A performance cluster comprises a collection of typical research activities as well as a set of specific KPI classes that relate to a common underlying process.
The underlying performance cluster processes can be characterized by the following properties: • Company internal vs. company external focus; • Formal vs. informal processes; and • Time pattern. Some of the underlying processes of the performance clusters are strongly linked to internal processes. The internal focus allows for an easier assessment of their performance, since measurement can be organized alongside a formalized and fairly transparent company-wide process with well-defined steps. External
17
Chesbrough (2003), Chesbrough et al. (2007).
4.2 Performance Clusters
131
processes are often less formalized, or their individual steps are less transparent, regarding objective assessment. The degree of formalization becomes apparent in the way in which the processes of an organization, or its steps to process formation, are defined. In the case of informal/latent processes there is a lack of transparency, and as a consequence, the possibilities to assess the progress of the processes are limited. Another characteristic of the underlying process is the time-lag between execution of the activities and the manifestation of its results. From the performance management context, the following timing patterns are conceivable: short-term consideration is within one performance management cycle, long-term consideration is at least one cycle and often more; non-termed is a continuous occurrence, and is therefore not limited time-wise, neither is it bound to the performance management cycle. Finally we define a KPI class as follows: A KPI class summarizes a set of KPIs with at least one common denominator in order to assess certain activities of a performance cluster.
The common denominator could for example be the artifact class in terms of input, output, process and outcome, or the generic artifact property regarding quality or quantity. Table 4.5 lists the 11 performance clusters, which were identified via the creation process as described above. Table 4.5 Performance clustersa No Underlying process Cluster name Acronym 1. Transfer research results Technology Transfer TT 2. Identify future business opportunities Future Business Opportunities FBO 3. Define and manage the research Research Portfolio Management RPM portfolio 4. Create intellectual property Intellectual Property Creation IPC 5. Operate effectively and efficiently Operational Excellence OpEx 6. Develop a talent pool Talent Pool TP 7. Create a thought leadership image Image IMG 8. Publish research results Publications PUB 9. Show presence in the scientific Presence in Scientific Community PSC community 10. Collaborate with academia Collaboration with Academia CA 11. Collaborate with partners and Collaboration with Partners and CPC customers Customers a In the remainder of this document we omit the term ‘performance’ when we referring to concrete performance clusters (e.g., Technology Transfer cluster or TT cluster instead of Technology Transfer performance cluster)
In the following section we describe each performance cluster on a very detailed level. The performance clusters are characterized, according to our definition, from the following perspectives: • Underlying process which predominates the performance cluster; • Corresponding activities;
132
4 Performance Management: Analysis Approach
• A set of KPI classes with the relevant artifact classes and properties; • Examples of concrete KPIs with concrete artifacts and properties; • References to literature that validate the cluster.
4.2.2
Cluster: Technology Transfer18
The Technology Transfer (TT) cluster comprises all activities that are relevant to hand over research results to other groups of a company, typically development or production units. The transferred results must have an exploitation potential (commercial) and the recipient must have an interest and/or plan in place for how to exploit them. Exploitation is in general deemed to be commercial and has two facets: cost-saving and revenue generating. Therefore, the important aspect of the TT cluster is to capture and assess the output and/or outcome of research and transfer activities. Output includes the tangible artifacts that are transferred, their refinement during the transfer process, as well as all activities that help to convey the knowledge upon which the artifact is built or is based in order to ensure further (product) development by the receiving unit. In this regard we consider code as a tangible result in the software industry. The handover may happen in different ways, starting with explicitly-defined and formalized processes to simple informal knowledge exchange. The more formalized the processes specified, the easier it is to define and measure process parameters besides the above-mentioned output and outcome parameters. Since the focus is clearly on output and outcome, the details of the underlying process are of less importance. As mentioned above, the term “transfer” implies two parties (sender and receiver) being involved. The findings from our case studies show that from the organizational perspective, the activities around technology transfer are commonly organized within projects. The advantage of the “project” form is the formal character that forces both parties to define the transferred artifact and to estimate its value from each perspective. The assessment of the technology transfer will depend on how the projects are finally organized. Ultimately, all companies will organize the details of this process differently, depending upon their specific needs and organizational settings. As such, the concrete design of the process is less relevant. Of greater relevance are the process characteristics, which can be measured, or those that ideally would be measured, in terms of inputs, specific gates, outputs or outcomes. For the TT process to succeed, a number of generic steps need to be followed. Figure 4.9 depicts the steps involved in such a process. While each concrete case (transfer project) is different, some of the key steps and questions involved in bringing forward new technologies through the value chain of the company can be uniformly presented. These key steps are:
18
Here we consider intra-organizational technology transfer within a company – between departments, not between different organizations or institutions.
4.2 Performance Clusters
133
2 Business case
1 Identification of transfer potential
4 Sign-off
3 Terms & conditions
6 Closing
5 Implementation
Fig. 4.9 Generic Technology Transfer process19
• Identification of a need, or enthusiasm, for a new idea/concept/technology (research output) in e.g. the business or development unit; • Design of a pro-forma business case; • Definition of terms and conditions (resources/funding); • Sign-off (agreement); • Implementation of the transfer object (execution/monitoring); • Closing/transfer completion and cross-charge or acknowledgment/product launch. Typically, technology transfer20 in an industrial research environment can originate from two different impulses. The two directions are technology push21 and technology pull. Depending on the motivation, the process is accordingly different. In the case of technology push (where the scientific and technological knowledge and skills can be applied to invent a new product or process) the situation typically is as follows: the technology from the technical point of view might be ready and some example use cases identified, but a concrete business scenario is still needed to test the technology in real life. The latter case is different: there is a concrete request (recognition of a need or a potential for an invention) for a specific technology. Depending on the concrete case, the process may encompass steps 1–6 in the case of technology push, whereas in the case of technology pull, the process will most likely directly start at step 3. In the case of the technology transfer, the process typically starts towards the end of the value chain of research (with the completion of a result to be transferred), and reaches beyond the borders of the research department, ending with further development of the artifact by the receiving unit. Using the outlined underlying process and analyzing the original KPIs that were assigned to the TT performance cluster from the case studies, we derived five KPI classes that together cover all KPIs (with some of the original KPIs being split into two or more parts and assigning the parts accordingly): • Transfer volume: volume of the technology transferred to development or other (business) units; • Transfer result quality: quality of the research results transferred;
19
Edvinsson and Malone (1997), Pham (2008), p. 92. Taschler and Chappelow (1997), pp. 29–34. 21 Sometimes also referred to as “idea push”. 20
134
4 Performance Management: Analysis Approach
• Transfer process quality: quality of the transfer process or transfer activities; • Transfer significance: significance of the transferred research results for the receiving unit; and • Economic transfer value: economic transfer value of the transfer activity or transferred research results. The generic TT process and the mapping to KPI classes associated with this performance cluster are shown in Fig. 4.10. 2 Business case
1 Identification of transfer potential
4 Sign-off
3 Terms & conditions
6 Closing
5 Implementation
1. Transfer volume 2. Transfer result quality 3. Transfer process quality 4. Transfer significance 5. Economic transfer value
Fig. 4.10 Generic technology transfer process mapped with KPI classes
Examples for typical technology transfer KPIs are: • Transfer efforts from research into business units measured by number of person days delivered to BU’s; • Number of component deliveries; • Significance of research projects measured by the investment of the business units in research projects e.g. in terms of resources dedicated to jointlyconducted research projects; • Time spent by researchers in the development or business units; • Net present value (NPV) of the transferred artifact. Many literature sources highlight the importance of technology transfer22 for research organizations. This issue, in particular in the industrial research domain, is intensively studied and well-understood. The most famous example is probably the book on high-profile failures at Xerox PARC.23 Nowadays, many companies have a dedicated “Office for Technology Transfer” to identify research
22
Tipping et al. (1995), pp. 32–63, especially their fourth managerial factor IWB, which stands for “Integration with Business”; Schmoch et al. (2000), Sommerlatte (2006), Geschka (1988). 23 Smith and Alexander (1999).
4.2 Performance Clusters
135
output that has commercial potential and to develop strategies to exploit it. To sum up, there are many processes and cross-divisional activities within companies that address technology transfer. These aim to seamlessly support innovation by reducing barriers for the diffusion of new technologies between departments.
4.2.3
Cluster: Future Business Opportunities
The Future Business Opportunities (FBO) cluster summarizes activities around the variety of research output and outcome, from fuzzy ideas to concrete business plans, all of which have the potential to lead to new income streams. If the TT cluster focuses on the concrete transfer process of research results into the development phase, the FBO cluster deals with the evolution of ideas within the overall innovation process and the contribution of the research department to this process. Both the need and importance of innovation, and the fact that innovation takes place on the market and is closely connected to commercial success are undisputed. Therefore, the FBO performance cluster is fundamental to innovation management, especially in industrial research. It encompasses aspects of idea generation and incubation, as well as new business development. It also includes facets of entrepreneurship in response to identified opportunities.24 The Oslo Manual25 covers the subject of development and diffusion of new technologies addressing technological innovation data. The manual has become the reference work for numerous large-scale surveys that examine the nature and impacts of innovation in the business sector. The manual shows that these activities are essential for research departments and therefore endorses the existence of this performance cluster. Figure 4.11 represents a generic process26 underlying the FBO cluster. The names of the stages and their demarcations, and to a minor extent even their sequence, might slightly differ from company to company, however all stages are necessary in principle: • Idea generation: this stage can happen in multiple ways as there are many different techniques and methodologies to support this stage (see especially Geschka 2006 for details). • Ideas screening and evaluation regarding their attractiveness, potential business impact and compatibility with the objectives of a company. • Concept generation and feasibility studies: This stage develops a concept for the implementation of an idea and evaluates the feasibility of this concept.
24
Heuser (2006), pp. 271–290. OECD (2005), Oslo Manual. 26 Scholl (2006), pp. 163–194, Heuser (2006), pp. 271–290, Geschka (2006), pp. 217–248. 25
136
4 Performance Management: Analysis Approach
• Finding sponsorship and piloting: With the sponsorship of an internal or external investor this stage develops a pilot in order to validate the concept and its feasibility in a concrete business scenario. • Implementation and incubation: The transfer of the pilot results to the development organization and its further development into a final product/service takes place during this stage. • Commercialization: This stage comprises a go-to-market strategy and the product/service roll-out. Idea screening
finding sponsorship
2 & estimation
1 Idea generation
4 / piloting
3 concept generation
/ feasibility
6 commercialization
5 Implementation
/ incubation
Fig. 4.11 Generic process of the future business opportunities cluster
This process is often visualized as a funnel, or a series of funnels, which implies that there are several stages to pass with each stage having specific evaluation criteria. The number of ideas is reduced, stage by stage, and thereby the most promising ideas are developed into real business opportunities. It is important to mention that the departments and people involved may change several times during the process. This is especially true if the level of maturity of an idea from the technological perspective is quite low. Because of these characteristics, the FBO cluster belongs to the group of fairly formalized, company-internal processes. The duration needed for an idea to pass through the various stages may be several performance management cycles and therefore exhibit a long-term pattern. This is particularly true for ideas with a low level of technological maturity and therefore a high likelihood that the research department is involved. The three KPI classes of the FBO cluster are arranged along this process (see Fig. 4.11): • Intensity of input into the FBO process; • (Weighted) # of ideas moved to a certain or the next phase of the FBO process; and • (Achieved) business impact of an idea in terms of its economic value. The three KPI classes comprise measures for the input (1), process (2), and output/outcome (3) of the FBO process. Using the throughput as the process measure seems to be more appropriate than measuring the mean “production” time, as the variance of the production times may be quite high due to the aforementioned different technology maturity levels. Therefore, ensuring and assessing a continuous stream of ideas seems to be the more appropriate choice. The generic FBO process and the mapping to KPI classes associated with this performance cluster are shown in Fig. 4.12.
4.2 Performance Clusters
2
1 Idea generation
137
Ideas screening & evaluation
4
finding sponsorship / piloting
3 concept generation
/ feasibility
input input intensity into the innovation process
6 commercialization
5 Implementation
/ incubation
throughput # of ideas moved to a certain phase of the innovation process or (weighted) # of ideas moved to the next phase
output / outcome (achieved) business impact of an idea in terms of its economic value
Fig. 4.12 Generic process of the future business opportunity cluster with KPI classes
The following KPI examples have been collected in case studies and are summarized within the FBO cluster: • • • • • • •
Number of ideas; Number of ideas per researcher; Number of ideas with business case developed; Number of ideas with demonstrator developed; Number of ideas handed over to development; Number of start-ups created in research; Number of products/business scenarios with a positive rating by the Advisory and Investment Board; • % Of key technology trends uncovered 3 years ago that are considered strategic for today’s product development.
4.2.4
Cluster: Research Portfolio Management
The cluster called Research Portfolio Management (RPM) characterizes a certain maturity level of the research department, which is reflected in the extent to which the portfolio of the research department is organized and/or managed. The RPM cluster encompasses all kinds of activities: from structuring and organizing the content areas that a research department is active in, describing the visions of these areas, through to developing and maintaining concrete implementation plans to achieve these visions. The activities of the RPM performance cluster are partially classified as management activities according to the Frascati Manual (FM). Generally, for the inhouse (not governmental) management activities, a distinction is made between: “. . . direct support for R&D by persons such as R&D managers closely associated with individual projects, who are included in both the personnel and expenditure series, and persons such as financial directors, whose support is indirect or auxiliary and who are included in the expenditure series only as an element of overheads”.27 The argument used by the FM to include these activities into R&D is as follows: “. . .when these contribute directly to R&D projects and are undertaken
27
OECD (2002), Frascati Manual, p. 19, clause 26.
138
4 Performance Management: Analysis Approach
exclusively for R&D, then they are part of R&D”.28 The FM lists “. . . the R&D manager who plans and supervises the scientific and technical aspects of the project or the person who produces the interim and final reports containing the results of the project”29 as a typical example of such activities. Furthermore, a clear overlap with our performance cluster RPM is given by Tipping et al. (1995) second Managerial Factor30: the Portfolio Assessment (PA). This factor is identified as communicating the total R&D program across various dimensions of interest, including time horizon, level of risk, core competency exploitation, and new/old business. The RPM is a dynamic decision process within which the number of active research areas, programs or projects is constantly being revised. This includes new research areas, programs or projects being evaluated, selected, and prioritized as well as existing ones that might be accelerated, killed, or re-prioritized, for example by allocating and reallocating. “The portfolio decision process is characterized by uncertain and changing information, dynamic opportunities, multiple goals and strategic considerations, interdependences among projects, and multiple decision makers and locations”.31 The following five generic steps represent a simplified portfolio management process within an industrial research department (Fig. 4.13).
Elaboration and asessment of a
2 vision for a research area/program
1 Identification of research
areas / programs –a portfolio
4
3 Elaboration and validation
of a strategic roadmap
Execution and implementation of the determined portfolio
5
Evaluation of the Implementation roadmaps
Fig. 4.13 Generic Research Portfolio Management process
• Identification: this process step includes trend-scouting and initial valuation of the relevance regarding the business of a company. • Elaboration and assessment of the vision tackles the next granularity level after a research area is determined. At this stage an individual area is defined and agreed upon in terms of a longer-term vision. • Elaboration and validation of a strategic roadmap: This step involves activities to define a concrete roadmap that is going to be implemented to achieve the vision agreed upon in the previous step. It also contains a consolidation of the individual roadmaps into an overall research portfolio. • Execution and implementation of the determined portfolio.
28
OECD (2002), Frascati Manual, p. 91, clause 293. OECD (2002), Frascati Manual, pp. 45–46, clause 132. 30 Altogether Five Managerial Factors have been identified by Tipping et al. (1995), pp. 32–63; compare Sect. 3.3.2, pp. 107–108. 31 Cooper et al. (2001), p. 3. 29
4.2 Performance Clusters
139
• Evaluation and readjustment of the implementation roadmaps: This step reflects the resulting portfolio for a certain point in time as it is an on-going (live) process, which also includes the closing and phasing out research areas. Within these five steps, which are presented on a high level here, numerous activities represent the detailed and lower-level decision-making process: optimal allocation of resources, selecting the right and killing poor projects, deciding on spending in the right areas, eliminating redundancies, aligning projects in terms of an overall strategy, managing gaps in the existing portfolio, planning the next periodic review of the total portfolio. The RPM cluster can be assessed with the following four KPI classes: • • • •
Structure and quality of the research (i.e. certain technology areas); Visions related to the individual parts of the research portfolio and their quality; Roadmaps to achieve the visions and their quality; and Implementation of the roadmaps and the quality of contributions from research projects.
These four KPI classes sequentially assess the progress within the process. The generic RPM process and the mapping to KPI classes associated with this performance cluster are shown in Fig. 4.14. Elaboration and asessment of a
2 vision for a research area/program
1 Identification of research
structure and quality of the research
Execution and implementation of the determined portfolio
3 Elaboration and validation
5 Evaluation of the Imple-
of a strategic roadmap
mentation roadmaps
areas / programs –a portfolio
Structuring
4
Vision visions related to the individual parts of the research portfolio and their quality
Roadmap roadmaps to achieve the visions and their quality
Implementation implementations of the roadmaps and the quality of contributions from research projects
Fig. 4.14 Generic Research Portfolio Management process mapped with KPI classes
The following KPI examples have been collected in case studies that were summarized within the RPM cluster: • Vision paper for Research Programs from Program Manager available and approved by superior manager; • Extent to which projects contribute to the vision of a Research Program (subjective evaluation); • External subjective evaluations from the recognized experts in the research field: “normal” accomplishment (solved an unsolved problem/generated visibility around his/her research); • Percentage of maximum score in Project Portfolio Score Sheet averaged over all program areas (creating and maintaining a well-balanced research project); • Subjective, qualitative evaluation by project lead and program manager;
140
4 Performance Management: Analysis Approach
• Subjective evaluation of the structure of the research work by internal or external reviewers (simplest way: the structure is in place or not); • Subjective evaluation of the vision documents of the individual research programs by internal or external reviewers (simplest way: vision documents are available or not); • Subjective evaluation of the roadmap to reach the vision by internal or external reviewers (simplest way: the roadmap document is in place or not); • Subjective evaluation of the scientific accomplishments, technical achievements, or technical impact by internal or external reviewers.
4.2.5
Cluster: Intellectual Property Creation
The Intellectual Property (IP) Creation cluster comprises KPIs that address research output and outcome in the form of ideas and inventions that are (1) created by researchers and that the company is trying to protect for a certain period of time; or (2) published by researchers in order to prevent others from protecting the idea or invention. The ideas and inventions are generally captured by invention disclosures. Depending on the IP strategy of the company these might for example be filed as patents or made public as defensive publications. In paragraph 75, the Frascati Manual states that patent work directly connected with R&ED projects is part of R&ED. For example, preparing initial invention disclosures together with the supporting technical documentation used as a starting point for the filing is an activity that is acknowledged as an activity conducted by the research department. This Frascati statement justifies the activities being grouped in this performance cluster. The manual also discusses the fact that there are several stages in the protection of IP and that statistics that consider these stages are more meaningful than statistics just reflecting granted patents, cf. Annex 7, paragraphs 5–7 and 9. This affirms the underlying process of the cluster as well as the KPI classes, which were identified. In addition, the Patent Statistics Manual32 deals with the measurement of scientific and technical activities. The fact that the OECD dedicated specific extra work on patents provides additional justification for subsuming the activities related to IP creation in a performance cluster of their own. The intellectual property creation process is most likely one of the most formalized processes in industry that secure competitive advantage. This is a result of the fact that the process has to be closely aligned with legislation regarding intellectual property protection and in turn with the modus operandi of patent offices and other related authorities. The most typical and relevant stages of the process are depicted in Fig. 4.15 below: • Invention Disclosure Form (IDF) is a written document which is presented to the internal legal or IP department. This document contains information that is often
32
OECD (2009), Patent Statistics Manual.
4.2 Performance Clusters
141
confidential. In general it is a brief description of the invention. It might be a new process, a device, a technology, a new application, or an improvement to an existing product or process. • First filing: the IP department typically drafts a patent application (first filing). This is then submitted to a national33 or international34 patent office. • Patent: the patent office eventually decides whether the patent will be granted or the application will be rejected. 1
Invention Disclosure Form
2 First filing
Patent granted / defensive publication
3
Fig. 4.15 Generic Intellectual Property Creation process
Our IP performance cluster has five KPI classes, which can be divided into two groups. The first summarizes quantitative KPI classes such as: • The volume of potentially protectable inventions submitted into the IP pipeline; • Volume of IDFs passing a certain intermediate stage in the external protection or publication process; • Volume of patents granted and/or defensive publications. The second group addresses qualitative KPI classes: • Quality of granted patents; • Alignment of research activities with the IP strategy of the company (Fig. 4.16).
qualitative
quantitative
1
Invention Disclosure Form
2 First filing
volume of potentially protectable inventions submitted into the IP pipeline
volume of first filings out of the IP pipeline
alignment of research activities with the IP strategy of the company
3
Patent granted / defensive publication
volume of patents granted and/or defensive publications
quality of granted patents and/or defensive publications
Fig. 4.16 Generic Intellectual Property Creation process mapped with KPI classes
33
For example: DPMA – Deutsches Patent- und Markenamt; INPI – Institut national de la proprie´te´ industrielle in France; USPTO – United States Patent and Trademark Office. 34 For example: WIPO – World International Property Organization; EPO – European Patent Office.
142
4 Performance Management: Analysis Approach
The following KPI examples have been collected in the case studies: • • • • •
Number of invention disclosures; Number of first filings, defensive publications, trade secrets; Number of patents granted to the company; Economic value of the granted patent; Number of submitted inventions addressing the IP strategy.
4.2.6
Cluster: Operational Excellence
The Operational Excellence (OPEX) cluster deals with the efficiency of process execution in an organization. It contains a set of auxiliary processes that support the operational structure of an organization as a whole, or its constituent parts. The first three KPI classes in the cluster: • Adherence to budget. • Adherence to timelines; and • Quality of the risk management in place need to be allocated to the operational units so they can easily be consolidated again for monitoring purposes. In fact, as consolidated figures they represent the actual status of the research organization in terms of operational achievements. As a consequence, those KPIs belonging to these classes are quite often collected and evaluated several times per performance management cycle. The KPI class • Quality of project management is in fact a consolidation of (1)–(3) in the sense that compliance at agreed deadlines is an important factor for the quality of a project. Additionally, it also deals with other project management tasks such as resource planning, conflict resolution, etc. The KPI class • General risk sharing is a partial answer to deal with the uncertainty of research. This is predominantly done by sharing the investment in research undertakings. Several studies35 show that the perceived quality of the working environment has an impact on the creativity of people and teams. Therefore, the KPI class • Quality of the working environment is an important operational factor, in particular within a research department. From a KPI perspective, this is supported by our case studies where the assessment of the working environment was present several times.
35
Amabile et al. (2003), p. 1157, Woodman et al. (1993), p. 295, Amabile and Conti (1994).
4.2 Performance Clusters
143
Based on these descriptions it becomes evident that the auxiliary processes of this cluster are mainly internal processes. Furthermore, they need to be fairly formal, especially as far as their breakdown and consolidation are concerned, and therefore the related KPIs are quite often evaluated several times within a single performance management cycle. The following examples for KPIs that were subsumed in the OpEx cluster have been collected in case studies: • Difference between the planned budget for each line item to the actual budget measured (least deviation is rated to be the best); • Successful (on-time) termination at a particular phase / gate; • Data compliance at an agreed deadline; • Scoring in employee satisfaction survey; • Percentage by which the potential damage has been lowered by adequate measures against the bottom-line; • Volume of third-party investment (in €), number of participations in externallyfunded programs.
4.2.7
Cluster: Talent Pool
The Talent Pool (TP) cluster summarizes activities that refer to recruiting people into the research department, their continuous development inside the department, and eventually their transfer into other parts of the organization or back into its eco-system. The Frascati Manual devotes the following paragraphs to discuss the measurement and classification of R&ED resources: 68, 92, 96-100, 149, 300-303, 306-309, 313 and annex 7: 41–43. The sheer number of paragraphs shows how important this matter is. Thus, the OECD saw the necessity to produce an extra document addressing human resources aspects within R&ED: the Canberra Manual.36 In order to obtain a complete picture of both the supply and demand for human resources in science and technology, the manual is based on two dimensions: qualification and occupation. The Manual states that highly-skilled human resources are essential for the development and diffusion of knowledge and that they constitute the crucial link between technological progress and economic growth, social development and environmental well-being. The manual argues that the lead time to train and to develop S&T skills are long and the costs high. The fact that the manual explicitly focuses on these topics highlights the importance of the TP cluster. As a consequence, it is legitimate to assess a research department upon metrics that deal with activities to train, retrain and utilize these expensive and often rare skills. Thus, our proposed cluster is justified. The processes within the Talent Pool Cluster are all people-related processes. Historically, labor law enforces many formal processes, which all companies have to comply with. These processes embrace, for example, the employment, deployment
36
OECD (1995), Canberra Manual.
144
4 Performance Management: Analysis Approach
and development of employees as well as the termination of work contracts. This explains the pronounced tendency of existent formalized processes in this cluster. On a generic level the following process stages can be identified (Fig. 4.17): 1 Inflow
2 Development
3 Outflow
Fig. 4.17 Generic Talent Pool process
• The Inflow step contains the initial contact between the company and potential employee, assessment of the personality profile and face-to-face interviews to attract talent. • Development comprises the steps that are necessary for development and deployment, management and evaluating talents, like 360 feedbacks, leadership diagnostics, coaching, mentoring, exposure to upper management or other special incentives/opportunities. • Outflow vs. retention of talent from the research department into the other departments of a company are the activities within this stage. Quite often, along with an idea/business opportunity being transferred or further developed, people are moving along the value chain and thereby accompanying the evolution from idea into a product or service. Within the Talent Pool cluster we identified four KPI classes: • Volume/quality of people hired into the research organization; • Volume/quality of people leaving the research organization to move to other parts of the company/ecosystem; • Volume/quality of development measures undertaken; and • Quality of the people in the research department. 2
Development
3
Outflow
quantitative
Inflow
volume of people hired into the research organization
volume of development measures undertaken
volume of people leaving the research organization into other parts the company / ecosystem
qualitative
1
quality of people hired into the research organization
quality of development measures undertaken; quality of the people in the research department
quality of people leaving the research organization into other parts the company / ecosystem
Fig. 4.18 Generic Talent Pool process mapped with KPI classes
As can be inferred from the names of the KPI classes, these could be further subdivided into two groups with the focus on quality and quantity. We explicitly did
4.2 Performance Clusters
145
not do this as we wished to avoid creating too many KPI classes in this performance cluster (Fig. 4.18). The following KPI examples, which are relevant for the TP cluster, have been collected in the case studies: • Number of externally-attracted senior scientists that match the profile for which they are hired; • Number of top talents moving to other parts of the company; • Number of job rotations, number of PhDs finished; • Subjective assessments through partners or stakeholders, team track records, 360 feedback.
4.2.8
Cluster: Image
The Image cluster (IMG) comprises all activities that contribute to the reputation of an industrial research department. In order to attract the best talent, the creation of a strong positive image is, for an industrial research department, at least as important as for the corporation. In fact, a research department may also contribute greatly to the image of a company by way of perceptions about its innovativeness. It is also worth mentioning that the image of a research department not only has to consider people external to the company but also to people within the company. Clusters such as TT and FBO might also contribute greatly to the goal achievement of the department and therefore its internal perception becomes a key success factor. Assessing the reputation, however, is a difficult and costly process.37 As an example for image assessment on the corporate level, recall the Fortune magazine which publishes an annual ranking of the World’s Most Admired Companies.38 This is a well-established measure for corporate image. Image and reputation are soft factors that need to be managed carefully. In practice, departments like “public relations” or “communication” take up the role of creating and managing the corporate image. Although the underlying process of the IMG cluster is considered as latent, less-formalized and to a large extent external, there are some explicit parts of an image “process”39 such as: • Identifying stakeholder groups (partners, customers, government institutions, specific research communities, media, financial institutions or public organizations); and determining its importance for the research department; • Choosing target audiences for certain image campaigns; • Defining a profile of a desirable research image via visual, verbal or behavioral elements;
37
Klein (1999). Fortune Magazine (2009). 39 Kotler (1996). 38
146
4 Performance Management: Analysis Approach
• Analyzing the present image, comparing it to the desirable image, identifying image gaps and generating a plan for how to fill the gaps and thereby improve the image. These internal steps are supposed to trigger the external part of the “process”. As a consequence of these considerations the KPI classes of the IMG cluster aim to capture the quality of both: • Internal visibility: internal perception or internal recognitions; • External visibility: external perception or external recognitions. The artifacts relevant for these KPI classes are more likely output artifacts for internal perception, and more likely outcome artifacts for external perception. The following examples of KPIs have been collected in our case studies: • Number of visits by key decision makers, subjective evaluation of collaborations with internal groups (BUs, PDs, LOBs); • Number of contributions to internal conferences or exhibitions; • Number of awards, speeches, invited keynotes, medals, prizes, fairs and exhibitions.
4.2.9
Cluster: Publications
The dissemination and communication of the progress of research work is an integral part of research as such. There are many forms of communicating the progress. One of these has already been described within the IP cluster. Another form is typically a publication ahead of which the actual scientific investigation of a researcher occurs. The Publications (PUB) cluster comprises activities that produce research output in the form of scientific and non-scientific publications, including journals, books, proceedings, etc. Journals typically refer to peer-reviewed publications devoted to disseminating new research results and developments within specific disciplines, sub-disciplines or fields of study. These include original articles, research letters, research papers, and review articles. There are different types of books such as monographs, which are relatively short books or treatises on a single scholarly subject written by a specialist(s). Chapters, which are one or more major divisions in a book, each complete in itself but related in theme to the division preceding or following it. Edited works are collections of scholarly contributions written by different authors on a related theme. Proceedings typically refer to a published record of a conference, congress, symposium or other meeting whose purpose is to disseminate new research results and new developments within specific disciplines, sub-disciplines, or fields of study. The Frascati Manual identifies, in paragraphs 69 and 149, the publication of research findings in scientific journals as activities of R&ED. Furthermore, in annex 7, paragraphs 21, 24 and 25 of the manual define bibliometrics as the generic term for
4.2 Performance Clusters
147
data on publications. It suggests that bibliometrics was originally limited to collecting data on the number of scientific articles and other publications, classified by author and/or by institution, field of science, country, etc., in order to construct simple “productivity” indicators for academic research. Subsequently, more sophisticated and multidimensional techniques based on citations were developed. The manual proposes the use of more sensitive measures of research quality using citation indexes and co-citation analyses. The visible part of a publication process is the submission to a journal or a conference. This stage, by nature, extends beyond the company boundaries and operates externally with the organizers of scientific conferences. Traditionally, such a journal or conference has a strict review process, often involving multiple blind peer reviews. The process of peer review has been well-described by Meadows (1998).40 When a researcher submits his/her paper to the editorial office of a journal, it is first scanned by the editor, who may reject it either because it is “out of scope” (not dealing with the right subject matter for that journal) or because it is clearly of such low quality that it cannot be considered at all. Papers that pass this first hurdle are then sent to experts in the field (usually two) who are generally asked to classify the paper as publishable immediately, publishable with amendments and improvements, or not publishable.41 The peer review process is formalized and well-organized when entering the journal or association’s boundaries. Within the industrial research department this process is on one hand a concomitant phenomenon, because it is in the nature of a researcher to publish his/her findings and insights of their own research. On the other hand, the volume and quality of publications represents the department’s ability to originate new concepts/solutions. This cluster captures the quality of the research department itself. In our classification (cf. Table 4.6) PUB cluster shows the following characteristics: external, formalized and non-termed. Non-termed applies to the PUB cluster, since there is no synchronization with the performance management cycle, besides the fact that the external process part of the publication cycle in general takes less than a year. The property of being formalized strongly refers to the external part of the process. The internal part depends heavily on the status of the research work and the availability of publishable results. The property “external” results from the fact that the overall process is mainly driven by external opportunities (conferences, journals, etc.) to publish research results. Two KPI classes can be distinguished within the PUB cluster: • Quantity of publication; • Quality of publications.
40 41
Meadows (1998), pp. 177–194. Rowland (2002).
148
4 Performance Management: Analysis Approach
The first KPI class is a quantitative count of published papers, for example: number of first or co-authorships of journal and conference papers, books, white papers, etc. The second KPI class captures only those that are most relevant for the corresponding department depending on the discipline and perceived quality standards. Examples for the second KPI class are the following KPIs: number of best paper awards, papers published in “A” journals, etc. Although we didn’t specifically search for concrete evidence, choosing the quality KPI class as opposed to the quantity KPI class might indicate a higher organizational maturity level. The following KPI examples have been collected from our case studies: • Number of first or co-authorships of journal and conference papers, books, white papers, technical reports etc. • Number of best paper awards, papers at selected conferences.
4.2.10 Cluster: Presence in Scientific Community Presence in Scientific Community (PSC): this cluster comprises KPIs measuring the activities of researchers in the scientific community, such as invited talks, keynotes, panels, chairmanship and editorship of conferences or scientific events. Paragraph 149 addresses the presence of R&ED in activities like organization of scientific conferences or involvement in scientific reviews and gives evidence for such a cluster. It is not a disputed fact that the participation of researchers in scientific communities belongs to the typical activities of researchers by nature. According to Chorafas42 affiliation with scientific societies or recognition within the scientific community are personal factors that need to be considered when assessing researcher performance. The Frascati Manual defines R&D activities for measurement purposes. It analyzes a whole range of R&D activities by classifying them and explicitly listing exclusions from R&D related activities. The manual furthermore defines criteria for distinguishing the boundaries of R&D and related activities, and discusses the problems at the border between the two. The presence of R&D activities is identified in several paragraphs; especially paragraph 14943 which identifies activities such as: “The publication of research findings in scientific journals, organization of scientific conferences or involvement in scientific reviews”. The first part of the clause addresses the Publications cluster and the second part explicitly mentions presence in the scientific community. When only considering the activities of industrial research, the participation in such communities is by nature more pronounced in research than in development.
42 43
Chorafas (1963), p. 67. OECD (2002), Frascati Manual, clause 149, p. 49.
4.2 Performance Clusters
149
This is explained by the fact that the findings of research projects need to be published and presented in the scientific community, which is often the only possibility to get qualified feedback from peers in the same area. Another reason is that this function accumulates expertise, which is then requested to act in an editorial or review capacity for publications, to be conference program committee members, to chair sessions, or to give talks, keynotes, presentations, etc. These activities are reflected in the two KPI classes of the PSC cluster: • Participation in scientific events (excluding publications); • Participation in advisory boards, or related bodies, honorary titles or memberships, or visiting scholarships. The KPI classes distinguish the seniority of people working in the research department that engage themselves in different scientific communities. The first KPI class summarizes activities of junior researchers’ involvement in scientific events excluding publications. Examples of KPIs summarized in the KPI class are: number of invited talks or keynotes, chairs in workshops or conferences, participation in program committees or journal reviews and editorships. The second KPI class addresses senior researchers’ involvement, such as participation in advisory boards or related bodies, honorary titles or memberships or visiting scholarships. KPI examples for this class include: number of invitations to governmental, professional, industrial, academic advisory boards or standardization committees, number of doctorates, fellowships, presidencies of scientific or technical societies. Again, the selection of the first or the second class can indicate the “maturity” of scientific expertise within a research department. The following KPI examples have been collected in the case studies: • Number of invited talks or keynotes, chairs in workshops or conferences, participation in program committees or journal reviews, editorships; • Number of invitations to governmental/professional/industrial/academic advisory boards or standardization committees, number of doctorates, fellowships, presidencies of societies, invitations for research sabbatical positions.
4.2.11 Cluster: Collaboration with Academia There are two clusters in our clustering concept that address collaboration aspects. These are related to each other, however they address different players in the ecosystem of a research department: academia on the one hand, and business partners and customers on the other. Collaborative research happens in many ways and is more common in some disciplines than in others. Working with others on a research project can have several benefits, but there can be drawbacks as well. When done in the right spirit, collaborative research can result in more reliable and powerful results that come to publication faster than they would if the research were
150
4 Performance Management: Analysis Approach
done independently.44 Researchers can pool their knowledge and critique each other’s work before starting the publication process. The Collaboration with Academia (CA) cluster KPIs evaluate the intensity or volume as well as the quality of input from, and activities related to, collaboration with academic partners. Input refers to output produced by academia and taken up by the research organization. Activities comprise joint work as well as actions trying to impact the roadmap of academic research. Collaboration between industry and academia is crucial to deliver a shared vision for the future of technology and education. Industry is, in general, committed to deepening its relationship with academia, because only by working together will the next generations of technologies be created. The connections of an industrial research department with academia can result in advantages for both sides. The nature of this linkage has a strong educational character and the spectrum of educational profiles involved in the collaboration is very broad. This collaboration typically involves postgraduate and postdoctoral research workers, but also other senior academic scientists, and can take the form of either short-term interaction via workshops, internships providing access to the real environment for studies and theses, or long-term co-sponsored studentships, or integration of academic staff into research projects. For students, such collaboration can result in higher qualifications, where eventually the students can take positions of responsibility in academia or industry, or just become a part of the scientific staff team. For the research department, collaboration can lead to the development and promotion of new methods or access to state of the art information. The benefits for an industrial research department to collaborate with academia are: • Brand and recruitment: better visibility (sponsored initiatives, access to real business environment to study, companies presentations, etc); • Student contacts and development of expertise: cooperation in teaching (learning tasks, entrepreneurial projects, theses, case studies, internships); • Development of expertise and competitiveness: shared expertise and networking (expert consultation, events, seminars on current research results, supplementary and management training, information services); • Innovative operational models and new inventions: cooperation in research (restricted and longer term research ventures, support functions for inventions and research activities, promotion of entrepreneurship). The drawbacks of collaborative research should not be underestimated. In the event of successful invention, it is often difficult to dismantle the ownership of the intellectual property rights to the invention. The CA cluster encompasses two KPI classes:
44
Fine (2010).
4.2 Performance Clusters
151
• Intensity of collaboration; • Quality of collaboration. The intensity KPI class assesses volume of involvement, e.g. investment at universities in relation to total budget, or number of guest researchers hosted within research coming from universities. The quality KPI class can, in general, only capture subjective opinions. These are the personal opinions of the people involved in the collaborative projects. The following KPI examples have been collected in our case studies: • Investment at universities in relation to total budget, number of guest researchers from universities (sabbaticals etc.); • Subjective evaluation of each university project from the supervising professor via a questionnaire.
4.2.12 Cluster: Collaboration with Partners and Customers The Collaboration with Partners and Customers (CPC) cluster addresses joint activities between the research organization and partners and customers, as well as the output resulting from these activities. The KPIs reflect the proximity to partners and customers (intensity of collaboration), as well as the quality of the activities. Such proximity reflects how well-aligned with partners and engaged with customers, the industrial research department is. The motivation for the CA cluster is entirely applicable to this cluster as well. The clusters are “somewhat” related, nevertheless they address different linkages in terms of the manuals, and therefore justify their separate existence. The differences lie especially in the nature of the conducted research: while the CA cluster focuses on basic research tackling theories or concepts, the CPC cluster points at applied research addressing, for example, prototyping in real business contexts. As an analogy to the CPC cluster, this cluster also consists of two KPI classes: • Intensity of collaboration; • Quality of collaboration. The intensity KPI class assesses the volume of involvement with partners. This KPI class is a quantitative count of collaborative projects. Example KPIs include the number of projects involving an external stakeholder versus the total number of projects within a department, or the number of joint research results like showcases, prototypes, etc. Again, analogous to the second KPI class of the CA cluster, the quality KPI class of the CPC cluster captures subjective assessment by customers or partners. The following KPI examples have been collected in the case study: • Number of projects in which an external customer and/or business partner is involved versus the total number of projects; number of joint research results like showcases, prototypes or demos; • Subjective assessment by customers and/or partners by means of a survey.
152
4 Performance Management: Analysis Approach
4.2.13 Performance Cluster Overview To conclude, this section introduced in-depth descriptions of performance clusters, which form the spectrum of activities found in industrial research organizations. Table 4.6 outlines three properties that allow for a categorization of performance clusters. Each property is discussed and thoroughly described within its respective performance cluster in the paragraphs above. Table 4.6 Performance cluster properties Performance cluster Company internal vs. company external focus 1 Technology Transfer I 2 Future Business Opportunities 3 Research Portfolio Management 4 Intellectual Property Creation 5 Talent Pool 6 Operational Excellence 7 Image I/E 8 Publications E 9 Presence in Scientific Community 10 Collaboration with Academia 11 Collaboration with Partners and Customers I internal, E external F formal, I informal L long-term, S short-term, N non-termed (not limited time-wise)
Formal vs. informal processes F F
Time pattern L L
F
L
F
L
F/I F I F F/I
L/N S N N N
F/I
N
F/I
N
The 11 performance clusters can be classified with the help of Table 4.6: the first classification property is connected with the internal or external focus. Clusters 1–6 are linked to internal company processes; whereas 8–11 extend beyond the boundaries of a company. The process of IMG cluster has both components: internal and external. Depending on the process, in some cases the whole activity chain has to be assessed, i.e. also the parts that go beyond the research department, in order to attain the outcome and to be able to assess it. This is particularly true for the following clusters: Technology Transfer, Future Business Opportunities, Technical Achievements and Intellectual Property Creation.45
45
Especially in the case of intellectual property, this might depend on the overall organizational setup of the individual company.
4.2 Performance Clusters
153
The second property is the degree of formalization of the processes (i.e. explicitly formalized processes or more informal or latent processes). The processes of clusters 1–4, 6 and 8 are more formal and the remaining clusters cannot be accurately assigned to one property. For example, a performance cluster with explicitly formalized processes is Intellectual Property Creation cluster, while examples of clusters with less explicit processes are Publications and Image. Processes of the Operational Excellence cluster are highly formalized. Operation Excellence e.g. refers to the fact that a large amount of research activities are embedded in a project context. It therefore embraces KPIs for assessing the entirety of all processes within a project context as well as ancillary processes carried out within a research department. The pattern of time can be seen quite clearly within the performance cluster processes. The processes of clusters 1–4 are for long-term periods, whereas 7–11 are more continuous and therefore not limited time-wise. The processes within performance cluster six are an exception and always considered to be short-term. Altogether, the performance clusters differ significantly from each other regarding their sharpness and the number of different KPIs that were collected in the case studies. For example, the nature of the Technology Transfer cluster allows for a variety of different KPIs for the measured object, whereas the Publications cluster mainly uses the number and quality of publications. The clusters: Publications and Presence in the Scientific Community are related to each other, but have been separated into two different clusters because we were able to unambiguously assign different KPIs to each one. The following table (Table 4.7) shows the resulting list of 37 classes, their relevant cluster affiliation, and examples. Table 4.7 Catalogue of KPI classes with corresponding properties and examples Cluster Nr KPI class description Artifact Artifact Examples classes property 1. Transfer volume: Technology Op, P # # Person days, amount of Transfer volume of services charged to technology transfer other departments activities to (€) development or other (business) units 2. Transfer result quality: Op Q Subjective evaluation by quality of the the receiving unit via research results questionnaire transferred 3. Transfer process quality: P Q Subjective evaluation of quality of the transfer the collaboration process or transfer with the receiving activities unit via questionnaire 4. Transfer significance: Op, Oc Q Subjective evaluation of significance of the the importance of the transferred research transfer for the results for the development/ receiving unit business of the receiving unit (continued)
154
4 Performance Management: Analysis Approach
Table 4.7 (continued) Cluster Nr KPI class description 5.
Future Business Opportunities
6.
7.
8.
Research Portfolio 9. Management
Artifact Artifact Examples classes property Op, Oc # Equivalent opportunity cost of using an external expert service, revenue generated by the transferred artifact I, P # # Ideas, # ideas per researcher
Economic transfer value: economic value of the transfer activity or transferred research results Input: intensity of input into the innovation process Innovation process: P (weighted) # of ideas moved to a certain or next phase of the innovation process
Outcome: (achieved) Oc business impact of an idea in terms of its economic value Structuring: structure P and quality of the research (i.e., certain technology areas)
10. Vision: visions related to P the individual parts of the research portfolio and their quality
#
#
Q
Q
P, Op
Q
P, Op 12. Implementation: implementation of the roadmaps and the quality of contributions from research projects
Q
11. Roadmap: roadmaps to achieve the visions and their quality
# Ideas with a business case developed + 2 # ideas with demonstrator developed + 3 # ideas handed over to development $, €
Subjective evaluation of the research portfolio by internal or external reviewers (simplest way – portfolio management is in place or not) Subjective evaluation of the vision documents of the individual research programs by internal or external reviewers (simplest way: vision documents are available or not) Subjective evaluation of the roadmap by internal or external reviewers (simplest way – in place or not) Subjective evaluation by internal or external reviewers of the scientific accomplishments, technical achievements, or technical impact (continued)
4.2 Performance Clusters Table 4.7 (continued) Cluster Nr KPI class description Intellectual Property Creation
13.
14.
15.
16. 17.
Operational Excellence
18.
19.
20.
21.
22.
23.
Talent Pool
24.
155
Artifact Artifact Examples classes property Input volume: volume of P, Op # # Invention disclosures potentially protectable inventions submitted into the IP pipeline Output volume: volume P, Op # # First filings, defensive of first filings from publications, trade the IP pipeline secrets Outcome volume: Op, Oc # # Patents granted to the volume of patents company granted Input quality: quality of Oc Q Economic value of the granted patents granted patent Outcome quality: Oc Q # Submitted inventions alignment of addressing the IP research activities strategy with the IP strategy of the company Budget: adherence to P # difference of planned budget budget of each line item to actual budget measured (less deviation is rated to be better) Q Successful (on-time) Timeliness: adherence to P termination at a timelines (phases, particular phase/gate gates) Oc # % By which the Risk management: potential damage has quality of risk been lowered by management adequate measures against the bottomline P Q Data compliance at Project management agreed deadlines quality: quality of project management I # Volume of third-party General risk sharing: investment (in €), # volume of (external) participations in investments in the externally-funded research organization programs (beyond seed funding) Working environment: I Q Scoring in employee satisfaction survey quality of the working environment Inflow: volume/quality I #Q # Externally-attracted of people hired into senior scientists with skill-sets compatible the research to the profiles organization required (continued)
156
4 Performance Management: Analysis Approach
Table 4.7 (continued) Cluster Nr KPI class description 25.
26.
27.
Image
28.
29.
Publications
30.
31.
Presence in Scientific Community
32.
33.
Artifact Artifact Examples classes property Outflow: volume/quality Oc #Q # Top talents moving to of people leaving the other parts of the research organization company to move to other parts of the company/ ecosystem Talent development: P #Q # Job rotations, # PhDs volume/quality of finished, training development measures undertaken Talent assessment: Op Q Subjective assessments quality of the people through partners or in the research stakeholders, team department track records, 360 feedback Internal visibility: Op Q # Visits by key decision internal perception or makers, subjective internal recognition evaluation of collaboration with internal groups, # contributions to internal conferences or exhibitions Oc Q # Awards, speeches, External visibility: keynotes, medals, external perception prizes, fairs and or external exhibitions recognition Quantity of publications: Op # # Journal & conference volume of papers, book publications contributions, # white papers, # technical reports Q # Best paper awards, Quality of publications: Op papers at selected quality of conferences publications Junior researcher Op Q # Invited talks or involvement: keynotes, chairs at participation in workshops or scientific events conferences, (excluding participation in publications) program committees or journal reviews, editorships Op Q # Invitations to Senior researcher governmental/ involvement: participation in professional/ industrial/academic advisory boards or (continued)
4.3 Performance Cluster Analysis Table 4.7 (continued) Cluster Nr KPI class description related bodies, honorary titles, memberships or visiting scholarships
Collaborationwith 34. Intensity: volume of Academia collaboration with academia
35. Quality: quality of collaboration with academia
Collaboration 36. Intensity: volume of withPartners collaboration with and Customers partners and customers
157
Artifact Artifact Examples classes property advisory boards or standardization committees, # doctorates, fellowships, presidencies in scientific/technical society Op # Investment at universities in relation to total budget, # guest researchers from universities (sabbaticals) Op Q Subjective evaluation for each university project from the supervising professor via a questionnaire Op, Oc # # Projects in which an external customer and/or partner in a business relationship is involved vs. total projects, # joint research results such as showcases and prototypes Op, Oc Q Subjective assessment by customers and/or partners via a survey
37. Quality: quality of collaboration with partners and customers Artifact classes: P process, I input, Op output, Oc outcome Artifact properties: Q quality, # quantity
4.3
Performance Cluster Analysis
A performance cluster is a set of semantically related subgoals/KPIs. Creating these clusters has two main benefits. We can: (1) determine the importance of each performance cluster based on the cumulative weight of its subgoals/KPIs, and thereby rank the clusters per company, (2) compare the weights and the rankings between the companies. Figure 4.19 depicts the cluster comparison for all eight case study companies. Clusters at the top are those with the highest weights and therefore most importance.
158
4 Performance Management: Analysis Approach Cluster ranking C1
C2
C3
C4
C5
C6
C7
C8
TT
IP
FBO
FBO
TT
CPC
OpEx
FBO
TP
TT
TT
TT
FBO
OpEx
PUB
RPM
RPM
CPC
TT
FBO
PSC
RPM
IP
IP
IP
IP
OpEx
IMG
OpEx
OpEx
IP
OpEx
TP
TT
CPC
RPM
FBO IP TT PUB
TP IP
IMG
RPM IMG
IMG
PUB
PUB PSC
TP
CA
PUB CPC
IMG
TP
RPM PUB PSC
CA
CA OpEx
Fig. 4.19 Cluster comparison from case studies
Based on this analysis we conclude that certain performance clusters have the same, or similar importance, for the organizations in question. For example, the Technology Transfer cluster is present in all companies examined and in the cases of C1, C2, C3, C4, C5 and C8 the TT cluster is ranked as first or second. In C6 it is ranked third and in C7 the TT cluster shares the fifth position with the Publications cluster. The Intellectual Property cluster is also omnipresent and is ranked fourth or fifth with two exceptions: C2 and then C6 where it shares second position with the Operational Excellence cluster. The Future Business Opportunities (FBO) cluster is also present in all but two companies and is always ranked in the top four, where present. To summarize, the ranking the clusters allows us to compare the number, presence and importance of performance clusters between the companies. The minimum amount of clusters used is four; for example in C2, while the maximum is ten, which is true for C5, C7 and C8.
4.4
Goal Decomposition Analysis
In the previous section we analyzed the KPIs dimension and the relative importance of the performance clusters. However, because the goal of this research is to build a comprehensive methodology, we now analyze the contribution to individual goals of the performance clusters. Whereas in the previous section we defined performance clusters by grouping semantically similar KPIs, our objective in this section is to determine whether companies measure similar sets of activities when similar goals are to be achieved. If we were able to provide a recommendation such as: if you have an organizational goal “X”, then a certain set of activities is necessary to achieve this goal,
4.4 Goal Decomposition Analysis
159
which are in turn most appropriately measured by a set of KPIs reflecting the corresponding set of performance clusters, it will greatly help practitioners building performance management systems. To make this example more concrete: if you have an organizational goal that is to “Create intellectual property”, then the activities summarized by the IP cluster are necessary in order to achieve this goal and therefore a set of KPIs grouped in the IP cluster is most appropriate to assess this goal. To be able to provide such guidelines we need to examine the goal dimension. We have to evaluate whether performance clusters are reasonable mediators between goals and KPIs. If this is the case, then we are in essence defining relations between individual goals, as well as between goals and KPIs, that are expressed by performance clusters. Within each case study the relationships between goals and KPIs appear to be independent to one another; however, in order to derive a general view across different companies, we are specifically interested in: 1. Similarities between the goals of the different research departments; 2. Relationships between the KPIs applied to similar goals; and 3. General connections between goals and KPIs. In the case studies, each company provided us with a set of organizational goals for their research department. These are, as such, not directly comparable across companies. As mentioned in Sect. 3.1.4, within the case studies, we collected 164 different KPIs and 45 goals. Of these 45 goals some of them are present in multiple cases: there are only 30 unique goals. These numbers show that the results of the case studies are difficult to interpret at an initial glance, where the lack of comparability is due to: (1) goals sounding similar or identical, yet having heterogeneous KPIs, making them likely to actually mean different things; (2) the sheer number of different KPIs being difficult to handle without any deeper analysis of the KPIs themselves. Therefore, recognizing general interdependencies between KPIs is difficult and, and as a result, the same is true for general connections between goals and KPIs. The situation is even more complex when comparing research goals only. The difficulty lies in the interpretation of language and the difficulty in this analysis step is to find an approach to systematically compare research goals. During the formation of our performance clusters, we identified which KPIs were associated with the cluster. Now, to compare the goals, we performed – what we called – goal decomposition analysis. We decomposed each goal into the performance clusters that contribute to the assessment of the goal Fig. 4.20. Using the notion of the performance cluster composition we are able to rephrase the recommendation from the beginning of this section as follows: if you have an organizational goal with specific characteristics, choose a set of KPIs that results in a performance composition as follows. In order to make such statements we need to be able to compare organizational goals. Expressing this in terms of performance clusters we therefore would ideally like to show: if goals are similar then with a high likelihood they have a similar cluster composition. In turn, this would require (a) a method to determine the extent
160
4 Performance Management: Analysis Approach
Fig. 4.20 Goal decomposition example
TT FBO
RESEARCH GOAL
RPM IP OpEx TP IMG PUB
PSC CA CPC
to which two goals are interpreted similarly and (b) a similarity measure for performance cluster compositions. It is obvious that we are not able to succeed with (a) in a general manner.46 As a consequence, we will proceed with our analysis the other way round and try to find reasonable similarity measures and define the similarity of goals based on the similarity of clusters: DEF_ G: Two goals are similar if they have similar performance cluster compositions.
If two cluster compositions have a “good” measure of similarity, there is high likelihood that if the goals are similar, based on our definition, they would be subjectively interpreted as being similar. As a result we have reduced our problem to finding a similarity measure for performance cluster compositions. When tackling this question we realized that we could define quite sophisticated47 similarity measures, such as geometric distance for instance, however from a pragmatic perspective the problem with determining their quality remains open as it is still bound to the evaluation of the semantic similarity of goals. We reduced this problem again by using a heuristic for the similarity measure that we call “similarity indicator” (SI). The similarity indicator reduces the similarity measure to a binary value and in this way abstracts from the “degree” of cluster compositions of “being similar”. The greater the correlation between the semantic
46
Semantic technologies would be an approach to go in this direction. However they fail due to the fact that for a general approach they need extensible ontologies and their pragmatic applicability is not (yet) given. 47 Other ways to define similarity of performance cluster spectra could be: the extent of deviation between clusters expressed in its threshold value, or to define the similarity relationship on a finegranular level taking into account not just one, but two or more maximum weighting clusters, etc.
4.4 Goal Decomposition Analysis
161
judgment of similitude, and the result of the similarity indicator, the better the indicator. In the remainder of this section we define two different similarity indicators and evaluate them by applying them to the goals of our case studies. Following our definition that a cluster composition (cf. Fig. 4.20) is the decomposition of a goal into the performance clusters that contribute to the goal assessment with a certain weight and our initial definition of the similarity of goals we further refine definition DEF_G using a specific similarity indicator48 to define similarity between performance cluster compositions as follows: SI-1: Two performance cluster compositions are similar if they have the same cluster with the highest weight.
The relationship between similar goals would then also implicitly define a relationship between goals and performance clusters in the sense that for goals with a similar semantic interpretation, there is, to a certain extent, a typical cluster composition. In the following we show examples of similar cluster compositions together with the title of the goal to which they are related. When applying this to our case studies we also have to consider that the selection of KPIs for some goals in our case studies might not be an ideal one in some of the cases. This could result in cluster compositions appearing as different even though the corresponding goals are semantically similar. In the following we show the considerations discussed above by applying the similarity indicator SI-1 to the goals collected in our case studies. Although we provide our own semantic interpretation of the goals for an evaluation of the usefulness of the indicator, we leave it to the reader’s own discretion to judge from his own semantic perspective whether the individual goals that are indicated as being similar are in fact similar – obviously solely based on the phrasing of the titles of the goals. The following overview comprises all goals that are assigned by SI-1, i.e. also those that from our subjective interpretation are not assigned appropriately Fig. 4.21. Figure 4.21 shows those goals where the TT cluster (dark green) has the highest weight. For example, the TT cluster contributes to the goal of company 2: “Scientific advisor” with 100%. The same is true for company 3 with 100% for both goals listed. The TT cluster contributes to the assessment of the goal “Top Results” of company 1 with 77%, the IP cluster with 16%, Publications (PUB) with 5%, and Operational Excellence (OpEx) with 2%. The “Alignment and Transfer” goal of company 8 shows a TT cluster contribution of 60%, Future Business Opportunities (FBO) of 30% and Talent Pool (TP) of 10%, etc. This example illustrates all aspects of the discussion in the previous paragraph: For the first six goals, the TT cluster contributes at least 50% to the assessment of the goals. Three out of nine goals listed have the same title “Alignment &
48
Similarity indicator is a function which compares two performance cluster compositions and provides a yes or no decision at the end.
162
4 Performance Management: Analysis Approach
[company 2] Scientific advisor to the development: researcher time spent on technology development [company 3] Alignment & Transfer TT
[company 3] Advanced Technology Solutions: building-out technology-leveraged solutions
FBO RPM
[company 1] Top Results: Project results have a significant impact on business [company 8] Alignment and Transfer: Give input to development to drive incremental innovation
OpEx CA IP
[company 6] Alignment and Transfer: Give input to Business Units by the transfer of research results [company 5] xPG Influence
TP IMG PUB PSC
[company 5] IT Strategy
CPC
[company 5] Enterprise Architecture [company 4] Technology innovation and leadership 0%
10% 20% 30%
40% 50% 60% 70%
80%
90% 100%
Fig. 4.21 Technology Transfer-oriented goals with SI-1 indicator
Transfer”. Additionally the “Scientific advisor to the development” goal can also be interpreted as a way of defining an “Alignment & Transfer” goal as can the “Top Results” goal of company 1 when looking at its high level description. The “xProduct Group influence” contains the notion of “transfer”: knowledge transfer or transfer of concrete artifacts. The first seven goals, from our semantic interpretation, are well-assigned to the group of “Technology Transfer-oriented goals”. The last three goals of the figure might be less related to the first seven goals in terms of semantic interpretation although SI-1 does indicate this. Out of the remaining 35 goals in our case studies, there are no additional goals that from our semantic interpretation would belong to the group of “Technology Transfer-oriented goals”. In summary SI-1 indicates technology transfer-oriented goals correctly 93% of the time. Figure 4.22 shows a selection of goals that are similar based on SI-1, since for all the goals the IPC cluster (marked in red) contributes to their assessment with at least 60%; in companies 2, 6 and 8 by 100%. IP
[company 2] IPR generation
TT FBO
[company 6] Intellectual Property
RPM
[company 8] Intellectual Property Creation
OpEx CA
[company 1] Protect Technology
TP
[company 5] Intellectual Property
IMG PUB
[company 3] Protect Technology
PSC CPC
0%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Fig. 4.22 Intellectual Property Creation-oriented goals with SI-1 indicator
4.4 Goal Decomposition Analysis
163
With respect to their titles, all six goals can be definitely interpreted as being similar. In our case studies we identified an additional goal “Scientific excellence” of company 2, which from our semantic interpretation belongs to the group of “Intellectual Property-oriented goals”, however the SI-1 did not detect this. In this case, out of the remaining 38 goals of our case studies there is one additional goal that we would assign to the group “Intellectual Property-oriented goals”. With this, the SI-1 identifies with 98% accuracy the intellectual property creation-oriented goals (Fig. 4.23). [company 1] Excellence OpEx
[company 3] Operational Excellence
TT
[company 7] People
FBO RPM
[company 7] Strategic partnerships
CA
[company 6] Operational Excellence
IP TP
[company 6] Collaborative Research
IMG
[company 6] Growth
PUB PSC
[company 8] Operational Excellence
CPC
[company 7] Operational Excellence 0%
10% 20% 30% 40%
50%
60% 70% 80%
90% 100%
Fig. 4.23 Operational Excellence-oriented goals with SI-1 indicator
Figure 4.23 shows goals that are related to operational excellence regarding their cluster composition, as the OpEx cluster delivers the highest contribution in terms of its weight. It is also interesting to mention that the Collaboration with Partners and Customers (CPC) cluster contributes in a significant way to the goal assessment; i.e., when present, which is the case in five of nine goals, the contribution is the second highest. This at least shows that more fine-grained criteria for the indicators could increase their quality. For the last two goals, the contribution of the OpEx cluster is below 50%, which might be less than expected; however, the cluster still represents the highest contribution for the assessment of the goal. With this, we believe our semantic assignment of the nine goals to the group of “Operational Excellence-oriented goals” is good. Of the remaining 36 goals in our case studies there are no additional goals that should belong to this group. As such, and in summary, this SI-1 assigns operational excellence-oriented goals 100% correctly. In Fig. 4.24 we selected goals where we expected a high contribution of the Talent Pool (TP) cluster marked in orange. This is true for three of the four goals shown, where the TP cluster contributes at least 50% to the goal achievement. In the case of company 5, a very diverse cluster composition is visible, which might indicate that either non-appropriate KPIs are used, or the goal title is misleading. In the case of the “Market savvy” goal, a goal strongly related to the TP-cluster would
164
4 Performance Management: Analysis Approach
TP OpEx
[company 6 ] Talent
TT
[company 1] Top people
FBO RPM
[company 1] Market savvy
CA IP IMG
[company 5] People
PUB PSC
0%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
CPC
Fig. 4.24 People Development-oriented goals with SI-1 indicator
not necessarily be expected at first glance. However, when analyzing the cluster composition, it becomes evident that this goal is actually pretty much a TP-oriented goal since “market savvy” has a central requirement that people be educated with an in-depth market knowledge. Furthermore, the high contribution of the TP cluster suggests that there is an expectation to transfer the insights from the knowledge, and advances related to new technology solutions, into the development organization. Out of the remaining 41 goals in our case studies there are no additional goals that from our semantic interpretation should belong to the group of “People Development-oriented goals”. In summary, SI-1 indicates people development-oriented goals 95% correctly. [company 1] Top ideas FBO
[company 3] New opportunities
TP OpEx
[company 4] Business impact
TT
[company 3] Technology evaluation
RPM CA
[company 3] Proof of concept development
IP IMG
[company 8] Technology creation
PUB
[company 4] Technology innovation and leadership
PSC CPC
[company 5] People 0%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Fig. 4.25 Idea Generation-oriented goals with SI-1 indicator
Figure 4.25 shows a selection of goals that are similar. This is because the FBO cluster, marked in yellow, dominates the contribution to the assessment of the goals. It is interesting to note that the RPM cluster seems to play an important role for the assessment of idea generation-oriented goals. The FBO cluster contributes to the goals of company 1: “Top ideas” and company 3: “New
4.4 Goal Decomposition Analysis
165
opportunities” with 100%; to the goals of company 4: “Business impact” with 70% and to the assessment of both goals of company 3: “Technology evaluation” and “Proof of concept development” with almost 60%. The contribution of the RPM cluster is also prominently represented in assessing the following goals: company 3: “Technology evaluation” and “Proof of concept development” with more than 25%; company 8: “Technology creation” with 50%; as well as company 4: “Technology innovation and leadership” with 20%. Common terms in the headings of the goals are as follows: ideas, opportunities, technology, concepts. The last goal of the Fig. 4.30 might be less related to the first seven goals in terms of semantic interpretation although SI-1 would indicate this to be the case. Out of the remaining 38 goals in our case studies there are no additional goals that from our semantic interpretation should belong to the group of the “Idea Generation-oriented goals”. In summary SI-1 correctly indicates 97% of technology transfer-oriented goals. IMG FBO
[company 1] 360 visibility
TT RPM
[company 8] Image
OpEx CA
[company 4] Thought leadership
IP
[company 5] Image
TP PUB PSC
0%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
CPC
Fig. 4.26 Image-oriented goals with SI-1 indicator
Figure 4.26 shows four goals from four different companies where the IMG cluster marked in purple has the highest weight. The IMG cluster contributes, for example, to the goal of company 1 “360 visibility” with 70%, 60% for company 8, and 57% for company 4. The contribution in the case of company 5 is 25%, which is still the highest amongst all clusters in the cluster composition. In addition, when looking at the headline of the goals and their semantic interpretation, one can conclude that the goals are in fact similar. In our case studies we identified an additional goal “Strategic partnerships” of company 7, which from our semantic interpretation belongs to the group of “Imageoriented goals”. However the SI-1 did not interpret this goal as such. As such, out of the remaining 40 goals in our case studies, there is one additional goal that we would assign to the group “Image-oriented goals”. With this, the SI-1 identifies, with 98% accuracy, the image-oriented goals. In sum, these examples demonstrate that the similarity indicator is a reasonable and pragmatic way to compare goals based on their cluster composition.
166
4 Performance Management: Analysis Approach
Nevertheless we additionally want to reaffirm this analysis by applying a slightly different second similarity indicator: SI-2: Two performance cluster compositions are similar if they have the same cluster with the highest average weight of the KPIs assigned to that cluster.
As with SI-1 we present the following groups of goals that should be considered similar but now based on SI-2. Figure 4.27 lists goals where the highest average weight of KPIs is given by the TT cluster (dark green). For the first five goals: from company 2, company 3 with two goals, and companies 6 and 1, the average weight of KPIs, which are assigned to the TT cluster is exactly 50%. The contribution of KPIs from the TT cluster to the next two goals, which are from company 4, is 40%. [company 2] Scientific advisor to the development: researcher time spent on technology development [company 3] Alignment & Transfer [company 3] Advanced Technology Solutions: building-out technology-leveraged solutions
TT FBO RPM
[company 6] Give input to Business Units by the transfer of research results to apply them in products
OpEx
[company 1] Market savvy
CA
[company 4] Contribute to critical standards
IP TP
[company 4] Technology innovation and leadership
IMG
[company 6] Image
PUB PSC
[company 8] Alignment and Transfer: Give input to development to drive incremental innovation
CPC
[company 1] Top results [company 5] IT strategy [company 5] People 0
10
20
30
40
50
60
70
80
90
100
Fig. 4.27 Technology Transfer-oriented goals with SI-2 indicator
Seven goals that are indicated by SI-2 are also, according to our semantic interpretation, contained within the “Technology Transfer-oriented goals” group. However, five goals are incorrectly assigned (Market savvy, Image, Top results, IT strategy and People). Of the remaining 33 goals from our case studies, there are no additional goals that should belong to the group of “Technology Transfer-oriented goals” according to our semantic interpretation. In summary, SI-2 assigns 88% of technology transfer-oriented goals correctly. From the semantic perspective, the similarity becomes at least partially evident when looking at the goals’ names: 8 of the 12 goals share terms such as: alignment, technology, transfer or standards. When comparing the results of SI-1 with those of SI-2 the following can be observed: (1) seven of the goals that are correctly indicated as being similar by SI-2 are also indicated as being similar by SI-1; (2) of the three goals that our semantic interpretation suggests are incorrectly assigned by SI-1, one is also incorrectly assigned by SI-2.
4.4 Goal Decomposition Analysis
167
[company 3] Protect technology IP
[company 8] Intellectual property creation
TT
[company 2] IPR generation
FBO RPM
[company 1] Protect technology
OpEx CA
[company 6] Intellectual property
TP
[company 5] Intellectual property
IMG PUB
[company 2] Scientific excellence
PSC
[company 7] Value creation
CPC
[company 5] People 0
10
20
30
40
50
60
70
80
90
100
Fig. 4.28 Intellectual Property Creation-oriented goals with SI-2 indicator
Figure 4.28 shows the goals that SI-2 assigns to the IPC cluster (red). For the first six of the nine goals, the title supports the similarity indicator result. The scientific excellence goal of company 2 might indicate that the IP protection strategy is driven by defensive publications for the following reasons: (1) There are two goals where the IPC cluster plays an important role and (2) one of those – the goal “Scientific excellence” – also includes the PUB cluster to a significant amount. Similar considerations are also valid for company 1. The last two goals are examples of where SI-2 does not deliver the expected results when we consider the names of the goals and we assume that the KPIs are well-chosen for those goals. Seven goals that are indicated by SI-2 are also, according to our semantic interpretation, well-assigned to the “Intellectual Property Creation-oriented goals” group. Two goals are falsely assigned (Value creation and People). Of the remaining 36 goals from our case studies there are no additional goals which, from our semantic interpretation, should belong to the “Intellectual Property Creation-oriented goals” group. In summary, SI-2 assigns 93% of IP creation-oriented goals correctly. When comparing the results of SI-1 with those of SI-2 the following can be observed: (1) six of the goals that are correctly indicated as being similar by SI-2 are also indicated as similar by SI-1; (2) one goal (Scientific excellence) has not been detected by SI-1 which in turn was correctly identified by SI-2. Figure 4.29 shows the selection of goals that SI-2 assigns to the OpEx cluster (beige). Five out of eight case studies have a goal entitled “(Operational) Excellence” and these are all assigned as being similar by SI-2. Companies 6 and 7 have three and four goals respectively. This might indicate that those companies have organizational goals are too focused on operational excellence and are therefore less appropriate reflections on the actual activities in the research departments. Or alternatively is could mean that process measures are preferred over KPIs that assess output or outcome.
168
4 Performance Management: Analysis Approach
[company 1] Excellence
[company 6] Operational excellence OpEx
[company 6] Growth
TT
[company 6] Collaboration with partners and customers
FBO RPM
[company 3] Operational excellence
CA
[company 8] Operational excellence
IP TP
[company 7] People
IMG
[company 7] Strategic partnerships
PUB PSC
[company 7] Value creation
CPC
[company 7] Operational excellence
[company 5] People 0
10
20
30
40
50
60
70
80
90
100
Fig. 4.29 Operational Excellence-oriented goals with SI-2 indicator
Nine goals that are indicated by SI-2 are also, according to our semantic interpretation, correctly assigned to the “Operational Excellence-oriented goals” group. Two goals are false positives (Value creation and People of company 5). Of the remaining 34 goals from our case studies there are no additional goals which, from our semantic interpretation, should belong to the “Operational Excellenceoriented goals” group. In summary SI-2 indicates 95% of operational excellenceoriented goals correctly assigned. When comparing the results of SI-1 with those of SI-2, the following can be observed: nine of the goals that are correctly indicated as being similar by SI-2 are also indicated as being similar by SI-1. TP
[company 1] Top people
OpEx
[company 1] Market savvy
TT FBO
[company 6] Talent
RPM CA
[company 7] Value creation
IP
[company 7] Operational excellence
IMG PUB
[company 5] People
PSC CPC
0
10
20
30
40
50
60
70
80
90
100
Fig. 4.30 People Development-oriented goals with SI-2 indicator
Figure 4.30 shows the selection of goals that SI-2 assigns to the TP cluster (orange). SI-2 correctly identifies three goals, which our semantic interpretation also assigned to the “People Development-oriented goals”.
4.4 Goal Decomposition Analysis
169
However, three goals are false positives (Value creation, Operational excellence and People of company 549). Of the remaining 39 goals from our case studies there are no additional goals that, according to our semantic interpretation, should belong to the “People Development-oriented goals” group. In summary, SI-2 assigns people-oriented goals correctly 93% of the time. When comparing the results of SI-1 with those of SI-2 the following can be observed: (1) three of the goals that are correctly indicated as being similar by SI2 are also indicated as being similar by SI-1; (2) one goal that, according to our semantic interpretation, is incorrectly assigned by SI-1 is also incorrectly assigned by SI-2. [company 3] Technology evaluation [company 1] Top ideas FBO
[company 3] New opportunities
TP OpEx
[company 8] Technology creation
TT
[company 4] Technology innovation and leadership
RPM CA
[company 4] Business impact
IP IMG
[company 5] xPG Influence
PUB
[company 7] Value creation
PSC CPC
[company 7] Operational excellence [company 5] People 0
10
20
30
40
50
60
70
80
90
100
Fig. 4.31 Idea Generation-oriented goals with SI-2 indicator
Figure 4.31 shows the goals that SI-2 assigns to the FBO cluster (yellow). It is interesting to note that companies 4, 5, 7 and 8 each have two goals related to idea generation. For the first four goals: from company 8, 1, 3 and 8, the average weight of KPIs that are assigned to the FBO cluster is 50% or higher. SI-2 identifies six goals that our semantic interpretation also assigned to the “Idea Generation-oriented goals” group. Three goals are false positives (xPG Influence, Value creation and Operational Excellence). Of the remaining 36 goals from our case studies, there are no additional goals that our semantic interpretation suggests should belong to the “Idea Generation-oriented goals” group. In summary, SI-2 predicts 91% of idea generation-oriented goals correctly. When comparing the results of SI-1 with those of SI-2 the following can be observed: (1) six of the goals that are correctly indicated as similar by SI-2 are also
49
Please see our explanations for the misleading title of this goal in our comments on the SI-1 indicator.
170
4 Performance Management: Analysis Approach
indicated as similar by SI-1; (2) one goal (People) which, according to our semantic interpretation is incorrectly assigned by SI-1, is also incorrectly assigned by SI-2. IMG FBO
[company 1] 360° visibility
TT
[company 4] Thought leadership
RPM OpEx
[company 7] Strategic partnerships
CA IP
[company 8] Image
TP
[company 5] Image
PUB PSC
0
10
20
30
40
50
60
70
80
90
100
CPC
Fig. 4.32 Image-oriented goals with SI-2 indicator
Figure 4.32 shows the goals assigned by SI-2 to the IMG cluster (purple). Five goals that are identified by SI-2 are also identified by our semantic interpretation as belonging to the “Image-oriented goals” group. Of the remaining 40 goals from our case studies there are no additional goals, which, according to our semantic interpretation, should belong to the “Image-oriented goals” group. In summary SI-2 assigns all the image-oriented goals correctly. When comparing the results of SI-1 with those of SI-2 the following can be observed: (1) four of the goals that are correctly indicated as being similar by SI2 are also indicated as being similar by SI-1; (2) one goal (Strategic partnerships) has not been detected by SI-1, but was correctly identified by SI-2 (Table 4.8). Table 4.8 Accuracy comparison of SI-1 and SI-2 for six goal groups No Goal group Accuracy of SI-1 (%) Accuracy of SI-2 (%) 1. Technology Transfer-oriented goals 93 88 2. Intellectual Property Creation-oriented goals 98 93 3. Operational Excellence-oriented goals 100 95 4. People Development-oriented goals 95 93 5. Idea Generation-oriented goals 97 91 6. Image-oriented goals 98 100
The performance of SI-2 is at least as good as SI-1.50 The composition analysis and similarity indicators applied to our case study data delivers, in a pragmatic way,
50
Regarding the analysis results, we would have expected that in each goal there is at least one dominant performance cluster present in the cluster spectrum.
4.5 Summary
171
reasonable results on the goal similarities. By turning this around, it can provide a simple way to formulate recommendations for practitioners. This is of course our underlying motivation, as we stated at the outset of this section. Additionally, we can conclude that the goal composition analysis seems to be a valid approach to compare organizational goals. The findings of this analysis enable us to construct recommendations for practitioners. More precisely, for a specific organizational research goal, we can suggest (1) the set of activities that might contribute to the achievement of the goal and (2) a set of KPIs to assess the results of these activities and hence the organizational goal. With this method, having implicitly affirmed our assumption that performance clusters are a suitable means to building the connection between goals and KPIs, we substantiated our performance clusters with a deep analysis of activities, as documented above. These analysis steps, like others described in this chapter, will eventually be applied to the data of a large quantitative survey in order to validate the developed procedures. Altogether, although our data from the case studies is less detailed than that which can be obtained in a large quantitative survey, we nevertheless have observed and described some patterns that are a partial answer to research question 1. They also help us generate a questionnaire for the survey.
4.5
Summary
In this chapter we introduced our analysis approach. We presented this method using the findings from an in-depth case study of more than 70 personal expert interviews. The data revealed the need for the formation of performance clusters. Initially, KPIs were summarized, from the content perspective, by grouping similar KPIs into performance clusters. The performance clusters we created allow for an analysis of organizational goals by decomposing the assessment via these performance clusters. The association of organizational processes within performance clusters yields KPI classes. Altogether, performance clusters, KPI classes and the final KPIs build the foundation for the measurement part of a performance management system. Based on these analysis steps we may conclude that we have succeeded in developing a preliminary approach to deal with the individual elements of performance management, i.e. KPIs, KPI classes, performance clusters and finally organizational department goals. These elements, among others, will be analyzed on a large scale by the quantitative survey. The design of the questionnaire, the execution of the enquiry, and the findings are presented in the next chapter.
.
Chapter 5
Survey Findings
Abstract This chapter presents the findings from the large quantitative survey, which tests the findings from the literature review and case studies. Both the qualitative case studies, as well as the results of a survey and its comparison, are the core of this chapter. The result of the evaluation reveals that industrial research organizations seem to have a set of shared organizational goals and a set of shared KPIs, which can be summarized in KPI classes. In a sum, the survey results endorse the relationships between KPIs and KPI classes, performance clusters and organizational research goals. This chapter is therefore dedicated to the examination of the qualitative results from the quantitative perspective. Keywords Analysis • Description of the results • Development of the survey • Sample design • Definition of the unit of analysis • Large quantitative survey • Statistics • Comparison of the results of quantitative survey with the qualitative analysis of the case studies
5.1
Introduction
In this section, we describe the development of our large quantitative survey and its findings.
5.2
Development of the Survey
The questionnaire1 is based on the findings of the case studies. As previously presented, the KPIs and goals have been classified, consolidated and condensed from individual facts into generic groups on a more abstract level. This format
1
See Appendix E.
T. Samsonowa, Industrial Research Performance Management, Contributions to Management Science, DOI 10.1007/978-3-7908-2762-0_5, # Springer-Verlag Berlin Heidelberg 2012
173
174
5 Survey Findings
enabled us to compose a questionnaire to gather information on a large scale with which to test/verify the findings of the case studies. The case studies essentially collated the five facts directly into one matrix.2 The survey also collated the five facts into one matrix, but rather than having the respondent complete a matrix, the questionnaire was administered via an online survey. The five facts are: (a) (b) (c) (d) (e)
Organizational research department goals Goal weightings in percent within the entire goal system (all goals ¼ 100%) KPIs KPI weightings in percent within each goal (one goal ¼ 100%) Mapping of KPIs contributing to a goal
Goals have been expressed in eight generic goals (a) KPIs have been expressed in 37 KPI classes (c) The weightings (b and d) in percentage are collected via a scale. The mapping (e) has been generated as a pre-selection based on the evaluation of case study data. The aggregation of goals and the mapping preparations are explained below. KPI consolidation (KPI classes) were presented in detail in Sect. 4.2.1. As previously presented the case studies identified 45 organizational research department goals. Thirty of the 45 were different. Eight generic organizational goals for industrial research departments were formulated for the questionnaire (Table 5.1): Table 5.1 Eight generic organizational goals in industrial research Goal Generic goals 1. Alignment with and transfer to internal development and other (business) units 2. Create and protect intellectual property 3. Improve the internal and external image of the research department and/or the company 4. Generate and evaluate future business opportunities 5. Recruit and develop excellent talent 6. Achieve a high standard of operational excellence 7. Establish and maintain strategic partnerships and/or collaborative research 8. Drive technology innovation and technology leadership
This list of eight generic goals is a result of a semi-objective, qualitative estimation, which results from iterative discussions with case study companies. The people selected for the process of sorting, classifying and formulating the goals held senior management positions and have experience in defining and cascading goals within their daily activity.
2
Compare the developed PMS matrix compilation presented earlier.
5.3 Sample Design
175
Table 5.2 represents the findings of the case studies from the cross-case analysis mapping of goals and KPI classes. Table 5.2 Mapping goals and KPIs Performance clusters Technology transfer Future business opportunities Technical achievements Intellectual property Operational excellence Talent pool Image Publications Presence in scientific community Collaboration with academia Collaboration with partners and customers Total KPI classes
Organizational research goals 1 x x
2 x x
3 x
x
x
x
4 x x
5 x
x
7
x
x x x
x
6
x x
x x x
8 x x x x
x x
x
x x
x x x
# KPI classes per performance cluster 5 3 4 5 6 4 2 2 2 2 2
15 15 16 13 17 17 14 17 37
This mapping table was used to build the questionnaire3 by pre-assigning KPI classes to the eight generic organizational research goals. The respondents had the possibility to specify additional KPIs that were used in the company and were not identified among the presented KPI classes.
5.3
Sample Design
A large quantitative survey was used as the research approach. The target population for the online survey was as follows: • Economically significant enterprises operating in the ICT industry; • Located worldwide, covering especially Europe, CIS, and USA; • With a formal research department in the organizational structure. The sample-design process consists of the following steps: 1. The ICT industry was defined according to the ISIC classification.4 2. The chosen codes based on ISIC were translated into the SIC.5
3
See Appendix E. Refer to the earlier definition of ISIC. 5 The reason for this translation step was the fact that the data available is still organized according the old standard (SIC). 4
176
5 Survey Findings
3. The sample was drawn from a total population of 2,161 companies. As a basis, the Schonfeld & Associates Database6 was used. This database was amended and refined with data from other economic areas.7 4. The initial list of 2,161 companies was reduced to 497 companies by the following instrument: R&D ratio.8 This implies that companies reporting R&D spending would have a research department. 5. An appropriate contact person9 was identified to complete the questionnaire in each company. 6. The identified person was then first contacted by phone. At the initial point of contact, the contact person was asked a series of questions to ensure: • That the company has a research department • That the contact person is involved in a decision-making process regarding research organization activities. 7. The person, who showed interest in the survey and met the screening requirements, received an invitation to participate in the survey by e-mail (Table 5.3).
Table 5.3 Definition of the unit of analysis Selection criteria Definition ICT manufacturing industries ICT sectoraa ICT trade industries ICT services industries Department The selection criteria is a dedicated research department within the overall organization (can even be an independent legal entity), the research department must have its own organizational goals Person The profile of the people who were addressed was: a person in charge of the research department (Head of Research). Alternatively, it COOs (Chief Operating Officers), persons responsible for controlling activities within research department, or somebody who is in charge of monitoring the achievement of goals and is able to propose countermeasures in case of deviations a The industry-based definition of the Information and Communication Technology (ICT) sector is based on International Standard Industry Classification (ISIC) Revision 4. The classification framework of ICT originates from industries that primarily produce ICT goods and services
6 Schonfeld & Associates Database: R&D Ratios & Budgets 2008. Note that this database contains worldwide data organized by industry sector. 7 EU: 2008 EU Industrial R&D Investment Scoreboard and Truffle 100 representing the top 100 European software vendors; CIS – ICT congress, China. 8 As a percentage of sales. 9 Different sources were used: Onesourse Database representing the largest business database, contact lists provided by the EU, contacts extracted by internet search, etc.
5.4 Analysis/Description of the Results
177
In total, 108 companies participated in the survey, two of whom were deselected as strong outliers resulting in 106, which meant a target response rate of 21.73%. The criteria for non-response were: • No appropriative research department (too few research activities, mostly R&D, etc.) • The form was returned, but the key questions were not answered • Strong inconsistencies in the data.
5.4 5.4.1
Analysis/Description of the Results General Data: Organizational Context and Research Environment
To first portray the sample formation of the participating companies in our investigation we used a frequency distribution, which shows that 106 companies responded to the survey. Figure 5.1 shows that 57 companies had headquarters in Europe or the U.S. and 49 companies were headquartered in the CIS. With such a clear divide in the sample, the ensuing analysis was done separately for the two groups where necessary. Fig. 5.1 Location of company headquarters: overview
Europe and USA 57
CIS